# REMOTE INIT FAILED error on pbs but manual submission works? **URL:** \ **Category:** Cylc Support\ **Created:** [June 3, 2021, 4:56pm UTC](https://cylc.discourse.group/t/remote-init-failed-error-on-pbs-but-manual-submission-works/344 "2021-06-03T16:56:54Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![bencash](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bencash](https://cylc.discourse.group/u/bencash)\ **Post date:** [June 3, 2021, 4:56pm UTC](https://cylc.discourse.group/t/remote-init-failed-error-on-pbs-but-manual-submission-works/344/1 "2021-06-03T16:56:55Z")
I am stuck trying to submit a basic suite via pbs on NCAR Cheyenne. If I submit via cylc play, I get the following message in job-activity.log: [jobs-submit cmd] (init cheyenne) [jobs-submit ret\_code] 1 [jobs-submit err] REMOTE INIT FAILED [jobs-submit cmd] (remote init) [jobs-submit ret\_code] 1 However, if I submit the job file that was created manually via ‘qsub job’ it runs fine. The suite will also execute correctly if I comment out the ‘platform’ line in my flow.cylc and run it on the login node. So the error seems confined to interacting with the batch system. My flow.cylc is: [meta] title = “Basic cylc suite for executing L96 model” [scheduling] [[dependencies]] graph = “L96” [runtime] [[root]] platform = cheyenne execution time limit = PT10M [[L96]] script = “python /glade/work/bcash/ssf/L96/L96\_roxie.py” [[[directives]]] -q = regular -A = UMIN0005 -l select=1:mpiprocs=1:ncpus=1 -M = bcash@gmu.edu And my global.cylc is: [platforms] [[cheyenne]] job runner = pbs ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [June 3, 2021, 6:50pm UTC](https://cylc.discourse.group/t/remote-init-failed-error-on-pbs-but-manual-submission-works/344/2 "2021-06-03T18:50:39Z")
Hello, Unfortunately that job activity log output doesn’t contain any information about why the remote init failed. The error should have been written to the workflow log `log/suite/log` (soon to be `log/workflow/log`). Taking a guess at the problem, are you able SSH directly into “cheyenne” from the Scheduler host (i.e. `ssh cheyenne hostname`) or do you have to go via a login node? If login nodes are required they will need to be configured like so: ```auto [platforms] [[cheyene]] hosts = cheyenne-01, cheyenne-02 # host will be picked at random from this list job runner = pbs ``` Otherwise if the problem is still not evident running the workflow in debug mode might yield some insights (`cylc play --debug`). The command that Cylc runs to initiate the “remote init” should be logged there (search for `remote-init`). ---
**Author:** ![bencash](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bencash](https://cylc.discourse.group/u/bencash)\ **Post date:** [June 3, 2021, 7:20pm UTC](https://cylc.discourse.group/t/remote-init-failed-error-on-pbs-but-manual-submission-works/344/3 "2021-06-03T19:20:57Z")
Thanks! I had’t spotted the information in log/suite - I reran with debug and it looks like it is what you suspected: 2021-06-03T13:08:08-06:00 DEBUG - [‘ssh’, ‘-oBatchMode=yes’, ‘-oConnectTimeout=10’, ‘cheyenne’, ‘env’, ‘CYLC\_VERSION=8.0b1’, ‘bash’, ‘–login’, ‘-c’, ‘‘exec “$0” “$@”’’, ‘cylc’, ‘remote-init’, ‘–debug’, ‘cheyenne’, ‘$HOME/cylc-run/debug\_submit/run1’] 2021-06-03T13:08:08-06:00 ERROR - cheyenne: initialisation did not complete: COMMAND FAILED (255): ssh -oBatchMode=yes -oConnectTimeout=10 cheyenne env CYLC\_VERSION=8.0b1 bash --login -c ‘’"’"‘exec “$0” "$@ "’"’"’’ cylc remote-init --debug cheyenne ‘$HOME/cylc-run/debug\_submit/run1’ COMMAND STDERR: ssh: Could not resolve hostname cheyenne: Name or service not known 2021-06-03T13:08:08-06:00 DEBUG - BEGIN TASK PROCESSING 2021-06-03T13:08:08-06:00 INFO - [L96.1] -submit-num=01, host=cheyenne 2021-06-03T13:08:08-06:00 ERROR - [jobs-submit cmd] (init cheyenne) [jobs-submit ret\_code] 1 [jobs-submit err] REMOTE INIT FAILED I’m a bit confused though - I am already on a login node when I ‘cylc play’, and if I try ‘ssh cheyenne1 hostname’ it asks me for a password. Do I need to setup passwordless ssh to the login nodes to have submission go through? Update: passwordless ssh won’t work because 2FA is still required. @Jim it seems like you have solved this for cheyenne and casper, what step am I missing here? ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [June 4, 2021, 5:48am UTC](https://cylc.discourse.group/t/remote-init-failed-error-on-pbs-but-manual-submission-works/344/4 "2021-06-04T05:48:00Z")
> [@bencash](#): > > I am already on a login node when I ‘cylc play’ But presumably you can’t submit to PBS from the login node that you’re already on, or if you can that would result in the job running (or attempting to run) in the wrong place? > [@bencash](#): > > Do I need to setup passwordless ssh to the login nodes to have submission go through? If you need to ssh to a remote platform to submit jobs, then the Cylc scheduler can do that for you, but yes you have to have passwordless ssh set up. Let’s see what @Jim says, but if 2FA is compulsory you might be able to get away with manually starting a persistent ssh tunnel to the job submission host. ---
**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\ **Post date:** [June 4, 2021, 12:31pm UTC](https://cylc.discourse.group/t/remote-init-failed-error-on-pbs-but-manual-submission-works/344/5 "2021-06-04T12:31:05Z")
If you are just trying to submit jobs on the same host where the cylc scheduler is running then you need to configure your platform to use localhost. i.e. ```auto [platforms] [[cheyenne]] job runner = pbs install target = localhost hosts = localhost ``` ---
**Author:** ![bencash](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bencash](https://cylc.discourse.group/u/bencash)\ **Post date:** [June 4, 2021, 2:27pm UTC](https://cylc.discourse.group/t/remote-init-failed-error-on-pbs-but-manual-submission-works/344/6 "2021-06-04T14:27:10Z")
> [@hilary.j.oliver](#): > > But presumably you can’t submit to PBS from the login node that you’re already on, or if you can that would result in the job running (or attempting to run) in the wrong place? If I understand the question correctly, you do in fact submit to PBS from the login node. If I am working on Cheyenne, I access the machine via a login node and unless I start up an interactive session on the compute nodes everything I do is on that node. That includes submitting to the batch system, which is just ‘qsub ’. I never directly access the compute nodes - the scheduler is in charge of that. Running via 7.8.3, which is centrally installed at NCAR, appears to work without a hitch. @Jim is also still using cylc7. Has something changed between 7 and 8 that would change the interaction with the job scheduler? Or is this likely to be an issue with how it is installed? ---
**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\ **Post date:** [June 4, 2021, 2:54pm UTC](https://cylc.discourse.group/t/remote-init-failed-error-on-pbs-but-manual-submission-works/344/7 "2021-06-04T14:54:34Z")
@bencash - you need to set up your `cheyenne` platform to submit locally as I’ve shown above. If you don’t specify `hosts` for a platform the default is to submit to a host of the same name as the platform which isn’t what you want in this case. ---
**Author:** ![bencash](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bencash](https://cylc.discourse.group/u/bencash)\ **Post date:** [June 4, 2021, 3:16pm UTC](https://cylc.discourse.group/t/remote-init-failed-error-on-pbs-but-manual-submission-works/344/8 "2021-06-04T15:16:56Z")
Thanks! I’ve made that modification and it seems to have worked. I’ll keep poking at it this morning to confirm.