Cylc remote cmd does not pass STDIN

Cylc Cylc Support
1

Hi there,

I am trying to run cylc-8.1.2 suite remotely.

suite installs files on a remote host, but fails to submit.
it seems to me, that for some reasons, the 'cat my_task/…/job | ’ that should appear in front of the ssh cmd is missing.
i tried to dig a bit and i got lost :wink:

  • the TaskJobManager submit_task_jobs() method puts an apropriate cmd [with 'cat …/job | '] to proc_pool
  • but the SubProcPool _run_command_init() has a cmd without 'cat …/job | ’ string, and stdin_files are empty

also, when i comment out cmd.append('--remote-mode') in submit_task_jobs() the cmd that appears in job log starts with 'cat …/job | ’ substring; [of course it fails later for other reasons, having remote_mode set to False]

I would be grateful for any hints.
maybe there is sth I am missing in global.cylc or suite flow.cylc?

2

Hello,

To help us debug, could you provide the error that Cylc reported.

You can find this in the “job-activity.log” file for the job which failed to submit. This command will help you to find the file:

$ cylc cat-log <workflow>//<cycle>/<task> -f a
3

hello,
here is the job-activity.log:

[jobs-submit cmd] ssh -oBatchMode=yes -oConnectTimeout=8 -oStrictHostKeyChecking=no okeanos env CYLC_VERSION=8.1.2 bash --login -c ''"'"'exec "$0" "$@"'"'"'' cylc jobs-s
ubmit --remote-mode --clean-env --path=/bin --path=/usr/bin --path=/usr/local/bin --path=/sbin --path=/usr/sbin --path=/usr/local/sbin -- '$HOME/cylc-run/exp9/run21/log/
job' 1/buy_ingredients/01
[jobs-submit ret_code] 1
[jobs-submit out] 2023-04-06T19:23:31+02:00|1/buy_ingredients/01|1|
2023-04-06T19:23:31+02:00 [STDERR] [Errno 2] No such file or directory: 'sbatch'

which reminds me, that when I tried to submit the above cmd manually, i needed to remove ‘–clean-env’; then, once the files were installed on the remote host, it worked [sbatch is not in any of the paths above]

4

It looks like the Slurm job submit command is not in your default environment on the job platform.

The --clean-env option above (which is internal to Cylc) stops the scheduler environment from being passed on to job platforms. It can be turned off with a global config setting, documented here (along with the rationale for why a clean environment is desirable):

https://cylc.github.io/cylc-doc/stable/html/reference/config/global.html#global.cylc[platforms][<platform%20name>]clean%20job%20submission%20environment

From what you say, it may be sufficient to turn that off. Or otherwise, make sure the correct environment is set in your login scripts on the job platform.

5

Hi,

thanks, that’s exactly what I wanted to ask - if I can pass the submission options from global.cylc.
Setting the clean-env to False worked.
I will try also to keep it True, and pass the contents of server path, although it is a very long path - it’s cray, with several modules loaded by default.

thanks for help!

1 Like