I am setting up running cylc on another new machine (Expanse at SDSC) and hit the error in the subject.
/home/bcash/cylc-run/P7b/run6/.service/etc/job.sh: line 113: cylc: command not found
/home/bcash/cylc-run/P7b/run6/.service/etc/job.sh: line 147: cylc: command not found
Looking in the job.sh file those are the two ‘cylc message’ commands. I can run cylc just fine from the command line, the cylc8b1/bin directory is in my $PATH, etc. Any suggestions as to why it is losing track of ./cylc in job.sh?
Hi @bencash,
Well, cylc must not be in your $PATH in the environment where the job script runs. We might need a bit more context to find out why. E.g.:
- presumably you have Cylc 8 installed into a virtual environment?
- is the job running on the scheduler host, or a remote host?
- are you using the central
cylc wrapper script, that allows you to dynamically select from multiple installed Cylc versions?
I do have it installed in a virtual environment. I used --prefix when I created it, so I have to give the full path when I activate it (maybe that is involved?).
This task is running on the scheduler - it is just running a set of bash scripts that copy some files.
I do not believe I am using the central script - I am just using ‘cylc play ___’, where cylc is pointing to: /expanse/lustre/projects/mia328/bcash/cylc8b1/bin/cylc
Thanks!
OK, if your tasks run as background jobs (which is the default), on localhost (i.e. the scheduler host; which is the default) then job shells, by default, should inherit the scheduler environment. So if you activated the conda environment before running cylc play (and presumably you did, because you said “cylc is pointing to …”) then the job should automatically have access to the cylc command.
You can prevent this automatic access by stopping the job environment from inheriting the scheduler environment, however, like this:
[platforms]
[[localhost]]
clean job submission environment = True # (not the default!)
(The reason for this is it makes local jobs consistent with remote ones, which can’t inherit the local scheduler environment)
Do you have that setting in your global config? Check with cylc config | grep "clean job submission environment")
(cylc8b1_v2) [bcash@login01 P7b]$ cylc config | grep “clean job submission environment”
clean job submission environment = False
clean job submission environment = False
clean job submission environment = False
I also tried installing my env in a different location and hit the same problem. I don’t remember seeing this problem running 8b1 on other machines.
Well that is mysterious. I just tried installing cylc-8.0b1 myself in a clean user account with a new conda environment (created with --prefix to get a non-standard env location). And this test trivial worfklow:
[scheduling]
[[graph]]
R1 = foo
[runtime]
[[foo]]
script = "echo HELLO"
It works exactly as I described above: with clean job submission environment = False the job runs fine, and with clean job submission environment = True it fails with cylc: command not found in the job.err file.
You could try putting printenv in your task scripting, and compare the environments of the job and the shell in which you run cylc play.
This is what I get as PATH if I use that same suite, substituting printenv for “echo Hello”
PATH=/home/bcash/cylc-run/path_test/run1/bin:/cm/shared/apps/slurm/current/sbin:/cm/shared/apps/slurm/current/bin:/cm/shared/apps/spack/cpu/opt/spack/linux-centos8-zen/gcc-8.3.1/gcc-10.2.0-n7su7jf54rc7l2ozegds5xksy6qhrjin/bin:/home/bcash/.local/bin:/home/bcash/bin:/home/bcash/cylc-run/path_test/run1/bin:/expanse/lustre/projects/mia328/bcash/miniconda3/bin:/home/bcash/cylc-run/P7b/run40/bin:/expanse/lustre/projects/mia328/bcash/miniconda3/condabin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/bin:/sbin
and this is PATH from the command line:
PATH=/cm/shared/apps/sdsc/1.0/bin:/cm/shared/apps/sdsc/1.0/sbin:/cm/shared/apps/slurm/current/sbin:/cm/shared/apps/slurm/current/bin:/expanse/lustre/projects/mia328/bcash/miniconda3/envs/cylc8b1_v2/bin:/cm/shared/apps/spack/cpu/opt/spack/linux-centos8-zen/gcc-8.3.1/gcc-10.2.0-n7su7jf54rc7l2ozegds5xksy6qhrjin/bin:/home/bcash/.local/bin:/home/bcash/bin:/expanse/lustre/projects/mia328/bcash/miniconda3/bin:/expanse/lustre/projects/mia328/bcash/miniconda3/condabin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin
Strange. Cylc runs local background jobs as normal subshell processes, so they should inherit the (exported) scheduler environment unless we actively prevent that (which is what the “clean job submission environment” setting does).
Next question, how did you get the cylc bin path in your terminal shell environment (where you ran cylc play)? Was it by conda activate? Just wondering if you manually set $PATH but didn’t export it.
It was from conda activate.
Edit: I’ve got it. I had conda config --set auto_activate_base true, and whenever my .bashrc was sourced it was reverting me to base environment - which does not have cylc installed. Once I set that to false the problem went away. 
1 Like