CYLC_* variables; cannot access in user script

Cylc Cylc Support
21

Thanks @hilary.j.oliver .

I found out from logs that it is not liking one of the qsub directive in job prepared by cylc. It does not tell, which qsub directive it is complaining about. I verified it outside cylc framework by copying directives from job file into a test script.

#PBS script ${MY_DEFAULT_PATH}/task1.scr …does not like it

<bap001@ppp8->more test5.scr
#!/bin/bash -l
#
# ++++ THIS IS A CYLC JOB SCRIPT ++++
# Workflow: ex10_useplatforms/run16
# Task: 1/task1
# Job log directory: 1/task1/01
# Job runner: pbs
# Execution time limit: 300.0

# DIRECTIVES:
#PBS -N pabla
#PBS -o job.out
#PBS -e job.err
#PBS -l walltime=300
#PBS -l select=1:ncpus=1:mem=2gb
#PBS -M balbir.pabla@ec.gc.ca
#PBS -m abe
#PBS -q development
#PBS script task5.scr
hn=`hostname`
echo "hn=${hn}"
me=`whoami`
echo "me=${me}"

<bap001@ppp8->qsub test5.scr
qsub: directive error: script task5.scr

Now I removed , #PBS script task5.scr
Resubmitted again, and it worked

<bap001@ppp8->qsub test5.scr
69356930.ppp8pbs-01-ib
<bap001@ppp8->hostname
ppp8login-01-ib      # ....submitting from login node, qsub submits to compute node named ppp8cn-115 

<bap001@ppp8->more job.out
(INFO) Using package (fully qualified) pixi_0.63.2_all from domain /fs/ssm/eccc/cmd/cmds/apps/pixi/202602/00
hn=ppp8cn-115  # ..compute node name, here "cn" stands for compute node
me=bap001

actual cylc job file /home/bap001/cylc-run/ex10_useplatforms/run16/log/job/1/task1/NN/job

#!/bin/bash -l
#
# ++++ THIS IS A CYLC JOB SCRIPT ++++
# Workflow: ex10_useplatforms/run16
# Task: 1/task1
# Job log directory: 1/task1/01
# Job runner: pbs
# Execution time limit: 300.0

# DIRECTIVES:
#PBS -N pabla
#PBS -o cylc-run/ex10_useplatforms/run16/log/job/1/task1/01/job.out
#PBS -e cylc-run/ex10_useplatforms/run16/log/job/1/task1/01/job.err
#PBS -l walltime=300
#PBS -l select=1:ncpus=1:mem=2gb
#PBS -M balbir.pabla@ec.gc.ca
#PBS -m abe
#PBS -q development
#PBS script ${MY_DEFAULT_PATH}/task1.scr
if [[ $1 == 'noreinvoke' ]]; then
    shift
else
    exec bash -l "$0" noreinvoke "$@"
fi
...

bap001@ppp8->more job.status
CYLC_JOB_RUNNER_NAME=pbs
<bap001@ppp8->more job-activity.log
[jobs-submit cmd] ssh -oBatchMode=yes -oConnectTimeout=10 ppp8 env CYLC_VERSION=8.6.3 bash --login -c ''"'"'ex
ec "$0" "$@"'"'"'' cylc jobs-submit --debug --remote-mode --path=/bin --path=/usr/bin --path=/usr/local/bin --
path=/sbin --path=/usr/sbin --path=/usr/local/sbin -- '$HOME/cylc-run/ex10_useplatforms/run16/log/job' 1/task1
/01
[jobs-submit ret_code] 1
[jobs-submit out] 2026-06-23T01:16:03Z|1/task1/01|1|None
2026-06-23T01:16:03Z [STDERR] qsub: directive error: script ${MY_DEFAULT_PATH}/task1.scr

My 2 cents: Instead of specifying the task name on PBS directive, you guys should append the contents of task1 script to job file.

22

Hi @bpabla50 - well this is progress, at least you’re now getting job submission errors from PBS, which means your Cylc installation and global config is basically working.

Actually Cylc does not automatically create an erroneous #PBS script directive - if it did, Cylc would not work with PBS anywhere. So you must have done it yourself, via the [directives] section in your task definition (where Cylc will create a directive out of any item you put there - and it leaves interpretation and validation of those directives to PBS).

I suspect you did it accidentally by mis-ordering items in workflow config. The flow.cylc file is a nested INI format. Indentation can be used for aesthetic reasons, but whitespace is ignored by the parser, so any item in the file is “nested” under the most recent heading above it in the file.

Here’s a valid working example (for my local PBS platform and queue):

[scheduling]
    [[graph]]
        R1 = "hello"
[runtime]
    [[hello]]
        platform = cascade-hob-pbs
        script = "echo hello"
        [[[directives]]]
            -koed =
            -q = hub_workerq
            -l walltime=00:01:00
            -l select=1:ncpus=1:mem=100mb

After running this workflow, I can see the following directives in the job file:

$ cylc cat-log -f j bug/run2//1/hello | grep '#PBS'
#PBS -N hello.1.bug-run2
#PBS -o cylc-run/bug/run2/log/job/1/hello/01/job.out
#PBS -e cylc-run/bug/run2/log/job/1/hello/01/job.err
#PBS -koed
#PBS -q hub_workerq
#PBS -l walltime=00:01:00
#PBS -l select=1:ncpus=1:mem=100mb

But the following is wrong and reproduces your error:

[scheduling]
    [[graph]]
        R1 = "hello"
[runtime]
    [[hello]]
        platform = cascade-hob-pbs
        [[[directives]]]
            -koed =
            -q = hub_workerq
            -l walltime=00:01:00
            -l select=1:ncpus=1:mem=100mb
        script = "echo hello"  # !!!! ERROR !!!! this is under [directives]!

Running this, the job fails to submit, and there is an erroneous #PBS script directive in the job file:

$ cylc cat-log -f j bug/run3//1/hello | grep '#PBS'
#PBS -N hello.1.bug-run3
#PBS -o cylc-run/bug/run3/log/job/1/hello/01/job.out
#PBS -e cylc-run/bug/run3/log/job/1/hello/01/job.err
#PBS -koed
#PBS -q hub_workerq
#PBS -l walltime=00:01:00
#PBS -l select=1:ncpus=1:mem=100mb
#PBS script echo hello

Actually the logs do say exactly what the problem is. As I said above, you need to look at the job activity log - that is a job log that records job-specific activity by the scheduler, such as the result of trying to submit the job. For my example above:

$ cylc cat-log -f a bug/run3//1/hello
[jobs-submit cmd] ssh -oBatchMode=yes -oConnectTimeout=10 login03.hob.hpc.niwa.co.nz env CYLC_VERSION=8.6.4 CYLC_ENV_NAME=cylc-8.6.4 bash --login -c ''"'"'exec "$0" "$@"'"'"'' cylc jobs-submit --remote-mode --path=/bin --path=/usr/bin --path=/usr/local/bin --path=/sbin --path=/usr/sbin --path=/usr/local/sbin -- '$HOME/cylc-run/bug/run3/log/job' 1/hello/01
[jobs-submit ret_code] 1
[jobs-submit out] 2026-06-23T21:11:48Z|1/hello/01|1|None
2026-06-23T21:11:48Z [STDERR] qsub: directive error: script echo hello

(The last line there says exactly what the problem is).

23

Thanks a lot @hilary.j.oliver for your help. Finally I am able to submit batch job to pbs cluster. Yes that erroneous PBS directive was due to non-compliance with INI format used in flow.cylc.

Just curious, at which stage, CYLC system uses qsub command to submit job script. I did not find qsub string in whole run directory.

bap001@hpcr8-vis->pwd
/home/bap001/cylc-run/ex10_useplatforms/run18
grep --text -ir qsub *

Is following ssh command uses  qsub command behind the scene. Please explain  little bit about the work done in this command.
From run18/log/scheduler/log :
2026-06-24T13:48:34Z DEBUG - ['ssh', '-oBatchMode=yes', '-oConnectTimeout=10', 'ppp8', 'env', 'CYLC_VERSION=8.6.3', 'bash', '--login', '-c', '\'exec "$0" "$@"\'', 'cylc', 'jobs-submit', '--debug', '--remote-mode', '--path=/bin', '--path=/usr/bin', '--path=/usr/local/bin', '--path=/sbin', '--path=/usr/sbin', '--path=/usr/local/sbin', '--', '$HOME/cylc-run/ex10_useplatforms/run18/log/job', '1/task1/01']
2026-06-24T13:48:36Z DEBUG - [jobs-submit cmd] cat /home/bap001/cylc-run/ex10_useplatforms/run18/log/job/1/task1/01/job | ssh -oBatchMode=yes -oConnectTimeout=10 ppp8 env CYLC_VERSION=8.6.3 bash --login -c ''"'"'exec "$0" "$@"'"'"'' cylc jobs-submit --debug --remote-mode --path=/bin --path=/usr/bin --path=/usr/local/bin --path=/sbin --path=/usr/sbin --path=/usr/local/sbin -- '$HOME/cylc-run/ex10_useplatforms/run18/log/job' 1/task1/01
24

The qsub command is hardcoded within the Cylc codebase itself.

  • It is called when the job is in the preparing state.
  • Once the submission has succeeded, the job enters the submitted state.
  • Once it starts running, the job enters the running state.

If you’re interested, here’s the location of Cylc’s PBS support module: cylc-flow/cylc/flow/job_runner_handlers/pbs.py at master · cylc/cylc-flow · GitHub. This is generic doesn’t need to be modified, any configuration required can be made in the global.cylc[platforms] section.

25

Thanks @oliver.sanders .