# CYLC\_\* variables; cannot access in user script

**URL:** <https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322>\
**Category:** Cylc Support\
**Created:** [May 25, 2026, 9:00pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322 "2026-05-25T21:00:51Z")\
**Posts on this page:** 5\
**Page:** 2

<div class="post-metadata">

**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\
**Post date:** [June 23, 2026, 2:17pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/21 "2026-06-23T14:17:50Z")

</div>

Thanks @hilary.j.oliver .

I found out from logs that it is not liking one of the qsub directive in job prepared by cylc. It does not tell, which qsub directive it is complaining about. I verified it outside cylc framework by copying directives from job file into a test script.

`#PBS script ${MY_DEFAULT_PATH}/task1.scr` …does not like it

```bash
<bap001@ppp8->more test5.scr
#!/bin/bash -l
#
# ++++ THIS IS A CYLC JOB SCRIPT ++++
# Workflow: ex10_useplatforms/run16
# Task: 1/task1
# Job log directory: 1/task1/01
# Job runner: pbs
# Execution time limit: 300.0

# DIRECTIVES:
#PBS -N pabla
#PBS -o job.out
#PBS -e job.err
#PBS -l walltime=300
#PBS -l select=1:ncpus=1:mem=2gb
#PBS -M balbir.pabla@ec.gc.ca
#PBS -m abe
#PBS -q development
#PBS script task5.scr
hn=`hostname`
echo "hn=${hn}"
me=`whoami`
echo "me=${me}"

```

```auto
<bap001@ppp8->qsub test5.scr
qsub: directive error: script task5.scr

```

Now I removed , `#PBS script task5.scr`  
Resubmitted again, and it worked

```bash
<bap001@ppp8->qsub test5.scr
69356930.ppp8pbs-01-ib
<bap001@ppp8->hostname
ppp8login-01-ib # ....submitting from login node, qsub submits to compute node named ppp8cn-115 

```

```bash
<bap001@ppp8->more job.out
(INFO) Using package (fully qualified) pixi_0.63.2_all from domain /fs/ssm/eccc/cmd/cmds/apps/pixi/202602/00
hn=ppp8cn-115 # ..compute node name, here "cn" stands for compute node
me=bap001

```

actual cylc job file `/home/bap001/cylc-run/ex10_useplatforms/run16/log/job/1/task1/NN/job`

```bash
#!/bin/bash -l
#
# ++++ THIS IS A CYLC JOB SCRIPT ++++
# Workflow: ex10_useplatforms/run16
# Task: 1/task1
# Job log directory: 1/task1/01
# Job runner: pbs
# Execution time limit: 300.0

# DIRECTIVES:
#PBS -N pabla
#PBS -o cylc-run/ex10_useplatforms/run16/log/job/1/task1/01/job.out
#PBS -e cylc-run/ex10_useplatforms/run16/log/job/1/task1/01/job.err
#PBS -l walltime=300
#PBS -l select=1:ncpus=1:mem=2gb
#PBS -M balbir.pabla@ec.gc.ca
#PBS -m abe
#PBS -q development
#PBS script ${MY_DEFAULT_PATH}/task1.scr
if [[$1 == 'noreinvoke']]; then
    shift
else
    exec bash -l "$0" noreinvoke "$@"
fi
...

```

```bash
bap001@ppp8->more job.status
CYLC_JOB_RUNNER_NAME=pbs
<bap001@ppp8->more job-activity.log
[jobs-submit cmd] ssh -oBatchMode=yes -oConnectTimeout=10 ppp8 env CYLC_VERSION=8.6.3 bash --login -c ''"'"'ex
ec "$0" "$@"'"'"'' cylc jobs-submit --debug --remote-mode --path=/bin --path=/usr/bin --path=/usr/local/bin --
path=/sbin --path=/usr/sbin --path=/usr/local/sbin -- '$HOME/cylc-run/ex10_useplatforms/run16/log/job' 1/task1
/01
[jobs-submit ret_code] 1
[jobs-submit out] 2026-06-23T01:16:03Z|1/task1/01|1|None
2026-06-23T01:16:03Z [STDERR] qsub: directive error: script ${MY_DEFAULT_PATH}/task1.scr

```

My 2 cents: Instead of specifying the task name on PBS directive, you guys should append the contents of task1 script to job file.

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [June 23, 2026, 9:29pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/22 "2026-06-23T21:29:19Z")

</div>

Hi @bpabla50 - well this is progress, at least you’re now getting job submission errors from PBS, which means your Cylc installation and global config is basically working.

> [@bpabla50](#):
>
> My 2 cents: Instead of specifying the task name on PBS directive, you guys should append the contents of task1 script to job file.

Actually Cylc does not automatically create an erroneous `#PBS script` directive - if it did, Cylc would not work with PBS anywhere. So you must have done it yourself, via the `[directives]` section in your task definition (where Cylc will create a directive out of any item you put there - and it leaves interpretation and validation of those directives to PBS).

I suspect you did it accidentally by mis-ordering items in workflow config. The `flow.cylc` file is a nested INI format. Indentation can be used for aesthetic reasons, but whitespace is ignored by the parser, so any item in the file is “nested” under the most recent heading above it in the file.

Here’s a valid working example (for my local PBS platform and queue):

```ini
[scheduling]
    [[graph]]
        R1 = "hello"
[runtime]
    [[hello]]
        platform = cascade-hob-pbs
        script = "echo hello"
        [[[directives]]]
            -koed =
            -q = hub_workerq
            -l walltime=00:01:00
            -l select=1:ncpus=1:mem=100mb

```

After running this workflow, I can see the following directives in the job file:

```bash
$ cylc cat-log -f j bug/run2//1/hello | grep '#PBS'
#PBS -N hello.1.bug-run2
#PBS -o cylc-run/bug/run2/log/job/1/hello/01/job.out
#PBS -e cylc-run/bug/run2/log/job/1/hello/01/job.err
#PBS -koed
#PBS -q hub_workerq
#PBS -l walltime=00:01:00
#PBS -l select=1:ncpus=1:mem=100mb

```

But the following is wrong and reproduces your error:

```ini
[scheduling]
    [[graph]]
        R1 = "hello"
[runtime]
    [[hello]]
        platform = cascade-hob-pbs
        [[[directives]]]
            -koed =
            -q = hub_workerq
            -l walltime=00:01:00
            -l select=1:ncpus=1:mem=100mb
        script = "echo hello" # !!!! ERROR !!!! this is under [directives]!

```

Running this, the job fails to submit, and there is an erroneous `#PBS script` directive in the job file:

```bash
$ cylc cat-log -f j bug/run3//1/hello | grep '#PBS'
#PBS -N hello.1.bug-run3
#PBS -o cylc-run/bug/run3/log/job/1/hello/01/job.out
#PBS -e cylc-run/bug/run3/log/job/1/hello/01/job.err
#PBS -koed
#PBS -q hub_workerq
#PBS -l walltime=00:01:00
#PBS -l select=1:ncpus=1:mem=100mb
#PBS script echo hello

```

> [@bpabla50](#):
>
> I found out from logs that it is not liking one of the qsub directive in job prepared by cylc. It does not tell, which qsub directive it is complaining about.

Actually the logs do say exactly what the problem is. As I said above, you need to look at the _job activity log_ - that is a job log that records job-specific activity by the scheduler, such as the result of trying to submit the job. For my example above:

```bash
$ cylc cat-log -f a bug/run3//1/hello
[jobs-submit cmd] ssh -oBatchMode=yes -oConnectTimeout=10 login03.hob.hpc.niwa.co.nz env CYLC_VERSION=8.6.4 CYLC_ENV_NAME=cylc-8.6.4 bash --login -c ''"'"'exec "$0" "$@"'"'"'' cylc jobs-submit --remote-mode --path=/bin --path=/usr/bin --path=/usr/local/bin --path=/sbin --path=/usr/sbin --path=/usr/local/sbin -- '$HOME/cylc-run/bug/run3/log/job' 1/hello/01
[jobs-submit ret_code] 1
[jobs-submit out] 2026-06-23T21:11:48Z|1/hello/01|1|None
2026-06-23T21:11:48Z [STDERR] qsub: directive error: script echo hello

```

(The last line there says exactly what the problem is).

---

<div class="post-metadata">

**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\
**Post date:** [June 24, 2026, 2:14pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/23 "2026-06-24T14:14:06Z")

</div>

Thanks a lot @hilary.j.oliver for your help. Finally I am able to submit batch job to pbs cluster. Yes that erroneous PBS directive was due to non-compliance with INI format used in flow.cylc.

Just curious, at which stage, CYLC system uses qsub command to submit job script. I did not find qsub string in whole run directory.

```auto
bap001@hpcr8-vis->pwd
/home/bap001/cylc-run/ex10_useplatforms/run18
grep --text -ir qsub *

Is following ssh command uses qsub command behind the scene. Please explain little bit about the work done in this command.
From run18/log/scheduler/log :
2026-06-24T13:48:34Z DEBUG - ['ssh', '-oBatchMode=yes', '-oConnectTimeout=10', 'ppp8', 'env', 'CYLC_VERSION=8.6.3', 'bash', '--login', '-c', '\'exec "$0" "$@"\'', 'cylc', 'jobs-submit', '--debug', '--remote-mode', '--path=/bin', '--path=/usr/bin', '--path=/usr/local/bin', '--path=/sbin', '--path=/usr/sbin', '--path=/usr/local/sbin', '--', '$HOME/cylc-run/ex10_useplatforms/run18/log/job', '1/task1/01']
2026-06-24T13:48:36Z DEBUG - [jobs-submit cmd] cat /home/bap001/cylc-run/ex10_useplatforms/run18/log/job/1/task1/01/job | ssh -oBatchMode=yes -oConnectTimeout=10 ppp8 env CYLC_VERSION=8.6.3 bash --login -c ''"'"'exec "$0" "$@"'"'"'' cylc jobs-submit --debug --remote-mode --path=/bin --path=/usr/bin --path=/usr/local/bin --path=/sbin --path=/usr/sbin --path=/usr/local/sbin -- '$HOME/cylc-run/ex10_useplatforms/run18/log/job' 1/task1/01
```

---

<div class="post-metadata">

**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\
**Post date:** [June 24, 2026, 2:56pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/24 "2026-06-24T14:56:55Z")

</div>

The `qsub` command is hardcoded within the Cylc codebase itself.

- It is called when the job is in the `preparing` state.
- Once the submission has succeeded, the job enters the `submitted` state.
- Once it starts running, the job enters the `running` state.

If you’re interested, here’s the location of Cylc’s PBS support module: [cylc-flow/cylc/flow/job\_runner\_handlers/pbs.py at master · cylc/cylc-flow · GitHub](https://github.com/cylc/cylc-flow/blob/master/cylc/flow/job_runner_handlers/pbs.py). This is generic doesn’t need to be modified, any configuration required can be made in the [`global.cylc[platforms]` section](https://cylc.github.io/cylc-doc/stable/html/reference/config/global.html#global.cylc%5Bplatforms%5D).

---

<div class="post-metadata">

**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\
**Post date:** [June 24, 2026, 7:01pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/25 "2026-06-24T19:01:01Z")

</div>

Thanks @oliver.sanders .

[Previous page](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322.md?page=1)