# CYLC\_\* variables; cannot access in user script

**URL:** <https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322>\
**Category:** Cylc Support\
**Created:** [May 25, 2026, 9:00pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322 "2026-05-25T21:00:51Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\
**Post date:** [May 25, 2026, 9:00pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/1 "2026-05-25T21:00:52Z")

</div>

Hi  
I have top level user script, which may be calling other script. I don’t see these CYLC\_\* variable passed to my top level script. Also I saw, when I submit the script with qsub having PBS directives, my script is not executed correctly . What I am doing wrong?

```auto
flow.cylc:
[[task1]]

 script = ord_soumet ${MY_DEFAULT_PATH}/task1.scr -mach xyz -cm 2G -t 300 -jn pabla -listing ${MY_DEFAULT_PATH}
#script = qsub -l host=xyz ${MY_DEFAULT_PATH}/task1.scr 

 [[[environment]]]
  MY_DEFAULT_PATH = /space/????

task1.scr
#!/bin/bash
set -x
#
# Change to the directory where the job was submitted:
echo "PBS_O_WORKDIR=${PBS_O_WORKDIR}"
echo "CYLC variables in top level script"
echo "CYLC_WORKFLOW_NAME=${CYLC_WORKFLOW_NAME}"
echo "CYLC_WORKFLOW_RUN_DIR=${CYLC_WORKFLOW_RUN_DIR} "
echo "CYLC_TASK_SHARE_DIR=${CYLC_TASK_SHARE_DIR} "

output :
++ echo 'CYLC variables in top level script'
CYLC variables in top level script
++ echo CYLC_WORKFLOW_NAME=
CYLC_WORKFLOW_NAME=
++ echo 'CYLC_WORKFLOW_RUN_DIR= '
CYLC_WORKFLOW_RUN_DIR=
++ echo 'CYLC_TASK_SHARE_DIR= '
CYLC_TASK_SHARE_DIR=
```

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [May 25, 2026, 10:52pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/2 "2026-05-25T22:52:50Z")

</div>

Hi @bpabla50

The job script generated for each task, such as `task1`, literally exports all of those CYLC variables, so there’s no way that they aren’t in your job environment.

Your job script (which you can look at - see the `job` file in the log directory for that task), will look something like this (omitting a lot):

```bash
...
export CYLC_WORKFLOW_NAME=whatever
...
ord_soumet ${MY_DEFAULT_PATH}/task1.scr -mach xyz ...
...

```

The only thing I can think of is, it’s not obvious that your `task1.scr` is being executed directly because you are not doing that - you’re passing it as a command line argument to `ord_soumet` and I have no idea what that does with its arguments.

As a first step, maybe change your `[task1]` script to this:

```ini
script = """
echo "CYLC variables in top level script"
echo "CYLC_WORKFLOW_NAME=${CYLC_WORKFLOW_NAME}"
# ...
ord_soumet ${MY_DEFAULT_PATH}/task1.scr ...
"""

```

You will see (I’m sure) that the CYLC variables are defined there, and because they are exported they will be available to subshells too.

If so, then `ord_soumet` must being doing something strange with your script - maybe it runs `task1.scr` detached from the original environment, e.g. by internal submission to PBS or something??

---

<div class="post-metadata">

**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\
**Post date:** [May 26, 2026, 8:19pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/3 "2026-05-26T20:19:42Z")

</div>

Thanks @hilary.j.oliver .  
As per your advice, I looked at jobfile, yes, CYLC\_\* variables are defined and exported there. Question is that job submitting tool(e.g. qsub, ord\_soumet…wrapper around qsub) does not pass these CYLC\_\* variable to task1.scr. Please note that here task1.scr is running in batch mode instead of running interactively.

```auto
job file:
    export CYLC_WORKFLOW_NAME="ex7_workflow"
    export CYLC_WORKFLOW_NAME_BASE="ex7_workflow"
    export CYLC_WORKFLOW_UUID="1a8e98a1-dd43-44de-af44-3719781438bb"

    # CYLC TASK ENVIRONMENT:
    export CYLC_TASK_COMMS_METHOD=zmq
    export CYLC_TASK_JOB="1/task1/01"
    export CYLC_TASK_NAMESPACE_HIERARCHY="root task1"
    export CYLC_TASK_TRY_NUMBER=1
    export CYLC_TASK_FLOW_NUMBERS=1
}
cylc __job__ inst__user_env() {
    # TASK RUNTIME ENVIRONMENT:
    export MY_DEFAULT_PATH
    MY_DEFAULT_PATH="/space/??"
}

cylc __job__ inst__script() {
# SCRIPT:
ord_soumet ${MY_DEFAULT_PATH}/task1.scr -mach xyz -cm 2G -t 300 -jn pabla -listing ${MY_DEFAULT_PATH}
}

CYLC_RUN_DIR="${CYLC_RUN_DIR:-$HOME/cylc-run}"
. "${CYLC_RUN_DIR}/${CYLC_WORKFLOW_ID}/.service/etc/job.sh"
cylc __job__ main

```

I tried qsub command outside cylc, and saw it is not passing CYLC\_\* variables to script..

```auto
$ export CYLC_WORKFLOW_NAME=ex8_workflow
$ qsub -l walltime=00:05:00 ./task1.scr_withpbs

$ more ./task1.scr_withpbs
set -x
#
# Job Name:
#PBS -l walltime=00:05:00
#PBS -N pabla
# Specify the output file:
#PBS -o task1_output.txt
# Specify the error file:
#PBS -e task1_error.txt
# Number of nodes and processors per node:
#PBS -l nodes=1:ppn=1
# Walltime (hh:mm:ss):
# Memory requirements:
#PBS -l mem=2gb

```

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [May 27, 2026, 1:52am UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/4 "2026-05-27T01:52:04Z")

</div>

> [@bpabla50](#):
>
> Question is that job submitting tool(e.g. qsub, ord\_soumet…wrapper around qsub) does not pass these CYLC\_\* variable to task1.scr. Please note that here task1.scr is running in batch mode instead of running interactively.

Right, that explains your problem.

PBS does not execute jobs in the same environment that you submit the job from. After all, it will almost certainly run the job on an entirely different node or platform that literally cannot see your environment. It is up to you to ensure that all dynamic variables (i.e. changing per job, as opposed to statically defined in your interactive shell environment) required by your PBS job are defined in the PBS job script itself.

Obviously I don’t know what your `ord_soumet` script does, or how complicated it is, but generally speaking it should not be doing internal job submissions to PBS. It is Cylc’s role, as a workflow manager, to determine where and how to run workflow jobs, and then to track their status.

Here’s a simple example to illustrate the problem. Say I have a script that runs a scientific model called `model` with a command line `model -xy $INPUT`.

To run it myself I might create a PBS job script `job.sh` like this:

```bash
#!/usr/bin/env bash
# PBS - l walltime=...
# PBS ...
INPUT=/path/to/input
model -xy $INPUT

```

and submit it to PBS: `qsub job.sh`

To run it with Cylc I would NOT do this (imagine `job1.sh` has the INPUT line removed):

```ini
[runtime]
    [[model]]
        script = """
            INPUT=/path/to/input
            qsub /path/to/job1.sh # ERROR - don't do this!
        """

```

That will NOT work because:

1. the PBS job runs detached on another host and will not see `$INPUT`
2. the `qsub` command exits with success status as soon as the job is submitted to PBS, so it will look to Cylc like the job finished before it even started running

Instead, get Cylc to handle job execution via PBS:

```ini
[runtime]
   [[model]]
       platform = my-pbs-platform # see Cylc platform docs
       script = """
           INPUT=/path/to/input
           model -xy $INPUT
       """

```

Now the Cylc-generated job script for task `model` is a PBS job script (like your hand-written `job.sh`) and Cylc will submit it to PBS for you, and track its progress from submitted to running to succeeded or failed.

---

<div class="post-metadata">

**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\
**Post date:** [May 28, 2026, 7:13pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/5 "2026-05-28T19:13:52Z")

</div>

Thanks @hilary.j.oliver for clarifying that my script should not have PBS directives, and job should not be submitted with system job submisson tools like qsub etc

I read about platforms here [pbs — Cylc 8.6.3 documentation](https://cylc.github.io/cylc-doc/latest/html/user-guide/task-implementation/job-runner-handlers/cylc.flow.job_runner_handlers.pbs.html)  
Things did not work, appreciate further help with complete flow.cylc example to submit job to compute node of HPC cluster using PBS PRO, and job requirement say a) memory= 2GB b)cpus=1 c)time limit=5min d) queue=development e) jobname=test f)listings\_dir=some\_path. Is it possible to define all the job related parametes including cluster name in flow.cylc without making use of global.cylc.

```auto
more job-activity.log
[jobs-submit cmd] (platform not defined)
[jobs-submit ret_code] 1
[jobs-submit err] No matching platform "my_frontend" found  

more ~/.cylc/global.cylc
 [platforms]
  [[my_frontend]]
  hosts = xyz
  login_command = ssh
  shell = /bin/bash
  batch_system = pbs  

more flow.cylc
[scheduler]
[task parameters]
  num = 1..6
 allow implicit tasks = True
 
[scheduling]
  initial cycle point = 1
    final cycle point = 2
  cycling mode = integer
# 
 [[graph]]
   R1 = """ task1 """
   
[runtime] 
  
 [[task1]]
   platform = my_frontend
   execution time limit = PT5M
   script = """
#load your packages here....
   ${MY_DEFAULT_PATH}/task1.scr
  """
 [[[environment]]]
  MY_DEFAULT_PATH = some_path
```

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [May 29, 2026, 1:01am UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/6 "2026-05-29T01:01:18Z")

</div>

> [@bpabla50](#):
>
> my script should not have PBS directives, and job should not be submitted with system job submisson tools like qsub et

To clarify that a bit more, for other readers - your task definitions should have PBS directives, and Cylc itself will use `qsub` to submit your tasks to PBS.

> [@bpabla50](#):
>
> `[jobs-submit err] No matching platform "my_frontend" found `

Your `global.cylc` is not being picked up by Cylc, because it isn’t in one of the legal locations for the global config file.

See [Global Configuration — Cylc 8.6.3 documentation](https://cylc.github.io/cylc-doc/stable/html/reference/config/global.html#global.cylc)

Move it to the right location, and then type `cylc config` to see if your global settings are being parsed.

Another point about your platform definition, you probably need to define `install target` - it is just an arbitrary label that needs to be the same for all Cylc platforms that see the same filesystem.

---

<div class="post-metadata">

**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\
**Post date:** [June 8, 2026, 5:32pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/7 "2026-06-08T17:32:36Z")

</div>

Hi @hilary.j.oliver  
What I am still doing wrong? in what file, one specify scheduler type is PBS? Thanks for any help?

```auto
->pwd
/home/user1/.cylc/flow
->more global.cylc
 [platforms]
  [[my_frontend]]
   hosts = mach1

->more job-activity.log
[jobs-submit cmd] (platform not defined)
[jobs-submit ret_code] 1
[jobs-submit err] No matching platform "my_frontend" found

->more task1.scr
#!/bin/bash
set -x
#
# Change to the directory where the job was submitted:
hn=`hostname`
echo "hn=${hn}"
echo "PBS_O_WORKDIR=${PBS_O_WORKDIR}"
echo "CYLC variables in top level script"
echo "CYLC_WORKFLOW_NAME=${CYLC_WORKFLOW_NAME}"
```

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [June 9, 2026, 1:46am UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/8 "2026-06-09T01:46:52Z")

</div>

Hi @bpabla50

Sorry if I was not quite clear enough above, but:

> [@hilary.j.oliver](#):
>
> Your `global.cylc` is not being picked up by Cylc, because it isn’t in one of the legal locations for the global config file.

From what you’ve said, you have put your global config in `~/.cylc/global.cylc` which is not one of the expected locations for the file. The link I gave above lists the expected locations. The usual one would be `~/.cylc/flow/8/global.cylc` (it goes under the `flow` sub-directory because this config file is part of the `cylc/flow` (scheduler) package. Cylc has other packages too such as `cylc/uiserver`…).

If you put the file in the right location, just type `cylc config` to verify that it is working - that command (with no arguments) will print out the parsed global configuration from all allowed locations.

> [@bpabla50](#):
>
> `batch_system = pbs`

Also: `batch_system` is not a valid global config item in Cylc 8, not sure where you got that from. It should be `job runner = pbs`. Check the platforms documentation again to make sure you have defined the right config items.

Hilary

---

<div class="post-metadata">

**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\
**Post date:** [June 9, 2026, 9:14pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/9 "2026-06-09T21:14:02Z")

</div>

Hi @hilary.j.oliver 🙂  
Getting close. Thanks for your help.

> [@hilary.j.oliver](#):
>
> Also: `batch_system` is not a valid global config item in Cylc 8, not sure where you got that from.

I think, I got it from chatgpt or some other AI tool.  
Until, I was not using install target, I was getting weird error message

```auto
026-06-09T17:25:21Z INFO - platform: my_frontend - remote init (on ppp8)
2026-06-09T17:25:23Z CRITICAL - An uncaught error caused Cylc to shut down.
    If you think this was an issue in Cylc, please report the following traceback to the developers.
    https://github.com/cylc/cylc-flow/issues/new?assignees=&labels=bug&template=bug.md&title=;
2026-06-09T17:25:23Z ERROR - argument of type 'NoneType' is not a container or iterable

```

Once I defined install target, and used “localhost”, it went further; is “localhost” can be replaced with my real cluster name say “xyz”, where my pbs job will run?  
instead of real cluster name “ppp”.

Last error message, is that it not liking one of PBS directive, where I am asking for 1cpu per node. Is all keywords in `[[directives]]` are same as PBS directives, but in different syntax.

```auto
<bap001@hpcr8-vis->pwd
/home/bap001/.cylc/flow/8

<bap001@hpcr8-vis->more global.cylc

 [platforms]
  [[my_frontend]]
# hosts = ppp8
     hosts = localhost
     job runner = pbs
     install target = localhost
flow.cylc :
[runtime]
  [[default]]
    platform = my_frontend
    execution time limit = PT5M
# 
 [[task1]]
   platform = my_frontend
   execution time limit = PT5M
        [[[directives]]]
            -N = pabla
            -o = task1_output.txt
            -e = task1_error.txt
            -l = nodes=1:ppn=1 ...looks like, it does not like it
            -l = mem=2gb
            -M = my_valid_email

```

task1.scr

```auto
!/bin/bash
set -x
#
echo "start of script..."
# Change to the directory where the job was submitted:
cd $PBS_O_WORKDIR
echo "PBS_O_WORKDIR=${PBS_O_WORKDIR}"
echo "CYLC variables in top level script"
echo "CYLC_WORKFLOW_NAME=${CYLC_WORKFLOW_NAME}"

```

Can you please clarify, when to use “loacalhost”, and when to use real cluster name, i prefer latter.

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [June 10, 2026, 12:28am UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/10 "2026-06-10T00:28:32Z")

</div>

> [@bpabla50](#):
>
> I think, I got it from chatgpt or some other AI tool.

Suggest consulting the Cylc docs 😁

> [@bpabla50](#):
>
> Until, I was not using install target, I was getting weird error message

You appear to have uncovered a bug (or at least, a config error that should be caught and reported gracefully) - can you reproduce it, and tell us your Cylc version and the `global.cylc` content (presumably) that triggers the error?

> [@bpabla50](#):
>
> Once I defined install target, and used “localhost”, it went further; is “localhost” can be replaced with my real cluster name say “xyz”, where my pbs job will run?  
> instead of real cluster name “ppp”  
> …  
> Can you please clarify, when to use “loacalhost”, and when to use real cluster name, i prefer latter.

The current documentation on `install target` is less than clear, according to some users (including myself, as it turns out). We have an in coming docs change to fix that. In the meantime, `install target` is just **an arbitrary label** , but the important thing is **it represents a filesystem** so all Cylc platforms (i.e. as defined under `[platforms]` in `global.cylc`) must have the same `install target` - this allows Cylc to determine whether or not it has already installed workflow files on a platform, in workflows that send jobs to multiple platforms.

As a minimal example, on an HPC cluster you might send some jobs to PBS (via one platform def), and run some trivial tasks as background jobs on your login node (via another platfrom def) - if the login node sees the same files as the compute nodes, both platforms should have the same `install target` label.

If you get this wrong and give several platforms on the same filesystem different `install target`s, you will get “platform init” errors in the workflow, when Cylc finds files that should not already be present when it first submits a job to a platform.

Long story short, you can use `install target = localhost` if you like, but Cylc just sees it in this context as an arbitrary label, not as a hostname, so IMO that’s a bit misleading. (And, you would have to use `localhost` for all other platforms that see the same filesystem too). So yes, better to use your cluster name, `install target = ppp` (so long as all nodes on the cluster see the same filesystem!).

> [@bpabla50](#):
>
> ```auto
> -l = nodes=1:ppn=1 ...looks like, it does not like it
> -l = mem=2gb
> 
> ```

Everything in a `flow.cylc` file is a config item of the form `key = value`. In a `[directives]` section these config items have to be translated into actual PBS directives. Here you have defined `-l` (as a config item) twice, so (as is the way with config items) the second definition overrides the first - so PBS will never see your `nodes` and `ppn` options.

How to handle PBS directives like this is explained here: [Job Submission and Management — Cylc 8.6.3 documentation](https://cylc.github.io/cylc-doc/stable/html/user-guide/task-implementation/job-submission.html#directives-section-quirks-pbs-sge)

---

<div class="post-metadata">

**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\
**Post date:** [June 10, 2026, 7:58pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/11 "2026-06-10T19:58:26Z")

</div>

Regarding potential bug I found, I will send later my examples from yesterday. Today, I slighly revised my global.cylc, and flow.cylc, I got the same error even with install target.

```auto
# scheduler/log:
2026-06-10T19:20:34Z INFO - platform: my_frontend - remote init (on ppp8)
2026-06-10T19:20:36Z CRITICAL - An uncaught error caused Cylc to shut down.
2026-06-10T19:20:34Z INFO - Cylc version: 8.6.3

    If you think this was an issue in Cylc, please report the following traceback to the developers.
    https://github.com/cylc/cylc-flow/issues/new?assignees=&labels=bug&template=bug.md&title=;
2026-06-10T19:20:36Z ERROR - argument of type 'NoneType' is not a container or iterable
    Traceback (most recent call last):
      File "/fs/site8/eccc/aq/r1/bap001/cylc_viapixi/.pixi/envs/default/lib/python3.14/site-packages/cylc/flow/scheduler_cli.py", line 753, in cylc_play
        asyncio.get_running_loop()

```

```ini
# global.cylc
<bap001@ppp8->pwd
/home/bap001/.cylc/flow/8
<bap001@ppp8->more global.cylc
 
[platforms]
  [[my_frontend]]
     hosts = ppp8
     job runner = pbs
# string right to = sign can be anything e.g. localhost, ppp8 etc.
# It is just a lable for FS used by my_frontend
     install target = sitestore8 

```

```ini
# flow.cylc:
[[graph]]
  R1 = """ task1 """
[runtime]
 [[task1]]
   platform = my_frontend
   execution time limit = PT5M
        [[[directives]]]
            -N = pabla
            -o = task1_output.txt
            -e = task1_error.txt
            -l = select=1:ncpus=1:mem=2gb
            -M = balbir.pabla@ec.gc.ca
            -m = abe
            -q = development
   script = """
#load your packages here....
   ${MY_DEFAULT_PATH}/task1.scr
[[[environment]]]
  MY_DEFAULT_PATH = /space/hall/?????
  """

```

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [June 10, 2026, 11:19pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/12 "2026-06-10T23:19:00Z")

</div>

OK so the error is occurring at runtime during “remote init”, presumably when the first job is submitted to your platform (which is when Cylc initializes the platform, and installs files if the install target has not already been done). Unfortunately, it looks like the exception has bubbled up to a higher level, obscuring the code block that it originated in. Can you reproduce again please, running the scheduler in debug mode (e.g. `cylc vip --debug`) and see if that gives us more information?

As an aside, you don’t need these directives, in fact they might be unhelpful:

> [@bpabla50](#):
>
> ```auto
> -o = task1_output.txt
> -e = task1_error.txt
> 
> ```

Cylc automatically tells PBS to use the standard Cylc job log directory `job.out` and `job.err` files for job stdout and stderr.

---

<div class="post-metadata">

**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\
**Post date:** [June 11, 2026, 2:41pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/13 "2026-06-11T14:41:07Z")

</div>

Thanks @hilary.j.oliver .  
Running with debug option gave useful information, cylc is not found.  
Looking into, why it is not finding “cylc” command with platform setup, while no issue running interactively. Please note that, I have installed cylc under pixi environment. Each cylc command should be preceeded by “pixi run”, e.g.  
pixi run cylc vip --debug .  
Any advice, how I can prepend “pixi run” for every cylc command, when things are run using platforms.

```auto
rundir/log/scheduler/log:
2026-06-11T13:59:36Z INFO - Scheduler: url=tcp://ppp8login-02-ib:43036 pid=2919507
2026-06-11T13:59:36Z INFO - Workflow publisher: url=tcp://ppp8login-02-ib:43099
2026-06-11T13:59:36Z INFO - Run: (re)start number=1, log rollover=1
2026-06-11T13:59:36Z INFO - Cylc version: 8.6.3
2026-06-11T13:59:36Z INFO - Run mode: live
2026-06-11T13:59:36Z INFO - Initial point: 1
2026-06-11T13:59:36Z INFO - Final point: 2
2026-06-11T13:59:36Z INFO - Cold start from 1
2026-06-11T13:59:36Z INFO - New flow: 1 (original flow from 1) 2026-06-11T13:59:36
2026-06-11T13:59:36Z DEBUG - Runahead: base point 1
2026-06-11T13:59:36Z DEBUG - Runahead limit: 1
2026-06-11T13:59:36Z DEBUG - [1/task1:waiting(runahead)] added to the n=0 window
2026-06-11T13:59:36Z INFO - [1/task1:waiting(runahead)] => waiting
2026-06-11T13:59:36Z INFO - [1/task1:waiting] => waiting(queued)
2026-06-11T13:59:36Z DEBUG - Loaded main loop plugin "health check":
    * health_check
2026-06-11T13:59:36Z DEBUG - Loaded main loop plugin "reset bad hosts":
    * reset_bad_hosts
2026-06-11T13:59:36Z INFO - [1/task1:waiting(queued)] => waiting
2026-06-11T13:59:36Z INFO - [1/task1:waiting] => preparing
2026-06-11T13:59:36Z INFO - platform: my_frontend - remote init (on ppp8)
2026-06-11T13:59:36Z DEBUG - main_loop [run] cylc.flow.main_loop.health_check:health_check
2026-06-11T13:59:36Z DEBUG - main_loop [end] cylc.flow.main_loop.health_check:health_check (0.000s)
2026-06-11T13:59:36Z DEBUG - main_loop [run] cylc.flow.main_loop.reset_bad_hosts:reset_bad_hosts
    * reset_bad_hosts
2026-06-11T13:59:36Z INFO - [1/task1:waiting(queued)] => waiting
2026-06-11T13:59:36Z INFO - [1/task1:waiting] => preparing
2026-06-11T13:59:36Z INFO - platform: my_frontend - remote init (on ppp8)
2026-06-11T13:59:36Z DEBUG - main_loop [run] cylc.flow.main_loop.health_check:health_check
2026-06-11T13:59:36Z DEBUG - main_loop [end] cylc.flow.main_loop.health_check:health_check (0.000s)
2026-06-11T13:59:36Z DEBUG - main_loop [run] cylc.flow.main_loop.reset_bad_hosts:reset_bad_hosts
2026-06-11T13:59:36Z DEBUG - main_loop [end] cylc.flow.main_loop.reset_bad_hosts:reset_bad_hosts (0.000s)
2026-06-11T13:59:36Z DEBUG - ['ssh', '-oBatchMode=yes', '-oConnectTimeout=10', 'ppp8', 'env', 'CYLC_VERSION=8.6.3', 'bash', '--login', '-c', '\'exec "$0" "$@"\'', 'cylc', 'remote-init', '-v', '-v', 'sitestore8', '$HOME/cylc-run/ex10_useplatforms/run2']
2026-06-11T13:59:37Z DEBUG - [remote-init cmd] cat '<tempfile.SpooledTemporaryFile object at 0x1468857fbcd0>' | ssh -oBatchMode=yes -oConnectTimeout=10 ppp8 env CYLC_VERSION=8.6.3 bash --login -c ''"'"'exec "$0" "$@"'"'"'' cylc remote-init -v -v sitestore8 '$HOME/cylc-run/ex10_useplatforms/run2'
    [remote-init ret_code] 0
    [remote-init out] (INFO) Using package (fully qualified) pixi_0.63.2_all from domain /fs/ssm/eccc/cmd/cmds/apps/pixi/202602/00
   [remote-init ret_code] 0
    [remote-init out] (INFO) Using package (fully qualified) pixi_0.63.2_all from domain /fs/ssm/eccc/cmd/cmds/apps/pixi/202602/00
    [remote-init err]
    === ordenv: version 4.5 ===
    === ordenv: master/default ===
    === ordenv: system/init ===
    === ordenv: system/site ===
    === ordenv: sourcing site:20260326 ===
    info: jobctl cell (ppp8)
    === ordenv: system/comm ===
    === ordenv: sourcing comm:eccc/20260205 ===
    === ordenv: system/group ===
    === ordenv: sourcing group:/fs/ssm/eccc/mrd/ordenv/20260326/profile/group ===
    === ordenv: sourcing group: SCIENCE domain (ppp8) ===
    === ordenv: system/user ===
    === ordenv: sourcing user:default/post ===
    === ordenv: completed ===
    (INFO) Profile load time: 0.476s
    cylc: line 1: exec: cylc: not found
    cleaning up temporary directories
    removing TMPDIR = /tmp/bap001/2919529
2026-06-11T13:59:37Z ERROR - platform: my_frontend - initialisation did not complete
    COMMAND:
        ssh -oBatchMode=yes -oConnectTimeout=10 ppp8 env \
            CYLC_VERSION=8.6.3 bash --login -c 'exec "$0" "$@"' cylc \
            remote-init -v -v sitestore8 \
            $HOME/cylc-run/ex10_useplatforms/run2
    RETURN CODE:
        0
    STDOUT:
        (INFO) Using package (fully qualified) pixi_0.63.2_all from domain /fs/ssm/eccc/cmd/cmds/apps/pixi/202602/00
    STDERR:
        === ordenv: version 4.5 ===
       === ordenv: completed === ...finished loading profile
        (INFO) Profile load time: 0.476s
        cylc: line 1: exec: cylc: not found
        cleaning up temporary directories
        removing TMPDIR = /tmp/bap001/2919529
2026-06-11T13:59:37Z DEBUG - [1/task1/01:preparing] host=ppp8
2026-06-11T13:59:37Z ERROR - [jobs-submit cmd] (init ppp8)
    [jobs-submit ret_code] 1
    [jobs-submit err] REMOTE INIT FAILED
2026-06-11T13:59:37Z ERROR - [jobs-submit cmd] (remote init)
    [jobs-submit ret_code] 1
2026-06-11T13:59:37Z ERROR - [1/task1/01:preparing] submission failed
2026-06-11T13:59:37Z INFO - [1/task1/01:preparing] => submit-failed
2026-06-11T13:59:37Z WARNING - [1/task1/01:submit-failed] did not complete the required outputs:
    ⨯ ┆ succeeded
2026-06-11T13:59:37Z DEBUG - 1/task1 -triggered off [] in flow 1
2026-06-11T13:59:37Z ERROR - Incomplete tasks:
    * 1/task1 did not complete the required outputs:
2026-06-11T13:59:37Z ERROR - Incomplete tasks:
    * 1/task1 did not complete the required outputs:
      ⨯ ┆ succeeded
2026-06-11T13:59:37Z CRITICAL - Workflow stalled
2026-06-11T13:59:37Z WARNING - PT1H stall timer starts NOW
2026-06-11T14:09:36Z DEBUG - main_loop [run] cylc.flow.main_loop.health_check:health_check
2026-06-11T14:09:36Z DEBUG - main_loop [end] cylc.flow.main_loop.health_check:health_check (0.000s)
2026-06-11T14:19:37Z DEBUG - main_loop [run] cylc.flow.main_loop.health_check:health_check
2026-06-11T14:19:37Z DEBUG - main_loop [end] cylc.flow.main_loop.health_check:health_check (0.000s)
2026-06-11T14:29:36Z DEBUG - main_loop [run] cylc.flow.main_loop.reset_bad_hosts:reset_bad_hosts
2026-06-11T14:29:36Z DEBUG - main_loop [end] cylc.flow.main_loop.reset_bad_hosts:reset_bad_hosts (0.000s)
2026-06-11T14:29:37Z DEBUG - main_loop [run] cylc.flow.main_loop.health_check:health_check
2026-06-11T14:29:37Z DEBUG - main_loop [end] cylc.flow.main_loop.health_check:health_check (0.000s)
```

---

<div class="post-metadata">

**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\
**Post date:** [June 11, 2026, 3:49pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/14 "2026-06-11T15:49:10Z")

</div>

Hi @bpabla50 👋,

#### Distributed Installation

Cylc is a distributed system.

You can run a Cylc workflow locally, and have it run all of it’s jobs locally, in which case you could just `pixi run cylc vip my-workflow` and away you go.

However, for distributed deployments where workflows are submitting jobs to remote platforms / batch systems, it is necessary to install Cylc in multiple places and find a way to intercept `cylc` command calls so you can redirect them towards your Cylc deployment(s).

This “interception” is done using a “wrapper script”. Various tools use/create them, including Pixi.

#### Wrapper Scripts

If you’re deploying Cylc via Pixi, then a basic “wrapper script” might look like this (untested!):

```auto
#!/bin/sh

exec pixi run cylc "$@"

```

You would name this script `cylc` and place it in a directory which is included in the default `$PATH`. That way any `cylc` call from any regular shell will be directed via this command, which invokes it via the Pixi environment.

This is a basic example of a wrapper script. In addition to Cylc, you may also require wrapper scripts for the `rose` and `isodatetime` commands if you are using them in your workflows.

#### The Cylc Wrapper Script

Distributed Cylc deployments are a common practice, so we have written a wrapper script which is designed to cater to most circumstances.

The Cylc wrapper script:

- Can support multiple parallel Cylc installations (which means you can install new Cylc versions without breaking running workflows). Switch between them using the `CYLC_VERSION` environment variable.
- Can work with Conda/Mamba environments or Python virtual environments.
- Handles some issues you might face if writing a wrapper script yourself.
- **Doesn’t** activate the environment, or use commands like `conda run` / `pixi run` (which active environments implicitly). This is actually the correct way to do it with Cylc because activating the environment would cause background jobs to run within it which is generally undesireable.

There are some details in the installation guide in the documentation:

> **[cylc - Installation — Cylc 8.6.3 documentation](https://cylc.github.io/cylc-doc/stable/html/installation.html#managing-environments)**
>
> For distributed and multi-user installation we recommend using Conda and installing Cylc components only where required.

Go through the comments at the top of the file and adjust any variables as necessary, then stick the script somewhere in `$PATH` and you’re done. If necessary, make a copies of the wrapper script called `rose` and `isodatetime` in the same directory.

The `CYLC_HOME_ROOT` variable should point at the directory which contains your Pixi Cylc environments (name them `cylc-<cylc-version>` to enable version switching).

For example, if you get this:

```auto
$ cylc version --long
8.6.4 (/site/apps/cylc-8.6.4-2/bin/cylc)

```

Then you would set:

```auto
CYLC_HOME_ROOT=/site/apps

```

We haven’t yet added explicit support for Pixi environments in this script yes, however, I suspect these are really just Conda environments in disguise so our existing support will probably work.

If Pixi installs `cylc` using a different directory pattern to:

```auto
$CYLC_HOME_ROOT/cylc-<version>/bin/cylc

```

Let us know what it is, and we’ll guide you through adapting the wrapper script.

---

<div class="post-metadata">

**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\
**Post date:** [June 17, 2026, 8:02pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/15 "2026-06-17T20:02:27Z")

</div>

Thanks @oliver.sanders

-Regrading distributed deployments

I don’t think, this is applicable to me. This is HPC cluster, where we login to login node for interactive work, and from there we submit jobs to compute nodes using qsub (uses pbs pro), Since we are using global parallel file system(GPFS), place I installed cylc under pixi is visible from all nodes of cluster( login as well as compute nodes).  
-also concept of wrapper script did not work for me either

```auto
I tried including following lines in my task1.scr, even it contains that cylc command not found. I grepped for "qsub" in run directory, it could not find it. Please clarify, what steps you take to prepare and submit job?
task1.scr:
#!/bin/bash
set -x
. ~/.profile
echo "pab..start of task1.scr"
 echo "pab...cding to cylc install directory"

echo "pab ..loading pixi ssm package"

. r.load.dot /fs/ssm/eccc/cmd/cmds/apps/pixi/202602/00/pixi_0.63.2_all ...loads pixi
#exec pixi shell
echo "pab..activating pixi env"
 eval "$(pixi shell-hook)"
echo "pab ..path of cylc"
which cylc
echo "pab ... printing cylc version"
cylc version --long

#
echo "start of script..."
# Change to the directory where the job was submitted:
cd $PBS_O_WORKDIR
echo "PBS_O_WORKDIR=${PBS_O_WORKDIR}"
echo "CYLC variables in top level script"
echo "CYLC_WORKFLOW_NAME=${CYLC_WORKFLOW_NAME}"

log/scheduler/log :
   TypeError: argument of type 'NoneType' is not a container or iterable
2026-06-17T19:52:34Z CRITICAL - Workflow shutting down - argument of type 'NoneType' is not a container or iterable
2026-06-17T19:52:34Z DEBUG - [remote-init cmd] cat '<tempfile.SpooledTemporaryFile object at 0x14ca939efb80>' | ssh -oBatchMode=yes -oConnectTimeout=10 ppp8 env CYLC_VERSION=8.6.3 bash --login -c ''"'"'exec "$0" "$@"'"'"'' cylc remote-init -v -v sitestore8 '$HOME/cylc-run/ex10_useplatforms/run14'
    [remote-init ret_code] 0
    [remote-init err]
    === ordenv: version 4.5 ===
 === ordenv: sourcing user:default/post ===
    === ordenv: completed ===
    (INFO) Profile load time: 0.382s ..finished loading my profile
   cylc: line 1: exec: cylc: not found ....is it coming from CYLC system, it cannot be from code
    cleaning up temporary directories
-please above ssh command, what you guys are trying to achieve here? why qsub was not used to submit job?

```

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [June 17, 2026, 9:07pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/16 "2026-06-17T21:07:55Z")

</div>

> [@bpabla50](#):
>
> don’t think, this is applicable to me. This is HPC cluster, …

Cylc is often (mostly!) used in HPC clusters like this. The shared filesystem means you only need to install Cylc once, which is nice, but you still need to make sure that the executable is automatically accessible (i.e., present in the executable search path `$PATH`) to non-interactive shells on all nodes.

Even if you just get run Cylc on a login node and get it to submit jobs to PBS locally, the task jobs - which are totally detached from your login session - still need to be able to run `cylc` commands, e.g. to send job status message back to their parent Cylc scheduler on the login node.

So the idea is you make a wrapper script called `cylc` that does whatever is needed to invoke the real `cylc` commands (e.g. `pixi run cylc` after loading whatever environment is needed to access `pixi`) and make sure the wrapper is automatically in your `$PATH`.

If you’re doing this yourself (as opposed to central deployment by system admins) you might put your wrapper in `$HOME/bin/cylc` and make sure that your login scripts (e.g. `.bashrc`) add `$HOME/bin` to your `$PATH`.

After doing this, you should be able to successfully invoke `cylc` commands via non-interactive remote shells e.g. `ssh <OTHER-NODE> cylc version --long`. (Or even `ssh localhost cylc version --long`).

---

<div class="post-metadata">

**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\
**Post date:** [June 22, 2026, 8:35pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/17 "2026-06-22T20:35:13Z")

</div>

Thanks @hilary.j.oliver .  
Sorry, I am still not able to submit task to PBS cluster. Getting remote init failed. Thanks for any suggestion.

```ini
<bap001@ppp8->more ~/.cylc/flow/8/global.cylc
 [platforms]
  [[my_frontend]]
     hosts = ppp8
     job runner = pbs
# string right to = sign can be anything e.g. localhost, ppp8 etc. It is just a lable for FS used by my_frontend
     install target = sitestore8 

```

```bash
<bap001@ppp8->more $HOME/bin/cylc
#!/bin/bash
# This is a wrapper script for cylc
cd /space/hall8/sitestore/eccc/aq/r1/bap001/cylc_viapixi
exec pixi run cylc "$@"

```

```bash
# == in my login profile script
# for cylc
. r.load.dot /fs/ssm/eccc/cmd/cmds/apps/pixi/202602/00/pixi_0.63.2_all
export PATH=${HOME}/bin:${PATH}

```

```ini
# === flow.cylc
[[task1]]
   platform = my_frontend
   execution time limit = PT5M
        [[[directives]]]
            -N = pabla
            -l = select=1:ncpus=1:mem=2gb
            -M = balbir.pabla@ec.gc.ca
            -m = abe
            -q = development
   script = """
   ${MY_DEFAULT_PATH}/task1.scr
 [[[environment]]]
  MY_DEFAULT_PATH = /space/hall8/sitestore/eccc/aq/r1/bap001/cylc_viapixi/ex10_useplatforms

```

```ini
# ===log/scheduler/log
2026-06-22T16:01:45Z DEBUG - ['ssh', '-oBatchMode=yes', '-oConnectTimeout=10', 'ppp8', 'env', 'CYLC_VERSION=8.6.3', 'bash', '--login', '-c', '\'exec "$0" "$@"\'', 'cylc', 'remote-init', '-v', '-v', 'sitestore8', '$HOME/cylc-run/ex10_useplatforms/run15']
2026-06-22T16:01:47Z DEBUG - [remote-init cmd] cat '<tempfile.SpooledTemporaryFile object at 0x151fbbb01cf0>' | ssh -oBatchMode=yes -oConnectTimeout=10 ppp8 env CYLC_VERSION=8.6.3 bash --login -c ''"'"'exec "$0" "$@"'"'"'' cylc remote-init -v -v sitestore8 '$HOME/cylc-run/ex10_useplatforms/run15'
    [remote-init ret_code] 0
    [remote-init out]
    (INFO) Using package (fully qualified) pixi_0.63.2_all from domain /fs/ssm/eccc/cmd/cmds/apps/pixi/202602/00
    REMOTE INIT FAILED
    Unexpected key directory exists: /home/bap001/cylc-run/ex10_useplatforms/run15/.service/client_public_keys Check global.cylc install target is configured correctly for this platform.
    [remote-init err]

```

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [June 22, 2026, 10:11pm UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/18 "2026-06-22T22:11:51Z")

</div>

I think you’re almost there!

> [@bpabla50](#):
>
> ```auto
> REMOTE INIT FAILED
> Unexpected key directory exists: /home/bap001/cylc-run/ex10_useplatforms/run15/.service/client_public_keys 
> Check global.cylc install target is configured correctly for this platform.
> 
> ```

This indicates Cylc unexpectedly found workflow files already installed on this platform.

It probably just means the platform that you’re installing and starting the workflow from (e.g. a login node) is on the same filesystem as your `my_frontend` job platform, but you have not defined the same install target for both.

If all your nodes see the same filesystem as `my_frontend` you can just add this in your `global.cylc`:

```ini
[platforms]
    [[localhost]]
        install target = sitestore8

```

(In Cylc global config, `localhost` just means local to the Cylc scheduler, i.e. where your schedulers run, such as on a login node if you don’t have pool of Cylc run hosts).

Now Cylc will know that (e.g.) your login node is on the same filesystem as your `my_frontend` job platform, so it won’t try to install files on the job platform when you first run a job there (and it won’t be surprised to find files installed there already).

I’m pretty sure that will fix it.

---

<div class="post-metadata">

**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\
**Post date:** [June 23, 2026, 1:28am UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/19 "2026-06-23T01:28:13Z")

</div>

Thanks, it went further, but failed at this point

```auto
===log/scheduler/log
   === ordenv: completed ===
    (INFO) Profile load time: 0.518s
2026-06-23T01:16:04Z ERROR - [jobs-submit cmd] ssh -oBatchMode=yes -oConnectTimeout=10 ppp8 env CYLC_VERSION=8.6.3 bash --login -c ''"'"'exec "$0" "$@"'"'"'' cylc jobs-submit --debug --remote-mode --path=/bin --path=/usr/bin --path=/usr/local/bin --path=/sbin --path=/usr/sbin --path=/usr/local/sbin -- '$HOME/cylc-run/ex10_useplatforms/run16/log/job' 1/task1/01
    [jobs-submit ret_code] 1
    [jobs-submit out] 2026-06-23T01:16:03Z|1/task1/01|1|None
2026-06-23T01:16:04Z ERROR - [1/task1/01:preparing] submission failed
2026-06-23T01:16:04Z INFO - [1/task1/01:preparing] => submit-failed
2026-06-23T01:16:04Z WARNING - [1/task1/01:submit-failed] did not complete the required outputs:
    ⨯ ┆ succeeded
2026-06-23T01:16:04Z ERROR - Incomplete tasks:
    * 1/task1 did not complete the required outputs:
      ⨯ ┆ succeeded
2026-06-23T01:16:04Z CRITICAL - Workflow stalled
2026-06-23T01:16:04Z WARNING - PT1H stall timer starts NOW

== all other files are same except global.cylc
bap001@ppp8->more ~/.cylc/flow/8/global.cylc
 [platforms]
  [[localhost]]
    install target = sitestore8
  [[my_frontend]]
     hosts = ppp8
     job runner = pbs
# string right to = sign can be anything e.g. localhost, ppp8 etc. It is just a lable for FS used by my_fronte
nd
     install target = sitestore8
```

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [June 23, 2026, 6:49am UTC](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322/20 "2026-06-23T06:49:48Z")

</div>

Ok - job submission failed - take a look at the job activity log to see why it failed

[Next page](https://cylc.discourse.group/t/cylc-variables-cannot-access-in-user-script/1322.md?page=2)