CYLC_* variables; cannot access in user script

Cylc Cylc Support
1

Hi
I have top level user script, which may be calling other script. I don’t see these CYLC_* variable passed to my top level script. Also I saw, when I submit the script with qsub having PBS directives, my script is not executed correctly . What I am doing wrong?

flow.cylc:
[[task1]]

 script = ord_soumet  ${MY_DEFAULT_PATH}/task1.scr -mach xyz -cm 2G -t 300 -jn pabla -listing ${MY_DEFAULT_PATH}
#script = qsub -l host=xyz  ${MY_DEFAULT_PATH}/task1.scr 

 [[[environment]]]
  MY_DEFAULT_PATH = /space/????

task1.scr
#!/bin/bash
set -x
#
# Change to the directory where the job was submitted:
echo "PBS_O_WORKDIR=${PBS_O_WORKDIR}"
echo "CYLC variables in top level script"
echo "CYLC_WORKFLOW_NAME=${CYLC_WORKFLOW_NAME}"
echo "CYLC_WORKFLOW_RUN_DIR=${CYLC_WORKFLOW_RUN_DIR} "
echo "CYLC_TASK_SHARE_DIR=${CYLC_TASK_SHARE_DIR} "

output :
++ echo 'CYLC variables in top level script'
CYLC variables in top level script
++ echo CYLC_WORKFLOW_NAME=
CYLC_WORKFLOW_NAME=
++ echo 'CYLC_WORKFLOW_RUN_DIR= '
CYLC_WORKFLOW_RUN_DIR=
++ echo 'CYLC_TASK_SHARE_DIR= '
CYLC_TASK_SHARE_DIR=
2

Hi @bpabla50

The job script generated for each task, such as task1, literally exports all of those CYLC variables, so there’s no way that they aren’t in your job environment.

Your job script (which you can look at - see the job file in the log directory for that task), will look something like this (omitting a lot):

...
export CYLC_WORKFLOW_NAME=whatever
...
ord_soumet ${MY_DEFAULT_PATH}/task1.scr -mach xyz ...
...

The only thing I can think of is, it’s not obvious that your task1.scr is being executed directly because you are not doing that - you’re passing it as a command line argument to ord_soumet and I have no idea what that does with its arguments.

As a first step, maybe change your [task1] script to this:

script = """
echo "CYLC variables in top level script"
echo "CYLC_WORKFLOW_NAME=${CYLC_WORKFLOW_NAME}"
# ...
ord_soumet ${MY_DEFAULT_PATH}/task1.scr ...
"""

You will see (I’m sure) that the CYLC variables are defined there, and because they are exported they will be available to subshells too.

If so, then ord_soumet must being doing something strange with your script - maybe it runs task1.scr detached from the original environment, e.g. by internal submission to PBS or something??

3

Thanks @hilary.j.oliver .
As per your advice, I looked at jobfile, yes, CYLC_* variables are defined and exported there. Question is that job submitting tool(e.g. qsub, ord_soumet…wrapper around qsub) does not pass these CYLC_* variable to task1.scr. Please note that here task1.scr is running in batch mode instead of running interactively.

job file:
    export CYLC_WORKFLOW_NAME="ex7_workflow"
    export CYLC_WORKFLOW_NAME_BASE="ex7_workflow"
    export CYLC_WORKFLOW_UUID="1a8e98a1-dd43-44de-af44-3719781438bb"

    # CYLC TASK ENVIRONMENT:
    export CYLC_TASK_COMMS_METHOD=zmq
    export CYLC_TASK_JOB="1/task1/01"
    export CYLC_TASK_NAMESPACE_HIERARCHY="root task1"
    export CYLC_TASK_TRY_NUMBER=1
    export CYLC_TASK_FLOW_NUMBERS=1
}
cylc__job__inst__user_env() {
    # TASK RUNTIME ENVIRONMENT:
    export MY_DEFAULT_PATH
    MY_DEFAULT_PATH="/space/??"
}

cylc__job__inst__script() {
# SCRIPT:
ord_soumet  ${MY_DEFAULT_PATH}/task1.scr -mach xyz -cm 2G -t 300 -jn pabla -listing ${MY_DEFAULT_PATH}
}

CYLC_RUN_DIR="${CYLC_RUN_DIR:-$HOME/cylc-run}"
. "${CYLC_RUN_DIR}/${CYLC_WORKFLOW_ID}/.service/etc/job.sh"
cylc__job__main

I tried qsub command outside cylc, and saw it is not passing CYLC_* variables to script..

$ export CYLC_WORKFLOW_NAME=ex8_workflow
$ qsub -l walltime=00:05:00 ./task1.scr_withpbs

$ more ./task1.scr_withpbs
set -x
#
# Job Name:
#PBS -l walltime=00:05:00
#PBS -N pabla
# Specify the output file:
#PBS -o task1_output.txt
# Specify the error file:
#PBS -e task1_error.txt
# Number of nodes and processors per node:
#PBS -l nodes=1:ppn=1
# Walltime (hh:mm:ss):
# Memory requirements:
#PBS -l mem=2gb
4

Right, that explains your problem.

PBS does not execute jobs in the same environment that you submit the job from. After all, it will almost certainly run the job on an entirely different node or platform that literally cannot see your environment. It is up to you to ensure that all dynamic variables (i.e. changing per job, as opposed to statically defined in your interactive shell environment) required by your PBS job are defined in the PBS job script itself.

Obviously I don’t know what your ord_soumet script does, or how complicated it is, but generally speaking it should not be doing internal job submissions to PBS. It is Cylc’s role, as a workflow manager, to determine where and how to run workflow jobs, and then to track their status.

Here’s a simple example to illustrate the problem. Say I have a script that runs a scientific model called model with a command line model -xy $INPUT.

To run it myself I might create a PBS job script job.sh like this:

#!/usr/bin/env bash
# PBS - l walltime=...
# PBS ...
INPUT=/path/to/input
model -xy $INPUT

and submit it to PBS: qsub job.sh

To run it with Cylc I would NOT do this (imagine job1.sh has the INPUT line removed):

[runtime]
    [[model]]
        script = """
            INPUT=/path/to/input
            qsub /path/to/job1.sh  # ERROR - don't do this!
        """

That will NOT work because:

  1. the PBS job runs detached on another host and will not see $INPUT
  2. the qsub command exits with success status as soon as the job is submitted to PBS, so it will look to Cylc like the job finished before it even started running

Instead, get Cylc to handle job execution via PBS:

[runtime]
   [[model]]
       platform = my-pbs-platform  # see Cylc platform docs
       script = """
           INPUT=/path/to/input
           model -xy $INPUT
       """

Now the Cylc-generated job script for task model is a PBS job script (like your hand-written job.sh) and Cylc will submit it to PBS for you, and track its progress from submitted to running to succeeded or failed.

5

Thanks @hilary.j.oliver for clarifying that my script should not have PBS directives, and job should not be submitted with system job submisson tools like qsub etc

I read about platforms here pbs — Cylc 8.6.3 documentation
Things did not work, appreciate further help with complete flow.cylc example to submit job to compute node of HPC cluster using PBS PRO, and job requirement say a) memory= 2GB b)cpus=1 c)time limit=5min d) queue=development e) jobname=test f)listings_dir=some_path. Is it possible to define all the job related parametes including cluster name in flow.cylc without making use of global.cylc.

more job-activity.log
[jobs-submit cmd] (platform not defined)
[jobs-submit ret_code] 1
[jobs-submit err] No matching platform "my_frontend" found  

more ~/.cylc/global.cylc
 [platforms]
  [[my_frontend]]
  hosts = xyz
  login_command = ssh
  shell = /bin/bash
  batch_system = pbs  

more flow.cylc
[scheduler]
[task parameters]
  num = 1..6
 allow implicit tasks = True
 
[scheduling]
  initial cycle point = 1
    final cycle point = 2
  cycling mode =  integer
# 
 [[graph]]
   R1 = """ task1 """
   
[runtime] 
  
 [[task1]]
   platform = my_frontend
   execution time limit = PT5M
   script = """
#load your packages here....
   ${MY_DEFAULT_PATH}/task1.scr
  """
 [[[environment]]]
  MY_DEFAULT_PATH = some_path
6

To clarify that a bit more, for other readers - your task definitions should have PBS directives, and Cylc itself will use qsub to submit your tasks to PBS.

Your global.cylc is not being picked up by Cylc, because it isn’t in one of the legal locations for the global config file.

See Global Configuration — Cylc 8.6.3 documentation

Move it to the right location, and then type cylc config to see if your global settings are being parsed.

Another point about your platform definition, you probably need to define install target - it is just an arbitrary label that needs to be the same for all Cylc platforms that see the same filesystem.

7

Hi @hilary.j.oliver
What I am still doing wrong? in what file, one specify scheduler type is PBS? Thanks for any help?

->pwd
/home/user1/.cylc/flow
->more global.cylc
 [platforms]
  [[my_frontend]]
   hosts = mach1

->more job-activity.log
[jobs-submit cmd] (platform not defined)
[jobs-submit ret_code] 1
[jobs-submit err] No matching platform "my_frontend" found

->more task1.scr
#!/bin/bash
set -x
#
# Change to the directory where the job was submitted:
hn=`hostname`
echo "hn=${hn}"
echo "PBS_O_WORKDIR=${PBS_O_WORKDIR}"
echo "CYLC variables in top level script"
echo "CYLC_WORKFLOW_NAME=${CYLC_WORKFLOW_NAME}"
8

Hi @bpabla50

Sorry if I was not quite clear enough above, but:

From what you’ve said, you have put your global config in ~/.cylc/global.cylc which is not one of the expected locations for the file. The link I gave above lists the expected locations. The usual one would be ~/.cylc/flow/8/global.cylc (it goes under the flow sub-directory because this config file is part of the cylc/flow (scheduler) package. Cylc has other packages too such as cylc/uiserver…).

If you put the file in the right location, just type cylc config to verify that it is working - that command (with no arguments) will print out the parsed global configuration from all allowed locations.

Also: batch_system is not a valid global config item in Cylc 8, not sure where you got that from. It should be job runner = pbs. Check the platforms documentation again to make sure you have defined the right config items.

Hilary

9

Hi @hilary.j.oliver :slight_smile:
Getting close. Thanks for your help.

I think, I got it from chatgpt or some other AI tool.
Until, I was not using install target, I was getting weird error message

026-06-09T17:25:21Z INFO - platform: my_frontend - remote init (on ppp8)
2026-06-09T17:25:23Z CRITICAL - An uncaught error caused Cylc to shut down.
    If you think this was an issue in Cylc, please report the following traceback to the developers.
    https://github.com/cylc/cylc-flow/issues/new?assignees=&labels=bug&template=bug.md&title=;
2026-06-09T17:25:23Z ERROR - argument of type 'NoneType' is not a container or iterable

Once I defined install target, and used “localhost”, it went further; is “localhost” can be replaced with my real cluster name say “xyz”, where my pbs job will run?
instead of real cluster name “ppp”.

Last error message, is that it not liking one of PBS directive, where I am asking for 1cpu per node. Is all keywords in [[directives]] are same as PBS directives, but in different syntax.

<bap001@hpcr8-vis->pwd
/home/bap001/.cylc/flow/8

<bap001@hpcr8-vis->more global.cylc

 [platforms]
  [[my_frontend]]
#  hosts = ppp8
     hosts = localhost
     job runner = pbs
     install target = localhost
flow.cylc :
[runtime]
  [[default]]
    platform = my_frontend
    execution time limit = PT5M
# 
 [[task1]]
   platform = my_frontend
   execution time limit = PT5M
        [[[directives]]]
            -N = pabla
            -o = task1_output.txt
            -e = task1_error.txt
            -l = nodes=1:ppn=1     ...looks like, it does not like it
            -l = mem=2gb
            -M = my_valid_email

task1.scr

!/bin/bash
set -x
#
echo "start of script..."
# Change to the directory where the job was submitted:
cd $PBS_O_WORKDIR
echo "PBS_O_WORKDIR=${PBS_O_WORKDIR}"
echo "CYLC variables in top level script"
echo "CYLC_WORKFLOW_NAME=${CYLC_WORKFLOW_NAME}"

Can you please clarify, when to use “loacalhost”, and when to use real cluster name, i prefer latter.

10

Suggest consulting the Cylc docs :grin:

You appear to have uncovered a bug (or at least, a config error that should be caught and reported gracefully) - can you reproduce it, and tell us your Cylc version and the global.cylc content (presumably) that triggers the error?

The current documentation on install target is less than clear, according to some users (including myself, as it turns out). We have an in coming docs change to fix that. In the meantime, install target is just an arbitrary label, but the important thing is it represents a filesystem so all Cylc platforms (i.e. as defined under [platforms] in global.cylc) must have the same install target - this allows Cylc to determine whether or not it has already installed workflow files on a platform, in workflows that send jobs to multiple platforms.

As a minimal example, on an HPC cluster you might send some jobs to PBS (via one platform def), and run some trivial tasks as background jobs on your login node (via another platfrom def) - if the login node sees the same files as the compute nodes, both platforms should have the same install target label.

If you get this wrong and give several platforms on the same filesystem different install targets, you will get “platform init” errors in the workflow, when Cylc finds files that should not already be present when it first submits a job to a platform.

Long story short, you can use install target = localhost if you like, but Cylc just sees it in this context as an arbitrary label, not as a hostname, so IMO that’s a bit misleading. (And, you would have to use localhost for all other platforms that see the same filesystem too). So yes, better to use your cluster name, install target = ppp (so long as all nodes on the cluster see the same filesystem!).

Everything in a flow.cylc file is a config item of the form key = value. In a [directives] section these config items have to be translated into actual PBS directives. Here you have defined -l (as a config item) twice, so (as is the way with config items) the second definition overrides the first - so PBS will never see your nodes and ppn options.

How to handle PBS directives like this is explained here: Job Submission and Management — Cylc 8.6.3 documentation

11

Regarding potential bug I found, I will send later my examples from yesterday. Today, I slighly revised my global.cylc, and flow.cylc, I got the same error even with install target.

# scheduler/log:
2026-06-10T19:20:34Z INFO - platform: my_frontend - remote init (on ppp8)
2026-06-10T19:20:36Z CRITICAL - An uncaught error caused Cylc to shut down.
2026-06-10T19:20:34Z INFO - Cylc version: 8.6.3

    If you think this was an issue in Cylc, please report the following traceback to the developers.
    https://github.com/cylc/cylc-flow/issues/new?assignees=&labels=bug&template=bug.md&title=;
2026-06-10T19:20:36Z ERROR - argument of type 'NoneType' is not a container or iterable
    Traceback (most recent call last):
      File "/fs/site8/eccc/aq/r1/bap001/cylc_viapixi/.pixi/envs/default/lib/python3.14/site-packages/cylc/flow/scheduler_cli.py", line 753, in cylc_play
        asyncio.get_running_loop()

# global.cylc
<bap001@ppp8->pwd
/home/bap001/.cylc/flow/8
<bap001@ppp8->more global.cylc
 
[platforms]
  [[my_frontend]]
     hosts = ppp8
     job runner = pbs
# string right to = sign can be anything e.g. localhost, ppp8 etc.
# It is just a lable for FS used by my_frontend
     install target = sitestore8 

# flow.cylc:
[[graph]]
  R1 = """ task1 """
[runtime]
 [[task1]]
   platform = my_frontend
   execution time limit = PT5M
        [[[directives]]]
            -N = pabla
            -o = task1_output.txt
            -e = task1_error.txt
            -l = select=1:ncpus=1:mem=2gb
            -M = balbir.pabla@ec.gc.ca
            -m = abe
            -q = development
   script = """
#load your packages here....
   ${MY_DEFAULT_PATH}/task1.scr
[[[environment]]]
  MY_DEFAULT_PATH = /space/hall/?????
  """
12

OK so the error is occurring at runtime during “remote init”, presumably when the first job is submitted to your platform (which is when Cylc initializes the platform, and installs files if the install target has not already been done). Unfortunately, it looks like the exception has bubbled up to a higher level, obscuring the code block that it originated in. Can you reproduce again please, running the scheduler in debug mode (e.g. cylc vip --debug) and see if that gives us more information?

As an aside, you don’t need these directives, in fact they might be unhelpful:

Cylc automatically tells PBS to use the standard Cylc job log directory job.out and job.err files for job stdout and stderr.

13

Thanks @hilary.j.oliver .
Running with debug option gave useful information, cylc is not found.
Looking into, why it is not finding “cylc” command with platform setup, while no issue running interactively. Please note that, I have installed cylc under pixi environment. Each cylc command should be preceeded by “pixi run”, e.g.
pixi run cylc vip --debug .
Any advice, how I can prepend “pixi run” for every cylc command, when things are run using platforms.

rundir/log/scheduler/log:
2026-06-11T13:59:36Z INFO - Scheduler: url=tcp://ppp8login-02-ib:43036 pid=2919507
2026-06-11T13:59:36Z INFO - Workflow publisher: url=tcp://ppp8login-02-ib:43099
2026-06-11T13:59:36Z INFO - Run: (re)start number=1, log rollover=1
2026-06-11T13:59:36Z INFO - Cylc version: 8.6.3
2026-06-11T13:59:36Z INFO - Run mode: live
2026-06-11T13:59:36Z INFO - Initial point: 1
2026-06-11T13:59:36Z INFO - Final point: 2
2026-06-11T13:59:36Z INFO - Cold start from 1
2026-06-11T13:59:36Z INFO - New flow: 1 (original flow from 1) 2026-06-11T13:59:36
2026-06-11T13:59:36Z DEBUG - Runahead: base point 1
2026-06-11T13:59:36Z DEBUG - Runahead limit: 1
2026-06-11T13:59:36Z DEBUG - [1/task1:waiting(runahead)] added to the n=0 window
2026-06-11T13:59:36Z INFO - [1/task1:waiting(runahead)] => waiting
2026-06-11T13:59:36Z INFO - [1/task1:waiting] => waiting(queued)
2026-06-11T13:59:36Z DEBUG - Loaded main loop plugin "health check":
    * health_check
2026-06-11T13:59:36Z DEBUG - Loaded main loop plugin "reset bad hosts":
    * reset_bad_hosts
2026-06-11T13:59:36Z INFO - [1/task1:waiting(queued)] => waiting
2026-06-11T13:59:36Z INFO - [1/task1:waiting] => preparing
2026-06-11T13:59:36Z INFO - platform: my_frontend - remote init (on ppp8)
2026-06-11T13:59:36Z DEBUG - main_loop [run] cylc.flow.main_loop.health_check:health_check
2026-06-11T13:59:36Z DEBUG - main_loop [end] cylc.flow.main_loop.health_check:health_check (0.000s)
2026-06-11T13:59:36Z DEBUG - main_loop [run] cylc.flow.main_loop.reset_bad_hosts:reset_bad_hosts
    * reset_bad_hosts
2026-06-11T13:59:36Z INFO - [1/task1:waiting(queued)] => waiting
2026-06-11T13:59:36Z INFO - [1/task1:waiting] => preparing
2026-06-11T13:59:36Z INFO - platform: my_frontend - remote init (on ppp8)
2026-06-11T13:59:36Z DEBUG - main_loop [run] cylc.flow.main_loop.health_check:health_check
2026-06-11T13:59:36Z DEBUG - main_loop [end] cylc.flow.main_loop.health_check:health_check (0.000s)
2026-06-11T13:59:36Z DEBUG - main_loop [run] cylc.flow.main_loop.reset_bad_hosts:reset_bad_hosts
2026-06-11T13:59:36Z DEBUG - main_loop [end] cylc.flow.main_loop.reset_bad_hosts:reset_bad_hosts (0.000s)
2026-06-11T13:59:36Z DEBUG - ['ssh', '-oBatchMode=yes', '-oConnectTimeout=10', 'ppp8', 'env', 'CYLC_VERSION=8.6.3', 'bash', '--login', '-c', '\'exec "$0" "$@"\'', 'cylc', 'remote-init', '-v', '-v', 'sitestore8', '$HOME/cylc-run/ex10_useplatforms/run2']
2026-06-11T13:59:37Z DEBUG - [remote-init cmd] cat '<tempfile.SpooledTemporaryFile object at 0x1468857fbcd0>' | ssh -oBatchMode=yes -oConnectTimeout=10 ppp8 env CYLC_VERSION=8.6.3 bash --login -c ''"'"'exec "$0" "$@"'"'"'' cylc remote-init -v -v sitestore8 '$HOME/cylc-run/ex10_useplatforms/run2'
    [remote-init ret_code] 0
    [remote-init out] (INFO) Using package (fully qualified) pixi_0.63.2_all from domain /fs/ssm/eccc/cmd/cmds/apps/pixi/202602/00
   [remote-init ret_code] 0
    [remote-init out] (INFO) Using package (fully qualified) pixi_0.63.2_all from domain /fs/ssm/eccc/cmd/cmds/apps/pixi/202602/00
    [remote-init err]
    === ordenv: version 4.5 ===
    === ordenv: master/default ===
    === ordenv: system/init ===
    === ordenv: system/site ===
    === ordenv: sourcing site:20260326 ===
    info: jobctl cell (ppp8)
    === ordenv: system/comm ===
    === ordenv: sourcing comm:eccc/20260205 ===
    === ordenv: system/group ===
    === ordenv: sourcing group:/fs/ssm/eccc/mrd/ordenv/20260326/profile/group ===
    === ordenv: sourcing group: SCIENCE domain (ppp8) ===
    === ordenv: system/user ===
    === ordenv: sourcing user:default/post ===
    === ordenv: completed ===
    (INFO) Profile load time: 0.476s
    cylc: line 1: exec: cylc: not found
    cleaning up temporary directories
    removing TMPDIR = /tmp/bap001/2919529
2026-06-11T13:59:37Z ERROR - platform: my_frontend - initialisation did not complete
    COMMAND:
        ssh -oBatchMode=yes -oConnectTimeout=10 ppp8 env \
            CYLC_VERSION=8.6.3 bash --login -c 'exec "$0" "$@"' cylc \
            remote-init -v -v sitestore8 \
            $HOME/cylc-run/ex10_useplatforms/run2
    RETURN CODE:
        0
    STDOUT:
        (INFO) Using package (fully qualified) pixi_0.63.2_all from domain /fs/ssm/eccc/cmd/cmds/apps/pixi/202602/00
    STDERR:
        === ordenv: version 4.5 ===
       === ordenv: completed ===    ...finished loading profile
        (INFO) Profile load time: 0.476s
        cylc: line 1: exec: cylc: not found
        cleaning up temporary directories
        removing TMPDIR = /tmp/bap001/2919529
2026-06-11T13:59:37Z DEBUG - [1/task1/01:preparing] host=ppp8
2026-06-11T13:59:37Z ERROR - [jobs-submit cmd] (init ppp8)
    [jobs-submit ret_code] 1
    [jobs-submit err] REMOTE INIT FAILED
2026-06-11T13:59:37Z ERROR - [jobs-submit cmd] (remote init)
    [jobs-submit ret_code] 1
2026-06-11T13:59:37Z ERROR - [1/task1/01:preparing] submission failed
2026-06-11T13:59:37Z INFO - [1/task1/01:preparing] => submit-failed
2026-06-11T13:59:37Z WARNING - [1/task1/01:submit-failed] did not complete the required outputs:
    ⨯ ┆  succeeded
2026-06-11T13:59:37Z DEBUG - 1/task1 -triggered off [] in flow 1
2026-06-11T13:59:37Z ERROR - Incomplete tasks:
    * 1/task1 did not complete the required outputs:
2026-06-11T13:59:37Z ERROR - Incomplete tasks:
    * 1/task1 did not complete the required outputs:
      ⨯ ┆  succeeded
2026-06-11T13:59:37Z CRITICAL - Workflow stalled
2026-06-11T13:59:37Z WARNING - PT1H stall timer starts NOW
2026-06-11T14:09:36Z DEBUG - main_loop [run] cylc.flow.main_loop.health_check:health_check
2026-06-11T14:09:36Z DEBUG - main_loop [end] cylc.flow.main_loop.health_check:health_check (0.000s)
2026-06-11T14:19:37Z DEBUG - main_loop [run] cylc.flow.main_loop.health_check:health_check
2026-06-11T14:19:37Z DEBUG - main_loop [end] cylc.flow.main_loop.health_check:health_check (0.000s)
2026-06-11T14:29:36Z DEBUG - main_loop [run] cylc.flow.main_loop.reset_bad_hosts:reset_bad_hosts
2026-06-11T14:29:36Z DEBUG - main_loop [end] cylc.flow.main_loop.reset_bad_hosts:reset_bad_hosts (0.000s)
2026-06-11T14:29:37Z DEBUG - main_loop [run] cylc.flow.main_loop.health_check:health_check
2026-06-11T14:29:37Z DEBUG - main_loop [end] cylc.flow.main_loop.health_check:health_check (0.000s)
14

Hi @bpabla50 :wave:,

Distributed Installation

Cylc is a distributed system.

You can run a Cylc workflow locally, and have it run all of it’s jobs locally, in which case you could just pixi run cylc vip my-workflow and away you go.

However, for distributed deployments where workflows are submitting jobs to remote platforms / batch systems, it is necessary to install Cylc in multiple places and find a way to intercept cylc command calls so you can redirect them towards your Cylc deployment(s).

This “interception” is done using a “wrapper script”. Various tools use/create them, including Pixi.

Wrapper Scripts

If you’re deploying Cylc via Pixi, then a basic “wrapper script” might look like this (untested!):

#!/bin/sh

exec pixi run cylc "$@"

You would name this script cylc and place it in a directory which is included in the default $PATH. That way any cylc call from any regular shell will be directed via this command, which invokes it via the Pixi environment.

This is a basic example of a wrapper script. In addition to Cylc, you may also require wrapper scripts for the rose and isodatetime commands if you are using them in your workflows.

The Cylc Wrapper Script

Distributed Cylc deployments are a common practice, so we have written a wrapper script which is designed to cater to most circumstances.

The Cylc wrapper script:

  • Can support multiple parallel Cylc installations (which means you can install new Cylc versions without breaking running workflows). Switch between them using the CYLC_VERSION environment variable.
  • Can work with Conda/Mamba environments or Python virtual environments.
  • Handles some issues you might face if writing a wrapper script yourself.
  • Doesn’t activate the environment, or use commands like conda run / pixi run (which active environments implicitly). This is actually the correct way to do it with Cylc because activating the environment would cause background jobs to run within it which is generally undesireable.

There are some details in the installation guide in the documentation:

Go through the comments at the top of the file and adjust any variables as necessary, then stick the script somewhere in $PATH and you’re done. If necessary, make a copies of the wrapper script called rose and isodatetime in the same directory.

The CYLC_HOME_ROOT variable should point at the directory which contains your Pixi Cylc environments (name them cylc-<cylc-version> to enable version switching).

For example, if you get this:

$ cylc version --long
8.6.4 (/site/apps/cylc-8.6.4-2/bin/cylc)

Then you would set:

CYLC_HOME_ROOT=/site/apps

We haven’t yet added explicit support for Pixi environments in this script yes, however, I suspect these are really just Conda environments in disguise so our existing support will probably work.

If Pixi installs cylc using a different directory pattern to:

$CYLC_HOME_ROOT/cylc-<version>/bin/cylc

Let us know what it is, and we’ll guide you through adapting the wrapper script.