Hi
I am trying to import work flow from another sequencer. I want to keep two directories from old sequencer as it is a) config …configuration files at high level b) modules directory, which contains a bunch of subdirectories called submodules, where each directory contains task configuration file, and execution script e.g.
old:
/home/bap001/.suites/osrun6/hub/ppp7/work/20180616000000/main/forecastcycle/10km_regional/gemmach/get_emissions
# here ppp7 is hostname
# main ..is name of module name(perhaos family in cylc world), which contains other modules (forcastcycle, 10km_regional, gemmach)
# get_emissions is the task name in gemmach module
My intention is to keep the path name as close to old sequencer, because task is written that way, which I don’t want to modify.
Other important difference is that in old sequencer, it is same directory, which contains everything …model scripts, hub directory, where model runs. In cylc, we have two separate directories, one which contains user tasks, and other cylc-run to install/run the stuff
How flow.cylc should look like to achieve that i.e. stick to old sequencer paths?
My current flow.cylc.
[scheduling]
initial cycle point = 20190211T00 # STARTDATE=2017071700 (UTC)
final cycle point = 20190211T00 # ENDDATE=2017072000
runahead limit = P0
[[graph]]
R1 = BuildDates => Launch => Next => getData => start_cycle
PT24H = prep_settings => get_emissions_10km
[run]
[[GEMMACH]] ..module aka family name
[[[environment ]]]
var_gm = 999
[[get_emissions_10km]]
inherit = GEMMACH
platform = my_frontend
execution time limit = PT5M
script = """
${MY_DEFAULT_PATH}/scripts/get_emissions.scr
"""
In case it’s not obvious, Cylc separates workflow source and run directories so that you can safely do things like use git to control your source without contaminating your repo with run files; work on your source files without affecting running workflow instances; and even run multiple instances of the same workflow at the same time from different source versions.
It’s not quite clear to me what your are asking. Taking a guess, it sounds like your old system defined tasks in sub-directories that contain task configuration and run scripts, and you want to know how to run those tasks in Cylc without changing the directory structure?
If so, that’s fine - Cylc task script items can do anything you like - it’s up to you - so just do whatever is needed to run the tasks with the desired configuration.
A quick example that I had lying around, which might be helpful.
My source dir looks like this:
$ tree bpa
bpa
├── bin
│ └── hello.sh
├── flow.cylc
└── taskdefs
├── bar
│ ├── config
│ └── run.sh
└── foo
├── config
└── run.sh
The workflow has 3 tasks:
hello runs bin/hello.sh with a command line argument
(the job will automatically have access to the workflow bin directory)
foo and bar are defined and configured under taskdefs
they both have run scripts called run.sh and config files called config
Taking foo for example - the config file:
$ cat taskdefs/foo/config
# source this file to "configure" your bash run script
COLOR=red
SHAPE=circle
and the run script:
$ cat taskdefs/foo/run.sh
#!/usr/bin/env bash
MY_CONFIG_FILE=$1
# configure me
. $MY_CONFIG_FILE
# run
echo "My color is $COLOR"
echo "My shape is $SHAPE"
The flow.cylc runs all three tasks in their native form:
Note this flow.cylc leverages the fact that foo and bar use the same filenames for run and config, but in locations that correspond to the task names. If they used different filenames (e.g. run-foo.sh and run-bar.py or whatever) just define them separately in the flow.cylc instead of in a shared family.
Note, when you cylc install or cylc vip this workflow, the source directory gets installed as-is, so at runtime your task jobs can access the taskdefs directory relative to the $CYLC_WORKFLOW_RUN_DIR (i.e. relative to the install location).
Run this:
~/cylc-src/bpa$ cylc vip --no-detach
$ cylc validate /home/oliverh/cylc-src/bpa
Valid for cylc-8.6.4
$ cylc install /home/oliverh/cylc-src/bpa
INSTALLED bpa/run4 from /home/oliverh/cylc-src/bpa
$ cylc play --no-detach bpa/run4
▪ ■ Cylc Workflow Engine 8.6.4
██ Copyright (C) 2008-2026 NIWA
▝▘ & British Crown (Met Office) & Contributors
...[SNIP]...
INFO - [1/bar/01:running] => succeeded
INFO - Workflow shutting down - AUTOMATIC
INFO - DONE
And check the results:
$ cylc cat-log bpa//1/hello/
Workflow : bpa/run4
Job : 1/hello/01 (try 1)
User@Host: oliverh@NIWA-1022450
Hello world!
2026-07-07T14:29:21+12:00 INFO - started
2026-07-07T14:29:22+12:00 INFO - succeeded
$ cylc cat-log bpa//1/foo
Workflow : bpa/run4
Job : 1/foo/01 (try 1)
User@Host: oliverh@NIWA-1022450
My color is red
My shape is circle
2026-07-07T14:29:24+12:00 INFO - started
2026-07-07T14:29:25+12:00 INFO - succeeded
$ cylc cat-log bpa//1/bar
Workflow : bpa/run4
Job : 1/bar/01 (try 1)
User@Host: oliverh@NIWA-1022450
My car is ferrari
My tree is pine
2026-07-07T14:29:27+12:00 INFO - started
2026-07-07T14:29:28+12:00 INFO - succeeded
Off topic, there’s nothing to stop you from running scripts in external locations, but it can be risky (what if you or someone else accidentally changes or deletes get_emissions.scr while your workflow is still running?).
We normally recommend either:
keep task scripts in the source directory, if they are unique to your workflow
(in which case they will be automatically installed to the run directory)
or use an initial task to deploy them into the workflow share directory at runtime, if they are not unique to your workflow
(then your workflow gets its own safe copy to use at run time, even though it is not part of the workflow source).
Thank you @hilary.j.oliver for sharing this example. A few questions:
-what --no-detach option does? Any suggestion to search smarter e.g. in doc , I searched for --no-detach, it gives me a few links, which did not explain, what this option does unless I missed.
-Just curious, we did not specify the path for hello.sh, but still it finds, because it is at the same level as flow.cylc.
-any example of flow.cylc to share, having family within family i.e. nested families. For example, I have /main/forecastcycle/10km_regional/gemmach/get_emissions…task name
The commands are supposed to be self-documenting, so we tend not to repeat that information in the user guide.
$ cylc play --help
...
-N, --no-detach, --non-daemon
Do not daemonize the scheduler
The scheduler normally runs as a daemon, i.e. it detaches from the parent process (e.g. your interactive shell session in a terminal) so that it can continue to run even if the parent process dies (e.g. you log out). The --no-detach option prevents this, so it stays attached to your terminal and all the output will appear in the terminal - it’s a nice way to follow small examples in real time, without having to start the GUI or TUI or follow the scheduler log.
If your workflow source has a bin directory, that will be installed to the run directory, and your jobs will automatically get access to it (i.e. Cylc will write export PATH=${CYLC_WORKFLOW_RUN_DIR}/bin:$PATH to every task job script).