Running two tasks inside loop

Cylc Cylc Support
1

Hi

Just curious, how can I run two tasks(submitted as a batch job) inside a loop; each task is making use of index loop variables.

for var in 72 144 216 288 360^last   or better if I can create these numbers
                                        with increment of 72
 run_taskA
 run_taskB
endfor
2

Depending on your needs, you can do this with integer cycling, or task parameters:

Integer cycling:

[scheduling]
    cycling mode = integer
    initial cycle point = 72
    final cycle point = 360
    [[graph]]
        R/^/P72 = "run_taskA & run_taskB"
[runtime]
    [[run_taskA]]
    [[run_taskB]]

This generates 5 cycles, and two task definitions that are used as templates for each cycle point instance:

$ cylc list -p,1000 . | sort -g
72/run_taskA
72/run_taskB
144/run_taskA
144/run_taskB
216/run_taskA
216/run_taskB
288/run_taskA
288/run_taskB
360/run_taskA
360/run_taskB

Task parameters:

[task parameters]
    m = 72..360..72
[scheduling]
    [[graph]]
        R1 = "run_taskA<m> & run_taskB<m>"
[runtime]
    [[run_taskA<m>]]
    [[run_taskB<m>]]

This generates 1 cycle and 10 task definitions:

$ cylc list -p 1,2 . 
1/run_taskA_m072
1/run_taskA_m144
1/run_taskA_m216
1/run_taskA_m288
1/run_taskA_m360
1/run_taskB_m072
1/run_taskB_m144
1/run_taskB_m216
1/run_taskB_m288
1/run_taskB_m360

For small workflows it doesn’t really matter which way you do it, but in general cycling is much more efficient because cycles are generated dynamically at run time, whereas with parameters ALL the tasks are pre-defined - so I’d recommend you go with cycling. See Task Parameters — Cylc 8.6.5 documentation
(Note you can also have parameterized sub-cycles within a cycling workflow, if you need that).

3

Thank you @hilary.j.oliver .
My external loop is on date, initial cycle point = 20180616T00, hence I cannot switch to cycling mode as integer, unless I am not clear that one can switch in between date mode to integer mode in same workflow.
I tried task parameters approach. I ran into issue, tasks those use, task parameters e.g. task_loop, “inherit” feature does not work, hence related task script is not executed. Thanks for any advice

scheduler]
  allow implicit tasks = True

[task parameters]
  num =  72..360..72

[scheduling]
  initial cycle point = 20180616T00             # STARTDATE=2017071700 (UTC)
  final cycle point = 20180616T00               # ENDDATE=2017072000
  runahead limit = P0
  [[graph]]
 R1 =  task1 => task_loop<num> => task_end

[runtime]
# GEM module tasks
[[GEM]]

 script = '''
 $TASK_RUN_SCRIPT $TASK_CONFIG_FILE
'''
         [[[environment]]]
[[task1]]           ...this works
  inherit = GEM
  platform = my_frontend
   execution time limit = PT02M
#
        [[[directives]]]
            -l = select=1:ncpus=1:mem=2gb
            -q = development
   [[[environment ]]]
       TASK_CONFIG_FILE = ${CYLC_WORKFLOW_RUN_DIR}/scripts/${CYLC_TASK_NAME}.config
       TASK_RUN_SCRIPT = ${CYLC_WORKFLOW_RUN_DIR}/scripts/${CYLC_TASK_NAME}.scr
#
[[task_loop<num>]]       ...does not work; script not executed, which is inherited from GEM family
  inherit = GEM
  platform = my_frontend
   execution time limit = PT02M
#
        [[[directives]]]
            -l = select=1:ncpus=1:mem=2gb
            -q = development
   [[[environment ]]]
       TASK_CONFIG_FILE = ${CYLC_WORKFLOW_RUN_DIR}/scripts/${CYLC_TASK_NAME}.config
       TASK_RUN_SCRIPT = ${CYLC_WORKFLOW_RUN_DIR}/scripts/${CYLC_TASK_NAME}.scr
4

Task parameters just generate normal tasks; it doesn’t affect inheritance in any way. Can you double-check that something else didn’t go wrong in your tasks at run time?

I copied your workflow, converted it run local background tasks, and replaced the GEM family task script with “echo hello” (so that I can run it locally; none of this affects inheritance) and it works as expected:

[scheduler]
    allow implicit tasks = True
[task parameters]
    num =  72..360..72
[scheduling]
    initial cycle point = 20180616T00             # STARTDATE=2017071700 (UTC)
    final cycle point = 20180616T00               # ENDDATE=2017072000
    runahead limit = P0
    [[graph]]
        R1 =  task1 => task_loop<num> => task_end
[runtime]
    [[GEM]]
        script = "echo hello"
    [[task1]]
        inherit = GEM
    [[task_loop<num>]]
        inherit = GEM

The whole thing runs in a minute. Result (for example):

$ cylc log apb//20180616T0000Z/task_loop_num144

Workflow : apb/run1
Job : 20180616T0000Z/task_loop_num144/01 (try 1)
User@Host: oliverh@NIWA-1022450

hello

2026-09-09T22:25:48Z INFO - started
2026-09-09T22:25:49Z INFO - succeeded
5

Thanks @hilary.j.oliver
I did couple of tests…
1.still run in batch mode using pbs directives; removed on R1=, and also on [[task_loop]] for task definition. It runs my script, and gives output

  [[graph]]
 R1 =  task1 => task_loop => task_end
[[task_loop]]
  inherit = GEM
...
pixi run cylc log ex12_loop//20180616T0000Z/task_loop
This is task script task_loop
num=    ....what will be CYLC environment variable for loop parameter num?
CYLC_TASK_CYCLE_POINT=20180616T0000Z

bap001@ppp8->more scripts/task_loop.scr
#!/bin/bash
set -x
echo "This is task script ${CYLC_TASK_NAME}"
echo "num= ${CYLC_TASK_PARAM_NUM}"
echo "CYLC_TASK_CYCLE_POINT=${CYLC_TASK_CYCLE_POINT}"
  1. Ran my original case with task_loop in graph defination as well as task_loop definition
    Did not run task_loop script
Workflow : ex12_loop/run7
Job : 20180616T0000Z/task_loop_num144/01 (try 1)
User@Host: username@hostname
                                                           ... no script output here; wondering , it is expectingtask_loop_144.scr as script name instead of task_loop.
2026-09-10T12:59:43Z INFO - started
  1. Ran case 2 , but scripts running interactively i.e. removed batch directives. It did not work.
  2. Ran case 2, forced to run “echo hello” by all tasks. it worked in batch mode.
[[GEM]]

 script = '''
#$TASK_RUN_SCRIPT $TASK_CONFIG_FILE
 echo hello

[[task_loop]]
inherit = GEM
platform = my_frontend
execution time limit = PT02M

    [[[directives]]]
        -l = select=1:ncpus=1:mem=2gb
        -q = development

[[[environment ]]]
TASK_CONFIG_FILE = ${CYLC_WORKFLOW_RUN_DIR}/scripts/${CYLC_TASK_NAME}.config
TASK_RUN_SCRIPT = ${CYLC_WORKFLOW_RUN_DIR}/scripts/${CYLC_TASK_NAME}.scr

I think, my conclusion is, when is used for task_loop, it has trouble in finding the correct script name, when decoding this line in {{GEM}} family
TASK_RUN_SCRIPT $TASK_CONFIG_FILE

6

That’s documented here in the User Guide… so in your case it will be $CYLC_TASK_PARAM_num.

Note, if job.out (i.e. stdout) is truncated like that it means the job aborted (failed), so look for error messages in job.err (i.e. stderr) - it should tell you exactly what went wrong.

Yes, I think that’s your problem. Looking back, you have this:

[runtime]
    [[GEM]]
        script = "$TASK_RUN_SCRIPT $TASK_CONFIG_FILE"
   [[task_loop<num>]]
       inherit = GEM
       # ...
       [[[environment ]]]
           TASK_CONFIG_FILE = ${CYLC_WORKFLOW_RUN_DIR}/scripts/${CYLC_TASK_NAME}.config
           TASK_RUN_SCRIPT = ${CYLC_WORKFLOW_RUN_DIR}/scripts/${CYLC_TASK_NAME}.scr

The parameterized task definition [[task_loop<num>]] generates tasks like (e.g.) task_loop_num144 with their own name in their job environment like (e.g.) CYLC_TASK_NAME=task_loop_num144.

If you need to run the same script for each of the parameterized tasks, you just need to tell Cylc to do that, e.g.:

   [[task_loop<num>]]
       # ...
       [[[environment ]]]
           TASK_RUN_SCRIPT = ${CYLC_WORKFLOW_RUN_DIR}/scripts/task_loop.scr
7

Thank you @hilary.j.oliver. It worked.

In summary, tasks with loop parameters have no longer original name for $CYLC_TASK_NAME, but it becomes : original name appended by loop index variable e.g. task_loop360. Other thing I found, for non-loop tasks e.g. task1, defining TASK_CONFIG_FILE is optional i.e. it works even .config file does not exist, but for loop type tasks, one must define both TASK_CONFIG_FILE and TASK_RUN_SCRIPT explicitly i.e. *.config file must exist