Using cylc submit with a non-scheduled task

Cylc Cylc Support
1

Hi. I’m running cylc-flow-7.8.3. I want to have a task in my suite that’s not scheduled for any cycle point and has no task dependencies; instead, other tasks which are in the suite graphs would tell the server to run the unscheduled task if and only if certain conditions are met.

From Section 9.9 of the docs, my understanding is that I need to do three things:

  1. define the task in [runtime] like I would any other task, e.g.:

    [[test_unsched_task]]
    script = “”"
    echo foo
    sleep 300
    “”"

  2. Add it to an exclude at start-up list, e.g.:
    [scheduling]
    [[special tasks]]
    exclude at start-up = test_unsched_task

  3. Use the “cylc submit” task to fire it up when desired – a manual example being:

$ cylc submit TESTSUITE test_unsched_task.20140119T0015Z

However, when I try this, the task doesn’t run, and the submit command returns a message saying “No task found for test_unsched_task.20140119T0015Z”.

The purpose of the “–initial-cycle-point” argument to the cylc submit command isn’t totally clear to me from the documentation; but I tried it, with

$ cylc submit --icp=20140119T0015Z TESTSUITE test_unsched_task.20140119T0015Z

No joy – the same result.

What am I getting wrong here?

Thanks!

2

Cylc submit won’t run tasks which aren’t in the graph.

I wouldn’t advise this approah for other reasons besides:

  • Submitting a task from within a task may have unforeseen consequences.
  • Cylc will not know about this task, it won’t appear in the GUI or logs.
  • The cylc submit command has been removed in Cylc 8.

For your use case you might be better off with custom outputs.

Here’s an example, the task d is either run or not depending on a custom output generated by the task a:

# the regular scheduled tasks
a => b => c

# if task a has finished and the condition is true then run the task "d"
a & a:condition-true => d

# otherwise remove the task "d" from the pool
a & a:condition-false => ! d

The condition is evaluated in task a which produces one of two outputs.

# untested
[runtime]
    [[a]]
       # you could do this in a post-script after the
       # main script has run
        script = """
            if [[ <some condition ]]; then
                cylc message -- 'condition-true'
            else
                cylc message -- 'condition-false'
            fi
        """
        [[[outputs]]]
            condition-true = condition-true
            condition-false = condition-false

There’s more information about these “custom outputs” in the message trigger documentation.

The exclamation mark in the graph (! d) is a suicide trigger, there are some worked examples in the Rose Documentation which might help.

3

Hi, and thanks for your reply. I’ve used custom outputs and suicide triggers in my suite to solve other requirements. However, I don’t think they’ll work for the problem I’m intending to solve. The particular example I wrote in my first post was just a toy model meant to help me understand how to use unscheduled tasks; and I agree that custom outputs would help me to accomplish the same thing as that toy model. But in the use case I actually have:

  1. Among many other things it needs to do, the suite will need to run a parallel/batch-queued binary a number of times which is nonzero but unknown at suite definition; so it can’t be hard-coded into suite.rc. Since the executions of this binary are sequential – each run depends on the previous – my intent was simply to include them in a single task which is scheduled in the graph. The script for that task would invoke the binary inside a loop whose length is determined at runtime. So far, no problem.

  2. At the end of each successful execution of the first task’s binary – at the end of each time through the first task’s loop – a second, separate task needs to start. It’s a separate task (it can’t be incorporated into the first task) because it must be asynchronous with the first one: it needs to run a parallel/batch queued binary that will be processing output from the first task’s first pass while the first task has moved on to running the second pass. It cannot wait until the first task is completely finished – it must run once for each time through the first task’s loop (each time the first task’s binary completes successfully). I could only think of two ways to accomplish what I need:

  • At the end of each pass through the first task’s loop, the second task is fired up via “cylc submit”. The second task name would presumably need to be parametric to avoid task ID conflicts from multiple invocations of the second task – which would mean the maximum range of the parameter would need to be big, to accommodate any possible number of times the second task would need to be invoked (the maximum number of times needed for the first task’s loop). That’s inelegant but I hadn’t thought of a better idea yet.

OR

  • Invoke the second task upon first task start-up, but have it sit around and do nothing, polling the first task’s output files until the first task completes a loop iteration, at which point it processes the output and then goes back to polling. In addition to being ugly, this is a bad solution because it means the second task will be spending most of its time doing nothing at all but polling, and occupying parallel processing cores (and chewing up batch allocation hours) while doing so.

If I knew how many times I’d need to run the first binary, this would all be trivial: I’d just graph out something like:

task01_iter01 => task01_iter02 => task01_iter03
task01_iter01 => postproc_iter01
task01_iter02 => postproc_iter02
task01_iter03 => postproc_iter03

. . .and in reality I’d do it with parametrized tasks to make this cleaner. But I do not know how many times it’ll need to run. However, your comment about “cylc submit” being removed in Cylc 8 is news to me – and that really makes me wonder how to accomplish this requirement.

Thanks.

4

I just thought of one other way to do it – it’s ugly but I’ve done something like this before.

Using parametrized tasks, define a graph like the one above with ~100 iterations (that’s about the maximum amount of times I imagine the first binary needing to be run). Precede the first task with a task that determines the number of times the first binary will need to be run, and uses a custom output to provide that count. Suicide off that portion of the graph that’s in excess of what that initial, number check task says. That’s really ugly, but I think it would work . . .

5

Another idea: sub-suites can be a more elegant way of handling dynamically determined parts of a workflow.

From a quick read of your post above, your “internal” workflow could be expressed simply as an integer cycling suite with just two tasks in it. Take the final cycle point value from an input variable (Jinja2). Then have a task in your main suite run the sub-suite (with —no-detach) with the right number of cycles supplied on the fly. Register the sub-suite each time with the current main suite cycle point in its name, so that each sub-suite run gets a new run directory.

Would that work for you?

6

Suicide off that portion of the graph that’s in excess of what that initial, number check task says. That’s really ugly, but I think it would work . . .

Bit ugly but that should be possible, solving this in the graph will be tricky, the best I could come up with is to reset the excess tasks as expired (moving the problem into the runtime section):

#!Jinja2    
                                                                   
{% set MAX_RUNS = 10 %}                                            
                                                                   
[cylc]                                                              
    [[parameters]]                                                  
        n = 1..{{ MAX_RUNS }}                                      
    [[parameter templates]]                                        
        n = %(n)d                                                  
                                                                   
[scheduling]                                                        
    initial cycle point = 1                                        
    cycling mode = integer                                          
    [[dependencies]]
        [[[R1]]]
            graph = """
                # run binary sequentially
                start => binary<n=1>
                binary<n> => binary<n+1>

                # run processing async when the "output" message is received
                binary<n>:output => processing<n>
             
                (binary<n> | binary<n>:expire) \
                & (processing<n> | processing<n>:expire) => end
            """
             
[runtime]    
    [[binary<n>]]
        script = """
            sleep 2
            cylc message -- output
            if [[ $run_number -gt 3 ]]; then
                # break the chain
                for num in $(seq $(( run_number + 1 )) $max_runs); do
                    cylc reset \
                        "$CYLC_SUITE_NAME" \                        
                        "binary${num}.$CYLC_TASK_CYCLE_POINT" \
                        -s expired
                    cylc reset \                                    
                        "$CYLC_SUITE_NAME" \
                        "processing${num}.$CYLC_TASK_CYCLE_POINT" \
                        -s expired                                  
                done                                                
            fi                                                      
        """                                                                        
        [[[parameter environment templates]]]                                      
            run_number = %(n)d                                                    
        [[[environment]]]                                                          
            max_runs = {{ MAX_RUNS }}                                              
        [[[outputs]]]                                                              
            output = output

The good news it that the new graph branching functionality of Cylc 8 may help out in the future. This Cylc 8 workflow does the same as the above example, but the excess tasks aren’t spawned so the problem can be more neatly contained in the graphing:

[scheduler]
    allow implicit tasks = true

[task parameters]
    n = 1..5
    [[templates]]
        n = %(n)d

[scheduling]
    [[graph]]
        R1 = """
            # run binary sequentially
            start => binary<n=1>
            binary<n> => binary<n + 1>

            # run processing async when the "output" message is received
            binary<n>:output => processing<n>

            # break the chain when the "finished" message is received
            binary<n>:finished => !processing<n + 1>
            binary<n>:finished => !binary<n + 1>

            # wait for scheduled tasks
            binary<n>:finished & processing<n> => end<n>
        """

[runtime]
    [[binary<n>]]
        script = '''
            sleep 2
            cylc message -- output
            if [[ $number -gt 3 ]]; then
                cylc message -- finished
            fi
        '''
        [[[outputs]]]
            output = output
            finished = finished
        [[[environment]]]
            number = %(n)d
7

Hi Hilary. I’m not sure whether sub-suites would help or not because I’m not sure I understand them, or understand the example framework you describe in your post! Is there an example of their use somewhere that I can look at? Thanks.

8

Both of these are cleaner than what I had in mind, which was something like (assuming 3 is the max possible iterations just for simplicity’s sake):

[scheduling]
    [[dependencies]]
        (other stuff omitted)
        [[[ ^+PT00H15M/PT00H15M ]]]
            earlier_stuff => count_checker

            count_checker:3 => task01_3
            task01_3 => task02_3

            count_checker:2 | task01_3 => task01_2
            task01_2 => task02_2

            count_checker:1 | task01_2 => task01_1
            task01_1 => task02_1

            count_checker:1 => !task01_2 & !task02_2 & !task01_3 & !task02_3
            count_checker:2 => !task01_3 & !task02_3

. . .which doesn’t look bad at all when there’s only 3, but gets nasty quickly when the maximum count gets big. Parametrized tasks help, but you have to use intermediate dummy tasks for the suicides because the list of tasks to suicide become crazy long and I’ve found that Cylc can have trouble with long dependency lines.

9

It’s pretty easy - here’s a working mock-up based roughly on your requirements above (if I’ve understood them).

Main suite, in ~/suites/main/suite.rc:

[cylc]
  UTC mode = True
[scheduling]
  initial cycle point = 2020
  [[dependencies]]
    [[[P1D]]]
      graph = "foo => run-subsuite => bar"
[runtime]
  [[foo, bar]]
    script = sleep 10
  [[run-subsuite]]
    script = """
      N_RUNS=$((1 + RANDOM % 5))  # determine no. of iterations on the fly
      SUBNAME=${CYLC_SUITE_NAME}-sub-${CYLC_TASK_CYCLE_POINT}
      cylc register ${SUBNAME} ${CYLC_SUITE_DEF_PATH}/sub
      cylc run --no-detach --set="FINAL_CYCLE_POINT=$N_RUNS" ${SUBNAME} 
    """

Sub-suite, in ~/suites/main/sub/suite.rc:

#!Jinja2
[scheduling]
  cycling mode = integer
  initial cycle point = 1
  final cycle point = {{FINAL_CYCLE_POINT}}
  [[dependencies]]
    [[[P1]]]
      graph = "task[-P1] => task => postproc"
[runtime]
  [[task, postproc]]
     script = sleep 10

Your iterations are handled as integer cycles in the sub-suite. Each sub-suite run is represented by a task in the main-suite, which registers and runs the sub-suite, passing the number of iterations to it as the final cycle point. Run sub-suites with --no-detach so that the main-suite task does not finish until the sub-suite is done. Multiple sub-suites can run concurrently if the main-suite dependencies allow that.

Note Cylc does not actually have special built-in support for sub-suites; we’re just running one suite inside of a task in another suite. Consequently each running sub-suite appears as (and in fact is) a full suite in its own right. Stopping the main suite will not automatically stop any child sub-suites that are running at the time (however you can monitor and manage them just like any other suite).

If you run this, the suite run directories will look like:

$ ls -lrt cylc-run
total 0
drwxr-xr-x 6 oliverh unix-users 105 May 11 12:17 main
drwxr-xr-x 6 oliverh unix-users 105 May 11 12:17 main-sub-20200101T0000Z
drwxr-xr-x 6 oliverh unix-users 105 May 11 12:18 main-sub-20200102T0000Z
drwxr-xr-x 6 oliverh unix-users 105 May 11 12:18 main-sub-20200103T0000Z
drwxr-xr-x 6 oliverh unix-users 105 May 11 12:18 main-sub-20200104T0000Z

If your main-suite has many cycle points you will probably want to add a housekeeping task (to the main suite) that extracts the important results from finished sub-suites and then deletes their run directories.

Personally I think sub-suites are by far the most elegant way to handle dynamically determined iterative processes in every main suite cycle point - so long as you realize they are actually first-class suites in their own right, and manage them accordingly.

Hilary

1 Like
10

(Sub-suites are described in the current Cylc 7 user guide, but no examples are given there: 13. Running Suites — The Cylc Suite Engine 7.9.3 documentation)

11

In trying out other approaches to this via toy models, I did something that isn’t mentioned in the Cylc docs but seems to work. I had created a task with output messages that corresponded to a defined parameter, and used those output messages to trigger a parametrized task:

[cylc]
    [[parameters]]
        fs = 1..3
    [[parameter templates]]
        fs = %(fs)01d

[scheduling]
    [dependencies]
        [[[R1]]]
            test_checker:<fs> => test<fs>

[runtime]
    [[test<fs>]]
        script = """
            sleep 10
                """

    [[test_checker]]
        script = """
            cylc message -- ${CYLC_SUITE_NAME} ${CYLC_TASK_JOB} 2            
                """
        [[[outputs]]]
            1 = "1"
            2 = "2"
            3 = "3"

This surprisingly worked – the “test_checker” task exited with a “2” output message, which was a defined output that was covered by the parameter. And then task “test2” was kicked off, but the other tasks “test1” and “test3” were not. But I hadn’t known that parameters could be used in suite.rc not only in references to task names, but also to task output messages. Is this intended to be available? As near as I can tell, it’s undocumented; and undocumented features could go away at any time, so I don’t want to depend on something that’s an unintentional capability . . .

Thanks!

12

Ooh, I like this a lot – and it is more elegant, I agree. Yes, there’d be multiple such subsuite run directories – probably 24 a day, 365 days a year – so I agree that a housekeeping task would be necessary. But I’m going to give this a try, thanks!

1 Like
13

Your example (after correcting a couple of typos) doesn’t validate for me at cylc-7.8.3, I get a bad graph node format error. But even if it did, no that’s not intentional (and you’d want to generate the outputs in the task definition section by parameter too, not write them out one by one). We have discussed extension to custom message outputs, but decided to punt the issue to the future Python API: parameterization: make parameters avaliable to other suite.rc sections · Issue #2453 · cylc/cylc-flow · GitHub

So, built-in parameters are intended for cleanly generating task names only. If you need more general parametrisation you can do it with Jinja2 loops:

#!Jinja2
{% set N=4 %}

[scheduling]
  initial cycle point = 2020
  [[dependencies]]
    [[[R1]]]
      graph = """
{% for fs in range(1,N) %}
         foo:{{fs}} => bar{{fs}}
{% endfor %}
              """
[runtime]
  [[foo]]
    [[[outputs]]]
{% for fs in range(1,N) %}
      {{fs}} = "{{fs}}"
{% endfor %}

{% for fs in range(1,N) %}
    [[bar{{fs}}]]
        # ...
{% endfor %}

Note you can combine the two approaches too, to eliminate some of the loops. E.g. for the above, you could use the Jinja2 variable N in the built-in parameter range definition.

14

A post was split to a new topic: Suite and task environment variables