Run task at last cycle point after all instances of another task are successful

Cylc Cylc Support
1

Hi!

I am using cylc 8.3.3 and I’m having trouble finding how to specify my complex task dependence.

In short, I have task A and task B. Task A happens “every year” (P1Y) while task B should be run only on the last cycle point and after all instances of task A are successful.

The simple way, would be to make all instances de A depend on the previous one, and B would depend on the last A. However, this would forbid running multiples instances of A in parallel, which is possible and would improve performance in my case. So is there something alike:

R1/$ = """
    task_a[*] => task_b
""""

?

2

No, nothing like that.

This is a fairly common question. We’ve had a solution implemented (for two years!) - separate start-up and shutdown graphs that don’t need to be connected to the main cycling graph by dependencies - but it hasn’t been merged and released yet because of higher priorities during Cylc 8 development.

For the moment, here’s an integer cycling example showing how to make b depend on only the last-cycle a:

[scheduler]
    allow implicit tasks = True
[scheduling]
    cycling mode = integer
    initial cycle point = 1
    final cycle point = 5
    [[graph]]
        P1 = a  # every cycle
        R1/$ = a => b  # once at last cycle only
[runtime]
    [[root]]
        pre-script = sleep 5

If I run this, it does the right thing:

 cylc log bug | grep '=> submitted'
2024-11-02T09:08:11+13:00 INFO - [1/a/01:preparing] => submitted
2024-11-02T09:08:11+13:00 INFO - [2/a/01:preparing] => submitted
2024-11-02T09:08:11+13:00 INFO - [3/a/01:preparing] => submitted
2024-11-02T09:08:11+13:00 INFO - [4/a/01:preparing] => submitted
2024-11-02T09:08:11+13:00 INFO - [5/a/01:preparing] => submitted
2024-11-02T09:08:20+13:00 INFO - [5/b/01:preparing] => submitted

So 5/b can only run after 5/a succeeds.

However (warning!) there are no dependencies to prevent 5/b running before 3/a (say) finishes if there’s any chance on your system that instances of a could actually run (or at least finish) out of order.

As you note, P1 = "a[-P1] => a", would do it, but then instances of a can’t run in parallel.

To enforce ordered execution without preventing parallel running you could do this:

P1 = "a[-P1]:started => a"

… but there might still be a small risk that a instances start in order but finish out of order (unlikely if a has a consistent run time).

If it really matters, you could make task b depend on an xtrigger that interrogates the workflow DB to check that all instances of a succeeded.

3

Thanks for the answer!

I came up with this other solution, it’s bit verbose, but seems to do the job of having task_b only be run once all task_a instances are done.

P1Y = """
    task_a => task_a_done
    task_a_done[-P1Y] => task_a_done
"""

R1/$ = """
    task_a_done => task_b
"""

.

4

Yep, good solution! You’ll want to make those dummy tasks run as local background jobs. Well soon have a new feature (skip mode) that will allow them to run as simulated tasks.

1 Like