Adding new task to succeeded workflow, explicit insertion?

Cylc Cylc 8 Migration
1

I am developing a workflow and I add a new task on to the end of the sequence. I have already run the existing tasks, and it all succeeded and shut down.

I cylc reinstall, cylc play --pause. Then I can trigger the new task.
Given that insertion is now implicit and we can just trigger the new task, why do I need to pause the workflow in order to do this. Why does the workflow not notice the new task and immediately shut down if I don’t pause it? Is this how it is intended to work? It was the opposite of what I expected.

Relatedly if I re-trigger the preceding task, which succeeds, tui now appears to show the new task. However, once the preceding task has finished, the workflow shuts down again, so the scheduler hasn’t registerd the new task.

I am now running cylc 8.0.0

This does occur with a simple example. e.g.
R1="a => b", [[a,b,c]]script=sleep 10
Then
R1="a=>b=>c"
cylc reinstall, cylc play --pause, cylc trigger ...//1/b (or c)

2

Given that insertion is now implicit and we can just trigger the new task, why do I need to pause the workflow in order to do this. Why does the workflow not notice the new task and immediately shut down if I don’t pause it?

Yes, this is the intended behaviour, here’s an explanation as to why:

The Cylc scheduling algorithm follows a single logical execution of the graph. This path begins with the start task (or tasks) and evolves naturally in response to task outputs. When tasks generate outputs, Cylc follows these to downstream tasks and “inserts” them as appropriate.

You can shut down/restart pause/play a workflow as many times as you like, Cylc will always preserve it’s place in this logical execution of the graph and continue where it left off.

Eventually the pathway Cylc is following runs out and there is nothing more for it to do, no more tasks to schedule, so Cylc shuts down, job done.

If you restart a workflow after it has completed, Cylc loads its state from the database, sees that there is nothing more to do and shuts down because the execution pathway has run out. The workflow cannot continue unless you tell it where you want it to continue from.

Consider this example:

P1 = a => b

If you run this workflow, then shut it down and make the following change to the graph:

P1 = a => b => c

What would you expect to happen when you restart the workflow?

  • Should the task c be inserted and run in every cycle?
  • Or just the last one?
  • Or just to new cycles?

Because Cylc is following a single execution of the workflow this new task would only apply to new/active cycles, not to old ones. The rules are exactly the same if the workflow has run to completion.

For Cylc to “continue” the workflow at this stage involves creating a new logical execution of the workflow starting from a new start task (or start tasks). This is a new feature of Cylc 8 implemented via the --flow option. Cylc cannot safely “guess” the start tasks for you (as it does when you first start the workflow).

You can think of Cylc’s execution of a graph like a wavefront. Everything ahead of the wave is in the future, everything behind is in the past. Only the wavefront itself is “active”. Once the wavefront dies out it’s gone. This is why you need to “trigger” the new task, the act of triggering the task creates a new wavefront for Cylc to process.

It was the opposite of what I expected.

We’re open to collaboration, it would be good if we could make this situation more intuitive.

Relatedly if I re-trigger the preceding task, which succeeds, tui now appears to show the new task. However, once the preceding task has finished, the workflow shuts down again, so the scheduler hasn’t registerd the new task.

I think Cylc should continue on in this case will take a look.

3

Yes, as per @oliver.sanders’ reply, if you add a new task to the graph, after the upstream task that would trigger it has already finished, then you restart the scheduler … in general Cylc can’t guess if you want “past” instances of the new task to trigger immediately off of past instances of the upstream task that already finished (as opposed to kicking in only in future cycle points).

If there happens to be only an R1 graph (i.e., no cycling) then I suppose the need to run the new task could be inferred, but not in general.

Note that in the cycling case, subsequent instances of the new task will be triggered automatically by the upstream parent.

Here’s one idea: if a completed workflow gets restarted, we could keep the scheduler alive on a timeout (say 10 minutes) to allow the user to manually trigger some new action.

A manually triggered task belongs by default to all existing flows. (If you watch the scheduler log, you’ll see flow=['all'] in logged as a command argument.). However, because your workflow had already run to completion, there are no existing flows, and the triggered task ends up with flow=none, i.e. a one-off task run that will not trigger an ongoing flow. So your new task c does not get triggered!

What you can do is tell the scheduler that the triggered task belongs to flow 1, and then the new task c will be triggered, because it has not yet run in flow 1.

cylc trigger --flow=1 test//1/b

We’ll consider whether or not the default flow assignment needs to be tweaked in the case of a restarted already-completed workflow…