Cylc 8 migration - :succeed or :finish?

Cylc Tips
1

Quick Overview

Sometimes the :finish, qualifier (or its :finish-all & :finish-any counterparts for families) were used erroneously in Cylc 7 workflows where :succeed (or its :succeed-all & :succeed-any counterparts) would have been more appropriate.

This may cause validation failures when you try to upgrade to Cylc 8, similar to this one like this:

GraphParseError: Output foo:succeed can't be both required and optional

If you encounter errors like this where :finish triggers are present, consider whether you want to permit the task to fail and allow the workflow to move on to the next task.

If you don’t, then :finish was probably a mistake, switch it for :succeeded and the error will go away.

Otherwise have a look at the graph branching section of the migration guide for upgrade advice.


Full Explanation

:succeeded vs :finished

In Cylc graphs, if you don’t specify a qualifier, Cylc assumes you mean :succeed.

For example this:

foo => bar

Is shorthand for this:

foo:succeed => bar

There is a special qualifier called :finished which is shorthand for :succeed or :fail.

For example this:

foo:finished => bar

If equivalent to this:

foo:succeed | foo:fail => bar  # | means "or"

The same goes for the :finished-all and :finished-any qualifiers which can be used with families.

Misconception

In Cylc 7, :finished was sometimes used by mistake in situations where :succeed would have been more appropriate. Especially with family triggers (e.g. FAMILY:finish-all) where the difference between :finish-all and :succeed-all might not have been so obvious to the person who wrote the workflow.

Issues Upgrading :finish to Cylc 8

Cylc 8 is smart enough to know that :succeed and :fail are opposite outcomes, so it doesn’t allow them to both be required.

For example this workflow:

foo:succeed => bar  # foo must succeed
foo:failed => baz   # foo must fail

Will fail validation:

$ cylc validate .
GraphParseError: Opposite outputs foo:succeeded and foo:failed must both be
optional if both are used

To fix the error, we put question marks after the outputs:

foo:succeed? => bar  # foo may succeed
foo:failed? => baz   # foo may fail

Putting a question mark after a task output tells Cylc that it is optional (i.e. this output may or may not be generated when the task runs).

At Cylc 8 :finish is equivalent to :succeed? | fail?, so this:

foo:finish => bar  # foo may either succeed or fail

Is equivalent to:

foo:succeed? | foo:failed? => bar  # | means "or"

Which means that this workflow:

foo:finish => bar  # foo MAY either succeed or fail
foo => baz         # foo MUST succeed

Will fail validation:

$ cylc validate .
GraphParseError: Output foo:succeed can't be both required and optional

Because one of the foo:succeed triggers has a question mark after it and the other doesn’t.

How To Fix Issues & Upgrade To Cylc 8

If you encounter this situation, there are two possible fixes.

  1. If the :finish trigger was used by mistake, swap it for :succeed.

    foo => bar  # foo must succeed
foo => baz  # foo must succeed

  2. If the :finish trigger was used purposefully, add question marks to allow the task to fail.

    foo:finished => bar  # foo is permitted to fail
foo? => baz          # foo is permitted to fail

    See graph branching for more information about those question mark symbols and how to direct Cylc to take different paths through the graph according to runtime events.

2

Is this captured in cylc lint for example, in case it is used but isn’t causing validation errors?

3

It is not possible for us to catch this with cylc lint as the usage may be purposeful and often is. The issue is cases where :finished had been used in error in Cylc 7 workflows. An issue which will come up during Cylc 8 migration as graph validation is now stricter.

4

This is to cylc lint what “Duplicate” and “Duplicitous” are to a spell checker - both valid, but one of them is very, very, wrong if the other was intended.

1 Like