Optional outputs

Cylc Cylc Support
1

[cylc 8.1.4]

I am using ? to set optional output dependencies on a task, but got stuck for a while with the error:

GraphParseError: Output XXXXX:succeeded can't be both required and optional

Here’s a simple example of what I was doing:

[scheduling]
    [[graph]]
        R1 = """
            a => b
            b? | b:expired? => c
        """

This produced the above error. I can only get the error to go away if I add ? to the first dependency, i.e.:

[scheduling]
   [[graph]]
       R1 = """
           a => b?  # <--- why?
           b? | b:expired? => c
       """

but I am struggling to understand why I need to do this. This line is surely just setting the dependencies of b triggering based on a? It is not setting any dependencies on b’s output. Is this by design, or a bug? I can’t quite get my head around it!!

The Cylc8 documentation has an example like my code above here: Scheduling Configuration — Cylc 8.1.4 documentation, copied below for reference:

foo => bar
bar:fail? => recover
bar? | recover => baz

Thanks!

2

Unfortunately that example in the docs is wrong, it fails with the same error as your graph. We’ll fix it!

3

OK - thanks @hilary.j.oliver .

It wasn’t clear to me from the docs that the first dependency a => b needed an optional output on b.

I know the docs say that if one graph entry has an dependency on an optional output then it must be declared for all graph entries, but I can’t quite understand what that means in the context of a => b?.

a will always trigger b, regardless of whether it has optional outputs.

Sorry if I am being a bit dim here !!! :smiley:

4

Not dim at all, it’s quite a change from Cylc 7!

The arrows (rather obviously) set the dependence of the downstream task on a particular output of the upstream one.

But additionally, in Cylc 8, every appearance of a task in the graph states - whether explicitly or implicitly - the optional or required nature of the relevant task output, and that goes for the downstream task as well.

Every reference to a particular task output has to be marked optional if any are, to avoid ambiguity - so you can look at (say) b (or b?) anywhere in the graph and instantly see if that its success is required (or optional).

Even if there is nothing downstream of a task, this matters. This (below) says, if foo succeeds, trigger bar. It also says both foo and bar are required to succeed (if they run at all) in order for the workflow to finish successfully:

foo => bar

If foo fails, bar will not trigger, and foo will be marked as an incomplete task (that did not complete its required outputs), and the as a result the workflow will stall with a warning rather than shut down as finished if there is nothing else to do.

This (below) also says, if foo fails bar will not trigger and foo will be marked as incomplete. BUT this time, if foo succeeds and bar runs, success of bar is optional (i.e. we expect that sometimes bar might fail and that is OK - either it doesn’t matter, or we have another path in the graph to handle the failure) and the workflow can shut down as complete successfully either way:

foo => bar?

I hope that makes a bit more sense?

1 Like
6

Thanks @hilary.j.oliver for the detailed explanation.

Yes - I can see how that makes sense now.

The pertinent bit that I was not appreciating is that not having b as optional, even when it has no downstream tasks, was implying that b could not fail…

I think I have this straight in my head now. Many thanks for taking the time to answer! :slight_smile:

1 Like
7

One thing to remember is that foo => bar is short for foo:succeed => bar:succeed, so Cylc expects bar to succeed and therefore failure is not an option, unless you add the question mark to bar:succeed?

1 Like
8

Ha - that’s my take away from this thread! :grinning:

2 Likes