Temporarily make all optional outputs required

Cylc Cylc Support
1

Is there any way of temporarily making optional outputs in the graph required? Either a command line option or a config setting?

As background, one of my cylc7 suites is designed to do everything it can to continue cycling without intervention. However, that makes it somewhat hard to debug/develop, so it has a “DEVMODE’” where jinja2 is used to disable the suicide triggering so that the suite stalls if there is an error.

The jinja2 for this at cylc7 was fairly concise as all of the suicide triggering was centred around a single ‘handle_failures’ dummy task, replacing every single question mark in the cylc8 graph with jinja2 will get quite messy.

2

No, you can’t do that. E.G. for this graph:

a:succeeded? => x
a:failed? => y

Making optional outputs required would make both :succeeded and :failed required outputs which wouldn’t work because they are mutually exclusive. In this example it’s reasonable to presume that a:succeeded? is the intended success pathway, so you could argue that Cylc should ignore :failed? for this case, but in this example it’s less clear which the success pathway is:

a:x? => x
a:y? => y

However, there are some things that you can do to work around this.

E.G. I’m guessing you’re using :failed? to decide which pathway Cylc takes through the graph as in the example above. Another option would be to use a custom output to do this:

a:succeeded? => x
a:error? => y

[a]
  script = """
    do-something || true
    if [[ $? == 42 ]]; then  # detect a failure condition you are willing to tolerate
      cylc message -- error
    fi
  """
  [[outputs]]
    error = error

This is somewhat safer than using :failed? because it only applies to whitelisted cases whereas failure could be caused by something as simple as a syntax error where you would likely want the task to fail, even in production mode. E.G. If an operator broadcasted an environment variable which caused an unset variable error we would want the task to fail and the workflow to stall if needed.

With this approach you could swap out cylc message -- error for exit 42 to toggle between “the show must go on” production mode and “brittle” development mode.

We currently have some ideas in the works which would make this sort of pattern a bit more attractive and might possibly open up another alternative depending on the specifics.

3

Thanks Oliver,
I have some TODO notes scattered in my code about using message to send emails for non-cricital failures but I hadn’t thought about it for controlling the branching.