Preventing alternate and main task from running and inconsistent task status

Cylc Cylc Support
6

Here is the link to the earlier post Blocking a task being triggered if its recovery task is active - Cylc / Cylc Support - Cylc Workflow Engine

Thanks, that’s useful.

If I understand correctly, you are worried that operators might manually trigger the forecast task whilst the forecast_shortstep task is running causing problems?

If so, I would recommend using a single forecast task and using cylc broadcast to toggle the shortstep behavior on and off. This is how we work at the MO.

Here’s an over-simplified example:

[runtime]
    [[forecast]]
        script = my-model
        [[[environment]]]
            SHORTSTEP = false

To turn shortstep on, issue a broadcast e.g:

$ cylc broadcast <workflow> -p <cycle> -n forecast -s '[environment]SHORTSTEP=true'

Any subsequent submission of this task will now run with SHORTSTEP=true.

You can automate this using “retry event handlers”. Here’s an example where the task will run with SHORTSTEP=false the first time, then all subsequent attempts will run with `SHORTSTEP=true":

[runtime]
    [[forecast]]
        script = my-model
        execution retry delays = 4*PT1M
        [[[events]]]
            retry event handlers = cylc broadcast $CYLC_WORKFLOW_ID -p $CYLC_TASK_CYCLE_POINT -n $CYLC_TASK_NAME -s '[environment]SHORTSTEP=true'
        [[[environment]]]
            SHORTSTEP = false

It’s likely your example isn’t quite as simple as mine above:

  • There might be multiple environment variables to change.
  • You might need to extend the execution time limit.
  • You want to retry three times before falling back to shortstep.

To handle this extra complexity, you might want to move the logic into a short script (e.g. in the workflow’s bin/ directory)

# flow.cylc

[runtime]
    [[forecast]]
        script = my-model
        execution retry delays = 4*PT1M
        execution time limit = PT30M
        [[[events]]]
            retry event handlers = shortstep-handler %(workflow)s %(cycle)s %(task)s %(try_num)d
        [[[environment]]]
            SHORTSTEP = false

# bin/shortstep-handler

set -euo pipefail

WORKFLOW="$1"
CYCLE="$2"
TASK="$3"
TRY_NUM="$4"

if [[ $TRY_NUM -gt 3 ]];
    # the task is on its third retry
    cylc broadcast "${WORKFLOW}" \
        -p "${CYCLE}" \
        -n "${TASK}" \
        -s '[environment]SHORTSTEP=true' \
        -s '[environment]FOO=bar' \
        -s 'execution time limit=PT1H'
fi

For a list of the %(variable)s available to event handlers (e.g. retry event handlers), see task event template variables.

2 Likes