Dynamic Task Wall Clock Limit Depending on ${CYCLE_TASK_CYCLE_POINT}

Cylc Cylc Support
1

Hi There,

I’m curious if there is a way to cleanly represent a dynamic wall clock value in a flow.cylc file task definition as a function of the ${CYLC_TASK_CYCLE_POINT} variable in Cylc 8.3.4. My use case is the following where, in a retrospective reforecast case study, all extended forecasts are designed to terminate at a specific valid ISO date for verification of an event. In particular, I see that the isodatetime library can produce the number of forecast days until the specified valid date, e.g.,

isodatetime ${CYLC_TASK_CYCLE_POINT} {{EXP_VRF}} -f 'd'

where {{EXP_VRF}} is a specified Jinja2 ISO date for the forecast verification. I’d like to pass the returned value to an arithmetic operation to multiply this by the run time required per day of simulation, something like the following non-working code

execution time limit = $(isodatetime ${CYLC_TASK_CYCLE_POINT} {{EXP_VRF}} -f 'd')*{{DAY_WC}}

However, I’m unsure of what the valid syntax would look like to:

  • perform the multiplication arithmetic; and
  • format this value as a valid ISO duration.

Is there a clean way to perform this currently? I understand that I can always just set this to the max wall clock limit for any extended forecast in the case study, but to help the throughput of the extended forecasts on the HPC system, it would be nice to set this dynamically.

Thanks!
Colin

2

Hi @cgrudz

Dynamically varying job resource requests by cycle point - that’s interesting requirement that I have not seen before!

This won’t work as you intend regardless of formatting the result etc., because execution time limit is a config item (whose values are set when the workflow configuration is parsed), whereas $CYLC_TASK_CYCLE_POINT is a job runtime environment variable (and job files are created on the fly for each task instance, and not interpreted by the shell until the job actually executes).

In principle it would be possible to allow some kind of templating of batch system directives at job file creation time, at which point the specific cycle point of the task instance is known, but we don’t have that capability at the moment.
[UPDATE: wrong! see @wxtim’s comment on cylc broadcast below.]

3

[UPDATE: my suggestion below in this post is not very helpful - see the better cylc broadcast suggestion in the next post!]

A crazy idea, if you really need this capability: run your forecast model via a non-cycling single-task sub-workflow with execution time limit computed in main workflow task instances at runtime, and passed (via a Jinja2 parameter) to each sub-workflow instance at start-up.

Untested example, with some important details omitted:

# main workflow
...
[runtime]
   [[forecast]]
        job runner = background
        script = """
             LIMIT=$(isodatetime $CYLC_TASK_CYCLE_POINT blah ...)
             cylc play --no-detach sub --set="TIME_LIMIT=$LIMIT"
        """

#!Jinja2
# sub workflow
[scheduling]
    [[graph]]
        R1 = model
[runtime]
    [[model]]
        job runner = pbs
        execution time limit = {{TIME_LIMIT}}
        script = "model.exe"

In the unlikely event that you want to put up with the additional complexity of this approach just to get more parsimonious resource requests, you’ll need to read about managing sub-workflows. I don’t think we’ve added an example to the user guide yet, but I’ve documented it pretty comprehensively here: GitHub - hjoliver/cylc-subwf-example: Cylc 8 sub-workflow example

4

@hilary.j.oliver, all interesting stuff and thanks for explaining how the variable ${CYLC_TASK_CYCLE_POINT} is created on the fly versus how the configuration is parsed. This isn’t a major item for me, as the simple uniform wall clock limit for all extended forecasts will be totally adequate. Cheers!

1 Like
5

Your other option would be to use Cylc broadcast, something like this:

[scheduling]
    [[graph]]
        P1 = set_resource => do_thing

[runtime]
    [[set_resource]]
        script = """
            VALUE=$(whatever calculation you need)
            SETTING="'execution time limit=${VALUE}'"
            cylc broadcast "${CYLC_WORKFLOW_ID" \
                --point "${CYLC_TASK_CYCLE_POINT}" \
                --name "do_thing" \
                --setting ''${SETTING}"
        """
3 Likes
6

(Yes, a much better option than the one I gave!)

[Update, to explain how this works, for the curious:]

The cylc broadcast command tells the scheduler to update the values of given task config items on the fly, at run time. (And you can target all tasks, or all tasks in a given cycle point, or a family of tasks, or an individual task).

In this case, another task in the same cycle point (which therefore knows the cycle point value) can compute the forecast length and broadcast the corresponding new execution time limit value for the model task (so they scheduler will use the new value when it instantiates the next model task and writes its job script).

2 Likes
7

@wxtim and @hilary.j.oliver, thanks for this demonstration of the cylc broadcast functionality, that makes a lot of sense what you are describing and I think I can see how this can be used very effectively in handling exceptional cases a variety of non-standard cycles. Cheers!

1 Like