Why are expired/succeeded tasks counting towards runahead limit?

Cylc Cylc Support
1

Not a biggie for me now, as I’m intending to rework my suite to avoid having cycles with such different cadences. But posting for curiosity’s sake - and in case answers help anyone else in future!
NB can’t see answer there, but Discourse flags this q as similar to fredw’s post on Robust coding of expired tasks.

My suite wraps a ~nowcast-y model which continually consumes 1 min cadence input data (which also gets refreshed at 1 min intervals). The model itself is intended to run at a lower cadence - I’ve currently configured this to 15 min cycles in dev mode, but this is intended to be OoM ~1-3 hours when running operationally.

My current implementation is ~naive: a low-cadence cycle for the model, and a high-cadence cycle for the input data, where the latter is implemented as one-cycle-per-minute-of-input-data. So given 15 min model cycles, I have 15 x 1 min input data cycles.

As the model cycle includes some prerequisites for the input data cycle, which can take a little time to run, I’m using a clock-expire on input data cycle/tasks which attempt to start too early (before model itself has started running). This approach happily expires these over-early input data tasks, but these expired input data tasks stick around in the GUI “queue”, as do the subsequent input data tasks which ones which run later (once model setup complete) and hence succeed.

These expired/succeeded input data cycles build up until suite reaches the runahead limit I’ve configured: max active cycle points = 10. See e.g. screenshot below - note 10 cycles there, 9 of which are input data cycles (top green one = the 15 min model cycle, still running at this point).

And then things pause (note 21:39 timestamp in screenshot: my expected 21:38 & 21:39 input data cycle points haven’t appeared). This “pause” remains until my lower cadence model cycle completes. Once this happens, the input data “backlog” clears, and the remaining ~6 input data cycles finish (1st couple expiring as have become stale during pause; subsequent cycles running on the minute as expected).

These remaining input data cycles behave more normally: tasks/cycles which have expired/completed disappear off the GUI queue, and don’t accumulate.

For various reasons, I’m intending to overhaul my implementation, such that my input data processing is done as a single task, on the same cadence as the model cycle (so I don’t have multiple cadence cycles going on).
This input data task will run in parallel with the model task, with the per-minute input data processing achieved via an internal time-aware process/sleep while loop within the task).
This is mostly to avoid swamping the scheduler of target platform, but also to simplify suite a little - and hopefully avoid issues like this!

However, I’m curious as to why I’m seeing this behaviour - and why the difference: accumulation of expired/succeeded input data tasks (seemingly until my runahead limit used up) until initial cycle completes, then expected non-accumulation thereafter.

Can anyone shed any light? Happy to share a potted version of my suite graph if it helps (e.g. if it’s not immediately obvious what my misunderstanding/mistake is!)

image
image323×563 75.5 KB

2

In general terms, the runahead limit is anchored by the slowest (ie, oldest cycle point) waiting tasks, and finished tasks accumulate ahead of them in case they (the finished tasks) are needed to satisfy the prerequisites of future tasks that could yet be spawned by the slow ones.

If you have quite drastically different cycling intervals (cadences, as you put it) in the same suite, and the longest intervals correspond to the slowest tasks, then the accumulated finished tasks will be cleaned up suddenly en masse once the slow ones finish and spawn beyond them. And then the cycle of accumulation and clean-up should start again.

Does that correspond to what you’re seeing? The initial cycle point shouldn’t behave any differently from subsequent cycles, so maybe their is something else going on that isn’t immediately obvious to me (to do with the expired tasks?). If so it might help to see a cut down version of your suite definition that exhibits this behaviour.

It’s fair to say that runahead limiting is a fairly crude mechansism to limit workflow activity over multiple cycles, and it is particularly crude if you have very different cycling intervals in the same workflow. This is something else that is vastly improved by the new scheduling algorithm in Cylc 8 (there’s no accumulation of finished tasks at all across the window of active cycle points).

1 Like
3

Thanks Hilary, that sounds a close-enough description of behaviour I’m seeing that I don’t think it’s worth digging further. Have put stripped-down graph below purely for completeness, in case helps others. While R1 tasks are involved, it doesn’t seem like they’re behaving any differently from what you describe ^^^.

Thanks v m for the insight - most helpful! Can I just check my current understanding based on your reply please?

I think I’ve been playing in the ~simple shallows of graphing, so had basically assumed graph was ~contractual on prereqs of future tasks, and that cylc would be parsing the graph at startup → so would know a priori that the finished/expired tasks in question couldn’t be prereqs for any as-yet-to-be-spawned-tasks-from-slow-task.

However thinking about it more, given all the possible complex graphs I could be writing, with suicide triggers / failure routes etc, I guess that there isn’t necessarily a small set of paths through “graph space”, so it may be that there’s no point trying to do a priori analyses (as the set could blow up coping with all contingencies along the way). So the sensible approach is more empirical - accumulating actually-eventuated tasks/cycle points as you describe just in case needed later, then release on-the-fly when it turns out they’ve not been needed.

Is that roughly right for < Cylc8? (great to hear changing with new scheduling algo)

Stripped down graph. NB the main model cycle not yet implemented, but will be OoM ~1H; governed by duration of mod_run_model task; currently configured to take ~15 min to run.

    [scheduling]
    initial cycle point = now
    final cycle point = +PT15M
    max active cycle points = 10

    [[special tasks]]
    clock-trigger = mod_input_highfreq(PT0M)
    clock-expire = mod_input_highfreq(PT1M)

    [[dependencies]]
        [[[R1]]]
        # "Proper" cold start tasks @ initial cycle point
        # ("Proper": building model / retrieving built model from cache)
        graph = """
            {% if RUN_PERFORMS_BUILD == true %}
            mod_build_cache_cold =>
            {% endif %}
            mod_cache_load_cold
            """

        [[[R1]]]
        # Run cycle @ initial cycle point
        graph = """
            mod_cache_load_cold[^] => 
            mod_config_inputdir
            COLD_INPUTDATA_TRG[^]:succeed-all => 
            mod_run_model =>
            mod_end_archive =>
            mod_end_prune
        """

        [[[R1]]]
        # Input data provision @ initial cycle point
        graph = """
            mod_config_inputdir[^] => COLD_INPUTDATA_TRG
        """

        [[[+PT1M/PT1M]]]  # offset/recurrence
        # High cadence input data cycle @ subsequent cycle points
        graph = """
            mod_run_model[^]:start => mod_input_highfreq
        """
4

Haha, good question! The reason is historical. Early versions of Cylc had no dependency graph up front. A suite (/workflow) was just a bunch of task definitions, each knowing its own prerequisites and outputs (relative to its own cycle point) and its own cycling sequence (and hence how to spawn its own next cycle point instance). The scheduler then managed a “self-organising task pool” of task objects that individually spawned their own successors and asked other tasks in the pool for completed outputs to satisfy their prerequisites.

In this model, the life cycle of task objects has to be such that finished tasks stick around long enough to satisfy the prerequisites of any tasks that may not have been spawned yet, behind them. This has some nice properties. For instance, you could throw a new task definition into the running system and the associated tasks would kick off automatically if their prerequisites got satisfied by existing tasks.

But users struggled with the structure of their workflows not being immediately obvious, so we bolted a depedency graph on the front, and parsed that to automatically define task prerequisites and outputs - and fed those to the original scheduling algorithm.

Once you have a dependency graph, the process of matching unsatisfied prerequisites with completed outputs is technically unnecessary (as you’ve guessed) because the graph “knows” exactly who will do it. But, the difficulties of managing never-ending workflow graphs of repeating tasks (and other priorities) meant it took us quite a while to realize that potential.

Cylc 8 does it though: no more waiting tasks spawned ahead before they are needed, no more finished tasks accumulating to satisfy future-task dependencies, no more suicide triggers for alternate path branching, much more efficient, (… and more).

1 Like