Not a biggie for me now, as I’m intending to rework my suite to avoid having cycles with such different cadences. But posting for curiosity’s sake - and in case answers help anyone else in future!
NB can’t see answer there, but Discourse flags this q as similar to fredw’s post on Robust coding of expired tasks.
My suite wraps a ~nowcast-y model which continually consumes 1 min cadence input data (which also gets refreshed at 1 min intervals). The model itself is intended to run at a lower cadence - I’ve currently configured this to 15 min cycles in dev mode, but this is intended to be OoM ~1-3 hours when running operationally.
My current implementation is ~naive: a low-cadence cycle for the model, and a high-cadence cycle for the input data, where the latter is implemented as one-cycle-per-minute-of-input-data. So given 15 min model cycles, I have 15 x 1 min input data cycles.
As the model cycle includes some prerequisites for the input data cycle, which can take a little time to run, I’m using a clock-expire on input data cycle/tasks which attempt to start too early (before model itself has started running). This approach happily expires these over-early input data tasks, but these expired input data tasks stick around in the GUI “queue”, as do the subsequent input data tasks which ones which run later (once model setup complete) and hence succeed.
These expired/succeeded input data cycles build up until suite reaches the runahead limit I’ve configured:
max active cycle points = 10. See e.g. screenshot below - note 10 cycles there, 9 of which are input data cycles (top green one = the 15 min model cycle, still running at this point).
And then things pause (note 21:39 timestamp in screenshot: my expected 21:38 & 21:39 input data cycle points haven’t appeared). This “pause” remains until my lower cadence model cycle completes. Once this happens, the input data “backlog” clears, and the remaining ~6 input data cycles finish (1st couple expiring as have become stale during pause; subsequent cycles running on the minute as expected).
These remaining input data cycles behave more normally: tasks/cycles which have expired/completed disappear off the GUI queue, and don’t accumulate.
For various reasons, I’m intending to overhaul my implementation, such that my input data processing is done as a single task, on the same cadence as the model cycle (so I don’t have multiple cadence cycles going on).
This input data task will run in parallel with the model task, with the per-minute input data processing achieved via an internal time-aware process/sleep while loop within the task).
This is mostly to avoid swamping the scheduler of target platform, but also to simplify suite a little - and hopefully avoid issues like this!
However, I’m curious as to why I’m seeing this behaviour - and why the difference: accumulation of expired/succeeded input data tasks (seemingly until my runahead limit used up) until initial cycle completes, then expected non-accumulation thereafter.
Can anyone shed any light? Happy to share a potted version of my suite graph if it helps (e.g. if it’s not immediately obvious what my misunderstanding/mistake is!)