2023-12-18T20:42:54Z INFO - Cylc version: 8.2.2
(I have observed this behavior with every cylc8.x I have used)
suite.rc specifies [scheduling]runahead limit=PT96H
The suite starts out respecting this limit. But when the suite is running a long integration (multiple weeks) at some point it starts opening cycle points much farther ahead (30 days vs 4 days expected). It does not open all of the cycle points. What it does is open them incrementally, adding 1-7 open cycle points at intervals from 30 minutes to 2 hours. After a few days, there are hundreds of active cycle points, because whatever logic is causing cylc to open new points is doing so much faster than they are completed.
The runahead tasks are all very lightweight, so hundreds of active cycle points mostly slows down the GUI. It may have other impacts but that’s what I notice. But it sure seems like a bug.
That does seem like a bug, and not one we’ve seen before. It sounds like you are running in back-compat mode (suite.rc)? Are you able to give us a copy of the workflow config?
Does your workflow graph contain any “future triggers” - i.e. inter-cycle triggers with a positive (rather than the usual negative) offset?
foo[+PT6H] => bar # bar triggers off of foo in the NEXT cycle
Yes, we have PUBLISH and SCRUB tasks that are keyed on the suite arriving at a certain task in the next cycle point:
forecast_watch_log[+PT6H]:tau002 => scrub_ready & publish_ready
OK, I think we have a bug when there are future triggers present AND the runahead limit is expressed as a time interval rather than number of cycle points.
You can avoid the bug by changing the runahead limit (e.g. for a 6 hour cycle) from PT96H (96 hours) to P16 (16 cycle points).
How many cycle points you choose will depend on what cycling sequences you have in the workflow. P16 is equivalent to PT96H if it’s just a single 6-hour cycle.
[UPDATE] bug fix posted: Simplify and fix runahead computation. by hjoliver · Pull Request #5893 · cylc/cylc-flow · GitHub
Thanks! I will make that change and see what happens!
1 Like
@hilary.j.oliver I believe this change worked. Nearly as impactful was learning about the setting in the cylc8 GUI that puts the oldest active cycle point at the top, that’s a huge improvement in usability for me. Thanks, on both counts!
2 Likes