I was wondering if anyone had a clean way to have queue limits change based on some events/situations (or if Cylc could add this sort of functionality). For example
- Time of day - have different limits at different times of day based on expected HPC schedules. For example, limit to 10 jobs between 00-03Z and 20 jobs 03-06Z and then unlimited afterwards
- Other tasks - change the limit based on task events/messages
- Some other events - message broker messages, tasks events from other suites, etc
With larger ensembles competing for resources, there are times when it would be very useful to be able to adjust the queue limits based on conditions, whether that be time of day or when other systems start/finish running which are a higher priority. The best idea I’ve had would be to have tasks which check for times/events and then use a
broadcast to adjust them back and forth, but that feels quite clunky.