In my system I have many tasks A and a few tasks B. I have a limit for tasks A+B = 9, and I would also like a limit for A so that B tasks can proceed. I set
[[queues]]
[[[queueA]]]
members= A
limit=5
[[[queueAB]]]
members= A, B
limit=9
In this particular case, the queueAB limit is actually enforced by the PBS system. So I can actually get some submit-fail because cylc resubmits before the PBS system has actually cleared its own queue. That’s a minor thing, I just set the cylc limit to N-2 of the PBS limit and the problem goes away.
If queueAB is set it appears that queueA is ignored-- cylc will submit tasks A up to the queueAB limit, ignoring queueA.
Task A comes first and runs fewer times, so A is ready to advance to the next cycle point before B is finished. If I use separate queueA and queueB that is what happens. But if I set queueAB it appears that cylc will not submit A[T+1] until all B[T] tasks are complete.
So the questions:
Q1: Does cylc8 support overlapping queues?
Q2: If I manually trigger tasks, are the queues respected? (I feel like I read the answer to this here a while back but I cannot recall it)
Q3: Is the queue strictly first-in first-out, as in, will queued jobs be submitted in the order in which they became eligible to run (graph dependencies met), or is there some cycle-point logic used to determine how to drain cylc queues?
No, there’s no overlap. In your example above, A gets reassigned from queueA to queueAB, so queueAB gets both tasks and queueA ends up empty. Unfortunately validation does not currently warn about that. (I’ll create a ticket on the code repository to get that fixed).
Correct, as per the answer to Q1.
Yes:
if you trigger a waiting task that is subject to a queue limit, it will be queued
if you trigger a queued task it will be submitted, regardless of the queue limit
Yes, strictly FIFO.
Cylc can support different queue implementations, but at the moment we only have simple non-overlapping FIFO queues.
As Hillary mentioned, the queueing interface in Cylc has now been abstracted which means that Cylc can support multiple different queueing implementations. As a result we don’t have to worry about accidentally introducing breaking changes to the existing implementation which had stifled progress in the past.
We haven’t yet exposed the queue interface to Cylc plugins, we should one day but it would be a good idea to let it settle in first as there’s still a lot of change around this area of the code. I think there is an open question about whether we would/could/should support the use of multiple queue implementations within the same workflow which could potentially require changes to this interface too.
I think we would be happy to accept new queue implementations into the cylc-flow library so long as they’re sufficiently abstract (we don’t want to maintain loads of single-application implementations), an overlapping priority queue would find uses in a lot of different applications.
If interested, the queue abstraction can be found here. Note, the queue interface can easily get hammered by larger workflows so efficiency is key (i.e. make sure there’s no O(n) code lurking in there and that cross referencing between queues is fast).
I was just wondering if this had been looked at more? Like the original post, it would be useful being able to prioritise items within a queue, allowing tasks to jump to the front if they are higher priority.
We have not developed any alternative queue implementations as yet. There’s a lot of high priority Cylc development work going on at the moment, so we can’t promise anything anytime soon.
The Cylc queue abstraction is still undergoing some minor changes as we bed it in. We haven’t exposed this interface to plugins just yet. If you’re interested in developing a queue implementation, it should be possible, we could open up a “beta” entry point, but you might need to adapt to changes in cylc-flow as they occur.