Overriding queue for single task

Cylc Cylc Support
1

I have a large suite which uses internal queues to prevent overwhealming the scheduler. I am currently trying to fix a problem with one of the tasks via some manual intervention but manually triggering the tasks sends it to the back of the queue which is not ideal in this scenario - I need to know if my fix has worked.

Is there a way to force this one task to trigger immediately?

I thought I could perhaps use cylc trigger for this, but I can’t see what state to set the job to.

2

Have you tried holding the workflow (suite) before triggering off the failed task? I think this does what you want if my toy workflow duplicates your problem correctly.

…CLI instructions…

cylc hold <my_workflow_name>
cylc trigger <my_workflow_name> <my_task_name>.<cycle_point>

n.b.
<my_task_name>.<cycle_point> can be a glob, so you can trigger multiple tasks.

3

@nhsavage - @wxtim’s suggestion might work too (haven’t tried) but it should work like this:

  • if you trigger a waiting task, it will get queued
    • if the queue is not full, the task will be released to submit immediately, otherwise it is queued
  • if you trigger a queued task, it will submit immediately regardless of queue limiting

i.e. using cylc trigger once says “queue this task now”; and using it again says “submit this queued task now”.

This is deliberate in case the queue limits are really important.

2 Likes
4

Documented in cylc trigger --help (which of course you might not be aware of if doing it via the GUI):

NOTE waiting tasks that are queue-limited will be queued if triggered, to
submit as normal when released by the queue; queued tasks will submit
immediately if triggered, even if that violates the queue limit (so you may
need to trigger a queue-limited task twice to get it to submit immediately).

5

thank you - I think that the double triggering is the key thing I missed - I was using commands, but I hadn’t realised that triggering something in the queue made it trigger immediately. Holding the suite didn’t seem to help me (I had tried that) - it looked like the hold prevented my triggered tasks from running at all (although possibly double triggering would help with that too). All useful stuff thanks.

1 Like
6

I have just tested this in a running suite. The first time I triggered it, the job didn’t change state and just had a last message “waiting for job host selection”. The second time it did submit ok. This suite is probably pushing the limits of cylc 7 as it has 36 members with 4 nests running at once and has 1 hour cruns (I should probably push that longer tbh).