When cylc client commands run (e.g. cylc reset, cylc message), if memory serves me correctly they are queued and then processed internally by the suite server. When internally processed, are they executed in parallel (i.e. using the process pool) or are they executed serially? Specifically here we’re wondering about a race condition in an err-script that calls:
cylc kill $CYLC_SUITE_NAME $CYLC_TASK_CYCLE_POINT/other_related_task
cylc reset --state=waiting $CYLC_SUITE_NAME $CYLC_TASK_CYCLE_POINT/other_related_task
There’s a race condition here if the commands are potentially processed in parallel, but I don’t recall if that’s just for things like job submission.
Commands like cylc kill and cylc reset send requests to the suite server program over the network and immediately disconnect before the request is actioned. The requests are then handled concurrently by a web server with 12 threads (ish).
However, you’ve got a bigger problem with doing a reset immediately after kill (on the same task): a task reset is very quick to action: it simply forces the task status to change inside the server program; but a task kill is relatively slow to action: it has to (asynchronously) spawn a process (in the process pool) kill the real job.
So in your example even if the requests to the server were processed sequentially, in order, it is still almost certain (I think) that the task would get reset to waiting before the real job was killed (whereupon the task status would get set to “failed”).
For a series of scripted task manipulations you really need to poll for the desired result after each command, before moving on the next command. (This will probably change at Cylc 8 where Websockets will allow us to main persistent connections until the command result is achieved, rather than just queuing a request and disconnecting).
Hilary
OK got it - so it’s asynchronous on top of concurrence!
I’ll modify the script to ensure that the job is successfully killed prior to resetting the state.