When testing my cylc8 suite (using 8.2.1) I accidentally triggered a task way before its clock-trigger (it wasn’t due for a few days, but was already in the runahead). The task predictably failed (input files from other tasks don’t exist yet), but now it’s stuck in failed state (red in the GUI).
In cylc7 GUI I could simply set the state back to waiting and it would go back to waiting for the clock-trigger. I think that used cylc reset behind the scenes. The equivalent in cylc8 is cylc set-outputs, but the help text (cylc help set-outputs) doesn’t say which string to use to set it back to waiting. I tried cylc set-outputs ... --output=waiting, but that doesn’t seem to change the task state.
I find the output of set-outputs slightly misleading, because whatever you state you ask it to apply, it always reports back Done, even for invalid states (like XXXXX). Is that intended?
So what’s the correct way to reset a task back to its original waiting state, where it goes back to waiting for its clock trigger??
With the current release version you can’t use the Cylc CLI or GUI to restore a triggered parentless (i.e., no upstream tasks in the graph) clock-triggered task to waiting on its clock trigger.
The good news:
You will be able to in the upcoming 8.3.0 release, which has a new cylc set command that generalizes and extends the existing cylc set-outputs command.
For the moment, you can brute-force the rewind by tweaking the workflow database to make it look as if the task never got erroneously triggered
The DB tweak (only needed until 8.3.0 is released): if your task is 2031/foo in workflow test, say:
shut the workflow down: cylc stop test
edit its DB as follows, for twoTABLEs: task_pool and task_states:
$ sqlite3 ~/cylc-run/test/runN/.service/db \
"update $TABLE set status='waiting' where cycle is '2031' and name is 'foo' "
This sort of thing should be much clearer after the aforementioned 8.3.0 release (which primarily targets all runtime interventions under the new Cylc 8 scheduling algorithm).
Some quick explanations for the moment:
cylc set-outputs is quite different to the old “reset” command. It literally tells the new event-driven scheduler to carry on as if the referenced outputs had been generated.
“waiting” is a task state, but not a task output. Some task states such as “succeeded” occur in response to a corresponding output, but not “waiting”.
(In Cylc 7-, task states were not just diagnostic, they had a direct effect the scheduling algorithm; hence the need for state reset)
I find the output of set-outputs slightly misleading, because whatever you state you ask it to apply, it always reports back Done, even for invalid states (like XXXXX). Is that intended?
Yeah, the “Done” just indicates that the CLI client successfully submitted the command to the server (scheduler). We should use better wording for that. Commands like this are executed asynchronously when the scheduler is able, so you have to look at the scheduler log to see what happened if the expected result doesn’t appear in the GUI.
The Cylc 8 architecture allows us to do better than that, by “subscribing” to a command result feed - but that work is still pending.
Brilliant!! It’s not very elegant, but it does the job. I wrote myself a little script to issue those commands, should I need it again. But for now I have cleaned up my database state, without the need for restarting the suite from scratch. That was VERY helpful!
Patiently waiting for cylc 8.3
And thank you for the explanation of the internal states!
