With Cylc 7, users could - and sometimes did - overwrite important run data and wipe out their workflow state by accidentally cold starting (i.e. starting from scratch) when they meant to do a restart from previous/current state. (And additionally, cycling workflows typically get restarted many more times than they get cold started, except perhaps during development).
Cylc 8 workflow run semantics are designed to avoid that danger: cylc play starts a new workflow from scratch, and then restarts it if called again.
Excerpt from cylc play --help:
To avoid overwriting existing run directories, workflows that already ran
can only be restarted from prior state. To start again, "cylc install" a new
copy or "cylc clean" the existing run directory.
I don’t want to do clean and then install and play. I want to reinstall and then start from scratch.
Why do you not want to clean and install and play? The play bit, which is presumably the important thing, will be exactly what you want, but it will happen in a nice tidy run directory that isn’t contaminated with files from a previous run. Does install take a long time for this workflow?
b) In the meantime you can just delete the existing workflow .service directory, where the prior state is kept. After that, cylc play will start the workflow from scratch.
It shouldn’t matter whether or not you use a run-name.
There are valid use cases for re-running a workflow, e.g. make type problems where you want to preserve the contents of the work/ or share/ directories between runs in order to incrementally update them, but where cycling might not be appropriate e.g. because you only re-run occasionally.
If this does not apply, please clean the workflow or use numbered runs. This is the Cylc 8 solution for re-running workflows.
# run the workflow
cylc vip --workflow-name=foo
# run the workflow again
cylc vip --workflow-name=foo
# remove both runs
cylc clean foo/*
If you have a use case for incrementally re-running the workflow over the same data read on…
As Hillary said, the presence or absence of a run name does not make any difference to the way a workflow is restarted or rerun.
# with a run name / number
cylc vip --workflow-name=foo --run-name=bar
cylc vr foo/bar
# without a run name / number
cylc vip --workflow-name=foo --no-run-name
cylc vr foo
To re-run a workflow from the start:
# start the workflow in paused mode (this prevents it from shutting down)
cylc vr <workflow> --pause
# get the workflow to re-run by starting it off from selected task(s)
cylc trigger --flow=new <workflow>//<cycle>/<task>
# un-pause the workflow
cylc play <workflow>
As Hillary mentioned we aim to make the pause/un-pause and selection of the start-task(s) more automatic going forward.
Thanks. So first thing I see is, I need to update to v8.1.0 of Cylc to get some simpler commands!
We do sometimes have use cases, yes. Sometimes, it is just that cleaning the data up can be time consuming when its terrabytes and/or lots of small files involved.
@TomC I neglected to mention Oliver’s suggestion here
This triggers a new “flow” in the graph, which allows the whole graph to be traversed again from the trigger point, without losing the original flow history (unlike using clean to clean out the run directory, or deleting the .service directory).
However, it sounds to me like you really want to run again from scratch and you don’t care about the previous history, but you can’t be bothered waiting for the old files to be cleaned up first (which is fine!).
That is really exactly the use case for numbered runs. Just start again in a new run-directory that is closely related to the original: my-workflow/run2/. Then you can clean out my-workflow/run1 while the new one is running.
There are still valid use cases for re-running in the same dirty run directory.
Another is, I would say, repeatedly re-running from scratch during development without having to redo the install process each time, if that is time consuming.
So we do intend to make that easier. In the meantime you can achieve the same thing by manual .service deletion.
Thanks. Yes, using numbered runs I knew would achieve the result, that is why I specifically asked if --no-run-name is used. I’ve learnt a lot from this discussion, and am continu
The file-driven workflow example intrigues me, so thanks for pointing that one out.
If you don’t care about the previous data this is definitely the way to do it.