# Restarting failed workflow **URL:** \ **Category:** Cylc Support\ **Created:** [June 14, 2022, 2:29pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491 "2022-06-14T14:29:11Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![bencash](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bencash](https://cylc.discourse.group/u/bencash)\ **Post date:** [June 14, 2022, 2:29pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/1 "2022-06-14T14:29:11Z")
I have a situation where a task in my workflow failed because a file was not staged in the right location. I moved the file, and would now like to just restart the workflow from where it ran into the problem. My understanding is I should just be able to run cylc play , but when I do that as far as I can tell nothing happens. Am I missing a step? This is using cylc8rc3. ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [June 14, 2022, 2:52pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/2 "2022-06-14T14:52:25Z")
Yes, `cylc play` should always pick up where the previous run left off. By failed I guess you mean that some of the tasks within the workflow failed rather than the Cylc scheduler failed? If the workflow is shutting down immediately after starting up it usually means that the workflow has already run to completion. Try inspecting the workflow log file, if there were any internal issues they will be reported there. If your workflow has already run to completion but you would like to “rewind” it back a bit and rerun try the following: ```bash # restart the workflow (the --pause will stop it shutting down straight away) cylc play --pause workflow # trigger a new "flow" with the first task you wish to rerun # for more info see https://cylc.github.io/cylc-doc/latest/html/user-guide/running-workflows/reflow.html cylc trigger --flow=new workflow//cycle/task # unpause the workflow cylc play workflow ``` New “flows” allow you to have multiple “logical runs” of the same workflow running under a single scheduler. They are useful for situations where you need to re-run part of a workflow e.g. due to data issues. They are a brand-new feature of Cylc 8. [https://cylc.github.io/cylc-doc/latest/html/user-guide/running-workflows/reflow.html](https://cylc.github.io/cylc-doc/latest/html/user-guide/running-workflows/reflow.html) ---
**Author:** ![bencash](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bencash](https://cylc.discourse.group/u/bencash)\ **Post date:** [June 14, 2022, 2:55pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/3 "2022-06-14T14:55:21Z")
It definitely hadn’t run to completion, because one of the intermediate tasks exited with an error and no other tasks ran after that. I’ll take a look at the logs and see if I can figure out what was going on with the restart attempt. ---
**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\ **Post date:** [June 14, 2022, 3:11pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/4 "2022-06-14T15:11:55Z")
When you restart a workflow it continues in the same way it would have if you never shut it down. So, if a task was in the failed state before shutdown it will remain in the failed state after the restart. To rerun the failed task use cylc trigger on that task. ---
**Author:** ![bencash](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bencash](https://cylc.discourse.group/u/bencash)\ **Post date:** [September 8, 2022, 3:22pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/5 "2022-09-08T15:22:36Z")
Hey everyone - a blast from the past! I got pulled away from this problem for a while but am now back messing with it and still struggling. It seems like no matter which combination of steps I take if a workflow fails my only recourse is to create a brand new workflow. Does someone have a really basic “hello world” example of restarting a failed workflow that I could try? ---
**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\ **Post date:** [September 9, 2022, 8:00am UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/6 "2022-09-09T08:00:17Z")
Here’s a trivial `flow.cylc`: ```ini [scheduling] [[graph]] R1 = hello [runtime] [[hello]] # fail the first time script = "((CYLC_TASK_SUBMIT_NUMBER > 1)) && echo Hello" ``` If you use `cylc play workflow` to run this it will stall (the scheduler remains running with task “hello” in the failed state). If you then trigger the failed task (`cylc trigger workflow//1/hello`) the task will run again, succeed and the scheduler will shutdown. It makes no difference if you stop the workflow (`cylc stop workflow`) after the task has failed and then restart it again (`cylc play workflow`) - the workflow just starts up and stalls with task “hello” in the failed state. ---
**Author:** ![bencash](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bencash](https://cylc.discourse.group/u/bencash)\ **Post date:** [September 9, 2022, 1:12pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/7 "2022-09-09T13:12:08Z")
> [@dpmatthews](#): > > ```auto > [scheduling] > [[graph]] > R1 = hello > [runtime] > [[hello]] # fail the first time > script = "((CYLC_TASK_SUBMIT_NUMBER > 1)) && echo Hello" > > ``` Thanks! Dumb question time - does the //1/ in this case indicate that we want to restart from the first attempt to run hello? ---
**Author:** ![bencash](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bencash](https://cylc.discourse.group/u/bencash)\ **Post date:** [September 9, 2022, 1:37pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/8 "2022-09-09T13:37:34Z")
I was able to get the test case to work, but I am running into an error with my production suite. In this case I had to cancel a job I submitted because I had the run length wrong. I now just want to restart the flow and rerun that step. When I go through the steps listed here, I get errors like: (cylc8) login2.frontera(1171)$ cylc trigger --flow=new prototype-p8/p8.benchmark.24h.x8.y8//1/run\_model\_ldate2012010100\_mem001 Done (cylc8) login2.frontera(1172)$ cplay prototype-p8/p8.benchmark.24h.x8.y8 CylcError: Cannot determine whether workflow is running on 129.114.63.99. /home1/02441/bcash/.conda/envs/cylc8/bin/python /home1/02441/bcash/.conda/envs/cylc8/bin/cylc play --pause prototype-p8/p8.benchmark.24h.x8.y8 ---
**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\ **Post date:** [September 9, 2022, 2:04pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/9 "2022-09-09T14:04:09Z")
> [@bencash](#): > > Thanks! Dumb question time - does the //1/ in this case indicate that we want to restart from the first attempt to run hello? It’s the cycle point . See [https://cylc.github.io/cylc-doc/stable/html/7-to-8/major-changes/cli.html#cylc-8-standardised-ids](https://cylc.github.io/cylc-doc/stable/html/7-to-8/major-changes/cli.html#cylc-8-standardised-ids) ---
**Author:** ![bencash](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bencash](https://cylc.discourse.group/u/bencash)\ **Post date:** [September 9, 2022, 2:24pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/10 "2022-09-09T14:24:56Z")
Thanks. My workflow isn’t cycled (for the most part) so I don’t tend to think of things in terms of cycle points. Since I don’t have any cycling information specified it defaults to 1? ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [September 9, 2022, 3:32pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/11 "2022-09-09T15:32:51Z")
> Since I don’t have any cycling information specified it defaults to 1? Yes. * * * > In this case I had to cancel a job I submitted because I had the run length wrong. I now just want to restart the flow and rerun that step. You do not need to restart the workflow to re-run a task. When tasks fail in Cylc the workflow remains running (note you can list running workflows with `cylc scan` or watch them live in the GUI). You can tell Cylc to re-run a task by running `cylc trigger`: ```auto # "cycle_point" is "1" for your example $ cylc trigger /// ``` (Note: you can also do this from the context menu opened by clicking on the task in the GUI) * * * > (cylc8) login2.frontera(1172)$ cplay prototype-p8/p8.benchmark.24h.x8.y8 > CylcError: Cannot determine whether workflow is running on 129.114.63.99. I’m guessing `cplay` is `cylc play`? The error message you are seeing here is an installation / system issue most likely caused by an SSH failure. Cylc can’t tell whether the workflow is running or not because it cannot SSH to `129.114.63.99` from `login2.frontera`. ---
**Author:** ![bencash](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bencash](https://cylc.discourse.group/u/bencash)\ **Post date:** [September 9, 2022, 4:11pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/12 "2022-09-09T16:11:47Z")
Sorry, yes, cplay is “cylc play”. I’ve aliased a bunch of the commands that get used over and over like that. Frontera is a pretty standard hpc setup with login and compute nodes. Is there something I need to change in my global.cylc or elsewhere to make this work? ---
**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\ **Post date:** [September 9, 2022, 4:36pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/13 "2022-09-09T16:36:05Z")
Does `ssh 129.114.63.99` work? ---
**Author:** ![bencash](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bencash](https://cylc.discourse.group/u/bencash)\ **Post date:** [September 9, 2022, 4:42pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/14 "2022-09-09T16:42:17Z")
> [@dpmatthews](#): > > ssh 129.114.63.99 It did, but first I had to answer a fingerprint prompt. Maybe that was causing it to hang? ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [September 12, 2022, 4:05am UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/15 "2022-09-12T04:05:29Z")
For automated ssh access to other hosts, by Cylc or otherwise, you need to configure ssh to use public/private keys, with no interactive passphrase entry. Even if you do that, by default ssh presents an interactive prompt the very first time you connect to a host (I suspect this is what you mean?). Options are: - initiate a connection manually the first time, then Cylc is good to go - tell ssh not to do the initial “strict host key check”: e.g. `ssh -o stricthostkeychecking=no` (Note you can configure the form of the ssh command used by Cylc, in `global.cylc`) We have a pending issue, to document this for users. ---
**Author:** ![bencash](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bencash](https://cylc.discourse.group/u/bencash)\ **Post date:** [September 19, 2022, 9:01pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/16 "2022-09-19T21:01:27Z")
> [@hilary.j.oliver](#): > > stricthostkeychecking=no Back again - I am finding that even when I have done an initial manual connection to a host and can subsequently connect without a prompt I am getting errors like the following: (cylc8) login3.frontera(1180)$ cylc play prototype-p8/restart\_test.3 CylcError: Cannot determine whether workflow is running on 129.114.63.100. /home1/02441/bcash/.conda/envs/cylc8/bin/python /home1/02441/bcash/.conda/envs/cylc8/bin/cylc play prototype-p8/restart\_test.3 I have also turned off strict host checking in my global.cylc. Is there another possible explanation for this issue? ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [September 19, 2022, 11:21pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/17 "2022-09-19T23:21:05Z")
A few questions, to help debug this: - have you checked that your global config is being picked up? (e.g. run `cylc config` and check the output) - when you get the error message above, are you saying that you can successfully ssh to the reported host (129.114.63.100), and without a password prompt? - have you configured scheduler “run hosts” or is 129.114.63.100 just the IP of the host you were logged on to when you started the previous run of the workflow (that you are now trying to restart)? - also try `cylc play --debug --no-detach` to see you get any additional information The only time I’ve seen this error myself recently was this: some external users get onto our HPC via a JupyterHub-launched interactive Slurm session. The session gets allocated to some available compute node, and their Cylc schedulers run locally (i.e. in the same node) if they don’t configure specific scheduler run hosts. If a scheduler is still running when the Slurm session ends, it gets killed by Slurm. Then, if the user logs back in and tries to restart the workflow, the scheduler “contact file” records that it is/was running on the former interactive node, so Cylc tries to ssh there to see if it is still running or not. But the system does not let users ssh directly to these nodes if they have no processes running there … hence the above error from `cylc play`. … however, I suppose that can’t be your problem, if you say you can ssh to the host in the error message? ---
**Author:** ![bencash](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bencash](https://cylc.discourse.group/u/bencash)\ **Post date:** [September 20, 2022, 1:20pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/18 "2022-09-20T13:20:00Z")
- Yes, global config is being picked up - Yes, I can ssh to that host without a password prompt - I have not specified anything in “run hosts”, so this is just whatever I got. - cylc play --debug --no-detach came back with some very interesting info! This is what it returned. (cylc8) login3.frontera(1260)$ cstop prototype-p8/x12y12.med300.reduced.cpn48 CylcError: Cannot determine whether workflow is running on 129.114.63.100. /home1/02441/bcash/.conda/envs/cylc8/bin/python /home1/02441/bcash/.conda/envs/cylc8/bin/cylc play prototype-p8/x12y12.med300.cpn48 (cylc8) login3.frontera(1261)$ cylc play --debug --no-detach prototype-p8/x12y12.med300.reduced.cpn48 2022-09-20T08:14:39-05:00 DEBUG - Loading site/user config files 2022-09-20T08:14:39-05:00 DEBUG - Reading file /home1/02441/bcash/.cylc/flow/8.0.1/global.cylc 2022-09-20T08:14:40-05:00 DEBUG - $ ssh -oBatchMode=yes -oConnectTimeout=10 129.114.63.100 env CYLC\_VERSION=8.0.1 bash --login -c ‘exec “$0” “$@”’ cylc psutil # returned 255 To access the system: 1) If not using ssh-keys, please enter your TACC password at the password prompt 2) At the TACC Token prompt, enter your 6-digit code followed by . [bcash@129.114.63.100](mailto:bcash@129.114.63.100): Permission denied (keyboard-interactive). However, ssh [bcash@129.114.63.100](mailto:bcash@129.114.63.100) goes through just fine if I enter it manually from the command line. One strange thing from that debug info is that I set stricthostkeychecking=no in my global.cylc but it doesn’t seem to have been picked up here: (cylc8) login3.frontera(1267)$ cylc config [scheduler] [[host self-identification]] method = address [platforms] [[cheyenne]] job runner = pbs install target = localhost hosts = localhost [[expanse]] job runner = slurm install target = localhost hosts = localhost [[stampede2]] job runner = slurm install target = localhost hosts = localhost [[frontera]] job runner = slurm install target = localhost hosts = localhost ssh command = ssh -oBatchMode=yes -oConnectTimeout=10 -ostricthostkeychecking=no Do I have that specified incorrectly somehow? ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [September 20, 2022, 2:43pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/19 "2022-09-20T14:43:08Z")
Hi there, ### Reported Traceback > Permission denied (keyboard-interactive). I think this confirms that SSH required an interactive prompt so has not been configured to work non-interactively. Unfortunately this is very hard for us to debug from our end as we don’t have access to the system or understanding of its configuration. It’s likely that this can be resolved in your SSH config file, but sadly I can’t tell you how from here. ### SSH > However, ssh [bcash@129.114.63.100](mailto:bcash@129.114.63.100) goes through just fine Could you try running this command (exactly as written, don’t add `bcash@` to the start of the host IP): ```auto ssh -oBatchMode=yes -oConnectTimeout=10 129.114.63.100 hostname ``` Then try with `-oStrictHostKeyChecking` turned off ```auto ssh -oStrictHostKeyChecking=no -oBatchMode=yes -oConnectTimeout=10 129.114.63.100 hostname ``` - If only the second command works then we need to work at adding `-oStrictHostKeyChecking` onto the SSH command. - If neither work you’ll likely need to do some work to your SSH configuration. - If both work [I’m confused](https://xkcd.com/2259/). The really strange thing is that this SSH should _only_ happen if: - More than one host has been configured by `[scheduler][run hosts]available`. - OR if host ranking has been configured by `[scheduler][run hosts]ranking`. So it shouldn’t happen at all for you if your `[scheduler][run hosts]` section is blank. ### StrictHostKeyChecking > One strange thing from that debug info is that I set stricthostkeychecking=no in my global.cylc but it doesn’t seem to have been picked up here: It looks like you have configured the `ssh command` for the “frontera” platform, however, I don’t think the SSH being performed is to the frontera platform, I think it is actually an SSH to localhost, can you confirm this? It is not presently possible to configure the `ssh command` used for the “run hosts” in this way (only the platforms where jobs are submitted), I’ll try to prioritise this. ---
**Author:** ![bencash](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bencash](https://cylc.discourse.group/u/bencash)\ **Post date:** [September 20, 2022, 2:54pm UTC](https://cylc.discourse.group/t/restarting-failed-workflow/491/20 "2022-09-20T14:54:38Z")
Hi @oliver.sanders - Neither worked, so I did some poking around and the options that is causing the failure seems to be -oBatchMode=yes and including “hostname” at the end. Setting BatchMode to no or leaving it out avoid the “permission denied” error. Including “hostname” just causes the ssh to fail with no message. [Next page](https://cylc.discourse.group/t/restarting-failed-workflow/491.md?page=2)