Can't restart workflow

Cylc Cylc 8 Migration
1

hey,

i can’t seem to restart a cylc 8 workflow.

when i run cylc play WORKFLOW/run1 the cylc log command gives me…

2023-04-19T05:25:07Z WARNING - No active tasks matching: */*
2023-04-19T05:25:31Z INFO - [command] resume
2023-04-19T05:25:32Z INFO - RESUMING the workflow now
2023-04-19T05:25:32Z INFO - Command succeeded: resume()
2023-04-19T05:25:32Z INFO - Workflow shutting down - AUTOMATIC
2023-04-19T05:25:33Z INFO - [('workflow-event-handler', 'shutdown') cmd]
    mail -s '[workflow shutdown] u-cv956/run1' -r notifications@w-cylc02.maui.niwa.co.nz williamsjh <<<'event: shutdown
    message: AUTOMATIC
    workflow: u-cv956/run1
    host: w-cylc02.maui.niwa.co.nz
    port: 43057
    owner: williamsjh
    '
    [('workflow-event-handler', 'shutdown') ret_code] 0
2023-04-19T05:25:33Z INFO - DONE

any idea what’s going on? it has several years still to run!

thanks for any ideas!

j

2

The AUTOMATIC shutdown message means that there is nothing left to run, according to the workflow state recorded in the run DB, i.e. the scheduler has determined that it already finished

How did it shut down exactly, before you tried the restart? Eg. did it shut down automatically as finished then, or did you tell it to stop early?

You should be able to find the scheduler log from that run, in the log/scheduler/ directory.

Whatever the cause, you can get it to carry on by restarting with cylc play --pause, then triggering the next tasks before un-pausing. But it would be good to check that some kind of erroneous early stop did not occur.

(Note, in a near-future release the scheduler will automatically stall on a timeout to allow intervention, without needing --pause, if you restart an already-finished workflow).

1 Like
3

We have a new issue up on the repository, to make shutdown reasons clearer in the logs.

1 Like
4

cool thanks,

using cylc play --pause allowed me to trigger the tasks and it’s running again now.

one thing to note (and bearing in mind that this particular run has been monumentally messed around with by me) is that it seems to be running everything in the none flow! i don’t know precisely how this has happened but it might account for the scheduler not thinking there was anything else to run.

cylc log output…

image
image1848×298 42.8 KB

cheers

5

OK, you may find that the it stops again when those tasks have completed!

The “flow=none” is not a flow - i.e. those tasks won’t spawn any action downstream in the graph.

In fact this explains why your workflow automatically shut down as finished - you had only (re)triggered tasks with --flow=none.

If you want it to carry on at this point, you actually need to trigger a new flow at the appropriate point in the graph.

1 Like
6

thanks! yeah that all makes sense and is done now. from a previous comment…

in a near-future release the scheduler will automatically stall on a timeout to allow intervention, without needing --pause , if you restart an already-finished workflow…

yeah this would be ace.

cheers,

j

1 Like