Automatic cylc clean?

Cylc Cylc Support
1

hey there,

i have a postprocessing/visualisation workflow (let’s call it foo) which runs after every year of my climate simulation (i think many others do to but not sure) and so for a standard 65 year run, i get foo/run1, foo/run2, ..., foo/run65.

I don’t and won’t need to keep any of them so is there a way i can either automatically cylc clean them after they’ve successfully finished or maybe always get it to repeat run1?

any ideas/suggestions welcome!

@hilary.j.oliver this will help a lot to reduce my installed workflow count! :stuck_out_tongue:

cheers,

jonny

2

You can get your workflow to delete itself at shutdown, with a shutdown event handler.

Obviously this is dangerous - you’d better be sure that the workflow has copied its results to some other location first!

[scheduler]
    [[events]]
        shutdown handlers = "cylc clean -y %(workflow)s"
[scheduling]
    [[graph]]
        R1 = "foo"
[runtime]
    [[foo]]

Test run:

...
INFO - [1/foo running job:01 flows:1] => succeeded
INFO - Workflow shutting down - AUTOMATIC
INFO - [('workflow-event-handler-00', 'shutdown') cmd] cylc clean -y xtr/run1
    [('workflow-event-handler-00', 'shutdown') ret_code] 0
    [('workflow-event-handler-00', 'shutdown') out]
    INFO - Removing directory: /home/oliverh/cylc-run/xtr/run1
    INFO - Removing directory: /home/oliverh/cylc-run/xtr/_cylc-install
    INFO - Removing directory: /home/oliverh/cylc-run/xtr
INFO - DONE

(Note you have to use a shutdown event handler, not script the clean in a final task, because clean will refuse to remove a running worklow).

1 Like
3

hi @hilary.j.oliver thanks a lot for this, looks like exactly what i was after!

so a shutdown handler is kind of like a post–post-script then?!

cheers,

jonny

4

Well, maybe, if you really want a strained analogy!

A post-script is a fragment of task scripting that executes in a running task job.

Event handlers are run by the scheduler (not by tasks) when certain events occur.

Shutdown handlers are run by the scheduler at the last instant before shut down, when the workflow is finished. So they have nothing to do with task scripting.

1 Like
5

Worth noting that if you want a safer (manual) way of still doing this relatively easily

cylc clean 'foo/*'

Will clean all these workflows.

If you are kicking these workflows off from another program you can use the -y option to skip the confirmation step of Cylc Clean.

Again, Hilary Speaks words of great wisdom:

Obviously this is dangerous - you’d better be sure that the workflow has copied its results to some other location first!

1 Like
6

haha year @hilary.j.oliver i love a strained analogy! :laughing:

thanks for the info re the distinction!

cheers,

7

ah ok yeah that’s good to know thanks @wxtim ,

in this particular example the workflow isn’t creating anything inside the cylc-run directory that i need to keep so your suggestion is safe but i appreciate that this isn’t true in general.

Cheers

1 Like
8

The cylc clean command is quite flexible in terms of what it does (not?) delete. Worth looking the --help docs. But ultimately it’s a deletion command. A sharp axe is still an axe.

1 Like