How to pre-prepare data in share before starting a workflow?

Cylc Cylc 8 Migration
1

If I want to warm-start a workflow by pre-preparing the share area on disk, I am getting a failure.

2024-06-03T07:43:26Z DEBUG - ['ssh', '-oBatchMode=yes', '-oConnectTimeout=10', 'remote.host', 'env', 'CYLC_VERSION=8.2.7', 'CYLC_ENV_NAME=cylc-8.2.7', 'bash', '--login', '-c', '\'exec "$0" "$@"\'', '/cylc-bin/cylc', 'remote-init', '-v', 'dev:b:aps4_g', '$HOME/cylc-run/my_workflow', 'share=/g/sc/lustre/cylc-run/my_workflow/share', 'work=/g/sc/lustre/cylc-run/my_workflow/work']
2024-06-03T07:43:29Z ERROR - platform: my_platform - initialisation did not complete
    COMMAND:
        ssh -oBatchMode=yes -oConnectTimeout=10 \
            remote.host env \
            CYLC_VERSION=8.2.7 CYLC_ENV_NAME=cylc-8.2.7 bash --login \
            -c 'exec "$0" "$@"' \
            /cylc-bin/cylc \
            remote-init -v something \
            $HOME/cylc-run/my_workflow \
            share=/g/sc/lustre/cylc-run/my_workflow/share \
            work=/g/sc/lustre/cylc-run/my_workflow/work
    RETURN CODE:
        1
    STDERR:
        WorkflowFilesError: Symlink dir target already exists: (/home/me/cylc-run/my_workflow/share ->) /g/sc/lustre/cylc-run/my_workflow/share
        Tip: in future, use 'cylc clean' instead of manually deleting workflow run dirs.

In Cylc7/rose suite-run, we were able to pre-setup the share area on disk, and then install the suite. In Cylc8, it seems, if the share directory already exists when the first task tries to run, the workflow fails. Looking at the source code I cannot see any way to override this behaviour.

In Cylc8, how do we pre-prepare data on disk before a workflow starts? I don’t really want to have a new task to copy data into place.

2

I think it should work if you create the symlink ($HOME/cylc-run/my_workflow/share) before starting the workflow.

3

I’ve confirmed @dpmatthews suggestion works.

Also, I think it’s reasonable to have to do that. Cylc 8 is stricter about workflow installation for good reasons - to avoid accidentally overwriting old workflow data with a new run. If you manually create the target data directory, but not the symlink that associates it with your new run, then Cylc can’t know it is safe to write there.

1 Like
4

Thanks, that is what I thought the answer would be, but I wanted to make sure there wasn’t a variable or option which overrides the behaviour.