Particular task deleting lib and bin directories in install directory

Cylc Cylc Support
1

hey there,

i have a workflow which installs lots of files in ~/cylc-run/$CYLC_WORKFLOW_ID/bin and ~/cylc-run/$CYLC_WORKFLOW_ID/bin.

however when i run cylc vr the lib and bin directories get deleted…

> cylc vr $(basename $PWD)/run1
$ cylc validate --against-source u-cy032/run1

Valid for cylc-8.2.1

$ cylc reinstall u-cy032/run1

REINSTALL u-cy032/run1 from /scale_wlg_persistent/filesets/home/williamsjh/roses/u-cy032

del. lib/python/irislib/utils/etc/fields.rc
[snip]
del. bin/zonal_mean.py
[snip]

NOTE: Files created by Rose file installation will show as deleted.
      They will be re-created during the reinstall process.

the key point here is that they are not re-created in the reinstall.

any ideas on how i can fix this? currently i have to manually rerun a task to get them back.

thanks!

jonny

2

Workflow reinstallation updates the run directory to be compatible with the source directory. That includes deleting files that are no longer present in the source directory (or in your case, never existed there!). That’s deliberate, because files that should not be there could in principle cause trouble at run time.

If your workflow generates files at run time that should not be deleted during reinstall, they should go under the share directory, which is reserved for files generated at run time. So executables and library files installed by an “install task” at run time should go in share/bin/ and /share/lib/. Top level bin/ and lib/ are for executables and library files installed from the source directory, and subject to update or removal by the reinstall process.

Documentation:

1 Like
3

ok roger that, thanks.

is this a new cylc 8 thing? i don’t remember this ever being an issue before. the reason i ask is that climate workflows very frequently check out files from the MOSRS repository at install time.

cheers,

4

Well it’s definitely new in the sense that Cylc 7 did not have built-in workflow installation capability at all.

Somewhat less definitely, I think rose suiite-run (for Cylc 7) did not do it either.

1 Like
5

ok thanks for this. i think this is going to involve some refactoring of some of the workflows that we use but it sounds like a pretty straightforward fix (i.e. installing stuff in share rather than bin).

cheers!

6

Use share/bin/ - that’s in $PATH in your job environment just like bin/

1 Like
7

hey @hilary.j.oliver, will do :slight_smile: thanks :pray: