In Cylc7/rose, when using rose suite-run, files are copied over to the remote. Old files which no longer (or never have) existed in the local area do not get touched. In Cylc8, you are using the --delete option to remove files on a remote platform that don’t exist in the install location.
Is there a way to stop this happening, or make sure it does not delete specific files? Perhaps like there is an [scheduler]install option there could be a [scheduler]exclude option too? I know of several cases in our systems which do some linking/copying in of files via an install_cold task to the app or bin area.
Running with unknown files present in the run directory is potentially dangerous, so they get deleted by default on reinstall from source.
Of course cylc reinstall can’t know the origin of legit files generated by the workflow at run-time, so there are two solutions for that:
ideally, all files written by workflow tasks at run time should go in the share directory, which is there for that purpose and is ignored during re-installation (Cylc 8 automatically supports a share/bin/ directory, BTW)
less ideally, if you insist on having tasks write files to the top-level run directory, you can handle that with a .cylcignore file in the source directory.
cylc reinstall --help: see How to prevent reinstall deleting files
and cylc reinstall spits out a run-time reminder too:
$ cylc install
INSTALLED bug/run1 from /home/oliverh/cylc-src/bug
$ touch ~/cylc-run/bug/run1/stuff # create a foreign file
$ cylc reinstall bug
REINSTALL bug/run1 from /home/oliverh/cylc-src/bug
del. stuff
TIP: You can "exclude" files/dirs to prevent Cylc from installing or overwriting
them by adding them to the .cylcignore file. See cylc reinstall --help.
Reinstall would make the above changes.
Continue [y/n]:
Thanks. I did not know about the share/* and .cylcignore pathways. I had thought of doing share/bin and manually adding it to a PATH, but it already being that should help. I’ll pass this information to the people who asked me.
Note that the .cylcignore file only applies to the local installation of the workflow source files to ~/cylc-run performed by cylc install and cylc reinstall. Remote installation is different and, by default, only installs a limited set of directories. If a remote tasks write files to any of these directories then they will get removed if the remote installation is repeated which happens after a reload - there is no way to prevent this. See https://cylc.github.io/cylc-doc/stable/html/user-guide/running-workflows/scheduler-start-up.html#remote-initialization
If you want to add files to the app or bin directories you can do this on localhost (but make sure you add them to the .cylcignore file to avoid them being removed and make sure you do this before running any remote task). For files only required on the remote platform, using share/bin is best. We’re intending to support use of share/app in Rose as well (https://github.com/metomi/rose/issues/2745).