I have a few workflows handling data from climat models simulations. From one simulation to another, only very few parameters of the workflows will differ, so I work with this structure:
I have one git-versionned repo containing the workflow definitions (bin/, flow.cylc, etc). All of these workflows use rose to parametrize flow.cylc, so each have a rose-suite.conf file. In each workflow I also have an opt/ folder where I put the simulation-specific parameters, so files named like rose-suite-SIMNAME.conf. This folder is git-ignored.
So when I start a workflow I do : cylc vip wkflname -O SIMNAME and everything is good.
However, if I need to start another workflow of the same type but for another simulation, then typing the same command : cylc vip wkflname -O OTHERSIM won’t work as expected. This is because the first time cylc install was called, rose created opt/rose-suite-cylc-install.conf which remembers to apply the SIMNAME options. I then get a new running workflow that merges the options of both simulations, and it sometimes screws things up. Especially when I need to stop and reload (cylc vr) a running workflow.
For the moment, each time I create a new workflow instance, I remember to remove the rose-suite-cylc-install.conf file before. But is there a way to prevent the file from being created ? Or a way to pass options (-O SIMNAME) and override previous ones instead of merging them ?
I’ve tried to replace the file with a non-writable one, but that makes cylc install fail. I’ve also tried replacing it with a symlink to /dev/null, but that fails too by making rose unable to detect changes.
From this situation, I guess I’m not using cylc and rose the way they were intended. Does anyone have a similar process, and how do you handle this ?
I’m not sure if I’ve misunderstood your scenario, but I don’t see that behaviour.
If you are using numbered runs (the default) then each time you run cylc vip it installs a new instance of the workflow in, e.g. wflow/run1/, wflow/run2/ etc., and each of these runs is entirely separate except for the link back to the source directory at the wflow/ level.
this is because the first time cylc install was called, rose created opt/rose-suite-cylc-install.conf which remembers to apply the SIMNAME options.
That file gets created inside the runX directory for each run X and it should not affect other runs in any way. It is only there to remember options for subsequent restarts of the same run.
Maybe you aren’t using numbered runs (cylc vip --no-run-name), in which case the worklfow will be installed to wflow/ (not wflow/runN) but if so, cylc vip will abort with an error after the first time because the target run directory already exists. Then you could reinstall (or cylc vr instead of vip) but Cylc will want to restart the existing workflow from where it got to in the first run, not start again from scratch.
Like Hilary, I’m not sure that I understand why you are seeing this - and I’m not sure I can replicate it. However, one thing that I don’t understand, which may be relevent:
This folder is git-ignored.
I’m not sure that makes sense. Why do you need to git-ignore your opt folder?
You should be able to have all your different setups as different rose-suite-experiment-one.conf, rose-suite-experiment2.conf files. If you are changing a single rose-suite-SIMNAME.conf file and reinstalling the workflow your changes will be applied.
So I am not using different runs for each simulation but rather making them whole new workflows.
My git-versionned folder looks like:
GIT/
workflow-template/
bin/
opt/
rose-suite-sim1.conf
rose-suite-sim2.conf
rose-suite-cylc-install.conf <- this is the one I don't want
flow.cylc
rose-suite.conf
When I start a workflow I go within workflow-template and do: cylc vip -n wk-sim1 -O sim1. Then later for the second simulation I do again cylc vip -n wk-sim2 -O sim2.
This creates new folders in my cylc-run which now looks like:
The problem being that cylc vip creates the rose-suite-cylc-install.conf within the GIT/workflow-template/opt/ directory and subsequent cylc vip or cylc vr pick it up. This is not the location you are talking about, is something misconfigured on my side ?
I don’t remember exactly why I am creating a new workflow each time instead of using runs. One reason would be that I want the simulation name sim1 to appear somewhere in the GUI, so if using runs I think I would want to use named runs and not numbered ones.
And Oliver is right : the opt/ folder is git-ignored so that I don’t have to manage versionning in the individual simulations configurations, it’s not useful to me and creates noise. In fact, the opt/ folder is a symlink to some place where all my colleagues have read permissions, creating some kind of shared archive of configurations. I also print the state of the git repo (branch + commit) as a comment in the top of the conf file so one can see on which version this conf was run.
In my example, the first run used -O foo and the second -O '(bar)'. Cylc didn’t mix them together.
The rose-suite-cylc-install.conf file live inside the installed run.
The cylc reinstall and cylc vr commands update existing installations and read in this file (so remember previously specified options).
The cylc install and cylc vip commands create new installations and do not read in this file (so do not remember options that were specified to create previous runs).
Can you provide some instructions to help us to replicate your problem.
Where is your workflow’s working copy, is it in ~/cylc-src/ or did you git checkout directly into ~/cylc-run/?
Ah! I got it. The issue was that my opt/ directory is a symbolic link to outside the git. When installing files, cylc copies the symbolic link information, thus the opt/ dir in the cylc-run directory is a symlink as well, to the same location.
I can see that the rsync command used to copy files uses the “–links” option of rsync to preserve symlinks (through the use of -a in the default rsync options here).
After a few tries, I haven’t found a working option for rsync command in my global.cylc, my guess is that any option I put there is overridden by the -a added by cylc later on. I will change my ways of doing things to avoid having opt/ as a symlink!