Seeking recommendations for flow reuse

Cylc Cylc Support
1

Hi all,

I am using the latest cylc with the rose-suite.conf option. Currently my setup looks like this:

git-versioned-folder/
    workflow_template/
        [ ... ]
        rose-suite.conf.template
workflow-folder/
    workflow-instance1/
        [ ... ] 
        rose-suite.conf    
    workflow-instance2/
        [ ... ] 
        rose-suite.conf

In short, my workflow is developed in a git repo. When I want to run an instance of said workflow, I copy the whole folder to somewhere else, I edit the rose-suite.conf file and then I install the workflow.

While it seems safe to have each instances in independent folders, it becomes a hassle when I need to make some tweaks to an instance and than import those change back into the template. At this point, only the rose-suite.conf file contains information that changes from instance to instance.

To avoid the duplication of folders, I would like to be able to do something like:

cylc install -n wf-name --conf wf-instances/instance1.conf ~/git-versioned-folder/workflow_template/

So, having only one folder for workflow development, but being able to change the name/location of the rose-suite.conf in the install call. So my multiple workflow instances would all be installed from the same folder, except for the configuration itself.

I don’t think this is possible. Would it be an interesting addition ?

How do people manage this kind of usage ? I read a bit about rose itself and it looks like it could address this issue, but it also seems much larger in scope, adding a lot more configurations and moving parts that I don’t think I need ?

2

Once you have used cylc install you can edit your source without causing any changes to the running workflow. That is why we have cylc install.

This means that you could use Git to make these copies:

git checkout -b "my_branch"
# Make changes
cylc vip --run-name "$(git br --show-current)"

It’s worth pointing out too, that if you were to cylc install without making a commit, the diff is recorded in cat ~/cylc-run/<workflow>/log/version/uncommitted.diffand the commit has is stored in ~/cylc-run/<workflow>/log/version/vcs.json .

A second way of doing something similar is using rose-suite optional configs:

mkdir opt
cat > opt/rose-suite-myoption.conf <<__HERE__
[env]
myvar="This value will override myvar from rose-suite.conf"
__HERE__

cylc vip -O myoption --run-name "whatever-you-want-to-call-it"
1 Like
3

Nice! Thanks for the quick answer, I knew there were ways I didn’t know about.

I’ll look into both options. In my use case, I often have multiple instances of the same workflow running at the same time and I currently didn’t think useful to have the instance’s rose-suite.conf committed in the repo. The second option really looks like what I was imagining.

4

Just to be more explicit - your copying the source file to a new location duplicates the functioning of cylc install, which is mostly just rsyncing your source file to a new file in ~/cylc-run.

cd myworkflow
cylc  install ~/cylc-src/myworkflow   # Will install # ~/cylc-run/myworkflow/run1
sed -i 's@x=42@x=84@g' rose-suite.conf
cylc install ~/cylc-install/myworkflow   # Will install # ~/cylc-run/myworkflow/run2

When you use cylc play it is only interested in the specific copy in the cylc-run directory, and not in the other copies or the source directory. This means that installing gives you a snapshot of the source directory at the time you install.

If in one of the tasks you were to have script = echo {{x}} then that task in run1 will always return 42, and in run2 will always be 84.