Hi everyone,
I simply can’t wrap my mind around that simple question: Why would I use Rose in addition to Cylc? What is it about? What is it good for?
I did not find the documentation to be that explanatory, other than “It runs suites (?)” and “It uses Cylc as an engine”.
High Level Explanation:
Rose is a toolkit for writing, editing and running application configurations.
Practical Explanation:
Cylc tasks can sometimes get a little complex, e.g. they might have a bunch of environment variables, command line arguments, namelists, file dependencies and so on.
Rose provides a configuration system you can use to define these things more formally. E.G. you can define the environment variables a task requires, provide metadata for these variables and validate a given configuration against this metadata to ensure it’s compliant.
These Rose “applications” are typically used to configure complex models such as the Unified Model (UM) which have large numbers of inter-dependent configurations and complex validation requirements.
There’s a Rose tutorial that walks through setting up a basic application configuration for a Cylc task: Rose Applications — Rose Documentation 2.1.0 documentation
Rose isn’t actually bound to Cylc, you can run Rose applications on the command line, however, Rose is most commonly used with Cylc. If you have a task configuration that’s getting a little complex, Rose might be useful for you.
Rose can also configure Cylc workflows, e.g. you can define Jinja2 variables that the workflow requires, document them with metadata, validate them, etc.
Rose also provides a bunch of utilities beyond this.
Wow, that actually sound quite useful. Thank you !
we should probably pin Oliver’s blurb to the top of the Rose category here.
Prior to Cylc 8, Rose also handled installation of Cylc 7 workflows (formal separation of source and run directory, etc.). That’s now built in to Cylc, leaving the core app configuration functionality to Rose - which is super-useful, especially for large scientific models.
Over time, shouldn’t the rest of Rose be migrated into Cylc as well?
In my very personal opinion, it’s weird to have user config not being implemented in the main software. I’m used to Snakemake and Nextflow where everything is bundled.
The more Cylc aligned utilities like rose_prune and rose_bunch will likely get re-implemented in Cylc in the future as time allows.
The Rose configuration system, however, will remain separate. People are free to use whatever configuration system they want with Cylc. Some people use Rose, some use YAML/JSON-Schema, some people may use both to configure different tasks within the same workflow (because the configuration system a task uses is often the decision of the model developer rather than the workflow orchestrator).
Cylc strives to be general purpose so isn’t going to enforce any particular choice.
I’m really just restating what Oliver said, but Cylc is generic in the sense that it needs to be able to run any command, script, or executable as a task in the workflow, whatever the means by which it is natively configured. So, the Cylc config file gives you complete freedom in the task definitions to configure the job environment and arbitrary job scripting to execute whatever you like.
Every scientific model (for instance) has its own preferred (and probably required) way of being configured. Cylc needs to work with that. It would not make sense to package, and especially not to require use of, one particular app configuration system.
I’m not super-familiar with Snakemake or Nextflow, but I’d be very surprised if they did that.