Parsing and unparsing flow.cylc

Cylc Cylc Support
1

Hi,

I’m new to Cylc and trying to work with variable substitution within flow.cylc. This is snippet I have, say:

     [[HOFXRun]]
        script = "pywalker RunJediExecutable $config -d $datetime"
        platform = $(platform)
        execution time limit = $(execution_time_limit)
        [[[directives]]]
            --account = $(account)
            --qos = $(qos)
            --constraint = $(constraint)
            --job-name = HOFXRun
            --nodes=$(nodes)
            --ntasks-per-node=$(ntasks_per_node)

When the user creates the experiment these would be substituted by some variables of their choosing. I have a dictionary ready to go (that changes with task) containing the values of the variables so it seems a natural thing to do is:

from cylc.flow.parsec import fileparse
target = fileparse.parse("/path/flow.cylc")

Then substitute $() in target using keys from the other dictionary and then rewrite the flow.cylc. But I don’t see anything for writing the dictionary to back to the cylc format INI. Perhaps I’m just missing it or perhaps I’m thinking about this substitution problem the wrong way and missing a feature elsewhere in Cylc. Perhaps I should be utilizing the platform differently? I could go line by line in the flow.cylc reading as a string and do the substitution but I need to know the name of the task in order to perform the correct substitution so using the dictionary approach seems more sensible and saves me writing my own parser. Anyway would appreciate any advice you have for a newbie!

2

Hi @Dan_Holdaway

While it’s possible to parse and unparse a flow.cylc file (the cylc config command essentially does that) - we wouldn’t recommend it, at least not until Cylc 9 where a supported API is planned.

For substituting variables in a workflow definition, just use Jinja2 (or Empy) templating: Jinja2 — Cylc 8.0b3 documentation

A simple example:

#!Jinja2

%include "task_props.jinja2"  # literal include

[scheduling]
   [[graph]]
       R1 = "foo => bar"
[runtime]
   [[foo]]
       script = echo {{task_props["foo"]["greeting"]}}
   [[bar]]
       script = echo {{task_props["bar"]["greeting"]}}

Where the “task properties” are defined in task_props.jinja2 as a dictionary:

{% set task_props =
   {
     "foo": {
        "greeting": "HELLO"
      },
     "bar": {
        "greeting": "GOODBYE"
      }
   }
%}

To see the result:

$ cylc view --jinja2 --stdout flow.cylc
[scheduling]
   [[graph]]
       R1 = "foo => bar"
[runtime]
   [[foo]]
       script = "echo HELLO"
   [[bar]]
       script = "echo GOODBYE"

or:

$ cylc config -i '[runtime]foo' flow.cylc
script = echo HELLO

Note my example uses %include to literally insert the content of the input file into the workflow definition. You can also use Jinja2 {% include %} and {% import %} for more sophisticated template wrangling. And Cylc’s Jinja2 support allows direct import of Python modules too (see the Cylc User Guide).

You can also supply Jinja2 inputs on the command line with all flow.cylc-parsing commands, with the --set and --set-file options. However, that is intended more for single value variables (one per line) and would be a bit ugly for dictionaries.

Hope that helps!

3

Many thanks @hilary.j.oliver this is very helpful insight.