Using arbitrary Python modules via Jinja2

Cylc Tips
1

Did you know you Cylc extends Jinja2’s import mechanism to allow direct use of arbitrary Python modules?

Here’s an example that creates tasks at start-up, to process each file in a directory:

$ ls /path/to/data/
cat  dog  fish

#!Jinja2
{# import Python's os module: #}
{% from "os" import listdir %}

{% set DATADIR = "/path/to/data" %}

[task parameters]  {# use os.listdir() #}
   id = {{ listdir(DATADIR) | join(", ") }}
[scheduling]
   [[graph]]
      R1 = "prep => proc<id> => results"
[runtime]
   [[prep]]
      # ...
   [[proc<id>]]
      script = """
         do-something.py {{DATADIR}}/${CYLC_TASK_PARAM_id}
      """
   [[results]]
      # ...

Result after Jinja2 processing (use cylc view -j):


[task parameters]
   id = fish, cat, dog
[scheduling]
   [[graph]]
      R1 = "prep => proc<id> => results"
[runtime]
   [[prep]]
      # ...
   [[proc<id>]]
      script = """
         do-something.py /path/to/data/${CYLC_TASK_PARAM_id}
      """
   [[results]]
      # ...

And (use cylc graph):

image
image648×542 48.3 KB

Documented here in the User Guide: Jinja2 — Cylc 8.1.4 documentation

3 Likes
2

If you have a workflow which would need to perform a lot of logic in Jinja2, you can use this import mechanism to implement the logic in Python. Python modules in a workflow’s lib/python directory are added to the path automatically.

Note that Jinja2 and any modules you import with it will run under the same Python environment that Cylc is installed under.

1 Like