Use Jinja2 to check for directory existence

Cylc Cylc Support
1

I am using Cylc version 7.9.3. My suite will use configuration files from a preset and premade subdirectory in ${CYLC_SUITE_DEF_PATH}/etc/experiments matching the name of the registered ${CYLC_SUITE_NAME}. If the suite cannot find a matching name, it will use ${CYLC_SUITE_DEF_PATH}/etc/default. I am trying to add in a conditional Jinja2 assert statement that notifies the user upon validation if the suite cannot find a matching experiment name and is going to use default but I cannot think of a way to do this. Normally, I would think of using an embedded shell command within Jinja, storing a True or False within a variable and then using an if statement. But I found out that shell statements are processed after Jinja2 rendering, so the if statement ends up being a string containing the shell command instead of the result of the shell command. Is it possible to do what I need with the tools on hand or is the only way to do this to add a new Jinja2Filter that checks for the existence of ${CYLC_SUITE_DEF_PATH}/etc/experiments/${CYLC_SUITE_NAME}?

2

I’m not entirely sure I understand what you’ve said here. Jinja2 is a template preprocessor - all it does (in Cylc) is programmatically manipulate text to generate the final config file content (suite.rc for Cylc 7) that the Cylc config parser sees.

The Cylc config file format does not support “shell statements” in any way, it is simply a bunch of key = value config items that configure what the Cylc scheduler program does. Certain config item values (e.g. script = "<value>") are written verbatim to task job scripts, where they are evaluated by the shell interpreter when the job executes, but they are never evaluated by Cylc itself.

Jinja2 does not support shell statements either - it is a Python-like language with its own statements etc… Unless you mean to call out to a subprocess in which you run shell commands - is that what you’re trying to do?

3

OK, on second read your intention seems clear, it’s just the “Normally, I would …” bit that I found confusing.

With Cylc 8 I can do this sort of thing:

#!Jinja2
{% from "os" import path %}
{% from "cylc.flow" import LOG %}

{% if path.exists("/expts/name") %}
    {% do LOG.warning("Using expt config dir") %}
{% else %}
    {% do LOG.warning("Using default config dir") %}
{% endif %}

[scheduling]
    [[graph]]
        R1 = "foo"
[runtime]
    [[foo]]

When I validate this, I get:

$ cylc validate test
WARNING - Using default config dir
Valid for cylc-8.3.3

And (just tested) this works back at Cylc 7.9.9 with a minor adjustment to the import statement:

{% from "cylc" import LOG %}

I have not tested as far back as 7.9.3 but I would expect it to work.

p.s. Cylc 7 is old (and Python 2 based!) - do you have Cylc 8 upgrade plans?

Relevant docs: Jinja2 — Cylc 8.3.0 documentation

4

Thanks for the reply and sorry for the confusion in the original post, I sometimes don’t entirely understand the extents and limits of Jinja2 within Cylc.

I tried what you posted and it works but only if I hardwire what is in the path.exists() function. Is it possible to take this one step further and use Cylc variables (i.e. CYLC_SUITE_DEF_PATH and CYLC_SUITE_NAME) within that function? The idea is the suite name is also the name of the subdirectory within /expts/name (note, the root directory such as /expts/name will also vary by user based on where they clone their code) and if a subdirectory named the same way as the suite is found, then one statement will be printed and if it is not found, a second statement will be printed. Hence, the root dir name and suite name should be defined based on the aforementioned Cylc variables. Is such a thing possible?

I tried {% if path.exists(" ${CYLC_SUITE_DEF_PATH}/etc/experiments/${CYLC_SUITE_NAME}") %} but always got the “Using default config dir”

Yes, we are planning on converting our suite to 8.0.0 in the near future but for now, 7.9.3 suits our needs

5

You wouldn’t be the first :slight_smile: It’s simpler than you might think. Jinja2 is just a template preprocessor: the code gets “processed out” at start-up to generate the final plain-text Cylc configuration that “Cylc proper” sees. That’s done whenever the config file gets parsed, e.g. first thing at validation, or first thing at scheduler start-up. You can use cylc view -p to see the result of Jinja2 preprocessing.

Those are shell environment variables. Cylc imports the shell environment into a Jinja2 dictionary called environ. You need to be aware of the context (i.e., when/where the file is being parsed) because some CYLC_ variables are only meaningful, and only defined, in the scheduler runtime environment, and not when (e.g.) validating the source directory.

That said, you’re good to go with those two variables. This works at validation and runtime:

#!Jinja2
{# Note the following code needs tweaking for Cylc 8 #}
{% from "cylc" import LOG %}
{% do LOG.warning("CYLC_SUITE_NAME is " + environ['CYLC_SUITE_NAME'] | default('not defined')) %}
{% do LOG.warning("CYLC_SUITE_DEF_PATH is " + environ['CYLC_SUITE_DEF_PATH'] | default('not defined')) %}
[scheduling]
  [[dependencies]]
       graph = "foo"
[runtime]
   [[foo]]

Note the | default("not defined") bit is required to allow cylc view -p to work, because no CYLC_ variables are defined in that context. (And it is good practice anyway, for variables that aren’t globally defined).

Cylc 8

Cylc 8 defines native Jinja2 CYLC_ variables so you don’t even have to read the environment Jinja2 — Cylc 8.3.0 documentation