Using a Jinja2 variable in an "%include" statement

Cylc Cylc Support
1

Hi. I apologize profusely if the answer to this is spelled out in the docs – cylc.github.io is timing out for me right now.

My suite.rc uses platform-specific include files to handle things like batch queueing systems and parameters that vary from platform to platform. The way I’ve been handling it is to have a Jinja2 variable set at the top of the suite.rc, e.g.

{% set J_PLATFORM = "foo" %}

and then, near the bottom of the suite.rc, some lines that act based on the variable value:

# Platform dependence
{% if J_PLATFORM == "foo" %}
%include inc/platform_foo.cylcrc
{% elif J_PLATFORM == "bar" %}
%include inc/platform_bar.cylcrc
{% elif J_PLATFORM == "baz" %}
%include inc/platform_baz.cylcrc
{% else %}
    {{ raise('User-specified configuration variable J_PLATFORM has invalid value.') }}
{% endif %}

While this form has worked fine for me, it also meant that every time someone installed the system on a new platform, after creating an include file (let’s say, inc/platform_mysys.cylcrc), it wasn’t enough that to just edit the top of suite.rc to set J_PLATFORM to “mysys”. They also had to add
an “elif” clause near the bottom to cover the new case as well.

So I wanted to replace the entire if/elif/endif block with something like this:

# Platform dependence
%include inc/platform_{{J_PLATFORM}}.cylcrc

Then, the only suite.rc change that would be required of the user installing our suite onto a new system would be to change what the J_PLATFORM variable was set to. However, this doesn’t validate: I get a FileParseError message telling me "Include-file not found: inc/platform_{{J_PLATFORM}}.cylcrc, as if the Jinja2 engine is not substituting in the value for {{J_PLATFORM}} prior to cylc examining the suite.rc contents. I don’t understand this, because I do it in other contexts in the same suite.rc, e.g.

{% set J_CP_CADENCE = "PT15M" %}

(stuff snipped)

[scheduling]
    [[dependencies]]
        (stuff snipped)
        [[[ ^+{{J_CP_CADENCE}}/{{J_CP_CADENCE}} ]]]

and that works fine.

What am I missing?

Thanks!

2

OK, some more investigation has shown me that it’s all about the include. If I just put a line that says nothing but

include inc/platform_{{J_PLATFORM}}.cylcrc

(i.e. no “%” at the beginning), then when the suite.rc fails to validate, it tells me

FileParseError:
Invalid line 1436: include inc/platform_foo.cylcrc

So in that case, it does the Jinja variable substitution. But if I precede that with a “%” sign, so that it becomes an actual include statement, the variable substitution doesn’t happen.

I’m guessing that what’s going on is that even though the docs said that the Jinja engine goes first, what really goes first is the includes, because maybe the include files have Jinja code in them. If the Jinja engine ran first, and then the includes happened, then Jinja code in the include files would never get acted on. And that makes sense.

So now I’m wondering if what I want to do is even possible.

3

Yes you’re right. We should mention this in the docs, where %include is documented, and in the Jinja2 section. (Sorry for that omission!).

Fortunately, Jinja2 has its own template inclusion mechanism to do what you need:

#!Jinja2

{% set inc1 = "include.flow" %}
[scheduling]
...
{% include inc1 %}
4

(The native Cylc %include mechanism also predates our Jinja2 support, BTW)

5

Note: Jinja2 has both import and include blocks, you may want to add the with context keywords: