Cylc submit and jinja2 template variables

Cylc Cylc Support
1

version 7.8.8
cylc submit -s myVar1=bob -s myVar2=fred --dry --debug mySuite myTask.1

...snip
2021-07-17T14:21:39Z DEBUG - Processing with Jinja2

2021-07-17T14:21:39Z DEBUG - Setting Jinja2 template variables:

+ myVar1=bob

+ myVar2=fred

2021-07-17T14:21:39Z DEBUG - Expanding [runtime] namespace lists and parameters
...snip

i was trying to replace values in a task script.
suite.rc

#!jinja2
[scheduling]
    cycling mode        = integer
    initial cycle point = 1
    final cycle point   = 1

    [[dependencies]]
        graph = "myTask"

[runtime]
    [[root]]

    [[myTask]]
        script = echo myVar1 = {% set myVar1 = myVar1 %} myVar2 = {% set myVar2 = myVar2 %}

script validates.
cylc submit ā€¦produces:
[myTask.1] Job ID: 49915

however, cylc get-config --sparse mySuite
shows

[scheduling]
    cycling mode = integer
    initial cycle point = 1
    final cycle point = 1
    [[dependencies]]
        graph = myTask
[runtime]
    [[root]]
    [[myTask]]
        script = echo myVar1 =  myVar2 =

am i going down the wrong path here?

show a use case where the
"-s " switch in ā€˜cylc submitā€™ is used successfully?

2

I made a small change to your suite.rc to use the Jinja variable, instead of declaring it.

#!jinja2

[scheduling]
    cycling mode        = integer
    initial cycle point = 1
    final cycle point   = 1

    [[dependencies]]
        graph = "myTask"

[runtime]
    [[root]]

    [[myTask]]
        script = echo myVar1 = {{ myVar1 }} myVar2 = {{ myVar2 }}

Now using {{ myVar1 }} syntax and also specifying the Jinja variable values in the command line (-s). After that, I think your example should work.

kinow@ranma:~/cylc-run/cylc7test$ /opt/cylc/usr/bin/cylc get-config -s myVar1=bob -s myVar2=fred --sparse cylc7test
[scheduling]
    cycling mode = integer
    initial cycle point = 1
    final cycle point = 1
    [[dependencies]]
        graph = myTask
[runtime]
    [[root]]
    [[myTask]]
        script = echo myVar1 = bob myVar2 = fred

Not sure why validation didnā€™t fail, although I think your suite was valid, but with empty variables.

Cheers
Bruno

3

thank you bruno.

i did try your instance initially without 'get-configā€™ing it, e.g., i always, register, validate, ā€¦, get-config, etcā€¦
the validation fails for meā€¦

Jinja2Error:
  File "<template>", line 15, in top-level template code
UndefinedError: 'myVar1' is undefined
Context lines:

    [[myTask]]
        script = """
            echo myVar1 = echo myVar1 = {{myVar1}} myVar2 = {{myVar2}}	<-- Jinja2Error


[mach]$ cylc --version
7.8.4-10-gde7d

FWIW; using --debug in cylc submit, ie.

cylc submit -s myVar1=bob -s myVar2=fred --dry --debug mySuite myTask

does not differentiate the cases of using {{myVar1}} vs. {%set myVar1=myVar1 %}.

hence, i was not able to diagnose using --debug.

4

Hi!

the validation fails for meā€¦

Ah, yes. I tested the validate command before changing the suite and it worked. But now it fails due to Jinja not finding the variables defined in the suite.rc file (the error is similar to referencing a variable before it is declared in Python).

This should work:

kinow@ranma:~/cylc-run/cylc7test$ /opt/cylc/usr/bin/cylc validate -s myVar1=bob -s myVar2=fred cylc7test
Valid for cylc-7.9.3

So you will always have to define the variables in your cylc sub-commands that need to parse the suite.rc file, like submit, validate, get-config, etc.

I have not used much Rose, besides reading the documentation and doing a few code reviews for the new code, but I think one of the features of Rose is to allow you to save your variables so you donā€™t have to keep repeating them in your command line.

But others can confirm if thatā€™s the case, or if there are other workarounds.

Note too that functionalities of Rose are being incorporated in Cylc 8, either directly in the code or as plug-ins.

does not differentiate the cases of using {{myVar1}} vs. {%set myVar1=myVar1 %}.

I think your previous example is a valid suite.rc. The generated echo command is valid, but does not print anything.

And the Jinja statements you used were valid. They were declaring variables, but not outputting anything.

Cheers
Bruno

5

Hi @schaferk ,

@kinow pretty much nailed it, but I can add a couple of comments.

It helps to think of Jinja2 as a preprocessor that generates the final suite.rc file that the Cylc scheduler loads. The Jinja2 {% ... %} syntax is for statements that set and manipulate Jinja2 variables but donā€™t print anything to your suite.rc file. The {{ ... }} syntax is the print statement, which writes text out to your suite.rc file (which @kinow is calling the template ā€œoutputā€).

If your suite.rc contains Jinja2 variables that have to take their values from command line inputs, then you will have to supply those inputs on every command line that parses the suite.rc file (including cylc validate) otherwise the command will fail due to referencing undefined variables. However, you can set defaults to be used if you donā€™t supply values on the command line:

{{ VAR | default("blah") }}

Yes, rose suite-run takes Jinja2 variables defined in the rose-suite.conf file and writes the variable definitions to the suite.rc at install time. However, that functionality has been migrated into Cylc 8, so donā€™t bother with it now unless you are already using rose suite-run with Cylc 7.

Hilary

1 Like