Cylc 8 has upgraded from Jinja2 to version 3
In May 2021 a new version of Jinja2 was released, confusingly named version 3.
Cylc 8.0.0 uses Jinja2 version 3.0
Most Cylc workflows work with version 3 out of the box with no changes, however, there are a couple of issues it’s worth being aware of. This post outlines what’s changed and what we are doing to minimise the disruption and help the transition go smoothly.
Integers with leading zeros
There is one significant change which will impact a some workflows.
Jinja2 version 2 supports writing integers with leading zeros e.g. you can write 001 rather than 1. Support for this has been removed at version 3.
This change was unexpected, there were no deprecation warnings in version 2.
Example
The following is valid with version 2 (used by Cylc 7) but invalid with version 3 (used by Cylc 8.0.0):
{% set my_integer = 001 %}
{{ my_integer }} # this prints the number '1' - leading zeros are ignored
Implications for Cylc 8
Leading zeros in workflow configurations will now result in the following error:
Jinja2Error: expected token 'end of print statement', got 'integer'
File ~/cylc-run/leading-zero-example/flow.cylc
#!Jinja2
{{ 001 }} <-- TemplateSyntaxError
To fix, remove the leading zeros.
- {% set integer = 001 %}
+ {% set integer = 1 %}
{{ integer + 2 }}
Jinja2 ignored leading zeros before so this will not impact your workflow. This example will now work with both versions 2 (Cylc 7) and 3 (Cylc 8).
Implications for Rose 2
This also applies to any template variables defined in the rose-suite.conf file within the following sections:
[jinja2:suite.rc][empy:suite.rc][template variables]
A large number of Rose configurations use integers with leading zeros, since this change was unexpected we have found a way to extend support for zero-padded integers in Rose 2 to avoid disruption.
Rose will now automatically strip any leading zeros and issue a warning that the deprecated syntax is in use:
WARNING - Support for integers with leading zeros was dropped in Jinja2 v3. Rose
will extend support until a future version.
Please amend your Rose configuration files e.g:
* 03 => 3
* 02 => 2
* 04 => 4
* 01 => 1
* 05 => 5
This support is for a limited time to assist with upgrading to Jinja2 version 3 so please take action on these warnings.
In the future we will upgrade again from Jinja2 version 3.0 to 3.1, support for integers with leading zeros will be retired at this point. After this you will see the following error:
PluginError: Error in plugin cylc.pre_configure.rose
ConfigProcessError: template variables=BAR={04: 05}: Invalid template variable: {04: 05}
Must be a valid Python or Jinja2 literal (note strings "must be quoted").
To avoid issues please upgrade these promptly (remember to check any optional configurations).
Upgrade Schedule
Jinja2 version 2.11 → 3.0
Cylc has upgraded to Jinja2 version 3.0 at Cylc 8.0.0.
Jinja2 version 3.0 changelog: Changes — Jinja Documentation (3.1.x)
Jinja2 version 3.0 removes support for Python 2 and deprecates some legacy functionality.
If your workflows use legacy syntax you will start to see Jinja2 warnings whenever you validate or run a workflow using Cylc 8.
- If you need you workflow to continue to run under Cylc 7, you may want to wait before acting on these warnings as upgrading could break Cylc 7 compatibility.
- If you are happy to drop Cylc 7 support please take action on these warnings.
Jinja2 version 3.0 → 3.1
We will upgrade to Jinja2 version 3.1 in a subsequent Cylc 8 release.
Jinja2 version 3.1 changelog: Changes — Jinja Documentation (3.1.x)
All legacy features deprecated at version 3.0 have been removed in Jinja2 version 3.1.
To make the transition easier we are first upgrading Cylc to version 3.0 to give us time to upgrade legacy Jinja2 in our workflows rather than jumping straight to version 3.1.
Questions & Issues
If you have any questions please comment below.
If you encounter any issues upgrading to Jinja2 version 3.0 please open new thread in the “Cylc 8 Migration” category.