# Cylc 8 with LFRic Rose stems and Jinja2 issues

**URL:** <https://cylc.discourse.group/t/cylc-8-with-lfric-rose-stems-and-jinja2-issues/525>\
**Category:** Cylc 8 Migration\
**Created:** [August 26, 2022, 6:19am UTC](https://cylc.discourse.group/t/cylc-8-with-lfric-rose-stems-and-jinja2-issues/525 "2022-08-26T06:19:30Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![DaveLee](https://avatars.discourse-cdn.com/v4/letter/d/a6a055/32.png) [@DaveLee](https://cylc.discourse.group/u/DaveLee)\
**Post date:** [August 26, 2022, 6:19am UTC](https://cylc.discourse.group/t/cylc-8-with-lfric-rose-stems-and-jinja2-issues/525/1 "2022-08-26T06:19:30Z")

</div>

Hi, I’m trying to run an LFRic rose stem using Cylc 8 as:

```auto
%> rose stem --group=developer --opt-conf-key=$platform --source=lfric=$lfric $PWD/rose-stem
%> cylc play lfric_multi_thread

```

This rose stem runs without problem in Cylc 7, but in Cylc 8 Jinja2 throws up an error when processing some of the macros. The first error is given as:

```auto
ERROR - Workflow shutting down - Jinja2Error: 'target'
    File /home/565/dl9118/cylc-run/lfric_multi_thread/run7/suite.rc
          [[compile_{{projectName}}_with_{{label}}]]
              inherit = {{family}}, TARGET
              {{ scripting( <-- KeyError

```

This error occurs in the file:  
[https://code.metoffice.gov.uk/trac/lfric/browser/LFRic/trunk/gungho/rose-stem/suite.rc](https://code.metoffice.gov.uk/trac/lfric/browser/LFRic/trunk/gungho/rose-stem/suite.rc)  
With the `scripting()` macro defined in:  
[https://code.metoffice.gov.uk/trac/lfric/browser/LFRic/trunk/infrastructure/rose-stem/inc/common\_macros.rc](https://code.metoffice.gov.uk/trac/lfric/browser/LFRic/trunk/infrastructure/rose-stem/inc/common_macros.rc)

As pointed out by my colleague Scott Wales, the error goes away if the macro argument `setup` is modified as:

```auto
- ['base', 'compiler.'+compiler, 'build'],
+ ['base', 'build'],

```

I’m not sure if this is an issue with Jinja2 (v3.0.1) or Cylc8. Any advice would be greatly appreciated.

Cheers, Dave.

---

<div class="post-metadata">

**Author:** ![wxtim](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/wxtim/32/151_2.png) [@wxtim](https://cylc.discourse.group/u/wxtim)\
**Post date:** [August 26, 2022, 8:41am UTC](https://cylc.discourse.group/t/cylc-8-with-lfric-rose-stems-and-jinja2-issues/525/2 "2022-08-26T08:41:12Z")

</div>

Hi, thank you for dropping us a line on this forum.

Can you compare what values `rose stem` is setting between Cylc 7 and Cylc 8?

To do this, if you run the same `rose stem <your args> -n test-cylc-<version>` command

```bash
grep set ~/cylc-run/rose-stem-cylc-7/suite.rc

cat ~/cylc-run/rose-stem-cylc-8/runN/opt/rose-suite-cylc-install.conf

```

---

<div class="post-metadata">

**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\
**Post date:** [August 26, 2022, 10:47am UTC](https://cylc.discourse.group/t/cylc-8-with-lfric-rose-stems-and-jinja2-issues/525/3 "2022-08-26T10:47:31Z")

</div>

Hello,

I’ve taken a look at this, it seems that Cylc 8 is providing the same Jinja2 variables to the workflow as Cylc 7 was so it doesn’t look like an issue with Cylc.

The error is coming from within the Jinja2 logic contained in the workflow itself. We’ve recently added some advice for debugging Jinja2 code into the Cylc documentation which might help.

- [debugging Jinja2](https://cylc.github.io/cylc-doc/nightly/html/user-guide/writing-workflows/jinja2.html#debugging)
- [logging Jinja2](https://cylc.github.io/cylc-doc/nightly/html/user-guide/writing-workflows/jinja2.html#logging)

With debug/log statements you might be able to locate the source of the problem.

The main changes as far as Jinja2 is concerned between Cylc 7 and 8 are:

- Python 2.{6,7} =\> 3.7+
- Jinja2 v2 =\> v3

There’s some information on the Jinja2 change in this post - [Cylc 8 & Jinja2 v3](https://cylc.discourse.group/t/cylc-8-jinja2-v3/512), it might also be worth having a quick skim of the [Jinja2 changelog](https://jinja.palletsprojects.com/en/3.1.x/changes/#version-3-0-0).

Cheers,  
Oliver

---

<div class="post-metadata">

**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\
**Post date:** [August 26, 2022, 12:38pm UTC](https://cylc.discourse.group/t/cylc-8-with-lfric-rose-stems-and-jinja2-issues/525/4 "2022-08-26T12:38:23Z")

</div>

Took a look into this myself,

I tracked the problem down to this line:

```auto
subtree = context.vars['target'][target_dets['platform']]

```

In this function `Jinja2Globals/get_target_property.py::get_target_property`.

After a bit of investigation it looks like `context.vars` is empty in certain situations. The offending call is coming from the macro called `scripting`. The `target` variable is set in this macro, however, it doesn’t seem to be present in `context.vars`. This might be a Jinja2 bug, I don’t know enough about `@contextfunction` to say for sure.

It appears that the `target` variable is present directly in the `context` variable so I managed to shift the error by making the following change:

```diff
 @contextfunction
 def get_target_property(context, target_dets, prop_path=None, compiler=None):
     if prop_path is not None:
         path_parts = prop_path.split('.')
     else:
         path_parts = [] 
- subtree = context.vars['target'][target_dets['platform']]    
+ subtree = context['target'][target_dets['platform']]    
     while len(path_parts) > 0:

```

---

<div class="post-metadata">

**Author:** ![DaveLee](https://avatars.discourse-cdn.com/v4/letter/d/a6a055/32.png) [@DaveLee](https://cylc.discourse.group/u/DaveLee)\
**Post date:** [August 29, 2022, 6:46am UTC](https://cylc.discourse.group/t/cylc-8-with-lfric-rose-stems-and-jinja2-issues/525/5 "2022-08-29T06:46:23Z")

</div>

Thanks for your responses @oliver.sanders , @wxtim , this gives me something to work with. Great to know that this is most likely a Jinja2, not Cylc8 issue. Is there a list of known Jinja2 bugs somewhere that I could look through?

Cheers, Dave.

---

<div class="post-metadata">

**Author:** ![wxtim](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/wxtim/32/151_2.png) [@wxtim](https://cylc.discourse.group/u/wxtim)\
**Post date:** [August 30, 2022, 7:05am UTC](https://cylc.discourse.group/t/cylc-8-with-lfric-rose-stems-and-jinja2-issues/525/6 "2022-08-30T07:05:06Z")

</div>

I don’t have any definitive knowledge, but a quick google suggests on possible thing to try changing:

[This note in the jinja2 docs](https://jinja.palletsprojects.com/en/3.0.x/api/#utilities):

> New in version 3.0.0: [pass\_context] Replaces `contextfunction` and `contextfilter`.

---

<div class="post-metadata">

**Author:** ![DaveLee](https://avatars.discourse-cdn.com/v4/letter/d/a6a055/32.png) [@DaveLee](https://cylc.discourse.group/u/DaveLee)\
**Post date:** [September 5, 2022, 1:51am UTC](https://cylc.discourse.group/t/cylc-8-with-lfric-rose-stems-and-jinja2-issues/525/7 "2022-09-05T01:51:30Z")

</div>

I just tried loading Jinja2 v3.1.2 (as opposed to version 3.0.3, which I gather is the default for Cylc 8.0.1), and got the following error when running `cylc check ...`:

`Jinja2Error: cannot import name 'contextfilter' from 'jinja2' (/g/data/dp9/dl9118/jinja2/3.1.2/lib/python3.9/site-packages/jinja2/ __init__.py)`

I can see that the `filter` functions that are imported in the v3.0.3 ` __init__.py` file from `filters.py` no longer exist in Jinja2 v3.1.2. I guess more recent versions of Jinja2 are not supported yet for Cylc8, is this correct?

---

<div class="post-metadata">

**Author:** ![wxtim](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/wxtim/32/151_2.png) [@wxtim](https://cylc.discourse.group/u/wxtim)\
**Post date:** [September 5, 2022, 7:07am UTC](https://cylc.discourse.group/t/cylc-8-with-lfric-rose-stems-and-jinja2-issues/525/8 "2022-09-05T07:07:36Z")

</div>

Cylc 8 is currently pinned to 3.0.\*.

but… it’s been pinned to keep the back-compatibility available at 3.0.\* - there are more changes to Jinja2 at 3.1.\* of which the removal of `contextfilter` is one - you’ve just previewed the issues you’ll have with Jinja2 changes when you change from version 3.0 to 3.1.

---

<div class="post-metadata">

**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\
**Post date:** [September 5, 2022, 8:37am UTC](https://cylc.discourse.group/t/cylc-8-with-lfric-rose-stems-and-jinja2-issues/525/9 "2022-09-05T08:37:31Z")

</div>

Jinja2 deprecated some old interfaces at version 3.0 (and added warning messages), they have removed back-support at version 3.1.

Cylc is currently sticking with Jinja2 `v3.0.x` in order to give everyone a chance to adapt to the Jinja2 warning messages. There’s more about the update here:

> [@Cylc 8 & Jinja2 v3](https://cylc.discourse.group/t/cylc-8-jinja2-v3/512#upgrade-schedule-6):
>
> Cylc 8 has upgraded from Jinja2 to version 3 In May 2021 a new version of Jinja2 was released, confusingly named version 3. Cylc 8.0.0 uses Jinja2 version 3.0 Most Cylc workflows work with version 3 out of the box with no changes, however, there are a couple of issues it’s worth being aware of. This post outlines what’s changed and what we are doing to minimise the disruption and help the transition go smoothly. Integers with leading zeros There is one significant change which will impact a …

---

<div class="post-metadata">

**Author:** ![DaveLee](https://avatars.discourse-cdn.com/v4/letter/d/a6a055/32.png) [@DaveLee](https://cylc.discourse.group/u/DaveLee)\
**Post date:** [September 5, 2022, 11:46pm UTC](https://cylc.discourse.group/t/cylc-8-with-lfric-rose-stems-and-jinja2-issues/525/10 "2022-09-05T23:46:37Z")

</div>

Thank you both for the confirmation Oliver and Tim. I will keep an eye out for when Cylc8 is at Jinja2 v3.1.x and then revisit. Cheers, Dave.

---

<div class="post-metadata">

**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\
**Post date:** [September 6, 2022, 9:46am UTC](https://cylc.discourse.group/t/cylc-8-with-lfric-rose-stems-and-jinja2-issues/525/11 "2022-09-06T09:46:04Z")

</div>

Is Jinja2 v3.1 a blocker for your work?

---

<div class="post-metadata">

**Author:** ![DaveLee](https://avatars.discourse-cdn.com/v4/letter/d/a6a055/32.png) [@DaveLee](https://cylc.discourse.group/u/DaveLee)\
**Post date:** [September 6, 2022, 10:02pm UTC](https://cylc.discourse.group/t/cylc-8-with-lfric-rose-stems-and-jinja2-issues/525/12 "2022-09-06T22:02:16Z")

</div>

No, not a blocker. I was just testing to see if I could run an LFRic rose stem with Cylc8, there is no immediate rush for me to get this working. I am happy to wait. Thanks again for all your help Oliver and Tim.