Cylc 8 with LFRic Rose stems and Jinja2 issues

Cylc Cylc 8 Migration
1

Hi, I’m trying to run an LFRic rose stem using Cylc 8 as:

%> rose stem --group=developer --opt-conf-key=$platform --source=lfric=$lfric $PWD/rose-stem
%> cylc play lfric_multi_thread

This rose stem runs without problem in Cylc 7, but in Cylc 8 Jinja2 throws up an error when processing some of the macros. The first error is given as:

ERROR - Workflow shutting down - Jinja2Error: 'target'
    File /home/565/dl9118/cylc-run/lfric_multi_thread/run7/suite.rc
          [[compile_{{projectName}}_with_{{label}}]]
              inherit = {{family}}, TARGET
              {{ scripting( <-- KeyError

This error occurs in the file:
https://code.metoffice.gov.uk/trac/lfric/browser/LFRic/trunk/gungho/rose-stem/suite.rc
With the scripting() macro defined in:
https://code.metoffice.gov.uk/trac/lfric/browser/LFRic/trunk/infrastructure/rose-stem/inc/common_macros.rc

As pointed out by my colleague Scott Wales, the error goes away if the macro argument setup is modified as:

-               ['base', 'compiler.'+compiler, 'build'],
+               ['base', 'build'],

I’m not sure if this is an issue with Jinja2 (v3.0.1) or Cylc8. Any advice would be greatly appreciated.

Cheers, Dave.

2

Hi, thank you for dropping us a line on this forum.

Can you compare what values rose stem is setting between Cylc 7 and Cylc 8?

To do this, if you run the same rose stem <your args> -n test-cylc-<version> command

grep set ~/cylc-run/rose-stem-cylc-7/suite.rc

cat ~/cylc-run/rose-stem-cylc-8/runN/opt/rose-suite-cylc-install.conf
3

Hello,

I’ve taken a look at this, it seems that Cylc 8 is providing the same Jinja2 variables to the workflow as Cylc 7 was so it doesn’t look like an issue with Cylc.

The error is coming from within the Jinja2 logic contained in the workflow itself. We’ve recently added some advice for debugging Jinja2 code into the Cylc documentation which might help.

With debug/log statements you might be able to locate the source of the problem.

The main changes as far as Jinja2 is concerned between Cylc 7 and 8 are:

  • Python 2.{6,7} => 3.7+
  • Jinja2 v2 => v3

There’s some information on the Jinja2 change in this post - Cylc 8 & Jinja2 v3, it might also be worth having a quick skim of the Jinja2 changelog.

Cheers,
Oliver

4

Took a look into this myself,

I tracked the problem down to this line:

subtree = context.vars['target'][target_dets['platform']]

In this function Jinja2Globals/get_target_property.py::get_target_property.

After a bit of investigation it looks like context.vars is empty in certain situations. The offending call is coming from the macro called scripting. The target variable is set in this macro, however, it doesn’t seem to be present in context.vars. This might be a Jinja2 bug, I don’t know enough about @contextfunction to say for sure.

It appears that the target variable is present directly in the context variable so I managed to shift the error by making the following change:

 @contextfunction
 def get_target_property(context, target_dets, prop_path=None, compiler=None):
     if prop_path is not None:
         path_parts = prop_path.split('.')
     else:
         path_parts = [] 
-    subtree = context.vars['target'][target_dets['platform']]    
+    subtree = context['target'][target_dets['platform']]    
     while len(path_parts) > 0:
5

Thanks for your responses @oliver.sanders , @wxtim , this gives me something to work with. Great to know that this is most likely a Jinja2, not Cylc8 issue. Is there a list of known Jinja2 bugs somewhere that I could look through?

Cheers, Dave.

6

I don’t have any definitive knowledge, but a quick google suggests on possible thing to try changing:

This note in the jinja2 docs:

New in version 3.0.0: [pass_context] Replaces contextfunction and contextfilter.

7

I just tried loading Jinja2 v3.1.2 (as opposed to version 3.0.3, which I gather is the default for Cylc 8.0.1), and got the following error when running cylc check ...:

Jinja2Error: cannot import name 'contextfilter' from 'jinja2' (/g/data/dp9/dl9118/jinja2/3.1.2/lib/python3.9/site-packages/jinja2/__init__.py)

I can see that the filter functions that are imported in the v3.0.3 __init__.py file from filters.py no longer exist in Jinja2 v3.1.2. I guess more recent versions of Jinja2 are not supported yet for Cylc8, is this correct?

8

Cylc 8 is currently pinned to 3.0.*.

but… it’s been pinned to keep the back-compatibility available at 3.0.* - there are more changes to Jinja2 at 3.1.* of which the removal of contextfilter is one - you’ve just previewed the issues you’ll have with Jinja2 changes when you change from version 3.0 to 3.1.

1 Like
9

Jinja2 deprecated some old interfaces at version 3.0 (and added warning messages), they have removed back-support at version 3.1.

Cylc is currently sticking with Jinja2 v3.0.x in order to give everyone a chance to adapt to the Jinja2 warning messages. There’s more about the update here:

10

Thank you both for the confirmation Oliver and Tim. I will keep an eye out for when Cylc8 is at Jinja2 v3.1.x and then revisit. Cheers, Dave.

1 Like
11

Is Jinja2 v3.1 a blocker for your work?

12

No, not a blocker. I was just testing to see if I could run an LFRic rose stem with Cylc8, there is no immediate rush for me to get this working. I am happy to wait. Thanks again for all your help Oliver and Tim.