# Jinja variables in rose stem cylc8 compatibility mode **URL:** \ **Category:** Cylc 8 Migration\ **Created:** [October 11, 2023, 11:48pm UTC](https://cylc.discourse.group/t/jinja-variables-in-rose-stem-cylc8-compatibility-mode/780 "2023-10-11T23:48:07Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![srennie](https://avatars.discourse-cdn.com/v4/letter/s/8edcca/32.png) [@srennie](https://cylc.discourse.group/u/srennie)\ **Post date:** [October 11, 2023, 11:48pm UTC](https://cylc.discourse.group/t/jinja-variables-in-rose-stem-cylc8-compatibility-mode/780/1 "2023-10-11T23:48:07Z")
I’m trying to run UM (13.0) rose-stem and the suite is failing to validate or play due to undefined Jinja2 variables. I haven’t used compatibility mode before, and I also haven’t run UM rose-stem before, so I don’t know if I’m seeing bugs, features, issues with my UM config or what else. The documentation does not answer all my questions! Is rose-stem fully supported at cylc 8.2.1? or are there still things to be done? From the documentation, I can see that the “defined on the fly at installation time” jinja2 variables are in opt/rose-suite-cylc-install.conf. It contains this: ```auto # This file records CLI Options. !opts= [env] # ROSE_ORIG_HOST set by cylc install. ROSE_ORIG_HOST=gadi-login-06.gadi.nci.org.au [jinja2] HOST_SOURCE_UM="gadi-login-06.gadi.nci.org.au:/g/data/dp9/sjr548/um/r120067_um13_add_bom_cfg" HOST_SOURCE_UM_BASE="gadi-login-06.gadi.nci.org.au:/g/data/dp9/sjr548/um/r120067_um13_add_bom_cfg" RUN_NAMES=['fcm_make'] SITE="nci" SOURCE_UM="/g/data/dp9/sjr548/um/r120067_um13_add_bom_cfg" SOURCE_UM_BASE="/g/data/dp9/sjr548/um/r120067_um13_add_bom_cfg" SOURCE_UM_MIRROR="fcm:um.xm/branches/dev/susanrennie/r120067_um13_add_bom_cfg@120401" SOURCE_UM_REV="" [jinja2:suite.rc] # ROSE_ORIG_HOST set by cylc install. ROSE_ORIG_HOST=gadi-login-06.gadi.nci.org.au ``` - I’m not sure what is generating the entries under `[jinja2]`, but they aren’t recognised when trying to play the workflow. This causes failure with cylc validate and cylc play. - If I use `-S SITE='"nci"'` with the rose-stem command then it shows up under `[jinja2:suite.rc]` - SITE is defined in global rose.conf in the cylc installation as `[rose-stem]automatic-options=SITE=nci` but I’m not sure if this is is the source of the SITE variable. - I don’t know where the rest of the stuff under `[jinja2]` is defined, but I guess somewhere in the UM rose-stem source code - blame my lack of knowledge on that one. - If I similarly add a variable to my own ~/.metomi/rose.conf, it does not get picked up at all - Should it? Why do I get stuff under `[jinja2]` which probably should be under `[jinja2:suite.rc]`? ---
**Author:** ![srennie](https://avatars.discourse-cdn.com/v4/letter/s/8edcca/32.png) [@srennie](https://cylc.discourse.group/u/srennie)\ **Post date:** [October 12, 2023, 2:23am UTC](https://cylc.discourse.group/t/jinja-variables-in-rose-stem-cylc8-compatibility-mode/780/2 "2023-10-12T02:23:55Z")
Okay, after some RTSC I found utilities.py in cylc-rose that does this: ```auto if templating != 'template variables': config['templating_detected'] = templating.replace(':suite.rc', '') else: config['templating_detected'] = templating ``` So now I know why I get jinja2 sans :suite.rc. But clearly that doesn’t explain how to get the workflow to recognise the jinja2 variables defined therein. ---
**Author:** ![wxtim](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/wxtim/32/151_2.png) [@wxtim](https://cylc.discourse.group/u/wxtim)\ **Post date:** [October 12, 2023, 8:17am UTC](https://cylc.discourse.group/t/jinja-variables-in-rose-stem-cylc8-compatibility-mode/780/3 "2023-10-12T08:17:21Z")
### Why do I get stuff under `[jinja2]` which probably should be under `[jinja2:suite.rc]`? Because there is a bug. Thank you for bug-spotting and identifying exactly where and what the bug is. As it happens we’ve already fixed this for the next release ([this PR](https://github.com/cylc/cylc-rose/pull/248)). > But clearly that doesn’t explain how to get the workflow to recognise the jinja2 variables defined therein. The top level `rose-suite.conf` in the installed workflow is modified to include `opts=(cylc-install) ` so that the variables saved in `rose-suite-cylc-install.conf` are made available when Cylc parses the `flow.cylc` file. Your best bet for a practical fix is to change `[jinja2:suite.rc]` for `[template variables]` in your `rose-suite.conf` files. > If I similarly add a variable to my own ~/.metomi/rose.conf, it does not get picked up at all - Should it? I don’t think so - have you found some documentation suggesting that you might expect this to happen? ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [October 12, 2023, 9:34am UTC](https://cylc.discourse.group/t/jinja-variables-in-rose-stem-cylc8-compatibility-mode/780/4 "2023-10-12T09:34:17Z")
As a workaround you can rename the `jinja2:suite.rc` section (Rose 2019/2) to `template variables` (Rose 2 only). ---
**Author:** ![srennie](https://avatars.discourse-cdn.com/v4/letter/s/8edcca/32.png) [@srennie](https://cylc.discourse.group/u/srennie)\ **Post date:** [October 12, 2023, 10:54pm UTC](https://cylc.discourse.group/t/jinja-variables-in-rose-stem-cylc8-compatibility-mode/780/5 "2023-10-12T22:54:24Z")
Thanks for the workaround! My apologies for going to the bother of reading the source code (I thought I was looking at the most recent version yesterday but didn’t register the bugfix) and not searching through the issues for the problem. I hadn’t seen the documentation mention user rose.conf with respect to rose-stem, but assumed any rose call might parse it. But I’m not very familiar with rose2 yet. A quick search of the documentation just now didn’t really clear things for me, unless the statement that rose-stem only wraps cylc install implies that things like parsing rose.conf are not included. ---
**Author:** ![srennie](https://avatars.discourse-cdn.com/v4/letter/s/8edcca/32.png) [@srennie](https://cylc.discourse.group/u/srennie)\ **Post date:** [October 12, 2023, 11:57pm UTC](https://cylc.discourse.group/t/jinja-variables-in-rose-stem-cylc8-compatibility-mode/780/6 "2023-10-12T23:57:50Z")
Spoke too soon. If I use [template variables] in the rose.conf, then the suite fails to validate. ```auto gadi-login-07:dp9 run6 $ cylc validate . WARNING - Backward compatibility mode ON FileParseError: Invalid line (line 1): {# The UM test suite is currently in a transitional state; the original #} (line numbers match 'cylc view -p') ``` It seems to be failing to parse all the jinja2 before validating? If I use jinja2:suite.rc and manually define all the variables with `-S` when calling rose stem, then the suite validates fine ```auto gadi-login-07:dp9 run5 $ cylc validate . WARNING - Backward compatibility mode ON INFO - nci WARNING - deprecated items were automatically upgraded in "workflow definition" WARNING - * (8.0.0) [cylc]abort if any task fails - DELETED (OBSOLETE) Valid for cylc-8.2.1 ``` The only difference of note between run5 and run6 seems to be in the .conf files generated when installing. ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [October 13, 2023, 8:37am UTC](https://cylc.discourse.group/t/jinja-variables-in-rose-stem-cylc8-compatibility-mode/780/7 "2023-10-13T08:37:09Z")
Hi, I think this error can be resolved (without changing `[template variables]`) by adding a `#!Jinja2` shebang. I.E. make `#!Jinja2` the first line of the `suite.rc` file. This is what tells Cylc to run Jinja2 on the file. I’ve spoken to the UM people, whilst Cylc 8 supports the rose-stem framework, the UM rose-stem workflow has not yet been upgraded to be Cylc 8 compatible on trunk. So sadly I think you’re going to find yourself fighting through the upgrade errors one at a time. There is a branch here [https://code.metoffice.gov.uk/trac/um/changeset?reponame=&new=119530%40main%2Fbranches%2Fdev%2Fjamesbruten%2Fvn13.3\_um\_cylc8\_update&old=118802%40main%2Ftrunk](https://code.metoffice.gov.uk/trac/um/changeset?reponame=&new=119530%40main%2Fbranches%2Fdev%2Fjamesbruten%2Fvn13.3_um_cylc8_update&old=118802%40main%2Ftrunk) but it isn’t fully functional just yet. It should validate and launch though. Note this isn’t on trunk, so it’s not supported and may change in the future.