Jinja2 variable differences in config files - new ones are not being respected?

Cylc Cylc Support
1

Hi,

The below is using CYLC_VERSION=8.2.4 and the UI looks like it is 2.1.0.

I have a workflow installed, then compare the flow-processed.cylc file to cylc config name, and they are not consistent.

e.g. installation

$ cylc vip -S JINJA_VAR=1
$ cylc vr -S JINJA_VAR=12

$ cylc config basic-workflow
[scheduler]
    allow implicit tasks = True
[scheduling]
    initial cycle point = 20200101T0000Z
    final cycle point = 20210101T1200Z
    [[graph]]
        PT6H = a => b
[runtime]
    [[root]]
        script = echo "12"
        platform = localhost
$ cat ~/cylc-run/basic-workflow/runN/log/config/flow-processed.cylc
[scheduler]
    allow implicit tasks=True
[scheduling]
    initial cycle point = 20200101T0000Z
    final cycle point = 20210101T1200Z
    [[graph]]
        PT6H = a => b
[runtime]
    [[root]]
        script = echo "1"
        platform = localhost

When I do edit runtime in the UI for the task a, it is consistent with the flow-processed.cylc file.

image

Letting one of the tasks run, and looking at the script in the job file, the script is "1", not what is in cylc config.

$ grep -A1 SCRIPT ~/cylc-run/basic-workflow/runN/log/job/20200101T1200Z/a/NN/job
# ++++ THIS IS A CYLC JOB SCRIPT ++++
# Workflow: basic-workflow/run1
--
# SCRIPT:
echo "1"

So a couple of questions,

  1. Is it intentional the flow-processed.cylc is not updated always?
  2. Where is the UI Edit Runtime getting its values from that they are different to what is in cylc config?
  3. Where is the job script generation coming from such that I get the old value, and not the latest that is available via cylc config?
  4. Most importantly (for me) - how do I get the latest values respected?
2

Hi @TomC

I followed your procedure with 8.2.4 and I don’t see the same problem.

  • the first command (cylc vip) installs and starts the workflow; after which cylc config and cat flow-processed.cylc match
  • the second command (cylc vr) doesn’t work:
WARNING - Template variables (from --set/--set-file)
          can only be changed if the workflow is stopped.

After that, the two queries still match, but still with the original value.

Can you double check? I guess one of us has made a mistake.

  1. the flow-processed.cylc should be updated if it changes on re-install
  2. template values are “processed out” during file parsing, so what the UI edit runtime page sees is the parsed result. (However they are also stored in the run DB for reuse if not repeated on the command line e.g. for a restart)
  3. ditto
  4. (maybe you just missed the error message above - that reload, unlike restart, doesn’t accept template changes?)
3

My example is with S not s.

4

Ah right, my bad - you’re talking about Rose templates.

For others watching this discussion, this is about capability injected into Cylc by a plugin.

I’m out of time for now, but best to leave this one to the Rose devs anyway.

[they might consider moving this to the Rose Support category]

5

Hi Tom -
I think that you ought to be getting the output Hilary has, and it’s a bug that you don’t.

The answer here is almost certainly to ensure that you have stopped the workflow before reinstalling and reloading it.

I’ve created an issue to fix it.

6

I am uncertain why you need to stop and start a workflow to change a jinja2 variable? I’m pretty sure (haven’t checked), I could change the graph and reinstall a workflow without needing to stop it. Similarly, I could change hardcoded stuff inside a flow.cylc without needing to stop/start it. So why does -[sS] need a special case when others do not?

EDIT:
Also, this seems to be a loss of functionality if this is the case. In Cylc7 with rose to install things at least, you could update jinja2 variables without issue whilst a workflow is running. If Cylc7 was similar to rose, able to update jinja2 variables without stop/starting, has this functionality change been called out in the 7to8 documentation?

7

Yeah I was wondering about that myself @TomC (although didn’t mention it, as failing to reproduce your problem seemed more salient at the time).

I suspect it might be an accidental omission in migrating rose suite-run into Cylc 8. Jinja2 templating is worked out when the flow.cylc is parsed, but that has to been done for a runtime reload as well for a restart. Let’s see what @wxtim says.

8

The -S option applies to (re)installation. It should work with the cylc vr command irrespective of whether the workflow is running or not, I think this is just a bug somewhere.

The -s, -z and --set-file options are not currently supported by the cylc reload command (they weren’t supported at Cylc 7 either). I don’t think there’s any technical reason for this, we just never implemented it.

9

Confirmed, it’s a simple bug: vr: plugin options not respected when workflow is running · Issue #5968 · cylc/cylc-flow · GitHub

There’s a one-line hack there that will suppress the issue for now if you’re willing to patch your deployment, otherwise, we’ll aim to get this into a bugfix release as soon we’re able to.