Complex conditionals

Cylc Cylc Support
1

Hi there! In operations we run a hindcast and a nowcast everyday. On wednesdays we additionally run a forecast and a “daily forecast”.
They can be run in a parallel, with the only requirement being that the model run must wait for the appropriate restart files produced by the “previous” model in the sequence hindcast → nowcast → forecast/daily forecast
The way I am approaching this is using Jinja2 in the [runtime] portion flow.cylc, like this:

[ runtime ]
{% set CASTS = [ ('hindcast', 'ana'), ('nowcast', 'ana'), ('forecast', 'pre'), ('daily_forecast', 'daily_pre')] %}
{% from "datetime" import datetime %}
{% for exemode, typerun in CASTS if exemode == 'forecast' or exemode == 'daily_forecast' and datetime.now().weekday() == 2 %}

.... all my tasks
{% endfor %}

I’ve tried doing this in the graph portion but I could not figure it out. Moreover, given that there are a bunch of variables that vary with the type of “cast” I figured this is a better approach.
Any thoughts?
Cheers!
Gaby

2

Hi,

Jinja2 is all evaluated before Cylc reads in the workflow configuration file, so you cannot use it to change the config on a cycle-by-cycle basis.

However, Cylc can handle this for you:

[scheduling]
  initial cycle point = 2025
    [[graph]]
        P1D = """
            # every day
            hindcast => nowcast
        """
        R/2025-W01-4/P7D = """
            # every thursday
            nowcast => forecast => daily_forecast
        """

[runtime]
    [[hindcast]]
    [[nowcast]]
    [[forecast]]
    [[daily_forecast]]

To see what this example would do, run the following command:

$ cylc graph . 20250101 20250110

The “every thursday” bit is a little tricky so the syntax is a bit unusual:

  • P1D - means a period of 1 day (i.e. run every day)
  • 2025-W01-4 means the 4th day of the first calenadr week of 2025 (i.e. the first Thursday of the year)
  • R/2025-W01-4/P7D - means a period of 7 days, starting on the first thursday of the year.
3

Just to expand on this statement a bit @gturek - Jinja2 templates are evaluated once, first thing when the file is parsed. It’s a preprocessor that generates the final “raw” workflow configuration that the scheduler sees. So for your example, datetime.now().weekday() will be evaluated once at start-up, end of story.

You can view the result of Jinja2 template processing with cylc view -p.

1 Like
4

Hi Oliver, thanx for the explanation.
Now, given that the graph for one “cast” is something like

      P1D = """

        # BUILD
        FAMBUILD[^]:succeed-all => archi<archiPre> => recup<recup> & fill_namelist<namtype>

        # PREP RUN
        FAMPREP:succeed-all & create_statics[^] => run_model

        # RUN
        run_model[-P1D] => run_model

        # POST RUN
        FAMRUN:succeed-all => concat_nc => obsopr
        obsopr => diafromola & archi_cdf & conv_moor

        # POST
        FAMPOSTRUN:succeed-all => archi<archiPost>
        """

with some small variations for each case TBD. It makes for a lot of code duplication for each cast. And how do you handle the dependence of run_model from one cast to the other?
I am guessing this is where Rose comes in, but that’s where we’re stuck, because it is not clear to us how to make use of it.

5

a lot of code duplication

This is where you might want to use jinja2 - In the example below I take a simplified version of your graph, and use Jinja2 to repeat it for each “cast”. There is an if section to demonstrate turning bits of graphing on and off depending on the name of the cast.

At the end I use @oliver.sanders graph to link the forecasts.

Note that there is no dependence on real world time or between cycles in this example.

    [[graph]]

{% set casts = {
    'hindcast': 'P1D',
    'nowcast': 'P1D',
    'forecast': 'P1D',
    'daily_forecast': 'R/2025-W01-4/P7D'
} %}
{% for cast, reccur in casts.items() %}
        {{reccur}} = """
            # This is where your graph goes.
            {{cast}}_prep => {{cast}}_forecast => {{cast}}_post

            # An example of adding an extra dependency for only one "cast"
            # n.b. style guide suggests that indentation should be separate
            # for Jinja and Cylc, but I think this is more readable.
            {% if cast == 'hindcast' %}
                {{cast}}_extra_prep => {{cast}}_forecast
            {% endif %}
        """
{% endfor %}

        # How to make the forecasts relate to each other.
        P1D = """
            # every day
            hindcast => nowcast
        """
        R/2025-W01-4/P7D = """
            # every thursday
            nowcast => forecast => daily_forecast
        """
{% endfor %}

To see how this would be interpreted by Cylc, use

> cylc config /path/to/workflow
> cylc graph /path/to/workflow
1 Like
6

Or cylc view -p /path/to/workflow

(The view command just processes out the Jinja2; config and graph actually parse the resulting raw Cylc config and work with that).