Setting runtime env variables from a script

Cylc Cylc Support
1

Kia ora! Another question regarding environment variables.
In our current pipeline, at the beginning of each run we do a number of calculations with dates and set an inordinate ( :scream:) number of env vars which are then used thruout the whole pipeline. I need to replicate this in our workflow at the beginning of each cycle, for each “cast” I am running in parallel (the vars will be different depending on what type of “cast” it is).
The way I thought to handle this is for run an initial task that produces a file with all the variables which is then [[[environment]]]%included in subsequent tasks. However this does not work, because the inclusion happens at flow.cylc parse time, at which time this file is empty.
Is there any other way I can set “dynamic” env variables at the beginning of each cycle?
I’ve looked at broadcasting, but don’t think it’s right for this case since it sets variables for all of runtime.
Any tips?
Cheers!

2

Hi,

You’ll want to take a look at the “cylc broadcast” command. This allows us to make changes to the configuration of a task, family or cycle of tasks, e.g. adding or updating environment variables.

A common pattern is to have one task at the start of the cycle, which configures the rest. Here’s a minimal example (untested):

[scheduling]
  initial cycle point = 2000
  [[graph]]
    P1D = configure => start => run => end

[runtime]
  [[configure]]
    script = """
      cylc broadcast \
        "${CYLC_WORKFLOW_ID}" \
        -p "${CYLC_TASK_CYCLE_POINT}" \
        -n root
        -s "[environment]START_TIME=$(isodatetime "${CYLC_TASK_CYCLE_POINT}" --format=CCYYMMDDThh)"
    """
  [[start, run, end]]
    script = echo $START_TIME

Broadcasts can be made manually via user intervention, but also automatically by embedding cylc broadcast commands in the workflow as done above. Broadcasts can also be adjusted or revoked on-the-fly. For a quick walk through, see this tutorial: Broadcast Tutorial — Cylc 8.4.0 documentation

If you have lots of environment variables that you want to broadcast at the same time, you can write a “broadcast file” which is like a snippet from a “.cylc” file. The cylc broadcast command can then apply all the changes in this file in one go.

There’s an example in the docs of how to do this. The example is for a different use case, but the broadcast part works the same:

https://cylc.github.io/cylc-doc/stable/html/user-guide/examples/event-driven-cycling/index.html#admonition-3

Final note. Cylc “xtriggers” can return data, this data is automatically broadcasted to all tasks in the cycle. So if you just need a simple (and fast to run) Python script to compute these variables, xtriggers can be a good option.

3

So this

#!Jinja2

{% set EXP_START_DATE = '20250113' %}
{% set EXP_END_DATE = '20250120' %}
{% set casts = ['hindcast', 'forecast', 'nowcast', 'daily_forecast'] %}
{% set casts = [
    {'exemode':'hindcast','recur':'R/2025-W01-1/P7D','cycle_length':'7','typerun':'ana'},
    {'exemode':'nowcast','recur':'R/2025-W01-3/P7D','cycle_length':'7','typerun':'ana'},
    {'exemode':'daily_forecast','recur':'P1D','cycle_length':'11','typerun':'daily_pre'},
    {'exemode':'forecast','recur':'R/2025-W01-3/P7D','cycle_length':'14','typerun':'pre'}
] %}

[scheduler]
    cycle point format = %Y%m%d

[ scheduling ]

    initial cycle point = {{ EXP_START_DATE }}
    final cycle point = {{ EXP_END_DATE }}

    [[ graph ]]

{% for cast in casts %}
        {{cast['recur']}} = """ {{cast['exemode']}}_pre => {{cast['exemode']}}_model => {{cast['exemode']}}_post """
{% endfor %}

[ runtime ]

{% for cast in casts %}

    [[ {{ cast['exemode'] }}_pre ]]
        script = """
            init_date_vars.sh ${CYLC_TASK_CYCLE_POINT} {{cast['exemode']}} 7
            cylc broadcast -p ${CYLC_TASK_CYCLE_POINT} -n {{cast['exemode']}} -F ${CYLC_TASK_WORK_DIR}/{{cast['exemode']}}.cylc ${CYLC_WORKFLOW_ID}
        """

    [[ {{ cast['exemode'] }}_model ]]

        script = """
            echo "{{cast['exemode']}}_model"
            echo "${JRUN}"
            echo "${JSTART}"
            echo "${JSTOP}"
            echo "${jul_JRUN}"
            echo "${jul_JRUN0}"
            echo "${julstart}"
            echo "${julstop}"
            echo "${jul_DATEINI}"
            echo "${ncycle_bias}"
            echo "${list_JBIAS_HTS}"
            echo "${list_julbias_HTS}"
            echo "${julmin_compute_HTS_bias_correction}"
            echo "${julmin_apply_HTS_bias_correction}"
            echo "${julmin_recup_TAIR}"
            echo "${julmin_create_TAIR}"
            echo "${julfirst_use_data}"
        """

    [[ {{ cast['exemode'] }}_post ]]
        inherit = {{ cast['exemode'] }}_pre

        script = """
            echo "{{cast['exemode']}}_post"
            echo "${JRUN}"
            echo "${JSTART}"
            echo "${JSTOP}"
            echo "${jul_JRUN}"
            echo "${jul_JRUN0}"
            echo "${julstart}"
            echo "${julstop}"
            echo "${jul_DATEINI}"
            echo "${ncycle_bias}"
            echo "${list_JBIAS_HTS}"
            echo "${list_julbias_HTS}"
            echo "${julmin_compute_HTS_bias_correction}"
            echo "${julmin_apply_HTS_bias_correction}"
            echo "${julmin_recup_TAIR}"
            echo "${julmin_create_TAIR}"
            echo "${julfirst_use_data}"
        """
{% endfor %}
does almost what I need it to do. 
But not quite: since the namespace is [root] it affects all subsequent tasks. I want to be able to tell it to broadcast to just the the tasks of the form { cast['exemode'] }}_*.
The -n flag only takes one namespace.
4

This line should target only tasks/families called cast['exemode'] (this won’t match all tasks which start cast['exemode']):

cylc broadcast -p ${CYLC_TASK_CYCLE_POINT} -n {{cast['exemode']}}

The -n option can be used multiple times to target multiple namespaces, e.g:

cylc broadcast -p ${CYLC_TASK_CYCLE_POINT} -n {{cast['exemode']}}_pre -n {{cast['exemode']}}_post

Alternatively, you could group these tasks together into a family so they can al be operated on with a single namespace:

[runtime]
    [[{{ cast['exemode'] }}]]

    [[{{ cast['exemode'] }}_pre]]
        inherit = {{ cast['exemode'] }}
    [[{{ cast['exemode'] }}_post]]
        inherit = {{ cast['exemode'] }}
    [[{{ cast['exemode'] }}_model]]
        inherit = {{ cast['exemode'] }}
5

Hi Oliver, yes, listing is not an option there will be too many.I tried the family thing, but I get the error:

WorkflowConfigError: undefined parent for hindcast_pre: [[hindcast]]
6

Ugh, never mind, it PEBKAC (and friday) :sweat_smile:. I kept the square brackets in the import statement