# Setting directives at runtime: --nodes from env-var

**URL:** <https://cylc.discourse.group/t/setting-directives-at-runtime-nodes-from-env-var/592>\
**Category:** Cylc Support\
**Created:** [February 1, 2023, 7:50am UTC](https://cylc.discourse.group/t/setting-directives-at-runtime-nodes-from-env-var/592 "2023-02-01T07:50:43Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![fredw](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/fredw/32/112_2.png) [@fredw](https://cylc.discourse.group/u/fredw)\
**Post date:** [February 1, 2023, 7:50am UTC](https://cylc.discourse.group/t/setting-directives-at-runtime-nodes-from-env-var/592/1 "2023-02-01T07:50:43Z")

</div>

Hello all,

Apologies, still using cylc v7.9.1 here. But my issue likely applies to cylc 8 as well.

I have a suite with dynamic parameters tat depend on the model domain which is defined by values is rose-suite.conf. Upon startup the suite creates a grid, extracts the horizontal grid dimensions, calculates how many processors are required, and finally calculates the required number of nodes to run the model.  
The startup tasks that compute these runtime parameters simply write those values into little files in ${CFG\_MPI\_PARAMS\_DIR} (under the suite’s share directory). In the root’s environment section I can import the values of these dynamic parameters, because shell substitution works well there:

```auto
    [[root]]
        [[[environment]]]
            MPI_PARAM_NPROCS = $(cat ${CFG_MPI_PARAMS_DIR}/MPI_PARAM_NPROCS.cfg || echo 0)
            MPI_PARAM_NUM_NODES = $(cat ${CFG_MPI_PARAMS_DIR}/MPI_PARAM_NUM_NODES.cfg || echo 0)

```

so any subsequent task can grab the values from the environment variables, e.g. MPI\_PARAM\_NPROCS.

In the runtime section I define the task that runs the model like this:

```auto
[runtime]
    [[run_model]]
        [[[environment]]]
            ROSE_TASK_APP = run_model
            MPI_OPTS = -np ${MPI_PARAM_NPROCS}
        [[[directives]]]
            --partition = defq
            --nodes = ${MPI_PARAM_NUM_NODES}

```

While this looks logical to me, the mechanism of using variables only works for the [[[environment]]] section of the task definition. The variable MPI\_OPTS will indeed take a value like “-np 103” if 103 was the content of the little file ${CFG\_MPI\_PARAMS\_DIR}/MPI\_PARAM\_NPROCS.cfg (written by the suite startup tasks).

But this doesn’t work for the [[[directives]]] and the job fails to start at all with an error reported in job-activity.log:

> [STDERR] sbatch: error: “${MPI\_PARAM\_NUM\_NODES}” is not a valid node count

It appears that no shell substitution is done, the text is interpreted verbatim as **–nodes ${MPI\_PARAM\_NUM\_NODES}** rather than **–nodes 4**.

So my question is:  
_ **How can I dynamically set the tasks directives, like --nodes ?** _

Thanks,  
Fred

---

<div class="post-metadata">

**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\
**Post date:** [February 1, 2023, 8:56am UTC](https://cylc.discourse.group/t/setting-directives-at-runtime-nodes-from-env-var/592/2 "2023-02-01T08:56:14Z")

</div>

For the moment, the only way to change the directives for a task is using `cylc broadcast`. For instance, in your startup tasks, rather than write the values into files, broadcast the required settings to the relevant tasks / families.

---

<div class="post-metadata">

**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\
**Post date:** [February 1, 2023, 10:39am UTC](https://cylc.discourse.group/t/setting-directives-at-runtime-nodes-from-env-var/592/3 "2023-02-01T10:39:21Z")

</div>

To give a little more detail on how you can make directives dynamic using broadcasts as suggested above.

This example sets the `--nodes` directive for all tasks in the `MPI` family on a per-cycle-point basis allowing `MPI_PARAM_NUM_NODES` to be set dynamically whilst the workflow is running:

```auto
[scheduling]
    [[graph]]
        P1D = """
            start_cycle => MPI
        """

[runtime]
    [[start_cycle]]
        script = """
            cylc broadcast \
                "${CYLC_WORKFLOW_ID}" \
                -p "${CYLC_TASK_CYCLE_POINT}"
                -n MPI \
                -s '[directives]--nodes=${MPI_PARAM_NUM_NODES}' \
                -s '[environment]MPI_PARAM_NUM_NODES=${MPI_PARAM_NUM_NODES}'
        """
        [[[environment]]]
            MPI_PARAM_NPROCS = $(cat ${CFG_MPI_PARAMS_DIR}/MPI_PARAM_NPROCS.cfg || echo 0)
            MPI_PARAM_NUM_NODES = $(cat ${CFG_MPI_PARAMS_DIR}/MPI_PARAM_NUM_NODES.cfg || echo 0)

    [[MPI]]

    [[run_model]]
        inherit = MPI

```

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [February 1, 2023, 11:02pm UTC](https://cylc.discourse.group/t/setting-directives-at-runtime-nodes-from-env-var/592/4 "2023-02-01T23:02:57Z")

</div>

> [@fredw](#):
>
> ```auto
> [[[directives]]]
> --partition = defq
> --nodes = ${MPI_PARAM_NUM_NODES}
> 
> ```
> 
> While this looks logical to me, the mechanism of using variables only works for the [[[environment]]] section of the task definition.

Hi @fredw - I’ll explain exactly why this doesn’t work, even though it might look sensible at first glance.

The workflow config file (`suite.rc` in Cylc 7, or `flow.cylc` in Cylc 8`) is not a shell script, of course, so Cylc itself does not parse bash syntax.

Particular task config items, namely the `script` and `[environment]` ones, get written verbatim to the task job scripts that get submitted to execute your tasks. The same goes for Slurm directives, which are written to the job script (as they are supposed to be) as shell comments. The written values are \*as defined in the `suite.rc` at start-up, and _nothing gets evaluated by the shell until the shell actually executes the job script, on the job host_. So **all** of your job scripts are going to look like this:

```bash
...
# --nodes=${NODES}
...
# Cylc user environment definitions:
NODES=$(cat nodes.txt)
...
# Cylc user script items
echo Hello World
...

```

The shell never sees the directives, because they’re protected inside a shell comment.

Note each of your tasks will actually define `NODES` by `cat`ing the file that the first task writes. They do not just inherit the resulting value from root. What they inherit from root is the literal environment variable definition including the `cat` statement.

Long story short, the values of batch system directives (like environment variables and script items) are defined at workflow start-up, and are written verbatim to the job script in the expected format - which is a shell comment for directives.

To support what you want we would have to make Cylc artificially interpret bash syntax in certain Cylc config item values before writing job scripts and handing them off to the shell (via Slurm or whatever) for execution. That is feasible but there are potential pitfalls and complications.

The correct way to do it, as suggested, is to use `cylc broadcast` to “broadcast” new/updated config item values to the scheduler at run time, to replace the values it parsed at start-up. The new values will then be used instead of the originals when new job scripts are written, for any tasks targetted by the broadcast command.

I hope that all makes sense!

---

<div class="post-metadata">

**Author:** ![fredw](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/fredw/32/112_2.png) [@fredw](https://cylc.discourse.group/u/fredw)\
**Post date:** [February 14, 2023, 4:11am UTC](https://cylc.discourse.group/t/setting-directives-at-runtime-nodes-from-env-var/592/5 "2023-02-14T04:11:45Z")

</div>

Thank you very much. This is the method I used in the end and it works fine.  
I inserted a new task, which always runs before the “run\_model” task, which broadcasts the directives.  
It’s a little bit ugly, because it’s not entirely obvious at first glance why this is needed, but I added a comment pointing to this thread, so future-me might remember.

As always, thanks for the prompt reply