Use bash arrays in environment variables

Cylc Cylc Support
1

Good Afternoon,

In a cylc workflow I am trying to set an environment parameter with a bash array, like this:

[task parameters]
      client_number = 0, 1
[ runtime ]
[[client_name<client_number>]]
        script = client_name.sh
        [[[environment]]]
           names_arr = ('Mark' 'Antony')
           client_name = ${names_arr[${CYLC_TASK_PARAM_client_number}]}

So that in the script I would have automathically the client name set (Mark, for client zero, and Antony for client one).

But this is not working, since when exported in the job the environment variables get wrapped in quotes, and so in I get

client_name="('Mark' 'Antony')"

Which is not a bash array, and my strategy does not work anymore.

I tried a different approach, with no better success.
I define an array in a json file:

[
{
"names" : ["Mark", "Antony"]
}
]

And in the workflow

{% set clients = load_json('env/clients.json') %}


{% for client in client %}
[task parameters]
      client_number = 0, 1
[ runtime ]
[[client_name<client_number>]]
        script = client_name.sh
        [[[environment]]]
           client_name = {{client['names'][${CYLC_TASK_PARAM_client_number}]}}

But it throws a Template Syntax error.
I tryed also the syntax %(client_number)d, or <client_number>, with no better luck.

(It works though if I hardcode 0 or 1:

{{client['names'][0]}}

).

Any suggestion on how to better manage this?

Many thanks in advance,
Stella

2

Hi,

Unfortunately, I think Cylc is hardwired to quote environment variables, so I don’t think you’re going to be able to achieve this at present.

You could embed your array in the task’s script, although that might not be convenient.

The env-script runs before the main script and can be a more convenient place to do things like this, e.g:

[runtime]
  [[mytask]]
    env-script = """
      export client_name=('Mark' 'Antony')
    """
    script = """
      # do something with $client_name
    """
1 Like
3

Hi Olvier,
thanks for the quick reply! I suspected as much 


A quick question on the second approach: why is it not working? The jinja code get processed before the tasks parameter expansion, isn’t it?

Stella

4

Jinja2 is evaluated first, then Cylc reads the config, then Cylc submits the job with this config.

For a closer look at what’s going on, take a look at the Cylc job script:

E.g, Lets run this workflow:

$ cat flow.cylc 
[scheduling]
  [[graph]]
    R1 = foo

[runtime]
  [[foo]]
    [[[environment]]]
      ARRAY = (1, 2, 3)

$ cylc vip -N -n myworkflow .
 â–Ș ■  Cylc Workflow Engine 8.5.0
 ██   Copyright (C) 2008-2025 NIWA
▝▘    & British Crown (Met Office) & Contributors
...

Then look at the job script that Cylc generated for foo:

$ cylc cat tmp.DQn9WUMcgp//1/foo -f j
#!/bin/bash -l
#
# ++++ THIS IS A CYLC JOB SCRIPT ++++
# Workflow: tmp.DQn9WUMcgp/run1
# Task: 1/foo
# Job log directory: 1/foo/01
# Job runner: background
if [[ $1 == 'noreinvoke' ]]; then
    shift
else
    exec bash -l "$0" noreinvoke "$@"
fi

CYLC_FAIL_SIGNALS='EXIT ERR TERM XCPU'
export CYLC_VERBOSE=false
export CYLC_DEBUG=false
export CYLC_VERSION='8.5.0'
export CYLC_ENV_NAME='cylc-8.5.0-1'
export CYLC_WORKFLOW_ID="tmp.DQn9WUMcgp/run1"

cylc__job__inst__cylc_env() {
    # CYLC WORKFLOW ENVIRONMENT:
    export CYLC_CYCLING_MODE="integer"
    export CYLC_UTC="False"
    export CYLC_WORKFLOW_FINAL_CYCLE_POINT="1"
    export CYLC_WORKFLOW_INITIAL_CYCLE_POINT="1"
    export CYLC_WORKFLOW_NAME="tmp.DQn9WUMcgp"
    export CYLC_WORKFLOW_NAME_BASE="tmp.DQn9WUMcgp"
    export CYLC_WORKFLOW_UUID="b77afc1e-0b76-41e0-b301-f7f1c492870c"

    # CYLC TASK ENVIRONMENT:
    export CYLC_TASK_COMMS_METHOD=zmq
    export CYLC_TASK_JOB="1/foo/01"
    export CYLC_TASK_NAMESPACE_HIERARCHY="root foo"
    export CYLC_TASK_TRY_NUMBER=1
    export CYLC_TASK_FLOW_NUMBERS=1
}

cylc__job__inst__user_env() {
    # TASK RUNTIME ENVIRONMENT:
    export ARRAY
    ARRAY="(1, 2, 3)"
}

CYLC_RUN_DIR="${CYLC_RUN_DIR:-$HOME/cylc-run}"
. "${CYLC_RUN_DIR}/${CYLC_WORKFLOW_ID}/.service/etc/job.sh"
cylc__job__main

#EOF: 1/foo/01

This script is what actually gets executed when Cylc submits a job.

All of the default environment variables Cylc provides the job with are defined in this file, e.g. CYLC_WORKFLOW_ID and CYLC_WORKFLOW_FINAL_CYCLE_POINT.

However, this file also contains all of the variables we define too, e.g. ARRAY in this case.

We didn’t define quotes, but Cylc added them for safety (otherwise, if we set ARRAY=a b c, it would break Bash syntax).

Because of this, no Jinja2 hacking will fix the issue.

1 Like
5

Thanks a lot for the explanation!

I will put here the solution I came up with, in case someone in the future will need to do something similar.
I reworked a bit my second solution. In the json I define a dictionary:

[
{
        "client_number" : [0, 1]
        "client_dic": [{"number":0,"name":"Mark"},{"number":1,"name":"Anthony"}]
}
]

And then I can define in the environment

{% set clients = load_json('env/clients.json') %}

{% for client in clients %}
[task parameters]
      client_number = {{ client["client_number"] | join(', ') }}
[ runtime ]
[[client_name<client_number>]]
        script = client_name.sh
        [[[environment]]]
           {% for i_client in client["client_dic"]  %}
                  client_name_{{i_client["code"]}} = {{i_client["name"]}}
           {% endfor %}
           client_name = $client_name_%(client_number)s
{% endfor %}

This is multiplying the number of variables exported in the env (as many as dict entries), but that’s the best I could come up with and it adapts automathically when I want to add or remove clients from the json.

Best,
Stella

6

That’s a clever solution @sparonuz

IMO probably a bit easier to do it in a script item rather than [environmentthen Cylc puts no restrictions on your bash syntax. But, that’s up to you.

7

Hi Hilary,
Thanks.

I would absolutely do that, if it were just one script. It would be cleaner and clearer.
In my case I prefer to put it into an env, so that all the scripts that needs it (8 so far) can just inherit it, and no code is duplicated!

Best,
Stella

1 Like