Use parameters as script's names

Cylc Cylc Support
1

Good afternoon/evening,

I am writing a long cylc workflow using cylc 8.3.3.
To try and make the flow.cylc file more readable I decided to put all my script in one folder, and to include them in the workflow using parameters

[task parameters]

    script_name = one, two

[[ script<script_name>]]
script = ${CYLC_WORKFLOW_RUN_DIR}/script/prep/${CYLC_TASK_PARAM_script_name}.sh

But soon enough I discovered that this represents a problem in the eventuality of errors: unless I set manually “set -e” inside my script_name.sh, cylc will not detect any problem and will mark the job as successful.

So the first question is, is there a workaround for this?

What I tried so far, is to use an %include statement instead. This way the script will be included and the set -e added automatically by cycl. But now I am stuck with using the CYLC_TASK_PARAM

[[ script<script_name>]]
%include  'script/prep/'${CYLC_TASK_PARAM_script_name}'.sh'

As I understand the task parameters are evaluated at run time, while the includes are processed at install time: so the validation ends with an

IncludeFileNotFoundError: script/'{{ CYLC_TASK_PARAM_script_name }}'.sh

Any help will be much appreciated!
Best,
Stella

1 Like
2

Hi Stella

To try and make the flow.cylc file more readable I decided to put all my script in one folder, and to include them in the workflow using parameters

You don’t need to do this (We recommend you don’t). If you put your scripts in bin/ Cylc automatically adds that folder to the path, so you can replace:

- script = ${CYLC_WORKFLOW_RUN_DIR}/script/prep/${CYLC_TASK_PARAM_script_name}.sh
+ script = script_name.sh

(Don’t forget to chmod the script).

unless I set manually “set -e” inside my script_name.sh, cylc will not detect any problem and will mark the job as successful.

This is expected.

We strongly recommend the use of set -euo pipefail in scripts [documentation] to prevent tasks succeeding if they shouldn’t.

3

Hi Tim!
Thanks a lot for the quick and helpful reply.

Just a clarification: are you suggesting to move all the scripts to the bin AND not to use the task parameter as scripts names? Or just to move the scripts?

Thanks again
Stella

4

Move the scripts and don’t use parameters.

You can use parameters as script names, but I’m not sure that’s super helpful. You can if you want though.

1 Like
5

Perfect,
Thanks again!

1 Like
6

Hi again from Europe,

I moved all my scripts to the bin, but I am not able to access them if they are in sub folders.

Explanation: We have a workflow that is logically subdivided in pre-run, run, post-run and post. And each of these steps runs between 5 and 10 scripts. So I would like to keep them separately in different folders. I tried to put them in bin’s subfolder and run

[[ recup ]]
        platform = login
        inherit = FAMPRE

        script = pre/recup.sh

But it end with a “command not found” error.
I do understand that it is strongly recommended to put the scripts in bin, but what I am trying to avoid is having 60 scripts + all the executables in one single folder.

Is there a way, for example, of adding other folders to the cycl path (like script/ for example). I tried with

[scheduler]
    install = script/pre, script/run, script/post_run, script/post

But this copies the folders to the remote, without adding them to the path. And so I had to go back to my old syntax of

[[ script_name ]]
script = ${CYLC_WORKFLOW_RUN_DIR}/script/pre/script_name.sh

Any suggestion will be much appreciated,
Stella

7

The best I can think of is:

    [[PRE]]
        pre-script = export PATH="${PATH}:${CYLC_WORKFLOW_RUN_DIR}/script/pre"
    [[recup]]
        platform = login
        inherit = FAMPRE, PRE
        script = recup.sh

BUT

Do you have Rose installed - this sounds like the use case for Rose: If you have Rose installed you can bundle your applications under the app/NAME/bin folder:

Quick example (This script will extract my example from heredocs to a temporary directory).

#!/bin/bash
TMP=$(mktemp -d)
echo Files extracted to ${TMP}
mkdir "${TMP}/."

mkdir "${TMP}/./app"

mkdir "${TMP}/./app/foo"

mkdir "${TMP}/./app/foo/bin"

cat > "${TMP}/./app/foo/bin/script.sh" <<__ICI__
#!/bin/bash
echo "Hello World"__ICI__

cat > "${TMP}/./app/foo/rose-app.conf" <<__ICI__
[command]
default=script.sh__ICI__

cat > "${TMP}/./flow.cylc" <<__ICI__
[scheduler]
    allow implicit tasks = True

[scheduling]
    [[graph]]
        R1 = foo

[runtime]
    [[foo]]
        script = rose task-run__ICI__

cat > "${TMP}/./rose-suite.conf" <<__ICI__
__ICI__

tree "${TMP}"
8

Thanks a lot Tim!

Well, what we are doing now, is porting an enormous mess of shell scripts to cylc.

So, for the moment I will use your first suggestion. Once everything will be ported and working, we already had the plan to introduce Rose, so we will use the final config you suggest!

Thanks again,
Stella