"hello world" workflow generates command not found error

Cylc Cylc Support
1

Hi all,

I am having trouble running a “hello world”-style workflow. It’s running in a new conda environment and it runs my hello_world job. But tui shows the job eventually getting to submit-failed state with “command not found error” in job.err. I worried I have a conda problem that is mucking up the operation.

I am running cylc-8.1.4. I set up a new conda environment to run it and installed via mamba

$> conda create --name cylc-8.1 python=3.9
$> conda activate cylc-8.1
$> conda install -c conda-forge mamba
$> mamba install -c conda-forge cylc-flow cylc-uiserver

My suite is pretty small:

#!jinja2
{% set HW_DIR = "/backup/josborne/projects/cylc-training/hello_world" %}
{% set NEWPATH = '/backup/josborne/toolbox/python/anaconda3/envs/cylc-8.1/bin:' + environ['PATH'] %}

[scheduling]
    initial cycle point = now
    final cycle point = 20311231T00
    [[graph]]
        PT2M = @wall_clock => hello_world

[runtime]
    [[root]]
        [[[environment]]]
        PATH = {{NEWPATH}}
        HW_DIR = {{HW_DIR}}
    [[hello_world]]
        script = """
        DTG=$(date +%Y%m%d%H%M%S)
        echo "hello world!" >> $HW_DIR/hello_world.log.$DTG 2>&1
        echo "check cylc"  >> $HW_DIR/hello_world.log.$DTG 2>&1
        which cylc  >> $HW_DIR/hello_world.log.$DTG 2>&1
        echo $PATH >> $HW_DIR/hello_world.log.$DTG 2>&1
        """

I’ve followed the “Managing Environments” directions at Installation — Cylc 8.1.4 documentation . I’ve tried CYLC_HOME_ROOT="/backup/josborne/toolbox/python/anaconda3/envs/cylc-8.1" and CYLC_HOME_ROOT="/backup/josborne/toolbox/python/anaconda3/envs/cylc-8.1/bin". In that bin dir, I’ve linked cylc to cylc-8.1.4:

$>  ls -l /backup/josborne/toolbox/python/anaconda3/envs/cylc-8.1/bin/cylc-8.1.4 
lrwxrwxrwx 1 josborne josborne 4 May 15 16:32 /backup/josborne/toolbox/python/anaconda3/envs/cylc-8.1/bin/cylc-8.1.4 -> cylc

I validate, install, and play the workflow. tui shows the job in submitted then submit-failed status. I check my script’s log and I get what I expect:

hello world!
check cylc
/backup/josborne/toolbox/python/anaconda3/envs/cylc-8.1/bin/cylc
/backup/josborne/toolbox/python/anaconda3/envs/cylc-8.1/bin:/backup/josborne/toolbox/python/anaconda3/envs/cylc-8.1/bin:/backup/josborne/toolbox/python/anaconda3/condabin:/common/krb5:/bin:/common/pgi/15.7/linux86-64/bin:/common/openmpi/pgi15.7/1.10.4/bin:/sbin:/usr/local/bin:/common/pkg/bin:/common/utilities/bin:/home/josborne/bin:.

But, in my job.err file, I get a command not found:

ERROR: cylc-8.1.4 not found in /backup/josborne/toolbox/python/anaconda3/envs/cylc-8.1
ERROR: cylc-8.1.4 not found in /backup/josborne/toolbox/python/anaconda3/envs/cylc-8.1

Any advice would be appreciated. Thanks in advance.

2

Hi @John_Osborne

The wrapper that comes with Cylc is designed to work with environments named with the full version string, e.g. cylc-8.1.4. At run time, the scheduler puts (e.g.) CYLC_VERSION=8.1.4 in the environment of its jobs, so that they (via the central wrapper) can use the same version to communicate job status back to their scheduler. (And similarly for users choosing their Cylc version on the fly).

It looks like you are using that wrapper but have not adapted it to handle a truncated version string like 8.1.

If you only intend to have a single Cylc environment, which you can upgrade in-place on demand, a much simpler wrapper will suffice. For example (using my environment path) this should work:

#!/usr/bin/env bash

# This script must be executable, called "cylc", and in your default $PATH
CYLC_ENV="$HOME/miniconda3/envs/cylc-8.1"
exec "${CYLC_ENV}/bin/cylc" "$@"

(Note you don’t actually have to “conda activate” the environment).

3

The wrapper will actually work with any environment named cylc-<label>. You can use that environment by setting CYLC_VERSION=<label> and then another environment variable (CYLC_ENV_NAME) is set in the job environment to ensure they use the same version.

CYLC_HOME_ROOT needs to be set to the directory containing your Cylc Conda environments:
CYLC_HOME_ROOT="/backup/josborne/toolbox/python/anaconda3/envs"
To make it the default version (selected by the wrapper if CYLC_VERSION is not set) you need to link cylc to cylc-8.1 in the envs directory.

The wrapper needs to be in your default PATH. Setting PATH = {{NEWPATH}} in your job environment won’t work and can be removed.

2 Likes
4

Thanks for the reminder @dpmatthews !

In fact (just tested) it seems you don’t need to set CYLC_VERSION=<label>. Just set CYLC_ENV_NAME=<label>. Then the job environment will get (e.g.) CYLC_ENV_NAME=8.1 and CYLC_VERSION=8.1.4 (the actual code version) but the ENV name takes precedence in the wrapper.

5

True. However, the preferred method is to use CYLC_VERSION (even though this then gets modified by the wrapper) since the wrapper will follow any symbolic links to ensure your workflow continues to use the same environment.

For example, at our site our latest environment is 8.1.4-1. We have a symlink cylc-8 -> cylc-8.1.4-1. To use the latest version of Cylc 8 we can simply set CYLC_VERSION=8. Then, the job environment will get CYLC_VERSION=8.1.4 (the version of Cylc we are using) and CYLC_ENV_NAME=8.1.4-1 (the environment we are using). This ensures that running workflows are unaffected when we update the cylc-8 symlink to point to a new version.

2 Likes
6

This worked, thank you very much! I got it to work two ways.

First, in my cylc wrapper, I updated CYLC_HOME_ROOT="/backup/josborne/toolbox/python/anaconda3/envs" and in my anaconda3/envs directory, I linked cylc-8.1.4 to to my cylc-8.1 environment,
josborne@osprey[envs]$ ls -l cylc-8.1.4 lrwxrwxrwx 1 josborne josborne 8 May 16 08:29 cylc-8.1.4 -> cylc-8.1. This worked.

Second, I set CYLC_VERSION="8.1" and renamed the linked directory in my anaconda3/envs directory (so that it wouldn’t work; I renamed it to _cylc-8.1.4). This approach also worked.

I also removed my NEWPATH variable.

Thanks so much!

1 Like