Accessing workflow source directory from within task

Cylc Cylc Support
1

Hi,

I am working on a Cylc 8 ensemble workflow where I need to initialise
per-member run directories by copying a sibling repository
(e.g. a shared “core” code base).

The setup on disk is:

<project_root>/
  ├── workflow-repo        (Cylc workflow)
  └── core-repo            (external code base to be copied per member)

The workflow is installed and executed via Cylc, so tasks run under:

$CYLC_WORKFLOW_RUN_DIR

which is a copy of the workflow, not the original repository.

In my initialisation task I need to locate the core repository relative
to the original workflow source directory (i.e. the Git checkout),
not the run directory.

Currently I am working around this by passing the path explicitly via
an environment variable:

[[[environment]]]
    CORE_REPO_DIR = /path/to/core-repo

This works, but feels a bit clunky and hard-coded.

My question:

Is there a Cylc-provided environment variable that points to the
original workflow source directory (i.e. where the workflow was
installed from), so that I could do something like:

$(dirname $SOME_CYLC_VAR)/../core-repo

I came across references to CYLC_WORKFLOW_SRC_DIR, but it does not
seem to be set in my environment.

Thanks for any suggestions!

2

Hi @BHFock

We don’t really encourage access to the workflow source directory at run time, because that’s easily abused and dangerous - the installed files record how the running instance is configured. Meanwhile, you might modify the source for the next run, or to launch other independent variations of the same workflow at the same time…

That said, if you really need to do it (your use case sounds reasonable, although it’s not obvious to me why you “need to locate the core repository relative to the original workflow source directory”) the source location is available as a Jinja2 template variable: {{CYLC_WORKFLOW_SRC_DIR}}.

3

Thanks Hilary!

Using

[[install_cold<m>]]
    inherit = INSTALL_COLD 
    script = install_cold.sh
    [[[environment]]]
        CORE_REPO_DIR = {{CYLC_WORKFLOW_SRC_DIR}}/../core-repo

allowed me to keep the following project layout:

<project_root>/
  ├── workflow-repo  
  └── core-repo

Thanks also for challenging the design — I will consider moving the core
repository inside the workflow repository, e.g.:

<project_root>/workflow-repo/lib/core-repo

In that case, the install task could be simplified to:

[[install_cold<m>]]
    inherit = INSTALL_COLD 
    script = install_cold.sh
    [[[environment]]]
        CORE_REPO_DIR = ${CYLC_WORKFLOW_RUN_DIR}/lib/core-repo

It’s great to have both options — this gives me a good basis for a design
discussion with my team on how we want to structure the project.

Cheers,
Bjoern

1 Like