Recommended approach to use shared plugins?

Cylc Cylc Support
1

Hi,

Is there a recommended approach to make use of shared plugins, assuming those shared plugins are available to install via conda. Perhaps that should be, how can we effectively use conda to distribute shared xtriggers and plugins (at least internally). For example, shared

For xtriggers, they seem to need to be in your PYTHONPATH or workflow/lib/python folder. If we do conda create -n my_workflow cylc-local-xtriggers then, that may require Cylc, so it will install Cylc, etc, so it seems inefficient. If we are working across multiple projects, I would prefer they just be installed in my shared cylc create -n cylc-8.2.0 cylc-flow==8.2.0 environment and Cylc will just find them. But that environment doesn’t need to be active at install time, so unless we activate it, we can’t make use of activate/deactivate.sh scripts to add to the PYTHONPATH (which is suboptimal as described in the provided bin/cylc file). Could the local cylc-local-xtriggers package be installed under cylc-flow itself or would that not work (I have not checked if multiple conda packages can put things in the same lib/cylc-flow area). Or could Cylc add some items to the PYTHONPATH when it runs itself (change the last line to: exec PYTHONPATH=${PYTHONPATH:+$PYTHONPATH:}$CYLC_HOME/share/cylc/lib/python "${CYLC_HOME}/bin/{$0##*/}" "$@" and it be documented that cylc will look in that spot too?

Similarly for Jinja2 globals/filters/tests, could there be an extension to how they are found to encorprate shared items installed in the conda environment, looking in $CYLC_HOME/share/cylc/Jinja2{Globals,Filters,Tests}?

Or have I over-thought this and got myself lost and confused?

Thanks.

2

Hi,

There’s nothing special about Cylc plugins, they should be managed, distributed and installed like any other Python project.

Cylc plugins must be installed into the Cylc environment (i.e. the environment which you installed cylc-flow into), they will be automatically loaded at the appropriate time if installed.

There are multiple types of Cylc plugin, the older plugins need to be installed under PYTHONPATH e.g:

  • Xtriggers.
  • Job runners.

The newer plugins must also be registered via entry-points e.g:

  • Main loop plugins.
  • Installation plugins.

In both cases, the best way to share and distribute a plugin is to turn it into a Python package which can be installed via pip or Conda. If the plugin is suitable for general consumption it can be hosted globally on PyPi and or conda-forge. If the plugin is closed-source or site-specific it can be hosted on an internal repository or just on the filesystem.

For an example, see the cylc-rose plugin which provides support for the rose-suite.conf file. Sites which wish to use cylc-rose should install it into their Cylc environment. E.G. you might use this Conda environment file to install Cylc with Rose support:

name: cylc-8.2
channels:
  - conda-forge
dependencies:
  - cylc-flow =8.2
  - cylc-rose
  - metomi-rose

Once cylc-rose is installed in the Cylc environment, it will be automatically loaded when Cylc is run.
There is no need for further configuration or activate scripts.

But that environment doesn’t need to be active at install time, so unless we activate it, we can’t make use of activate/deactivate.sh scripts to add to the PYTHONPATH

Activate/deactivate.sh scripts are not needed, any Python package which is installed with pip/conda is automatically present in the PYTHONPATH when Python is run.

Note: lib/python<version>/site-packages is automatically added to PYTHONPATH when python is run so the full path Python considers for finding packages is not necessarily what you see in the PYTHONPATH environment variable, use import sys; sys.path to inspect the full Python path at runtime.

Note: Conda activate/deactivate scripts are discouraged.

3

Thanks. I had read or anywhere in your Python library path as PYTHONPATH, so that is my bad interpretation.

Do you have a response about shared Jinja2{Globals,Filters,Tests} though? I don’t believe that is covered in your response. From the docs it states they need to be inside the suite itself. I wondered if this was true or there was a better way of sharing them via pip/conda?

Cylc also supports custom Jinja2 globals, filters and tests. A custom global, filter or test is a single Python function in a source file with the same name as the function (plus .py extension). These must be located in a subdirectory of the run directory called Jinja2Filters, Jinja2Globals or Jinja2Tests respectively.

4

I don’t have a particularly good response, there isn’t a plugin interface for Jinnja2 globals/filters/tests so at present they can only be loaded from hardcoded locations. From an inspection of the code this includes ~/.cylc which allows them to be shared between workflows under one user account, but at the expense of portability. I’ll put up an issue to switch to an entry point interface for our built-in Jinja2 extensions which would open up the potential for libraries of common Jinja2 filters.

Until then, one option for sharing Jinja2 functionality is to import Python libraries directly e.g you can import the Cylc log like so:

{% from "cylc.flow" import LOG %}

This allows you to distribute common functionality in Python modules but doesn’t give you the syntactic sugar of Jinja2 globals/filters/tests. Personally, I prefer to push logic out of Jinja2 (which isn’t really a programming language) into Python (which is) so don’t make much use of globals/filters/tests.

5

Thanks. We only have two shared Jinja2Globals, and I don’t think many use either of them. For now, maybe it can be advised for people when starting up a workflow to include something to copy them in if required.

6

Hi @oliver.sanders. I wanted to come back to this topic briefly. If xtriggers need to be added to the PYTHONPATH, I’m not sure how doing a proper package/install into site-packages works. Say a package was called foo and the xtrigger file was called bar.py and its function inside was called bar too. This would be installed into lib/python3.11/site-packages/foo/bar.py. The package foo would be in the PYTHONPATH, but the xtrigger bar is not. Am I missing something obvious? It’s feeling like xtriggers would need to be installed into somewhere and (I know not recommended, but…) an activate script used to add the install location to the PYTHONPATH. Or do you mean to install them into cylc/flow/xtriggers itself (I assume not)?

Thanks for the help on this one.

7

The docs for custom xtriggers mention:

they must be defined in a module with the same name as the function

So if you have installed site-packages/bar/__init__.py into the site-packages directory for the environment that Cylc uses, then you should be able to use the function bar in your xtrigger

Note that bar/ has not been added to PYTHONPATH, it is just included in sys.path in Python.

8

I worried that would be the approach recommened. I didn’t like that approach because it makes many small packages in site-config, and many have generic names so could conflict with other things. I discussed with @ScottWales and he suggested a new entry point for xtriggers. I have implemented that idea in Add a new entry_point for xtriggers by ColemanTom · Pull Request #5831 · cylc/cylc-flow · GitHub - hopefully it is an approach that is agreeable as I think it leads to a cleaner and better result.