I have a Python project that is dependent on a Conda environment and is itself normally built and installed into such environment before use. The work flow of the project is currently provided by a Cylc suite. So, imagine the root level of the project containing an environment.yml file, a setup.py file and a suite.rc file.
An initial task of the suite creates a Conda environment under the suite’s share directory and installs the Python project in it. Subsequent tasks would activate the Conda environment to gain access to the project’s console scripts.
The advantage of this set up is such that the Conda environment is self-contained within the suite run time, and so has no reliance on specific Conda environments outside the suite. As long as a user has Miniconda (or equivalent) installed, then this suite is portable and does not pollute their environments or generates any namespace clashes.
This Gist contains a copy of the script run by the initial task:
(Sorry, actual suite is under a private GitHub repository, so cannot link to it.)
My understanding is that we are still going to use Conda to manage the complex Cylc 8 environment? If so, can anyone think of any issue with the Conda environment set up described above?
1 Like
Hi Matt,
My understanding is that we are still going to use Conda to manage the complex Cylc 8 environment? If so, can anyone think of any issue with the Conda environment set up described above?
Yes, that’s still the case.
Your approach sounds to me like the best way to manage task jobs that are (or can exist in) conda packages (clean, and self-contained). However, I have not actually tried to do that yet, with Cylc 8. Maybe others have.
Conda docs suggest that nested environment activation should work fine:
Managing environments — conda 23.10.1.dev61 documentation
Hilary
Nested activation has worked for me though I don’t know enough about Conda to speculate.
You could always call out to a script which does a quick Conda deactivate for safety before initialising its own environment.
1 Like
The nested environment appears to work correctly in a simple set up:
- Cylc 8.0a1.
- Suite running on an AWS EC2 instance with Amazon Linux 2.
- Local background jobs only.
I would conda activate the environment with the installation of cylc-flow before starting up the suite. The suite will build and install the Conda environment for my project, then conda activate --stack ... the environment on demand. The important thing to note is that some scripts in the project environment would trigger cylc message to report a warning back to the suite - and that seems to have worked correctly with both Conda environments stack-activated. 
2 Likes