Cylc8 installation errors: I corrected importlib_metadata but the colorama problem stumps me!

Cylc Cylc Support
1

Dear Cylc Discourse Group,

Apologies for what may be an obvious question: I am not sure whether this problem is to do with Cylc in particular or my version of Python more generally (obviously, if the latter, I understand that I should post this question elsewhere).

I am trying to make the transition Cylc7 to Cylc8 at the same time as jumping Python v2.75 → Python v3.11.5. I have stripped out my Cylc7+Rose1+FCM installation and am installing everything from fresh.

I tried to install Cylc v8.4.2 using pip. Apparently, it did install (see SESSION1 below).

I then found a slight issue with importlib_metadata, but by modifying cylc.py a little bit I could get around that (from the code comment there, it’s a recognised problem but it seems the change was made slightly before Python 3.12 not at Python 3.12 as the code assumes):

If you get error “ModuleNotFoundError: No module named ‘importlib_metadata’” then do:
(1) Either “pip install importlib.metadata” (for Python >3.8) or “pip install importlib_metadata” (otherwise)
and then (2) If Python >3.8, change line 49 of ~/.local/lib/python3.11/site-packages/cylc/flow/scripts/cylc.py, change “if sys.version_info[:2] > (3, 11):” to “if sys.version_info[:2] > (3, 10):”

Flushed with success after having got around that one, I got stuck again when I tried to do “cylc --version” and “cylc check-software”. Strangely, these didn’t give me a simple “Command not found”, but gave me an error about a missing Python package Colorama. Therefore, I tried doing pip install colorama (see SESSION2 below) but it didn’t stop the error message.

For me, I think this is all a bit weird (!): the colorama bit would seem to be an error about Python, but all is occurring when I do a seemingly very standard installation of Cylc.

Can anyone give me any suggestions how to move forward? My suspicion is that there is an issue with colorama for this version of Python, and an IF like the one already in cylc.py to sort out importlib_metadata needs to be inserted there, but I don’t know how to sort that out.

Very many thanks anyone for any comments!

Toby
Oxford, UK.

==================SESSION1:
[tobmar@wlsc-polarln01 ~]$ python --version
Python 3.11.5
[tobmar@wlsc-polarln01 ~]$ pip install cylc-flow
Defaulting to user installation because normal site-packages is not writeable
Collecting cylc-flow
Using cached cylc_flow-8.4.2-py3-none-any.whl.metadata (11 kB)
Requirement already satisfied: ansimarkup>=1.0.0 in /home/tobmar/.local/lib/python3.11/site-packages (from cylc-flow) (2.1.0)
Requirement already satisfied: async-timeout>=3.0.0 in /home/tobmar/.local/lib/python3.11/site-packages (from cylc-flow) (5.0.1)
Requirement already satisfied: colorama<1,>=0.4 in /gpfs01/apps/mk2/Python-bundle-PyPI/2023.10-GCCcore-13.2.0/lib/python3.11/site-packages (from cylc-flow) (0.4.6)
Requirement already satisfied: graphene<3,>=2.1 in /home/tobmar/.local/lib/python3.11/site-packages (from cylc-flow) (2.1.9)
Collecting jinja2==3.0.* (from cylc-flow)
Using cached Jinja2-3.0.3-py3-none-any.whl.metadata (3.5 kB)
Requirement already satisfied: metomi-isodatetime<1!3.2.0,>=1!3.0.0 in /home/tobmar/.local/lib/python3.11/site-packages (from cylc-flow) (1!3.1.0)
Requirement already satisfied: packaging in /gpfs01/apps/mk2/Python-bundle-PyPI/2023.10-GCCcore-13.2.0/lib/python3.11/site-packages (from cylc-flow) (23.2)
Requirement already satisfied: protobuf<4.25.0,>=4.24.4 in /home/tobmar/.local/lib/python3.11/site-packages (from cylc-flow) (4.24.4)
Requirement already satisfied: psutil>=5.6.0 in /gpfs01/apps/mk2/Python-bundle-PyPI/2023.10-GCCcore-13.2.0/lib/python3.11/site-packages (from cylc-flow) (5.9.6)
Requirement already satisfied: pyzmq>=22 in /home/tobmar/.local/lib/python3.11/site-packages (from cylc-flow) (26.4.0)
Requirement already satisfied: importlib_metadata>=5.0 in /gpfs01/apps/mk2/Python-bundle-PyPI/2023.10-GCCcore-13.2.0/lib/python3.11/site-packages (from cylc-flow) (6.8.0)
Requirement already satisfied: urwid!=2.6.2,!=2.6.3,==2.* in /home/tobmar/.local/lib/python3.11/site-packages (from cylc-flow) (2.6.16)
Requirement already satisfied: rx in /home/tobmar/.local/lib/python3.11/site-packages (from cylc-flow) (1.6.3)
Requirement already satisfied: promise in /home/tobmar/.local/lib/python3.11/site-packages (from cylc-flow) (2.3)
Requirement already satisfied: MarkupSafe>=2.0 in /gpfs01/apps/mk2/Python-bundle-PyPI/2023.10-GCCcore-13.2.0/lib/python3.11/site-packages (from jinja2==3.0.->cylc-flow) (2.1.3)
Requirement already satisfied: typing-extensions in /home/tobmar/.local/lib/python3.11/site-packages (from urwid!=2.6.2,!=2.6.3,==2.->cylc-flow) (4.13.2)
Requirement already satisfied: wcwidth in /gpfs01/apps/mk2/Python-bundle-PyPI/2023.10-GCCcore-13.2.0/lib/python3.11/site-packages (from urwid!=2.6.2,!=2.6.3,==2.*->cylc-flow) (0.2.8)
Requirement already satisfied: six<2,>=1.10.0 in /gpfs01/apps/mk2/Python-bundle-PyPI/2023.10-GCCcore-13.2.0/lib/python3.11/site-packages (from graphene<3,>=2.1->cylc-flow) (1.16.0)
Requirement already satisfied: graphql-core<3,>=2.1 in /home/tobmar/.local/lib/python3.11/site-packages (from graphene<3,>=2.1->cylc-flow) (2.3.2)
Requirement already satisfied: graphql-relay<3,>=2 in /home/tobmar/.local/lib/python3.11/site-packages (from graphene<3,>=2.1->cylc-flow) (2.0.1)
Requirement already satisfied: aniso8601<=7,>=3 in /home/tobmar/.local/lib/python3.11/site-packages (from graphene<3,>=2.1->cylc-flow) (7.0.0)
Requirement already satisfied: zipp>=0.5 in /gpfs01/apps/mk2/Python-bundle-PyPI/2023.10-GCCcore-13.2.0/lib/python3.11/site-packages (from importlib_metadata>=5.0->cylc-flow) (3.17.0)
Using cached cylc_flow-8.4.2-py3-none-any.whl (1.0 MB)
Using cached Jinja2-3.0.3-py3-none-any.whl (133 kB)
Installing collected packages: jinja2, cylc-flow
Successfully installed cylc-flow-8.4.2 jinja2-3.0.3
[tobmar@wlsc-polarln01 ~]$

===================SESSION2:
#Initially I got:
[tobmar@wlsc-polarln01 ~]$ cylc --version
Traceback (most recent call last):
File “/home/tobmar/.local/bin/cylc”, line 5, in
from cylc.flow.scripts.cylc import main
File “/home/tobmar/.local/lib/python3.11/site-packages/cylc/flow/scripts/cylc.py”, line 61, in
from importlib_metadata import (
ModuleNotFoundError: No module named ‘importlib_metadata’
[tobmar@wlsc-polarln01 ~]$

#…but I did the correction in the text above and got past that one. However, then the Colorama problem arose:
[tobmar@wlsc-polarln01 ~]$ cylc --version
Traceback (most recent call last):
File “/home/tobmar/.local/bin/cylc”, line 5, in
from cylc.flow.scripts.cylc import main
File “/home/tobmar/.local/lib/python3.11/site-packages/cylc/flow/scripts/cylc.py”, line 70, in
from ansimarkup import parse as cparse
File “/home/tobmar/.local/lib/python3.11/site-packages/ansimarkup/init.py”, line 1, in
from .markup import AnsiMarkup, AnsiMarkupError, MismatchedTag, UnbalancedTag
File “/home/tobmar/.local/lib/python3.11/site-packages/ansimarkup/markup.py”, line 5, in
from colorama import Style
ModuleNotFoundError: No module named ‘colorama’
[tobmar@wlsc-polarln01 ~]$
[tobmar@wlsc-polarln01 ~]$
[tobmar@wlsc-polarln01 ~]$ pip install colorama
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: colorama in /gpfs01/apps/mk2/Python-bundle-PyPI/2023.10-GCCcore-13.2.0/lib/python3.11/site-packages (0.4.6)
[tobmar@wlsc-polarln01 ~]$ cylc --version
Traceback (most recent call last):
File “/home/tobmar/.local/bin/cylc”, line 5, in
from cylc.flow.scripts.cylc import main
File “/home/tobmar/.local/lib/python3.11/site-packages/cylc/flow/scripts/cylc.py”, line 70, in
from ansimarkup import parse as cparse
File “/home/tobmar/.local/lib/python3.11/site-packages/ansimarkup/init.py”, line 1, in
from .markup import AnsiMarkup, AnsiMarkupError, MismatchedTag, UnbalancedTag
File “/home/tobmar/.local/lib/python3.11/site-packages/ansimarkup/markup.py”, line 5, in
from colorama import Style
ModuleNotFoundError: No module named ‘colorama’
[tobmar@wlsc-polarln01 ~]$

2

Hi @soi521

Cylc 8 is very easy to install (far easier than Cylc 7!) BUT you need to install it into a virtual environment.

Otherwise pip tries to install it into your system environment, likely via ~/.local which extends the system environment. For a large application like Cylc, achieving version compatibility of all dependencies with everything else in your system environment is a big ask.

With virtual environments, Cylc will install out of the box, plus you’ll be able to install new Cylc versions whenever you like without having to upgrade existing workflows all at once.

Please read the installation docs.

For native Python virtual environments, you need to create the venv first with Python 3 then pip install Cylc into it.

We recommend Conda virtual environments because Conda can install Python itself, and non-Python components for the Cylc 8 web UI.

1 Like
3

Look like pip is trying to install into ~/.local as Hillary suggested above as indicated by this error:

Defaulting to user installation because normal site-packages is not writeable

This tells us that you do not have the permissions to install packages into the standard Python environment you’re using (located in /gpfs01/apps/mk2), so pip is instead installing locally (into /home/tobmar/.local). This might cause issues for distributed installations (e.g. if /home is not networked like /gpfs01 is) resulting in missing modules (i.e. it’s a pip usage issue, not a bug in the cylc-flow package).

Note, this local installation will get activated for ALL Python environments (using the same Python version), not just the system one which often leads to environment corruption (e.g. if cylc-flow dragged in a newer version of a dependency, but something in the base Python stack required an older version of the same dependency). Strongly recommend deleting your /home/tobmar/.local/lib/Python* directories to avoid further issues.

Note, you can use Cylc 7 & 8 in parallel, see the recommended approach of Conda environments and a wrapper script - Installation — Cylc 8.4.2 documentation.

4

Thank you very much for this response: I have managed to implement Cylc in a venv and therefore get around this problem. The details you provided were essential help! Best wishes. Toby.

1 Like