Cylc 8 with LFRic Rose stem and Python issues

Cylc Cylc 8 Migration
1

I am trying to get the LFRic “make test-suite” command in lfric_atm to work.

After eliminating the arguments --name, --config, --no-gcontrol, --new, --define-suite (used by LFRic but no longer supported by rose stem) from the LFRic build configuration, the command boils down to

rose stem -n=meta_diag-lfric_atm-meto-spice-developer --source=rose-stem --opt-conf-key=meto-spice --verbosity=1 --group=developer

The produces the following error:

pkg_resources.ContextualVersionConflict?: (setuptools 39.0.1 (/net/project/extrasoftware/packages.rhel7/python/3.7.0/site/lib/python3.7/site-packages), Requirement.parse(‘setuptools>=49’), {‘cylc-flow’})

This message doesn’t mean much to me.

Here are the relevant versions my session is using:

$ python --version
Python 3.7.0 (default, Oct 9 2018, 10:31:47)
[GCC 7.3.0]
$ cylc version
8.0.0
$ rose version
rose 2.0.0

I would appreciate any advice.

Cheers,

Andreas

2

Hello,

This error is happening because you have some Python 3.7 files in your PYTHONPATH which are messing with the Cylc environment.

To fix, try running Cylc in a clean environment.

# NOTE: Met Office specific
module purge

Unfortunately there isn’t much we can do to improve this error message as the Cylc wasn’t able to load.

The LFric rose-stem suites require a few small changes to make them Cylc 8 compatible. Andrew C has a branch where he’s made some progress on this.

For general info, here are the changes made so far:

  • rose-stem options
    • --name → --workflow-name
    • --define-suite → -S (this is a Rose bug, we will reinstate the --define-suite argument soon)
    • --config <path> → <path> (now a positional argument).
    • Remove --new & --no-gcontrol.
  • cylc play
    • The rose stem command now only installs the workflow.
    • Add cylc play <workflow-name> after the rose stem command to make it run.
  • cylc gscan
    • This GUI has been replaced by the web GUI (cylc gui).
    • For a light-weight CLI alternative try cylc scan.
  • Jinja2
  • ROSE_ORIG_HOST
    • Tasks for the SPICE platform had [remote]host set to ROSE_ORIG_HOST.
    • This tells Cylc to perform an unnecessary SSH. This doesn’t work at Cylc 8.
    • Easy fix, change ROSE_ORIG_HOST → 'localhost' for SPICE sections in a Jinja2 file somewhere.

A long list but it boils down to a small diff. With that the rose-stem suites tested so far validate and run under Cylc 8.

3

Thank you for your fast and detailed response. This is very useful.

Unfortunately, the module purge doesn’t solve our problem - I suspect this is because after module purge, we are left with python 2.7.5. In any case, I agree that this is a problem for the LFRic team to solve.

4

My own comments, in addition to Oliver’s:

  • -n is the short form of both --workflow-name and --name, so it works on both versions.
  • --new doesn’t make as much sense at Cylc 8 because rose stem wraps cylc install, and Cylc install defaults to installing workflows in ~/cylc-run/<workflow>/run<x> where x increments every time you run it. The nearest equivalent would be cylc clean <workflow>/run<x>, then running rose-stem.
  • You use cylc reinstall if you want to re-install a rose stem workflow.
5

There’s probably something left in your PYTHONPATH which is breaking the stack. Rose is installed in an isolated environment with it’s own Python stack but PYTHONPATH can still interfere with it.

6

Spot on - clearing PYTHONPATH fixed the problem. Thank you very much!