Cylc 8 with LFRic Rose stem and Python issues

I am trying to get the LFRic “make test-suite” command in lfric_atm to work.

After eliminating the arguments --name, --config, --no-gcontrol, --new, --define-suite (used by LFRic but no longer supported by rose stem) from the LFRic build configuration, the command boils down to

rose stem -n=meta_diag-lfric_atm-meto-spice-developer --source=rose-stem --opt-conf-key=meto-spice --verbosity=1 --group=developer

The produces the following error:

pkg_resources.ContextualVersionConflict?: (setuptools 39.0.1 (/net/project/extrasoftware/packages.rhel7/python/3.7.0/site/lib/python3.7/site-packages), Requirement.parse(‘setuptools>=49’), {‘cylc-flow’})

This message doesn’t mean much to me.

Here are the relevant versions my session is using:

$ python --version
Python 3.7.0 (default, Oct 9 2018, 10:31:47)
[GCC 7.3.0]
$ cylc version
$ rose version
rose 2.0.0

I would appreciate any advice.




This error is happening because you have some Python 3.7 files in your PYTHONPATH which are messing with the Cylc environment.

To fix, try running Cylc in a clean environment.

# NOTE: Met Office specific
module purge

Unfortunately there isn’t much we can do to improve this error message as the Cylc wasn’t able to load.

The LFric rose-stem suites require a few small changes to make them Cylc 8 compatible. Andrew C has a branch where he’s made some progress on this.

For general info, here are the changes made so far:

  • rose-stem options
    • --name--workflow-name
    • --define-suite-S (this is a Rose bug, we will reinstate the --define-suite argument soon)
    • --config <path><path> (now a positional argument).
    • Remove --new & --no-gcontrol.
  • cylc play
    • The rose stem command now only installs the workflow.
    • Add cylc play <workflow-name> after the rose stem command to make it run.
  • cylc gscan
    • This GUI has been replaced by the web GUI (cylc gui).
    • For a light-weight CLI alternative try cylc scan.
  • Jinja2
    • Tasks for the SPICE platform had [remote]host set to ROSE_ORIG_HOST.
    • This tells Cylc to perform an unnecessary SSH. This doesn’t work at Cylc 8.
    • Easy fix, change ROSE_ORIG_HOST'localhost' for SPICE sections in a Jinja2 file somewhere.

A long list but it boils down to a small diff. With that the rose-stem suites tested so far validate and run under Cylc 8.

Thank you for your fast and detailed response. This is very useful.

Unfortunately, the module purge doesn’t solve our problem - I suspect this is because after module purge, we are left with python 2.7.5. In any case, I agree that this is a problem for the LFRic team to solve.

My own comments, in addition to Oliver’s:

  • -n is the short form of both --workflow-name and --name, so it works on both versions.
  • --new doesn’t make as much sense at Cylc 8 because rose stem wraps cylc install, and Cylc install defaults to installing workflows in ~/cylc-run/<workflow>/run<x> where x increments every time you run it. The nearest equivalent would be cylc clean <workflow>/run<x>, then running rose-stem.
  • You use cylc reinstall if you want to re-install a rose stem workflow.

There’s probably something left in your PYTHONPATH which is breaking the stack. Rose is installed in an isolated environment with it’s own Python stack but PYTHONPATH can still interfere with it.

Spot on - clearing PYTHONPATH fixed the problem. Thank you very much!