Rose PYTHONPATH

Rose Rose Announce
1

As of version 2.2.0, Rose no longer respects the PYTHONPATH environment variable.

Quick Summary

  • Rose now ignores PYTHONPATH to avoid issues where the Rose software stack conflicts with the environment you are activating.

  • If you want to control the Rose PYTHONPATH, use the new ROSE_PYTHONPATH environment variable.

  • If something which previously worked now fails with import errors, this will fix the issue:

    export ROSE_PYTHONPATH="$PYTHONPATH"

  • Note, this dos not affect commands run by Rose e.g:

    [command]
default=python my_script.py  # this will continue to see PYTHONPATH as normal

  • Note, Conda/Mamba environment activation does not set PYTHONPATH so is not affected.

Explanation

The reason for this is that PYTHONPATH is often used to load environments required by the scripts that we configure Rose to run.

For example, this Cylc configuration adds some items to PYTHONPATH, then runs a Rose app:

# flow.cylc
[runtime]
    [[foo]]
        pre-script = module load my_environment  # add to PYTHONPATH
        script = rose task-run

This Rose app then runs in this environment:

# app/foo/rose-app.conf
[command]
default=python my_script.py  # this script needs those things in PYTHONPATH

This all works fine, however, Rose itself is a Python application. As a result changing PYTHONPATH will also change the environment that Rose runs in (not just the things you want Rose to run). This has lead to issues where the software stack being loaded conflicts with the stack that Rose is installed into.

To avoid these issues, we have isolated the stack that Rose runs in by getting it to ignore the PYTHONPATH environment variable.

There are some valid reasons why you might want to set the PYTHONPATH for Rose, e.g.:

  • Installing rose_ana analysis modules (or their dependencies).
  • Installing dependencies of custom Rose macros.

For these situations, you should now use ROSE_PYTHONPATH instead of PYTHONPATH e.g:

export ROSE_PYTHONPATH="/directory/to/add/to/pythonpath"

Or:

module load my_environment  # adds to PYTHONPATH
export ROSE_PYTHONPATH="$PYTHONPATH"

See also the release announcement.

2 Likes