Installing Cylc 7 on Mac OS "Big Sur" / Apple Silicon

Cylc Cylc Support
1 
$ git clone https://github.com/cylc/cylc-flow.git
$ cd cylc-flow
$ git checkout -b 7.8.x origin/7.8.x

# follow instructions in INSTALL.md

cylc-flow % bash
export PATH=$PWD/bin:$PATH

bash-3.2$ make
etc/dev-bin/create-version-file
(a cylc VERSION file is not needed in a cylc git repo)

$ make -C doc clean html

Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named cylc.version
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named cylc.version
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named cylc.version
/bin/sh: sphinx-build: command not found

bash-3.2$ cylc check-software

Individual results:
===============================================================
Package (version requirements)         Outcome (version found)
===============================================================
                 *REQUIRED SOFTWARE*
Python (2.6+, <3).....FOUND & min. version MET (2.7.16.final.0)

Meets core requirements.

Summary:
                  ****************************
                   Core requirements: ok
                   Full-functionality: not ok
                  ****************************

OK. so sphinx doc does not build.
move on and try a known working suites.

bash-3.2$ cylc register test test
REGISTERED test -> /Users/schaferk/cylc-suite/test

bash-3.2$ cylc validate test
Traceback (most recent call last):
  File "/Users/schaferk/cylc-flow/bin/cylc-validate", line 38, in <module>
    from cylc.config import SuiteConfig, SuiteConfigError
  File "/Users/schaferk/cylc-flow/lib/cylc/config.py", line 68, in <module>
    from cylc.xtrigger_mgr import XtriggerManager
  File "/Users/schaferk/cylc-flow/lib/cylc/xtrigger_mgr.py", line 28, in <module>
    from cylc.subprocpool import get_func
  File "/Users/schaferk/cylc-flow/lib/cylc/subprocpool.py", line 92, in <module>
    class SubProcPool(object):
  File "/Users/schaferk/cylc-flow/lib/cylc/subprocpool.py", line 120, in SubProcPool
    POLLREAD = select.POLLIN | select.POLLPRI
AttributeError: 'module' object has no attribute 'POLLIN'

who knows?

tried this with various branches, same thing

cylc-suite % python --version
Python 2.7.16

can Anyone recommend cylc for install on MacBook Air M1 Big Sur?

2

@schaferk - if you enclose multi-line code and console transcripts in triple back-quotes (```blah``` → blah), it’ll be formatted for readability on this forum. I edited your post above to do that.

The INSTALL.md file contains minimal instructions for installing Cylc release tarballs. Installing from a git clone may require additional steps that Cylc developers would be expected to know or figure out.

Either way, with Cylc 7 it is unfortunately up to you to install required software dependencies. It seems (for instance) that you don’t have Sphinx installed to build the documentation (but don’t bother with that, just use the online docs).

Cylc 8 (still in beta) has proper packaging for pip and conda and brings everything it needs with it.

AttributeError: 'module' object has no attribute 'POLLIN'

Huh. A web search shows some old-ish results that suggest select.POLLIN is not (or was not?) supported on MacOS. But I believe one of the Cylc team uses a Mac as his development platform and that evidently works … maybe @oliver.sanders can enlighten us…

Hilary

3

Hello,

Cylc 7 should be capable of running on Mac OS “Big Sur”, however, Cylc 7 never officially supported or tested on Mac OS. I’ve had Cylc 7 running on Mac OS up to Mavericks but haven’t tried on the latest and greatest. It should be possible but may take a bit of fiddling.

Cylc 8 will be Mac OS compatible, currently there is no Mac OS Conda-Forge build so Cylc 8 must be installed from PyPi or source with a small patch to the source code to handle Mac OS DNS issues.

Cylc 7 Installation On Mac OS / Apple Silicon

Apple Silicon is a brand new architecture and the community is still racing to support it. Legacy packages / versions are unlikely to be rebuilt for this new architecture so you might run into difficulties installing some of the more involved components, namely the GUIs.

However, I think it should be possible. If you want to go ahead and try to get Cylc 7 running on Big Sur, here’s some info that might help. Let us know how you get on!

AttributeError: ‘module’ object has no attribute ‘POLLIN’

Is this using the Python 2 version that is included with Big Sur?

Historically Apple’s Python builds have come with broken interfaces in the select module. Unfortunately Homebrew have now dropped Python 2 support, however, you may still be able to install Python 2 via Conda.

Otherwise, I think this patch to lib/cylc/subprocpool.py should bypass the polling functionality (not strictly necessary) allowing this part of the code to work with Apple compiled Python:

     ERR_WORKFLOW_STOPPING = 'workflow stopping, command not run'
     JOBS_SUBMIT = 'jobs-submit'
-    POLLREAD = select.POLLIN | select.POLLPRI
+    POLLREAD = None
     RET_CODE_WORKFLOW_STOPPING = 999

     def __init__(self):
         self.size = glbl_cfg().get(['scheduler', 'process pool size'])
         self.proc_pool_timeout = glbl_cfg().get(
             ['scheduler', 'process pool timeout'])
         self.closed = False  # Close queue
         self.stopping = False  # No more job submit if True
         # .stopping may be set by an API command in a different thread
         self.stopping_lock = RLock()
         self.queuings = deque()
         self.runnings = []
+        self.pipepoller = None
-        try:
-            self.pipepoller = select.poll()
-        except AttributeError:  # select.poll not implemented for this OS
-            self.pipepoller = None

Cylc Dependencies

Cylc has some dependencies that you will need to install, see cylc check-software and this page.

You don’t need to build your own copy of the documentation (make -C doc clean html) so don’t need to install the documentation dependencies (e.g. Sphinx).

I would suggest installing missing system dependencies via Homebrew e.g:

$ brew install graphviz

Followed by installing the Python dependencies via Pip e.g:

$ pip install pygraphviz

Mac OS Dependencies

On top of the regular Cylc dependencies you will need a few extra ones to make things run on Mac OS, I would suggest installing these via Homebrew:

$ brew install bash coreutils gnu-sed

Note: From Catalina onwards the default shell is now ZSH not Bash. The Bash version Apple provides is very old (Bash 3) so it’s a good idea to update to Bash 4+. You don’t need to change your default shell to Bash.

Note: You will need to prepend the coreutils and gnu-sed installations to your $PATH , follow the instructions in the brew install output to prepend these tools to your $PATH for the Bash shell.

GUI

The GUI will likely be the trickiest part to install Homebrew no long supports GTK2.

If you don’t have GTK2 preinstalled you may need to install from Conda Forge or compile yourself.

Note GTK2 must be installed before attempting to pip install pygtk.

The GUI (and PyGTK) is not required for general functionality, only the GUIs (cylc gui, cylc graph, cylc gscan). The cylc review web GUI can be used as a standin for monitoring purposes.

DNS Patch

From Mac OS Catalina onwards the socket.getfqdn interface is broken for the default DNS configuration. If on a personal machine applying this patch to lib/python/hostuserutil.py should do the trick.

SSH

From Mac OS Catalina onwards SSH has been disabled by default for security reasons.

You may need to enable it from system preferences for full Cylc functionality.

Atrun

Atrun (the at command) does not run out-of-the-box on Mac OS for security reasons and must be manually enabled.

4

Just to chuck in one last suggestion, if you don’t need Cylc to SSH around a network of hosts and the above looks like too much hassle you might want to consider installing Cylc via Docker Engine.

Docker works nicely on Mac OS, it runs a Linux kernel inside a VM that is surprisingly efficient.

5

downloaded Docker Desktop to initiate cylc via Docker.

6

update:
Docker installed on BigSur M1.
containers kinow/cylc-flow (version 8.0a1) and kinow/cylc-uiserver pulled and running verified with docker ps.
some custom suites installed in cylc-flow container and suites running.
how is the gui used?
‘’’’
[I 2021-07-04T12:12:23.159 JupyterHub app:2422] JupyterHub is now running at http://:8000
‘’’

now what?

7

Those Docker images were used to experiment running Cylc with Docker. alanbchristie (GH) did a much better work [1] that includes instructions for running the GUI, though the version packaged is a bit older.

We were working to get his image back under the cylc community, but the work was paused to focus on Cylc 8.

But maybe what Oliver suggested was for you to use an existing image like alpine, ubuntu, debian, etc., and install Cylc there as you would in a local Linux system?

Cheers
Bruno

[1] GitHub - alanbchristie/cylc-docker: Docker files

8

Yes, I think @oliver.sanders was suggesting the latter method. We haven’t got any up-to-date pre-configured Cylc Docker images, but it’s easy enough to do that yourself - just treat Cylc like anything else you want installed into the image.

9

I think this should do the trick:

FROM ubuntu:latest

RUN apt-get update && \
    apt-get install -y python3.9 pip nodejs npm && \
    npm install -g configurable-http-proxy && \
    pip install cylc-flow cylc-uiserver

For a full list of dependencies see the Conda feedstock files for cylc-flow and the cylc-uiserver:

10

i was able to get both
kinow/cylc-flow
kinow/cylc-uiserver

working to where i could log in the the web interface. as alluded to, these are somewhat a work in progress. The exercise was useful though.

11

i/ll work to build via a Dockerfile as suggested.