Exception: No available ports

‘’‘’
debugging a cylc7 container yesterday

today when running the suite we get:

Exception: No available ports
after many

Exception in thread HTTPServer Thread-X:
File “/opt/cylc/lib/cherrypy/wsgiserver/wsgiserver2.py”, line 1954, in start
raise socket.error(msg)
error: No socket could be created – ((‘MY_HOST_FQDN’, $PORT): [Errno 99] Cannot assign requested address)

I suspect sys admin work as the system was not available for login for a short period. in general I am not privy to the changes, only the effects.

if someone could provide a basic overview (and conjecture) as to what is going on, I don/t mind patching the source. I/m doing that now to enable debugging (import pub) and logging (import logging; import logging.basicConfig(level=logging.DEBUG)

sometimes I feel like the cat and sometimes I feel like the mouse.
‘’’

Each Cylc 7 workflow uses a single port for communication (Cylc 8 uses 2). By default the port range used is 43001 … 43100 so you can have up to 100 workflows running on a single server. The range is configurable: https://cylc.github.io/cylc-doc/7.9.3/html/appendices/site-user-config-ref.html#suite-servers-run-ports

If no ports are available that implies there are processes running which have all the ports in the range open. There are various tools you can use to check which ports are in use.
For example running ss -lp | grep ":43" shows all the allocated ports within the default cylc port range and if you run it as root you can see all the associated process id’s.

If you’re running Cylc in a container, you will likely have to open these ports to any systems or containers you want to be able to access them from, e.g. using the -P option to Docker.

Note, Cylc chooses ports from the configured range at random and will only fail after trying all ports within the range.

using the above, I see that there is now one IP address and one port
on the machine in question.

on another machine, there are more than 30 ports.

this is useful, thank you dp.

useful to know along with the ‘Note’ about the ‘randomness’ of the choice. thank you, os.