Long-time Cylc user here, previously at the Met Office. I’m now at a new organisation and working on my first proper Cylc installation (I’ve only ever used it as an end user, or run it on my personal laptop where everything worked out of the box).
I’m running into an issue where the TUI shows ‘loading…’ indefinitely and the GUI only shows workflows after they have completed. I’ve done quite a bit of digging and would appreciate fresh eyes.
Environment
Cylc 8.6.3, installed via conda
Single-server setup (all components on one machine)
Linux
Symptoms
Workflows run correctly and complete successfully
TUI shows the workflow as ‘running’ but displays ‘loading…’ instead of live task status
The uiserver is running (files exist under ~/.cylc/uiserver/) but it does not seem to be able to connect to the scheduler despite the port being open. The False in register_workflow suggests it finds the workflow but cannot establish a live connection.
Is there a known issue with uiserver ↔ scheduler authentication in 8.6.3 on a fresh single-server conda install? Could there be missing uiserver-side certificates that are not generated automatically? Is there any additional configuration which I need to add somewhere?
Happy to provide any additional logs or config output. Thanks in advance! Björn
I haven’t heard of these symptoms before, single-server setups are usually straight-forward. However, this is what I would expect if network communication was broken for some reason.
Is there a known issue with uiserver ↔ scheduler authentication in 8.6.3 on a fresh single-server conda install?
No.
Could there be missing uiserver-side certificates that are not generated automatically?
No (HTTPS certs are required for secure Cylc Hub use, but that doesn’t affect workflows).
Is there any additional configuration which I need to add somewhere?
Shouldn’t be.
It might be worth running a command against the workflow using the --debug option, e.g:
If everything is working as expected, we should see:
DEBUG - zmq:send and DEBUG - zmq:recv messages in the output of the cylc ping command.
DEBUG - ALLOWED (CURVE) and DEBUG - ZAP reply messages in the workflow log.
The scheduler port is open and reachable (confirmed with nc)
Note that each Cylc scheduler uses two ports, these ports are selected at random from the range configured in the global.cylc file:
It might be the case that some ports within the range are ok, but others are blocked.
~/.cylc/flow/global.cylc` is configured with:
If the hostname is in /etc/hosts and the hostname command is returning the expected value, then you shouldn’t need to configure [[host self-identification]].
There’s a chance (albeit a very small one) that this could be causing issues if localhost isn’t in /etc/hosts or is routing in a strange way for some reason.
If the above doesn’t help, here’s a simple test which might help to identify any network communication problems.
These two Python files implement a “hello-world” TCP networking example using ZMQ:
server.py
import zmq
import time
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")
while True:
# Wait for next request from client
message = socket.recv()
print(f"Received request: {message}")
# Do some 'work'
time.sleep(1)
# Send reply back to client
socket.send_string("World")
client.py
import zmq
context = zmq.Context()
# Socket to talk to server
print("Connecting to hello world server...")
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")
# Do 10 requests, waiting each time for a response
for request in range(10):
print(f"Sending request {request} ...")
socket.send_string("Hello")
# Get the reply.
message = socket.recv()
print(f"Received reply {request} [ {message} ]")
Run the test using two terminals, here’s a sample of the expected output:
terminal 1
$ python server.py
Received request: b'Hello'
Received request: b'Hello'
Received request: b'Hello'
terminal 2
$ python client.py
Connecting to hello world server...
Sending request 0 ...
Received reply 0 [ b'World' ]
Sending request 1 ...
Received reply 1 [ b'World' ]
Sending request 2 ...
Received reply 2 [ b'World' ]
Sending request 3 ...
Try changing the hostname and port in client.py to the values Cylc is using (e.g, look at the top of the log of a stopped workflow).
My example workflow is here: ctn05_clock-triggers/flow.cylc. It worked fine when I originally put it together as training material on a personal laptop, so hopefully it should run here as well.
From your suggestion, I understand that I should be able to remove my ~/.cylc/flow/global.cylc file entirely, and then check with our IT department whether the ports 43001–43101 are accessible.
When I run the server.py and client.py test programs, the output suggests that some security policy or a problem with the local ZeroMQ installation might be interfering:
./server.py
import-im6.q16: attempt to perform an operation not allowed by the security policy `PS' @ error/constitute.c/IsCoderAuthorized/426.
import-im6.q16: attempt to perform an operation not allowed by the security policy `PS' @ error/constitute.c/IsCoderAuthorized/426.
./server.py: line 4: syntax error near unexpected token `('
./server.py: line 4: `context = zmq.Context()'
Are there any additional things I should ask my IT department to check, or anything else in my installation that might cause this?
or anything else in my installation that might cause this?
If that hello-world example doesn’t work, it’s definitely a network issue. Once that example works, Cylc should be ok.
./server.py: line 4: syntax error near unexpected token ('`
Make sure you’re calling that script with Python (i.e, python server.py), you can activate the Cylc environment to pick up the installation of ZMQ it includes.
import-im6.q16: attempt to perform an operation not allowed by the security policy `PS’ @ error/constitute.c/IsCoderAuthorized/426.
I wouldn’t be surprised if security policy is the cause here. Firewall/blocked ports are a common issue, if you’re using Security Enhanced Linux (SELinux) that might also be a factor.
Good catch. Calling the scripts as “python server.py” and “python client.py” prints the hello world messages in the two terminals. So that part is working at least.
This is unclear to me. I tested all ports 43001–43101 with the client.py and server.py test scripts. The communication works. So, the issue might be something else.
Also, I noticed that I can clean finished workflows from the TUI but from the GUI where I get the error message “Command failed: clean – InputError: Operating on other users’ workflows is not supported”.
Thoughts and hints for continuing with my Cylc setup are very welcome.
I am not sure which Cylc log you mean. Replacing localhost in client.py with the output of the command hostname still allows me to run the hello world test.
2026-03-20T10:59:53Z INFO - Workflow: generic/run1
2026-03-20T10:59:53Z INFO - LOADING saved workflow parameters
2026-03-20T10:59:53Z INFO - + workflow UUID = 1d3883f1-dcf0-408c-9806-fecb705e9c92
2026-03-20T10:59:53Z INFO - + UTC mode = True
2026-03-20T10:59:53Z INFO - + initial point = 20191209T0900Z
2026-03-20T10:59:53Z INFO - + cycle point time zone = Z
2026-03-20T10:59:53Z INFO - + run mode = live
2026-03-20T10:59:53Z INFO - Pausing the workflow: Paused on start up
2026-03-20T10:59:53Z INFO - Scheduler: url=tcp://<HOST>:<PORT> pid=2251409
Spotted this in your debug output, I suspect it might have something to do with the issues you’ve been seeing, especially the “Operating on other users’ workflows is not supported” message in the GUI.
My guess is that there’s a long and a short form of your username, e.g. user123@domain.example.de and user123 with some system interfaces return one, and some the other? This might be tripping up Cylc authorisation/authentication.
(note that Tui/GUI will only display workflows running under the same user account)
This seems to show localhost, which is expected as I have not configured anything else in my workflow ctn05_clock-triggers/flow.cylc or elsewhere:
head -12 ~/cylc-run/ctn05_clock-triggers/run12//log/scheduler/log
2026-03-20T10:13:46Z INFO - Workflow: ctn05_clock-triggers/run12
2026-03-20T10:13:47Z INFO - Scheduler: url=tcp://localhost:43028 pid=2758971
2026-03-20T10:13:47Z INFO - Workflow publisher: url=tcp://localhost:43044
2026-03-20T10:13:47Z INFO - Run: (re)start number=1, log rollover=1
2026-03-20T10:13:47Z INFO - Cylc version: 8.5.4
2026-03-20T10:13:47Z INFO - Run mode: live
2026-03-20T10:13:47Z INFO - Initial point: 20260320T1013Z
2026-03-20T10:13:47Z INFO - Final point: 20260320T1033Z
2026-03-20T10:13:47Z INFO - Cold start from 20260320T1013Z
2026-03-20T10:13:47Z INFO - New flow: 1 (original flow from 20260320T1013Z) 2026-03-20T10:13:47
2026-03-20T10:13:47Z INFO - [20260320T1013Z/generate_forcing:waiting(runahead)] => waiting
2026-03-20T10:13:47Z INFO - [20260320T1013Z/install_cold:waiting(runahead)] => waitin
The Cylc default is to use the full hostname here (i.e, the name of the box), but it could be that localhost is the name of the box, or that host self-identification is hardwired to localhost.
Either way, you’ve confirmed that this works with that cylc ping command, so I think we’re good here. Would expect all the commands which interact with workflows to work given this (you would need to observe the workflow log to determine this due to the Tui / GUI issues you’ve reported).
My guess is that there’s a long and a short form of your username
Here’s an idea which might help to confirm this if it is the case.
Can you check if the output of these commands agree with each other?
[scheduler]
[[host self-identification]]
method = hardwired
host =localhost
in ~/.cylc/flow/global.cylc.
I have removed that file now and get
head ~/cylc-run/ctn05_clock-triggers/run13/log/scheduler/log
2026-03-20T12:07:24Z INFO - Workflow: ctn05_clock-triggers/run13
2026-03-20T12:07:25Z INFO - Scheduler: url=tcp://myhostname.srx.myorganisation.de:43003 pid=2869200
2026-03-20T12:07:25Z INFO - Workflow publisher: url=tcp://myhostname.srx.myorganisation.de:43006
With myhostname being the result of the command hostname.
If I put myhostname.srx.myorganisation.de into the client.py test programme and run python client.py the hello world greeting works as before.
Great, networking is working, you shouldn’t need that host self-identification going forwards.
Whoami gives me myuser@win.myorganisation.de
I think these two different forms of the same username are probably causing the issues you’ve been seeing. I don’t think this is something we have seen before.
The short form of the user name is the one I would expect. The long form (including the domain) is probably going to cause issues with some network tools like rsync and SSH.
For example rsync some-directory myuser@myhost is a valid command, but rsync some-directory myuser@win.myorganisation.de@myhost probably isn’t.
So I think we probably want to prevent the long form from getting into Cylc and standardise on the short form.
One last question, does this command return the long or short form of the username?
That gives the long form - myuser@win.myorganisation.de
I thought it would. Thanks for the info, I think I understand what’s going on here.
Two different parts of Cylc are using two different interfaces for requesting the user name. While these interfaces should return the same result, it would appear that in this case they are not.
This is causing the Tui/GUI to error during the connection process (workflow commands issued via the GUI may also fail).
If you are able to get the system reconfigured, I think that would probably be the best outcome as this issue could recur in other situations. However, I will raise the suggestion to standardise these interfaces within Cylc to make it more robust against such configurations.
,