# Cylc broadcast error: WorkflowStopped **URL:** \ **Category:** Cylc Support\ **Created:** [May 13, 2025, 1:54pm UTC](https://cylc.discourse.group/t/cylc-broadcast-error-workflowstopped/1167 "2025-05-13T13:54:14Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![sparonuz](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/sparonuz/32/495_2.png) [@sparonuz](https://cylc.discourse.group/u/sparonuz)\ **Post date:** [May 13, 2025, 1:54pm UTC](https://cylc.discourse.group/t/cylc-broadcast-error-workflowstopped/1167/1 "2025-05-13T13:54:14Z")
Good afternoon, I am seeking advice for debugging a broadcast. In particular, I have a task that, after computing a variable, tries to send it to another task. Let’s say that this is my task (named `hindcast_recup_SST`) ```auto #!/bin/bash set -euo pipefail # Count the number of files, that will equal the number of tasks needed to create superobs n_task=$(ls -lrt ${MY_PATH} | wc -l) # Send the number of tasks necessary to create superobs task cylc broadcast ${CYLC_WORKFLOW_ID} -p ${CYLC_TASK_CYCLE_POINT} \ -n hindcast_create_superobs_SST \ -s "[directives]--ntasks=${n_task}" ``` `hindcast_create_superobs_SST` is the task where I need to set the ntasks directive. Now the problem: the first task fails with the following error > WorkflowStopped: glo12\_cylc/run1 is not running > 2025-05-13T12:48:24Z CRITICAL - failed/ERR And the error is caused by the broadcast, since if I remove it, the task succeeds. I tried to run the broadcast by hand on the command line, and it works > $ cylc broadcast glo12\_cylc/run1 -p 20241111 -n hindcast\_create\_superobs\_SST -s “[directives]–ntasks=4” > Broadcast set: > > - [20241111/hindcast\_create\_superobs\_SST] [directives]–ntasks=4 Now, I made many test, and I am quite at loss. First of all, how does cycl check if a workflow is running? Normally I check my workflow it with `cycl tui`, and there is marked as running, but from the error message, I understand there must be some variable that is telling the opposite to cycl. Second: how do I debug this? I tryed to run the workflow in debug, but I see just an error like > DEBUG - [20241111/hindcast\_recup\_SST/01:submitted] (polled)failed > INFO - [20241111/hindcast\_recup\_SST/01:submitted] setting implied output: started > DEBUG - [20241111/hindcast\_recup\_SST/01:submitted] (internal)started > INFO - [20241111/hindcast\_recup\_SST/01:submitted] =\> running > DEBUG - [20241111/hindcast\_recup\_SST/01:running] health: execution timeout=None, polling intervals=PT1M,3\*PT2M,PT1M,… > INFO - [20241111/hindcast\_recup\_SST/01:running] =\> failed > WARNING - [20241111/hindcast\_recup\_SST/01:failed] did not complete the required outputs: > ⨯ ┆ succeeded I don’t know how to retrieve more information on the failure, I added a --debug to the broadcast command without much success. Also, I have previous broadcast in the workflow, and they work. It could be that some previous broadcast fails, and this one is just trapping the error? Any help will be highly appreciated! Best, Stella ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [May 13, 2025, 2:12pm UTC](https://cylc.discourse.group/t/cylc-broadcast-error-workflowstopped/1167/2 "2025-05-13T14:12:34Z")
> First of all, how does cycl check if a workflow is running? Cylc workflows create a special file when they are started containing information about where the scheduler is running. You can find this file in `~/cylc-run//.service/contact`. The file is removed when the scheduler shuts down. All Cylc utilities, from `cylc broadcast` to `cylc gui` use this file to determine if the workflow is running. > Second: how do I debug this? Cylc needs to keep track of the status of the jobs it submits. There are two options for this: - Push - Cylc jobs send messages back to the scheduler. - Pull - The scheduler routinely checks the`job.status` file that each job creates. Looking at this log snippet: ```auto DEBUG - [20241111/hindcast_recup_SST/01:submitted] (polled)failed ``` The `(polled)` bit means that Cylc is using “pull” based communication (aka “polling”) for this job. Unfortunately, that means you won’t be able to push data back to the workflow, e.g. using `cylc broadcast`. Push communication is the default, however, Cylc will fallback to pull communication if: - Cylc has been [configured to use pull communication](https://cylc.github.io/cylc-doc/stable/html/reference/config/global.html#global.cylc%5Bscheduler%5D%5Bhost%20self-identification%5Dmethod). - Or, if the scheduler doesn’t receive any update from the job within a configured time window. So either: - Cylc has been configured to use pull communication. - Or, there’s something wrong with the setup. To find out which, run the `cylc config -i '[platforms]'` and find the entry for the platform you are using to run this job. If it includes `method = poll` then Cylc has been configured for pull communication. If the platform is _not_ configured for pull communication, this section of the troubleshooting guide has some information on debugging: > **[Troubleshooting — Cylc 8.4.2 documentation](https://cylc.github.io/cylc-doc/stable/html/user-guide/troubleshooting.html#job-status-isn-t-updating)** ---
**Author:** ![sparonuz](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/sparonuz/32/495_2.png) [@sparonuz](https://cylc.discourse.group/u/sparonuz)\ **Post date:** [May 13, 2025, 2:36pm UTC](https://cylc.discourse.group/t/cylc-broadcast-error-workflowstopped/1167/3 "2025-05-13T14:36:00Z")
Hi Oliver, thanks a lot for your quick reply! In our platform file we have ```auto communication method = poll ``` And I can confirm that we configured the entire workflow for using the polling method, due to the special set-up (firewall problems) on the machine we are using. So, if I understand correctly, all broadcasts should fail then? And it makes a lot of sens now that you mention it. But it also confuses me, since … they work. And this one worked until I introduced another previous one (if I remove it, it restarts working). Also, it works from command line: could it be that it just fails silently? Stella ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [May 13, 2025, 2:59pm UTC](https://cylc.discourse.group/t/cylc-broadcast-error-workflowstopped/1167/4 "2025-05-13T14:59:22Z")
> So, if I understand correctly, all broadcasts should fail then? All broadcasts sent from the remote platform should fail. Broadcasts made locally to the scheduler will still work. E.g: ```cylc [scheduling] [[graph]] R1 = local & remote [runtime] [[local]] # this should work: script = cylc broadcast "$CYLC_WORKFLOW_ID" -s '[environment]PASS=1' [[remote]] platform = my-remote-platform # this will probably fail: script = cylc broadcast "$CYLC_WORKFLOW_ID" -s '[environment]FAIL=2' ``` ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [May 13, 2025, 3:04pm UTC](https://cylc.discourse.group/t/cylc-broadcast-error-workflowstopped/1167/5 "2025-05-13T15:04:20Z")
> [@sparonuz](#): > > due to the special set-up (firewall problems) on the machine we are using. This is a common problem with HPC systems, as a result, we provide a fallback option for push communication using SSH rather than TCP: ```auto communication method = ssh ``` With this configuration, Cylc will try to SSH _from_ the HPC, to the scheduler host. It will then use TCP to contact the scheduler. This requires SSH to be setup _from_ the HPC to the local platform, where normally we might only set it up in the other direction. It might be a bit more of a hassle to set up your user account to do this, however, it would enable push-based communication without coming into conflict with the firewall. All Cylc commands can be invoked over this interface, including `cylc broadcast`. More info here: > **[Global Configuration — Cylc 8.4.2 documentation](https://cylc.github.io/cylc-doc/stable/html/reference/config/global.html#global.cylc%5Bplatforms%5D%5B%3Cplatform%20name%3E%5Dcommunication%20method)** ---
**Author:** ![sparonuz](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/sparonuz/32/495_2.png) [@sparonuz](https://cylc.discourse.group/u/sparonuz)\ **Post date:** [May 13, 2025, 3:08pm UTC](https://cylc.discourse.group/t/cylc-broadcast-error-workflowstopped/1167/6 "2025-05-13T15:08:55Z")
Thanks Oliver for all this info! I think that @gturek went through all of this before, and the polling was the only option left for us for some reason. Tomorrow I will discuss with her again about it, but I think we will have to come up with a different workflow that avoids broadcasts from the remote platform … it is anyway very good to know why something fails 🙂 Thanks a lot again, Stella ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [May 13, 2025, 3:17pm UTC](https://cylc.discourse.group/t/cylc-broadcast-error-workflowstopped/1167/7 "2025-05-13T15:17:02Z")
We should do something to make the error from commands like `cylc broadcast` more helpful in this situation. ---
**Author:** ![Bruno\_P\_Kinoshita](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/bruno_p_kinoshita/32/369_2.png) [@Bruno\_P\_Kinoshita](https://cylc.discourse.group/u/Bruno_P_Kinoshita)\ **Post date:** [May 18, 2025, 3:53pm UTC](https://cylc.discourse.group/t/cylc-broadcast-error-workflowstopped/1167/8 "2025-05-18T15:53:39Z")
For BSC MN5, at least, only the poll communication method works. TCP or SSH both time out/fail. So if you have a similar HPC/environment, where HPC has no connectivity back to login nodes, then it might be needed to keep using the poll method. ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [May 18, 2025, 10:23pm UTC](https://cylc.discourse.group/t/cylc-broadcast-error-workflowstopped/1167/9 "2025-05-18T22:23:19Z")
It’s perfectly possible to configure the network to allow push TCP comms back from HPC compute nodes, of course, but whether or not you can persuade the platform admins to do it might be another matter, unfortunately. You might try to enlist the help of someone with power, to make the business case - efficient workflow orchestration makes for better use of the machine, and presumably your workflow products are important too. If you are stuck with job polling (pull), you should still be able to achieve the same result by adding a local task to your worklfow. Instead of: - remote task `a` broadcasts information to downstream tasks Do this: - remote task `a` writes information to disk - **local** task `b` triggers off of `a`, reads the information from disk and broadcasts it to downstream tasks - (this might involve ssh **to** the remote platform, if the filesystem is different, but that’s OK - ssh must work in that direction because it’s used for job submission) ---
**Author:** ![sparonuz](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/sparonuz/32/495_2.png) [@sparonuz](https://cylc.discourse.group/u/sparonuz)\ **Post date:** [May 19, 2025, 7:45am UTC](https://cylc.discourse.group/t/cylc-broadcast-error-workflowstopped/1167/10 "2025-05-19T07:45:50Z")
Hi Bruno (nice to hear from you ☺!) We have a slightly more complicated set-up than MN5 (but fundamentally the same issue): we have our dedicated login node on the machine we use (MeteoFrance), were we set up the cylc instance. And this login is not seen from the compute nodes (you can’t connect back). So you’re right, the only method we can use is polling, all the rest fails. @hilary.j.oliver Yes, I though about this intermediate set-up. But for the moment I decided to slightly change the workflow to avoid those broadcast, I thought it was simpler and more linear (it just create a bit of duplication between the tasks). If at a certain point we can’t avoid to broadcast, I will implement this “workaround” though, thanks! Best, Stella ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [May 19, 2025, 8:51am UTC](https://cylc.discourse.group/t/cylc-broadcast-error-workflowstopped/1167/11 "2025-05-19T08:51:59Z")
Cylc’s SSH communication method combined with SSH tunnelling (via the login node) might be an option in some difficult setups, but if the compute nodes can’t even see the login nodes, and TCP ports cannot be opened, then I think you’re stuck with polling ☹ . Unfortunately, it is hard to send data back to the scheduler in this case as there is no way of communicating with it. However, there is one method you can use to send simple pre-defined outcomes back to the scheduler using `cylc message`. These messages are written into the `job.status` file so are polled by the scheduler along with the job status. You can’t send data back via this mechanism, however, you can send back pre-defined messages and map them onto triggers that can then be used in the workflow’s graph. Tutorial: [Message Triggers — Cylc 8.4.2 documentation](https://cylc.github.io/cylc-doc/stable/html/tutorial/furthertopics/message-triggers.html) Example (with graph branching): [Scheduling Configuration — Cylc 8.4.2 documentation](https://cylc.github.io/cylc-doc/stable/html/user-guide/writing-workflows/scheduling.html#custom-outputs)