# Workflow consistently stalling

**URL:** <https://cylc.discourse.group/t/workflow-consistently-stalling/632>\
**Category:** Cylc Support\
**Created:** [April 14, 2023, 2:28am UTC](https://cylc.discourse.group/t/workflow-consistently-stalling/632 "2023-04-14T02:28:13Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![jonnyhtw](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jonnyhtw/32/232_2.png) [@jonnyhtw](https://cylc.discourse.group/u/jonnyhtw)\
**Post date:** [April 14, 2023, 2:28am UTC](https://cylc.discourse.group/t/workflow-consistently-stalling/632/1 "2023-04-14T02:28:13Z")

</div>

hey,

i have a workflow which is running fine but it keeps on stalling with messages like…

```auto
> cylc log u-cv956|grep -i 'is waiting on'
      * 20090701T0000Z/housekeeping is waiting on
      ['20090701T0000Z/postproc:succeeded']

```

this is unexpected since this task has definitely completed successfully and so i am having to use `cylc set-outputs` to fix these stalls. i have messed around with this suite _a lot_ workflow i am a `cylc` 8 noob but i was just wondering if anyone knows of a reason why this might be happening and if so how to fix it?

i have an identical (in `cylc` terms anyway) workflow which is running fine so i know it’s something that i must have broken manually without meaning to! 😄

btw i have manually run `cylc triger --flow=new ...` a couple of times to force a `housekeeping` task to rerun so that i could test a postprocessing workflow which is run as part of a `post-script`. this may well have messed things up.

cheers,

jonny

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [April 14, 2023, 4:13am UTC](https://cylc.discourse.group/t/workflow-consistently-stalling/632/2 "2023-04-14T04:13:44Z")

</div>

Hi @jonnyhtw

The fact that the identical workflow is running fine shows there’s nothing fundamentally wrong, which is good.

> this is unexpected since this task has definitely completed successfully and so i am having to use `cylc set-outputs` to fix these stalls. … btw i have manually run `cylc trigger --flow=new ...` a couple of times

I suspect you have inadvertently started a new flow, with a new flow number, (i.e. a new self-perpetuating run through the graph) and the apparently-unsatisfied prerequisites pertain to the other flow.

Can you stop your workflow, then restart it with `cylc play --debug` so we can see flow numbers in the log. You can show me the results offline, and I’ll post an update here later once we’ve confirmed what’s happening.

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [April 14, 2023, 4:30am UTC](https://cylc.discourse.group/t/workflow-consistently-stalling/632/3 "2023-04-14T04:30:38Z")

</div>

Actually, flow numbers should still be reported without debug mode on. Do you see this sort of thing:

```auto
WARNING - [20200101T0000Z/foo failed job:01 flows:1] did not complete required outputs:
    ['succeeded']
INFO - [20200101T1200Z/foo running job:01 flows:1] => failed

```

But with `flows` other than `1`?

---

<div class="post-metadata">

**Author:** ![jonnyhtw](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jonnyhtw/32/232_2.png) [@jonnyhtw](https://cylc.discourse.group/u/jonnyhtw)\
**Post date:** [April 18, 2023, 12:01am UTC](https://cylc.discourse.group/t/workflow-consistently-stalling/632/4 "2023-04-18T00:01:08Z")

</div>

hi @hilary.j.oliver, thanks for this and the offline chat, moving back here in case this is helpful for others in the future…

seems that by doing lots of this - - - \> `cylc trigger --flow=new...` launched lots of new flows which then were left hanging/orphaned (?).

i then stopped them with `cylc stop --flow=[number]...` and this seems to have allowed the workflow to proceed a expected.

so, the question now is, how would i run a task in an arbitrary cycle point, which may well be significantly behind the runahead limit and _not_ trigger any tasks further down the flow?

again this may seem odd in `cylc` 8 world but it is something that we need to do often-ish in climate modelling. an example might be to re-trigger a specific `postproc` task to generate a missing annual mean which wasn’t created for whatever reason.

cheers!

jonny

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [April 18, 2023, 12:58am UTC](https://cylc.discourse.group/t/workflow-consistently-stalling/632/5 "2023-04-18T00:58:51Z")

</div>

> [@jonnyhtw](#):
>
> seems that by doing lots of this - - - \> `cylc trigger --flow=new...` launched lots of new flows which then were left hanging/orphaned (?).

Yeah, I think what happened was you repeatedly launched new flows in parts of the graph with “off-flow” prerequisites that caused a stall: To illustrate what I mean by that:

![off-flow](https://global.discourse-cdn.com/free1/uploads/cylc1/original/1X/902e24fc5d07f935a881747454e1361148e5f4d9.png)

In this graph, if I trigger a new flow at `1/B`, it will flow on through the whole graph downstream of `1/B`, because success of `B` satisfies the prerequisites of both `C` and `D`, which satisfy `E`, which satisfies `F`.

But if I trigger a new flow at `1/C` the workflow will stall with `1/E` waiting on the “off-flow” task `1/D` (which I did not trigger in the new flow, and which does not getting automatically triggered by a parent task in the new flow).

No biggie, I can still use `cylc set-outputs` to make the workflow carry on as if `1/D` had succeeded, or I could manually trigger `1/E` to make it run even though its prerequisites are not satisfied.

Unfortunately, in your workflow there were **implicit** (not visible in the graph) off-flow prerequisites due to declaring all tasks to be `sequential` (an implicit dependence on their own previous-cycle instance). This made it harder to see what the problem was. _[Note we’ve recommended not using “sequential tasks” for a long time now]_.

> [@jonnyhtw](#):
>
> so, the question now is, how would i run a task in an arbitrary cycle point, which may well be significantly behind the runahead limit and _not_ trigger any tasks further down the flow?

Excellent question.

The ability to run multiple logically distinct flows through the same graph is a super-powerful new feature of Cylc 8.

If you manually trigger a task somewhere in the graph,

- by default it belongs to the current active flow(s) - which is usually flow=1
- but you can make it start a new flow (`--flow=new`)
- or you can make it not belong to any flow (`--flow=none`)

With `--flow=none` the triggered task will run without starting a flow or affecting any other flows that may pass over it in the future.

Otherwise the triggered task will flow on IF its child-tasks (i.e. downstream in the graph) have not already run in this flow.

To re-run a part of the past graph, you have to start a new flow there, because the tasks downstream of the triggered one have already run in the original flow.

You can read all about flows here: [Concurrent Flows — Cylc 8.2.2 documentation](https://cylc.github.io/cylc-doc/stable/html/user-guide/running-workflows/reflow.html)

---

<div class="post-metadata">

**Author:** ![jonnyhtw](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jonnyhtw/32/232_2.png) [@jonnyhtw](https://cylc.discourse.group/u/jonnyhtw)\
**Post date:** [April 18, 2023, 4:21am UTC](https://cylc.discourse.group/t/workflow-consistently-stalling/632/6 "2023-04-18T04:21:44Z")

</div>

this is great, thanks @hilary.j.oliver!

i’ve tested the `--flow=none` syntax and it works perfectly! thanks for the background info on this.

> … we’ve recommended not using “sequential tasks” for a long time now.

does `sequential` mean something `cylc`-specific here? in climate modelling, the main number crunching task in cycle point `N` will always need to wait for that at `N-1` to finish first since the starting state of `N` will be the same as the end state of `N-1`.

cheers,

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [April 18, 2023, 4:39am UTC](https://cylc.discourse.group/t/workflow-consistently-stalling/632/7 "2023-04-18T04:39:51Z")

</div>

By “sequential tasks” I mean this:

```ini
# (A)
[scheduling]
   [[special tasks]]
      sequential = foo, bar # <---- implicit previous-cycle dependence!
   [[graph]]
      P1D = """
         foo => bar
      """

```

That is exactly equivalent to this:

```ini
# (B)
[scheduling]
   [[graph]]
      P1D = """
         foo => bar 
         foo[-P1D] => foo # explicit, good!
         bar[-P1D] => bar
      """

```

But `(B)` is much better because the dependencies are explicit - you can see them in the graph, so you’re not going to forget about them and wonder why a task is waiting on a seemingly-invisible prerequisite. And you’re less likely to do it unnecessarily, again because all the dependencies are visible in the graph (unfortunately I’ve seen a lot of workflows where users seem to unnecessarily declare all the tasks as sequential “just in case”).

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [April 18, 2023, 4:41am UTC](https://cylc.discourse.group/t/workflow-consistently-stalling/632/8 "2023-04-18T04:41:11Z")

</div>

> [@jonnyhtw](#):
>
> in climate modelling, the main number crunching task in cycle point `N` will always need to wait for that at `N-1` to finish first since the starting state of `N` will be the same as the end state of `N-1`.

Yes, dead right - but my point is, it’s much better to make that dependence explicit in the graph.

---

<div class="post-metadata">

**Author:** ![jonnyhtw](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jonnyhtw/32/232_2.png) [@jonnyhtw](https://cylc.discourse.group/u/jonnyhtw)\
**Post date:** [April 18, 2023, 5:23am UTC](https://cylc.discourse.group/t/workflow-consistently-stalling/632/9 "2023-04-18T05:23:35Z")

</div>

OK got it, agreed!

fwiw i didn’t realise that ‘explicit’ meant ‘visible in the graph’, which has undoubtedly caused me to be even-slower-than-usual on the uptake here. 😄

cheers,

jonny

---

<div class="post-metadata">

**Author:** ![jonnyhtw](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jonnyhtw/32/232_2.png) [@jonnyhtw](https://cylc.discourse.group/u/jonnyhtw)\
**Post date:** [April 18, 2023, 5:34am UTC](https://cylc.discourse.group/t/workflow-consistently-stalling/632/10 "2023-04-18T05:34:00Z")

</div>

brilliant thanks,

i’ll have a go at changing the graph to incorporate this! i’ll launch it as a different workflow to avoid any further issues haha.

cheers,

j