# Previously completed task not disappearing from workflow

**URL:** <https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672>\
**Category:** Cylc Support\
**Created:** [June 20, 2023, 1:08am UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672 "2023-06-20T01:08:37Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![jonnyhtw](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jonnyhtw/32/232_2.png) [@jonnyhtw](https://cylc.discourse.group/u/jonnyhtw)\
**Post date:** [June 20, 2023, 1:08am UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/1 "2023-06-20T01:08:37Z")

</div>

hi there,

i have a `cylc` 8 workflow which is running fine but has a completed task hanging around, see screenshot.

the offending task (is ‘orphaned’ the right word?) completed fine and somehow doesn’t seem to be stopping the workflow from running but it’s puzzling to know why it’s there!

any ideas?

thanks

 ![image](https://global.discourse-cdn.com/free1/uploads/cylc1/original/1X/a1f050a69d5232bc2c5d68ef1876d7eb51a62285.png)

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [June 20, 2023, 1:35am UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/2 "2023-06-20T01:35:04Z")

</div>

Interesting. Is it there in the other views too (tree and table)?

Also, if you do `cylc dump -t u-cw633/run1`, is it listed there?

---

<div class="post-metadata">

**Author:** ![jonnyhtw](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jonnyhtw/32/232_2.png) [@jonnyhtw](https://cylc.discourse.group/u/jonnyhtw)\
**Post date:** [June 20, 2023, 1:48am UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/3 "2023-06-20T01:48:06Z")

</div>

yeah it’s there in all views, and `cylc dump`, strange!

```auto
> cylc dump -t u-cw633/run1
coupled, 19880601T0000Z, succeeded, not-held, not-queued, not-runahead
coupled, 19880701T0000Z, submitted, not-held, not-queued, not-runahead
coupled, 19880801T0000Z, waiting, not-held, not-queued, not-runahead
postproc, 19820501T0000Z, succeeded, not-held, not-queued, not-runahead
postproc, 19880701T0000Z, waiting, not-held, not-queued, not-runahead

```

 ![image](https://global.discourse-cdn.com/free1/uploads/cylc1/original/1X/92d90733aa22614bf0bb5facd312cba64aa03258.png)

 ![image](https://global.discourse-cdn.com/free1/uploads/cylc1/original/1X/a01ef87b4b796c7d523b54f650abd1eec001521c.png)

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [June 20, 2023, 2:39am UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/4 "2023-06-20T02:39:46Z")

</div>

Next thing, can you try stopping and restarting the workflow? (That’s safe to do, even if tasks are running at the time).

---

<div class="post-metadata">

**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\
**Post date:** [June 20, 2023, 10:43am UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/5 "2023-06-20T10:43:46Z")

</div>

Thanks, that `cylc dump` output is very helpful to us in narrowing down the source of the problem. We have now seen this but at present can’t reproduce it reliably ☹

---

<div class="post-metadata">

**Author:** ![jonnyhtw](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jonnyhtw/32/232_2.png) [@jonnyhtw](https://cylc.discourse.group/u/jonnyhtw)\
**Post date:** [June 20, 2023, 11:40pm UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/6 "2023-06-20T23:40:50Z")

</div>

ok just sent the `cylc stop` command. it’ll take a while to finish the active tasks so i’ll let you know how it gets on hen i restart it. thanks

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [June 21, 2023, 4:43am UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/7 "2023-06-21T04:43:39Z")

</div>

Note you don’t have to wait for running tasks to finish before shutdown:

```console
$ cylc stop --now <workflow_id>

```

This will shut the scheduler down and leave the active tasks running. On restart, Cylc will automatically reconnect with the orphaned tasks, or update itself appropriately if they succeeded or failed during the downtime.

---

<div class="post-metadata">

**Author:** ![jonnyhtw](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jonnyhtw/32/232_2.png) [@jonnyhtw](https://cylc.discourse.group/u/jonnyhtw)\
**Post date:** [June 23, 2023, 5:49am UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/8 "2023-06-23T05:49:04Z")

</div>

good to know! is this a new `cylc` 8 thing?

i’ve stopped and started the offending workflow now and the old task is still there, very odd!

cheers

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [June 23, 2023, 7:19am UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/9 "2023-06-23T07:19:58Z")

</div>

> [@jonnyhtw](#):
>
> good to know! is this a new `cylc` 8 thing?

Nope, Cylc has had that magic capability for a long time!

(Magic, because Cylc can figure out what happened to orphaned jobs during scheduler downtime even if they are long gone from the system when you start up again).

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [June 23, 2023, 8:32am UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/10 "2023-06-23T08:32:29Z")

</div>

> [@jonnyhtw](#):
>
> i’ve stopped and started the offending workflow now and the old task is still there, very odd!

Next, try killing your UI Server (`cylc gui`) and start that again.

(It is probably just an object - representing the old task - in the datastore that feeds the UI, that was not cleaned up when it should have been as the workflow moved on).

---

<div class="post-metadata">

**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\
**Post date:** [June 23, 2023, 8:39am UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/11 "2023-06-23T08:39:22Z")

</div>

If the task is visible in `cylc dump`, then the problem is at the cylc-flow end. You should see the same thing in `cylc tui` which doesn’t go through the UI Server.

We’ve recently seen a similar situation where a succeeded no-flow task in the database was causing tasks one edge out to display in the GUI. When we hacked the no-flow task away from the DB the spurious tasks disappeared on restarted.

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [June 23, 2023, 8:41am UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/12 "2023-06-23T08:41:46Z")

</div>

> [@oliver.sanders](#):
>
> If the task is visible in `cylc dump`, then the problem is at the cylc-flow end

Good point (so my UI Server restart suggestion won’t help). It is still, I think, in the datastore though, as opposed the scheduler task pool (not that that will make any sense to users).

---

<div class="post-metadata">

**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\
**Post date:** [June 23, 2023, 8:44am UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/13 "2023-06-23T08:44:58Z")

</div>

The task wasn’t in the pool in our example either, although poking at the DB there were a few tasks there which shouldn’t have been, this appeared to just be a [delayed database operation](https://github.com/cylc/cylc-flow/issues/5598).

---

<div class="post-metadata">

**Author:** ![jonnyhtw](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jonnyhtw/32/232_2.png) [@jonnyhtw](https://cylc.discourse.group/u/jonnyhtw)\
**Post date:** [June 26, 2023, 2:05am UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/14 "2023-06-26T02:05:51Z")

</div>

hi there,

i restarted the UI and that previous task has gone now but there’s a new one there now that shouldn’t be 😄!

it’s also there in the `cylc dump` output and by the looks of it, the new task which is showing up is coming from the time when i stopped workflow toward the end of last week so maybe that helps with the diagnosis?

cheers,

```auto
> cylc dump -t u-cw633/run1
coupled, 19900401T0000Z, succeeded, not-held, not-queued, not-runahead
coupled, 19900501T0000Z, running, not-held, not-queued, not-runahead
coupled, 19900601T0000Z, waiting, not-held, not-queued, not-runahead
postproc, 19881001T0000Z, succeeded, not-held, not-queued, not-runahead
postproc, 19900501T0000Z, waiting, not-held, not-queued, not-runahead

```

 ![image](https://global.discourse-cdn.com/free1/uploads/cylc1/original/1X/16244c251975adebf2f00d9fe9839631f06cbbc4.png)

---

<div class="post-metadata">

**Author:** ![MetRonnie](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/metronnie/32/125_2.png) [@MetRonnie](https://cylc.discourse.group/u/MetRonnie)\
**Post date:** [June 27, 2023, 2:15pm UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/15 "2023-06-27T14:15:11Z")

</div>

Looks like a common theme with these tasks showing when they shouldn’t may be they were present at the point of shutdown/stall?

---

<div class="post-metadata">

**Author:** ![jonnyhtw](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jonnyhtw/32/232_2.png) [@jonnyhtw](https://cylc.discourse.group/u/jonnyhtw)\
**Post date:** [June 27, 2023, 10:29pm UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/16 "2023-06-27T22:29:03Z")

</div>

@MetRonnie yeah looks that way to me

---

<div class="post-metadata">

**Author:** ![jonnyhtw](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jonnyhtw/32/232_2.png) [@jonnyhtw](https://cylc.discourse.group/u/jonnyhtw)\
**Post date:** [June 27, 2023, 11:14pm UTC](https://cylc.discourse.group/t/previously-completed-task-not-disappearing-from-workflow/672/17 "2023-06-27T23:14:48Z")

</div>

ok yeah that’s it…

1. initial ‘broken’ workflow state (with `19881001T0000Z/postproc` still there)…

 ![Screenshot 2023-06-28 105500](https://global.discourse-cdn.com/free1/uploads/cylc1/original/1X/8f5e43aba3f5dd88489c7ae3d991f6610dfda255.png)

1. after `cylc stop -k u-cw633/run1`…

 ![Screenshot 2023-06-28 105540](https://global.discourse-cdn.com/free1/uploads/cylc1/original/1X/cac581196eaf5fcdc786377b9fc592a11c6f185f.png)

1. after shutdown of `cylc` UI and `cylc play u-cw633/run1` all looks good again…

 ![Screenshot 2023-06-28 105808](https://global.discourse-cdn.com/free1/uploads/cylc1/original/1X/61848d48e06f92adac05bc2b3d0b033cc53ee0bc.png)
