# Execution retry not working, task state showing as waiting

**URL:** <https://cylc.discourse.group/t/execution-retry-not-working-task-state-showing-as-waiting/573>\
**Category:** Cylc Support\
**Created:** [December 14, 2022, 5:02am UTC](https://cylc.discourse.group/t/execution-retry-not-working-task-state-showing-as-waiting/573 "2022-12-14T05:02:23Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![prajeeshag](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/prajeeshag/32/212_2.png) [@prajeeshag](https://cylc.discourse.group/u/prajeeshag)\
**Post date:** [December 14, 2022, 5:02am UTC](https://cylc.discourse.group/t/execution-retry-not-working-task-state-showing-as-waiting/573/1 "2022-12-14T05:02:23Z")

</div>

Hi,

I have a task “check\_IC\_available” with “execution retry intervals” and “submission retry intervals” set to 20\*PT5M. But it does not resubmit after failing rather it goes to waiting state.

The following is the scheduler log:

2022-12-14T02:28:12+05:30 INFO - [20221214T0000Z/check\_IC\_available submitted job:03 flows:1] poll now, (next in PT15M (after 2022-12-14T02:43:12+05:30))  
2022-12-14T02:28:17+05:30 INFO - [20221214T0000Z/check\_IC\_available submitted job:03 flows:1] (polled)failed at 2022-12-14T02:14:16+05:30  
2022-12-14T02:28:17+05:30 INFO - [20221214T0000Z/check\_IC\_available submitted job:03 flows:1] =\> waiting  
2022-12-14T02:28:17+05:30 WARNING - [20221214T0000Z/check\_IC\_available waiting job:03 flows:1] retrying in PT5M (after 2022-12-14T02:33:17+05:30)  
2022-12-14T02:33:18+05:30 INFO - xtrigger satisfied: \_cylc\_retry\_20221214T0000Z/check\_IC\_available = wall\_clock(trigger\_time=1670965397.0793738)

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [December 14, 2022, 7:16am UTC](https://cylc.discourse.group/t/execution-retry-not-working-task-state-showing-as-waiting/573/2 "2022-12-14T07:16:42Z")

</div>

If a task fails with retries configured, it is supposed to go back to the `waiting` state, to wait on the configured delay before the next try.

> retrying in PT5M (after 2022-12-14T02:33:17+05:30)

Are you saying that the task did not resubmit its job after the PT5M (5 minute) delay?

Your log excerpt doesn’t say one way or the other, as the job submission would come after the last line.

---

<div class="post-metadata">

**Author:** ![prajeeshag](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/prajeeshag/32/212_2.png) [@prajeeshag](https://cylc.discourse.group/u/prajeeshag)\
**Post date:** [December 14, 2022, 7:32am UTC](https://cylc.discourse.group/t/execution-retry-not-working-task-state-showing-as-waiting/573/3 "2022-12-14T07:32:16Z")

</div>

Yes, the task was not resubmitted after the PT5M. There is nothing related to this task after this line in the log  
“2022-12-14T02:33:18+05:30 INFO - xtrigger satisfied: \_cylc\_retry\_20221214T0000Z/check\_IC\_available = wall\_clock(trigger\_time=1670965397.0793738)”

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [December 14, 2022, 8:13am UTC](https://cylc.discourse.group/t/execution-retry-not-working-task-state-showing-as-waiting/573/4 "2022-12-14T08:13:53Z")

</div>

Is this repeatable? And can you tell us what Cylc version you are running?

---

<div class="post-metadata">

**Author:** ![prajeeshag](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/prajeeshag/32/212_2.png) [@prajeeshag](https://cylc.discourse.group/u/prajeeshag)\
**Post date:** [December 14, 2022, 3:40pm UTC](https://cylc.discourse.group/t/execution-retry-not-working-task-state-showing-as-waiting/573/5 "2022-12-14T15:40:20Z")

</div>

It is cylc version 8.0.3. I tried a simplified version of the same workflow to recreate this issue, but that is resubmitting the job correctly.

Maybe I should try reinstalling the workflow.

2022-12-14T20:32:05+05:30 INFO - [20221209T0000Z/check\_IC\_available submitted job:04 flows:1] health: submission timeout=None, polling intervals=PT15M,…  
2022-12-14T20:47:05+05:30 INFO - [20221209T0000Z/check\_IC\_available submitted job:04 flows:1] poll now, (next in PT15M (after 2022-12-14T21:02:05+05:30))  
2022-12-14T20:47:09+05:30 INFO - [20221209T0000Z/check\_IC\_available submitted job:04 flows:1] (polled)failed at 2022-12-14T20:32:53+05:30  
2022-12-14T20:47:09+05:30 INFO - [20221209T0000Z/check\_IC\_available submitted job:04 flows:1] =\> waiting  
2022-12-14T20:47:09+05:30 WARNING - [20221209T0000Z/check\_IC\_available waiting job:04 flows:1] retrying in PT5M (after 2022-12-14T20:52:09+05:30)  
2022-12-14T20:52:09+05:30 INFO - xtrigger satisfied: \_cylc\_retry\_20221209T0000Z/check\_IC\_available = wall\_clock(trigger\_time=1671031329.4756932)  
2022-12-14T20:52:09+05:30 INFO - [20221209T0000Z/check\_IC\_available waiting job:04 flows:1] =\> waiting(queued)  
2022-12-14T20:52:09+05:30 INFO - [20221209T0000Z/check\_IC\_available waiting(queued) job:04 flows:1] =\> waiting  
2022-12-14T20:52:09+05:30 INFO - [20221209T0000Z/check\_IC\_available waiting job:05 flows:1] =\> preparing  
2022-12-14T20:52:09+05:30 INFO - [20221209T0000Z/check\_IC\_available preparing job:05 flows:1] host=10.119.14.15  
2022-12-14T20:52:13+05:30 INFO - [20221209T0000Z/check\_IC\_available preparing job:05 flows:1] (internal)submitted at 2022-12-14T20:52:47+05:30  
2022-12-14T20:52:13+05:30 INFO - [20221209T0000Z/check\_IC\_available preparing job:05 flows:1] submitted to elogin04:background[16273]

---

<div class="post-metadata">

**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\
**Post date:** [December 14, 2022, 3:44pm UTC](https://cylc.discourse.group/t/execution-retry-not-working-task-state-showing-as-waiting/573/6 "2022-12-14T15:44:33Z")

</div>

I can think of two things which are supposed to suppress retries:

- If the task was “held” (marked with a paused badge next to the task icon in the UI).
- If the workflow was “paused” (the word paused should appear in the toolbar in the UI).

---

<div class="post-metadata">

**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\
**Post date:** [December 14, 2022, 3:59pm UTC](https://cylc.discourse.group/t/execution-retry-not-working-task-state-showing-as-waiting/573/7 "2022-12-14T15:59:15Z")

</div>

Reinstalling shouldn’t make a difference (this only syncs files from `cylc-src` to `cylc-run`). Restarting could make a difference because it would wipe any cached state in the scheduler (the state is preserved in the database).

If it can be reproduced, for diagnosis you could try running the workflow in `--debug` mode. Then send us the log file from the failure up to the first entry after the task should have retried. This might contain something informative.

---

<div class="post-metadata">

**Author:** ![prajeeshag](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/prajeeshag/32/212_2.png) [@prajeeshag](https://cylc.discourse.group/u/prajeeshag)\
**Post date:** [December 14, 2022, 4:17pm UTC](https://cylc.discourse.group/t/execution-retry-not-working-task-state-showing-as-waiting/573/8 "2022-12-14T16:17:48Z")

</div>

Okay. I will try that.
