# Cylc job hanging and failing to submit

**URL:** <https://cylc.discourse.group/t/cylc-job-hanging-and-failing-to-submit/958>\
**Category:** Cylc Support\
**Created:** [May 20, 2024, 2:35am UTC](https://cylc.discourse.group/t/cylc-job-hanging-and-failing-to-submit/958 "2024-05-20T02:35:16Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Cameron.Mc](https://avatars.discourse-cdn.com/v4/letter/c/bb73d2/32.png) [@Cameron.Mc](https://cylc.discourse.group/u/Cameron.Mc)\
**Post date:** [May 20, 2024, 2:35am UTC](https://cylc.discourse.group/t/cylc-job-hanging-and-failing-to-submit/958/1 "2024-05-20T02:35:16Z")

</div>

Currently trying to run a workflow on Cylc 8.2.4.

When it gets to a certain job, it hangs on the job without submitting. I get the following messages in the job-activity.log file:

> [jobs-poll ret\_code] 0  
> [jobs-poll out] 2024-05-20T10:48:35+12:00|20140101T0000Z/atmos\_main/01|{“job\_runner\_name”: “slurm”, “job\_id”: “3987323”, “job\_runner\_exit\_polled”: 0, “time\_submit\_exit”: “2024-05-20T10:33:32+12:00”}  
> [jobs-poll ret\_code] 0  
> [jobs-poll out] 2024-05-20T11:03:35+12:00|20140101T0000Z/atmos\_main/01|{“job\_runner\_name”: “slurm”, “job\_id”: “3987323”, “job\_runner\_exit\_polled”: 0, “time\_submit\_exit”: “2024-05-20T10:33:32+12:00”}  
> [jobs-poll ret\_code] 0

This messages is repeating every 15m minutes with a new timestamp, as I’m assuming it tries to execute the job but fails? Any ideas what would be causing this issue?

I am also getting an error at the start of the job-activity.log file reading:

> 2024-05-19T22:33:32Z [STDERR] sbatch: error: plugin\_load\_from\_file: dlopen(/opt/cray/pe/atp/libAtpSLaunch.so): /opt/cray/pe/atp/libAtpSLaunch.so: cannot open shared object file: No such file or directory  
> 2024-05-19T22:33:32Z [STDERR] sbatch: error: spank: /opt/cray/pe/atp/libAtpSLaunch.so: Dlopen of plugin file failed

This error doesn’t happen every time I try submit and run the workflow.

---

<div class="post-metadata">

**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\
**Post date:** [May 20, 2024, 6:35am UTC](https://cylc.discourse.group/t/cylc-job-hanging-and-failing-to-submit/958/2 "2024-05-20T06:35:23Z")

</div>

> [@Cameron.Mc](#):
>
> This messages is repeating every 15m minutes with a new timestamp, as I’m assuming it tries to execute the job but fails?

No. Your job has been submitted sucessfully and Cylc is now waiting for the job to run - the UI should show the task in the [submitted state](https://cylc.github.io/cylc-doc/stable/html/user-guide/running-workflows/task-job-states.html#task-job-states). Cylc polls the job (by default every [15 minutes](https://cylc.github.io/cylc-doc/stable/html/reference/config/global.html#global.cylc%5Bplatforms%5D%5B%3Cplatform%20name%3E%5Dsubmission%20polling%20intervals)) to check whether it is still in the queue. You can see from the poll output that your job was submitted at 2024-05-20T10:33:32+12:00 and that the job id is 3987323. So, you need to query Slurm (using `squeue`) to see why your job isn’t running.

The `sbatch` errors need reporting to your Slurm administrators.

---

<div class="post-metadata">

**Author:** ![Cameron.Mc](https://avatars.discourse-cdn.com/v4/letter/c/bb73d2/32.png) [@Cameron.Mc](https://cylc.discourse.group/u/Cameron.Mc)\
**Post date:** [May 20, 2024, 7:02am UTC](https://cylc.discourse.group/t/cylc-job-hanging-and-failing-to-submit/958/3 "2024-05-20T07:02:44Z")

</div>

Thanks for the help, I’ll contact my local support