What could stop job.status updating?

Cylc Cylc Support
1

I’ve not seen this before, but I found a job.status file which never updated with INIT nor EXIT times. Nor is there a CYLC_JOB_EXIT=SUCCEEDED in there which I expected.

$ cat job.status
CYLC_JOB_RUNNER_NAME=pbs
CYLC_JOB_ID=2073588
CYLC_JOB_RUNNER_SUBMIT_TIME=2025-06-17T03:08:55Z

The scheduler log shows it starting at the right time and ending the right time.

2025-06-17T03:08:55Z INFO - [20250617T0000Z/ph_um_fcst_00/01:preparing] submitted to platform:pbs[2073588]
2025-06-17T03:08:55Z INFO - [20250617T0000Z/ph_um_fcst_00/01:preparing] => submitted
2025-06-17T03:09:06Z INFO - [20250617T0000Z/ph_um_fcst_00/01:submitted] => running
2025-06-17T03:19:06Z INFO - [20250617T0000Z/ph_um_fcst_00/01:running] poll now, (next in PT10M (after 2025-06-17T03:29:06Z))
2025-06-17T03:19:17Z INFO - [20250617T0000Z/ph_um_fcst_00/01:running] (polled)submitted
2025-06-17T03:19:17Z INFO - [20250617T0000Z/ph_um_fcst_00/01:running] submitted to platform:pbs[2073588]
2025-06-17T03:29:07Z INFO - [20250617T0000Z/ph_um_fcst_00/01:running] poll now, (next in PT10M (after 2025-06-17T03:39:07Z))
2025-06-17T03:29:09Z INFO - [20250617T0000Z/ph_um_fcst_00/01:running] (polled)submitted
2025-06-17T03:29:09Z INFO - [20250617T0000Z/ph_um_fcst_00/01:running] submitted to platform:pbs[2073588]
2025-06-17T03:37:38Z INFO - [20250617T0000Z/ph_um_fcst_00/01:running] => succeeded

This was Cylc 8.3.6.

2

If the job was run on a remote platform, check the job.status file on the remote filesystem. It might have been synced whilst a write operation was in progress?

Also check the job.err file for evidence of failure to write to the file (A Python traceback likely mentioning DiskIOError).

3

The file I looked at was on the remote machine.

I had not seen anything of in the job.err file, although I only looked at the end, around the succeeded cylc message. If the logs are still around in a couple of days I can look again, but I can’t remember if we keep those ones.