# Cylc not capturing failed jobs **URL:** \ **Category:** Cylc Support\ **Created:** [August 28, 2019, 7:36pm UTC](https://cylc.discourse.group/t/cylc-not-capturing-failed-jobs/135 "2019-08-28T19:36:18Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Jim](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jim/32/60_2.png) [@Jim](https://cylc.discourse.group/u/Jim)\ **Post date:** [August 28, 2019, 7:36pm UTC](https://cylc.discourse.group/t/cylc-not-capturing-failed-jobs/135/1 "2019-08-28T19:36:18Z")
I am trying to retry a failed job with cylc on a pbs system. But the cylc graphic and cat-state output both show the job as submitted even after the job has already aborted. ```auto 34: Lagrangian levels are crossing 9.999999999999999E-012 34: Run will ABORT! 34: Suggest to increase NSPLTVRM 34: ERROR: te_map: Lagrangian levels are crossing 34:Image PC Routine Line Source 34:cesm.exe 000000000375022D Unknown Unknown Unknown 34:cesm.exe 0000000002EBB682 shr_abort_mod_mp_ 114 shr_abort_mod.F90 34:cesm.exe 0000000000F253D9 te_map_mod_mp_te_ 512 te_map.F90 34:cesm.exe 0000000000569332 dyn_comp_mp_dyn_r 2621 dyn_comp.F90 34:cesm.exe 0000000000EE7CA7 stepon_mp_stepon_ 315 stepon.F90 34:cesm.exe 00000000004EC8FF cam_comp_mp_cam_r 243 cam_comp.F90 34:cesm.exe 00000000004DE7ED atm_comp_mct_mp_a 454 atm_comp_mct.F90 34:cesm.exe 0000000000425A44 component_mod_mp_ 728 component_mod.F90 34:cesm.exe 000000000040CABD cime_comp_mod_mp_ 3436 cime_comp_mod.F90 34:cesm.exe 00000000004256EC MAIN__ 125 cime_driver.F90 34:cesm.exe 0000000000408E5E Unknown Unknown Unknown 34:libc.so.6 00002B0DB7DC56E5 __libc_start_main Unknown Unknown 34:cesm.exe 0000000000408D69 Unknown Unknown Unknown 34:MPT ERROR: Rank 34(g:34) is aborting with error code 1001. 34: Process ID: 71751, Host: r8i5n12, Program: /glade/scratch/jedwards/70Lwaccm6.00/bld/cesm.exe 34: MPT Version: HPE MPT 2.19 02/23/19 05:30:09 34: -1: aborting job ^Ccheyenne6: /glade/work/jedwards/cases_S2S/70Lwaccm6.00 :) cylc cat-state testsuite run mode : live time : 2019-08-28T19:27:19Z (1567020439) initial cycle : None final cycle : None (dp0 . Begin task states st_archive.19990802T0000Z : status=waiting, spawned=False get_data.19990802T0000Z : status=succeeded, spawned=True get_data.19990809T0000Z : status=waiting, spawned=False run.19990802T0000Z : status=submitted, spawned=True run.19990809T0000Z : status=waiting, spawned=False ``` ```auto ----- suite.rc ------- [meta] title = test CESM CYLC workflow for 70Lwaccm6.00 [cylc] UTC mode = True [scheduling] initial cycle point = 19990802T0000Z final cycle point = 19990830T0000Z [[dependencies]] [[[R1]]] graph = "get_data => run => st_archive " [[[R/P1W]]] # Weekly Cycling graph = """ st_archive[-P1W] => get_data => run run => st_archive """ [runtime] [[get_data]] script = cd /glade/work/jedwards/sandboxes/CESM2-Realtime-Forecast/bin/; ./s2srun.py [[st_archive]] script = cd /glade/work/jedwards/cases_S2S/70Lwaccm6.00; ./case.submit --job case.st_archive; ./xmlchange CONTINUE_RUN=TRUE [[run]] script = cd /glade/work/jedwards/cases_S2S/70Lwaccm6.00; ./case.submit --job case.run [[[job]]] execution retry delays = PT0S, PT10S, PT20S, PT30S batch system = pbs batch submit command template = qsub -q regular -l walltime=04:00:00 -A P93300606 '%(job)s' [[[directives]]] -r = n -j = oe -V = -S = /bin/bash -l = select=15:ncpus=36:mpiprocs=36:ompthreads=1 [[[events]]] handlers = cp ../user_nl_cam.${CYLC_TASK_TRY_NUMBER} user_nl_cam handler events = retry, failed ``` ---
**Author:** ![Jim](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jim/32/60_2.png) [@Jim](https://cylc.discourse.group/u/Jim)\ **Post date:** [August 28, 2019, 7:42pm UTC](https://cylc.discourse.group/t/cylc-not-capturing-failed-jobs/135/2 "2019-08-28T19:42:35Z")
What I really want to happen is to copy a new inputfile into the run directory and then restart the run, I’m not sure if what I have in the suite.rc is going to do that correctly even if cylc correctly detects the failure. Any suggestions? thanks ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [August 28, 2019, 7:53pm UTC](https://cylc.discourse.group/t/cylc-not-capturing-failed-jobs/135/3 "2019-08-28T19:53:17Z")
Hi Jim, If your Cylc server program thinks a job is still “submitted” even though it is running or has completed (succeeded or failed) that implies that job status messages are not able to be sent back from the job host to the suite host. If you look at the job.err file for the task you should see evidence of a bunch of message send failures. If it’s just a naming issue (the suite host server has to self-identify as the same name or IP address that is seen from the job host) see [https://cylc.github.io/doc/built-sphinx/appendices/site-user-config-ref.html#suite-host-self-identification](https://cylc.github.io/doc/built-sphinx/appendices/site-user-config-ref.html#suite-host-self-identification) However, it is usually a network configuration issue: the suite host server (and Cylc ports) must be visible from the compute node that PBS runs your job on. You can test this with a job that tries to `ping` (or whatever) the suite host. If this is the problem, you either have to get you system admins to allow this job-to-suite communication, OR configure Cylc to use suite-to-job-host polling to update job status. Polling is not as good because you only find out true job status after each polling interval, but it may be good enough if you can’t get reverse comms back over the network. Hilary ---
**Author:** ![Jim](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jim/32/60_2.png) [@Jim](https://cylc.discourse.group/u/Jim)\ **Post date:** [August 29, 2019, 1:33pm UTC](https://cylc.discourse.group/t/cylc-not-capturing-failed-jobs/135/4 "2019-08-29T13:33:40Z")
I was able to fix that with suite-host-self-identification. The retry now occurs, but the handler that I want to move a new namelist into place doesn’t work, how do I go about that step? ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [August 29, 2019, 8:14pm UTC](https://cylc.discourse.group/t/cylc-not-capturing-failed-jobs/135/5 "2019-08-29T20:14:58Z")
That’s good. > The retry now occurs, but the handler that I want to move a new namelist into place doesn’t work, how do I go about that step? Are you saying that you’ve attached a handler to the retry event, but it doesn’t run at all, or that it doesn’t do what it is supposed to do before the retrying task runs? I’ll guess the latter, since event handler execution should not be broken (it is well tested). Note that event handlers are reactive - they are executed by the suite server program, on the suite host, in response to an event - not on the job host before the event occurs. So you can’t use a retry handler to modify the configuration of a task before it retries. You can do that sort of thing by accessing the try number in job scripting, as described here: [https://cylc.github.io/doc/built-sphinx-single/index.html#task-retry-on-failure](https://cylc.github.io/doc/built-sphinx-single/index.html#task-retry-on-failure) Hilary ---
**Author:** ![Jim](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jim/32/60_2.png) [@Jim](https://cylc.discourse.group/u/Jim)\ **Post date:** [August 29, 2019, 8:52pm UTC](https://cylc.discourse.group/t/cylc-not-capturing-failed-jobs/135/6 "2019-08-29T20:52:49Z")
Yes, I have it working now. Thanks,