We are trying to capture events from Cylc suite while running it with batch system as pbs. Even though we mentioned multiple events to capture (handler events = submitted, submission failed, submission timeout, submission retry, started, succeeded, warning, critical, failed, retry), we were able to capture only, submitted event.
Here is our suite.rc file.
[meta]
# Suite metadata.
title = TEST SUITE TITLE
url = www.testsuiteurl.com
uuid = 148ec4de-4a89-11e9-8646-d663bd873d84
priority = HIGH
color = blue
[cylc]
UTC mode = True
[scheduling]
initial cycle point = "now"
[[special tasks]]
clock-trigger = bar(+PT1M)
[[dependencies]]
[[[PT1M]]]
graph = bar
[runtime]
[[bar]]
script = "qsub -- /bin/sleep 1"
[[[job]]]
batch system = pbs
[[[meta]]]
# Task metadata.
title = TEST TASK TITLE
priority = LOW
url = www.testtaskurl.com
omit = false
color = red
[[[events]]]
handler events = submitted, submission failed, submission timeout, submission retry, started, succeeded, warning, critical, failed, retry
handlers = handler.py event=%(event)s suite=%(suite)s point=%(point)s task_name=%(name)s submit_num=%(submit_num)s id=%(id)s suite_url=%(suite_url)s batch_sys_name=%(batch_sys_name)s batch_sys_job_id=%(batch_sys_job_id)s start_time=%(start_time)s end_time=%(finish_time)s uuid=%(suite_uuid)s user_at_host=%(user@host)s msg=%(message)s
Below are events that were captured by handler.py.
{"event": "submitted", "suite": "Acc_upd", "point": "20190709T1207Z", "task_name": "bar", "submit_num": "1", "id": "bar.20190709T1207Z", "suite_url": "www.testsuiteurl.com", "batch_sys_name": "pbs", "batch_sys_job_id": "19.pbsserver", "start_time": "None", "end_time": "None", "uuid": "148ec4de-4a89-11e9-8646-d663bd873d84", "user_at_host": "pbsuser@localhost", "msg": "job submitted"}
{"event": "submitted", "suite": "Acc_upd", "point": "20190709T1209Z", "task_name": "bar", "submit_num": "1", "id": "bar.20190709T1209Z", "suite_url": "www.testsuiteurl.com", "batch_sys_name": "pbs", "batch_sys_job_id": "20.pbsserver", "start_time": "None", "end_time": "None", "uuid": "148ec4de-4a89-11e9-8646-d663bd873d84", "user_at_host": "pbsuser@localhost", "msg": "job submitted"}
{"event": "submitted", "suite": "Acc_upd", "point": "20190709T1210Z", "task_name": "bar", "submit_num": "1", "id": "bar.20190709T1210Z", "suite_url": "www.testsuiteurl.com", "batch_sys_name": "pbs", "batch_sys_job_id": "21.pbsserver", "start_time": "None", "end_time": "None", "uuid": "148ec4de-4a89-11e9-8646-d663bd873d84", "user_at_host": "pbsuser@localhost", "msg": "job submitted"}
When we tried with the batch system as background, we were able to capture all events.
Any ideas on where is it going wrong and what can I do to debug it further?
Amaresh C.