Some functionality I created in the xtriggers I’ve created is to only start really running it after a certain interval after the cycle point, and also to end after a certain time period (whether that be time from the first time it ran, or time from the cycle point). It works fine, but I feel some of this could be baked into xtrigger syntax and supplied by default by Cylc and could use similar syntax to cycles in graphs.
For example, something like
MY_TRIGGER = something(...):PT2M/PT5H/PT6H could be poll eveyr 2 minutes starting 5 hours after the cycle point until 6 hours after the cycle point. If only one interval is supplied, its the polling interval, if two options, polling interval and delay start.
If its an integer cycling suite, then it could be that only two items are allowed,
PT2M/PT5H could then mean poll every two minutes until 5 hours after the first time it polled.
I’m not sure the best way the return information could be handled in Cylc for downstream tasks, my xtrigger/decorator approach forces the success to be true if the timeout period has expired, and the result dictionary would have a
'success':False key/value pair so tasks can know that whilst the task was triggered, it was a timeout that caused it. Some other forced in variable could be used instead though (e.g.
'CYLC_XTRIGGER_TIMEOUT'=True which tasks could check for/value of).
Why did we need this sort of functionality?
- Delaying start - if you’re polling for files in a remote system (delivered there by another system which cannot talk to Cylc or any message broker you have access to), then you may not want to start until closer to the time you expect the data to arrive if its a routine schedule. Its low network load, but it is not required load.
- Timeout after an interval from a cycle starting - We have cyclic ondemand systems. They can only be activated during certain time windows each day. Whilst you can have a task/trigger looking for the request and another as a wallclock which triggers a task which suicides the other one, we felt that was unnecessary graph bloat when it could be handled simply in an xtrigger and a small if block in a script.
- Timeout period after first poll - this was just me being paranoid. If you’re again, polling a remote system, and something is wrong with the polling (e.g. maybe SSH keys have expired), you won’t know there is a problem as the xtrigger hides the information. So, expire the xtrigger periodically so the following task can check the access to the remote host is still there, and if not fail and alert support staff.
Whilst on this topic, I don’t know if/how xtriggers have changed in Cylc8, but obviously the
| operator would be nice to have available, but also being able to do
task[-P1] => @my_trigger => task would be useful too. In this case, what I’m trying to suggest is that task should only run if the preceeding task has finished, but, the xtrigger which triggers task should also only run once the previous task has finished. Why? Ondemand systems and making sure the xtrigger doesn’t accidently pick up on the same request as the previous cycle.
Anyway, just some thoughts, it would be good to see what others think, or if some of the issues we’ve worked through have already been resolved/improved in Cylc8.