We now have several applications with the following use case, where each cycle of the suite will contain the need to flexibly rewind a particular graph subsection several cycles ago.
- A data-assimilative cycling suite with dependencies on both incoming observations (divided between low and high latency) and upstream model forcing (low latency)
- Product delivery demands that the suite runs with wallclock delays consistent with the low latency observations and upstream model forcing.
- At the beginning of each cycle, go back a specified time window and cycle qc/da/forecast tasks incorporating the higher-latency observations
- The results of the “rewound” analyses including both high and low-latency data form the basis for the current cycle’s forecast initial condition.
This has come up both for specialized cycles with cycling intervals as short as 15 minutes (where the “high latency” is only several hours) to other more traditional PT6H-cycling suites with much longer latency (24+ hours) data.
What’s a canonical way that others have handled this? I’ve toyed with the idea of having a task that registers a separate suite that would have the suite controller run non-daemonized within a task and submit its own jobs, but that makes monitoring tough. I know there are some discussions about things in the works for future versions which should make this cleaner, but for now this is in the 7.x world.