Let’s say I want to run a simulation from 2000-01-01 to an arbitrary date, with a period of several days (here three days). After the simulation is finished, I want to post-process the results.
The flow may look like:
[scheduling]
initial cycle point = 20000101
final cycle point = {{end_date}}
[[graph]]
P3D = compute_period[-P3D] => compute_period
R1/$ = compute_period => post-proc
My problem: If the difference between the final point and the initial point is not a multiple of the cycle period (3D), then R1/$ will launch a compute_period task without any dependencies.
How can I have the task run at the final cycle, regardless of the period ?
You’ve spotted something that’s a little tricky to achieve with Cylc.
The simplest solution is to use final cycle point constraints to ensure that the final cycle point is always set to a value that works for your workflow. This avoids the issue of the “ragged” end cycle.
If the “ragged” end cycle is a requirement of the workflow, there is a sneaky solution that is suitable if the task is trivial, use the shutdown handler. Not generally reccomended, but sometimes used to generate workflow reports.
Beyond that there is a Jinja2 solution but it’s a little more advanced. Let me know if you would be interested in that, if so I’ll write it up tomorrow.
The simplest solution is to use final cycle point constraints to ensure that the final cycle point is always set to a value that works for your workflow.
I’m not sure I understand how that works (and the documentation doesn’t give an example). If the end date is several months/years in the future, even I don’t know what a valid end date will be.
Beyond that there is a Jinja2 solution but it’s a little more advanced. Let me know if you would be interested in that, if so I’ll write it up tomorrow.
That final inter-cycle trigger is what tells the final foo which previous task to wait on. If your final cycle point is variable, you’ll have to compute the final trigger offset using Jinja2 (I presume that’s what Oliver will suggest).
Another suggestion - leave your final point on the P3D sequence. At runtime the task job can check if it is at the final point or not (compare $CYLC_TASK_CYCLE_POINT with $CYLC_TASK_FINAL_CYCLE_POINT) and if it is, just use the “correct” value instead for any cycle point based computations, filenames, or whatever. (Of course the Cylc job log directory path will still reflect the automatic “wrong” value).
(As an aside, a future Cylc release will allow a distinct shutdown graph that runs after the main cycling graph finishes, regardless of dependencies - that work was done a while ago but it remains trumped by higher priorities for now).
There are examples for the initial cycle point constraints in the docs, but not the final cycle point constraints, but the syntax is the same. The constraints allow you to set rules which the initial/final cycle points will be validated against e.g:
# FCP must be on the first of Jan at 00:00
final cycle point constraints = 0101T00
# FCP must be the 1st, 4th or 7th of the month
final cycle point constraints = 01, 04, 07
Here’s the Jinja2 solution:
flow.cylc
#!Jinja2
{% set start_date = '2000-01-01T00Z' %}
{% set end_date = '2000-01-08T00Z' %}
{% set recurrence = 'P3D' %}
{% from "offset" import get_fcp_offset %}
{% set fcp_offset = get_fcp_offset(start_date, end_date, recurrence) %}
[scheduler]
allow implicit tasks = True
[scheduling]
initial cycle point = {{ start_date }}
final cycle point = {{end_date}}
[[graph]]
{{ recurrence }} = compute_period[-{{ recurrence }}] => compute_period
R1/$ = compute_period[-{{ fcp_offset }}] => post-proc
lib/python/offset.py
from metomi.isodatetime.datetimeoper import DateTimeOperator
def get_fcp_offset(start_date, end_date, recurrence):
"""Return the offset between the final cycle of the recurrence and the
final cycle point.
Args:
start_date:
The start point of the recurrence, e.g. the workflow's
initial cycle point, in ISO8601 format.
end_date:
The end point of the recurrence, e.g. the workflow's
final cycle point, in ISO8601 format.
recurrence:
The cycling recurrence in ISO8601 format, e.g. P3D
Returns:
An ISO8601 offset string.
"""
oper = DateTimeOperator()
start_date = oper.time_point_parser.parse(start_date)
end_date = oper.time_point_parser.parse(end_date)
cycle_generator = iter(
oper.recurrence_parser.parse(
f'R/{start_date}/{recurrence}'
)
)
date = last_date = start_date
while date <= end_date:
last_date = date
date = next(cycle_generator)
return end_date - last_date
