Parametrised cycle durations: consistency cf next([...])

Cylc Cylc Support
1

When developing to-become-operational suites, what do people do regarding any cycle duration wallclock limit caps on queues for non-operational work?

I’ve been happily developing a suite intending to target a 6 hour cycle - I’m now getting to stage where I’d like to match the target operational cycle length in some of my tests. I’ve just realised local caps mean this value isn’t catered for in suite dev/testing - I can only go up to 3 hours.

I think I could have options including:

  • Aim for a shorter operational cycle duration, which fits within acceptable dev durations (e.g. 3h)
    • Can do, but would prefer 6H for archiving retrieval - this is a nowcast model, not forecast, so want to minimise archive retrievals needed to reconstruct events
  • Try to parametrise run duration, such that I have Jinja magic letting me switch durations depending on context (3h for dev, changing to suck-it-and-see/pray 6h works when I throw it over to the operational team)

What do people with 6h cycling operational suites (e.g. UM, no?) do?

If it’s parametrised run duration, does anyone have a handy link to a suite implementing the required Jinja incantations - I’m particularly keen to see what people do about normalising (correct term?) start times wrt a presumably ~random start time - going from initial cycle point = now behaviour to initial cycle point = next([...]) behaviour.

For the actual cycle duration, I can see I can ~happily use Jinja to achieve sth like @funkapus’ parametrised run duration/cadence.

But I’m less clear if/how I can link this cadence up to the corresponding next() incantation like Matt’s example here.
Specifically, I can’t see if there’s a way to automagically generate:

  • next(T00, T06, T12,T18) from CYCLE_DURATION="PT6H"
  • next(T00, T03, ..., T21) from CYCLE_DURATION="PT3H"
  • … other next variants from other durations, with something to ensure this folds into 24H with no remainder

Is this technically achievable? E.g. via a cunning call in Jinja of cylc cyclepoint as discussed here by @hilary.j.oliver & @jonnyhtw ? Has anyone got a shareable example of doing this?

Not to worry if not possible - just trying to avoid my other non-DRY solution: defining / maintaining consistency between two implicitly linked Jinja vars, a la

  • CYCLE_DURATION="PT3H"
  • CYCLE_NEXT="next(T00, T03, ..., T21)"

MO people: I’m linking to this q on our internal channels in case there’s a specific internal answer to this!

Extra brownie points for examples which do duration arithmetic - if my cycle nominally lasts PT6H, I’d like to be able to define padded execution time limits, e.g. PADDED = (110% * NOMINAL) + PAD_C, with padding “slope” (110%) and offset being fine-tuneable once I understand how my model performs at different durations.

Thanks,
Edmund

2

Specifically, I can’t see if there’s a way to automagically generate:

Here’s a Jinja2 filter that can convert a duration (e.g. PT6H) into times of the day (e.g. T00, T06, T12, T18).

from isodatetime.data import Duration, TimePoint
from isodatetime.parsers import TimeRecurrenceParser, TimePointParser, DurationParser
    
        
def duration_to_time(duration, offset='00'):                       
    """Generates times of the day from a duration.

    Works with durations that are shorter than one day. 
    
    Arguments:
        duration (str):
            The duration as a string e.g. "PT3H".
        offset (str):
            The offset from which to count,          
            (i.e. the earliest time to consider). 
            e.g. "00" (midnight, the default)

    Returns:
        str
    
    Examples:
        # every six hours starting at T00
        >>> list(duration_to_time('PT6H'))
        ['T0000', 'T0600', 'T1200', 'T1800']
        
        # every six hours starting at T0030
        >>> list(duration_to_time('PT6H', '0030'))
        ['T0030', 'T0630', 'T1230', 'T1830']
    """
    tp1 = TimePointParser().parse('20000101T' + offset + 'Z')
    tp2 = TimePointParser().parse('20000102T00Z')
    dur = DurationParser().parse(duration)   
    while tp1 < tp2:
        yield 'T%02d%02d' % (tp1.hour_of_day, tp1.minute_of_hour)
        tp1 = tp1 + dur
1 Like
3

Fantastic, thanks Oliver!
I’ve not come across Jinja2 filters before - trying to mug up on them now as it’s looking like this isodatetime library can likely solve my secondary question about duration arithmetic for setting padded execution time limits.

Can I check:

  • Will this work with Cylc7 (I need this in an operational suite context, PS45)?
  • Does cylc give me access to the isodatetime library at runtime?
    • I see I can’t simply do an import isodatetime successfully from within the Met Office’s standard “scientific software stack” (at least the default version - assume same for operational flavours?)
    • I see I can install isodatetime from conda if I want to extend my own environment for Jinja filter development purposes - sadly though I can’t use my own env in operations
    • Guessing/hoping cylc/rose must rely on this library, so I can safely rely on this being available to my custom Jinja2 filters at runtime?
  • If I’ve written one of these Jinja2 filters for my suite, and want to put this in a file (rather than inlining in suite.rc), where should I put this?

* I think there may be something which has gone missing from the Cylc8 docs at that point:

Cylc also supports custom Jinja2 globals, filters and tests. A custom global, filter or test is a single Python function in a source file with the same name as the function (plus .py extension) and stored in one of the following locations:

In the argument list[…]

I.e. I think there’s a set of missing locations which were intended to be listed between

[…] locations:

and

In the argument list[…]

4

Aha - re path, think found the relevant sections of Cylc7 docs which suggests it’s ${my-suite}/lib/python/ (or poss ${my-suite}/lib/Jinja2Filters/ - only former mentioned in Cylc8 equivalent to 2. though, so assume preferred):

  1. Cylc7: Custom jinja2 filters
  2. Cylc7: Suite configuration dirs
5

Can I check:

  • Will this work with Cylc7 (I need this in an operational suite context, PS45)?

Yes

  • Does cylc give me access to the isodatetime library at runtime?

Yes

…

  • Guessing/hoping cylc/rose must rely on this library, so I can safely rely on this being available to my custom Jinja2 filters at runtime?

:+1:

  • If I’ve written one of these Jinja2 filters for my suite, and want to put this in a file (rather than inlining in suite.rc), where should I put this?..
    I.e. I think there’s a set of missing locations which were intended to be listed between …

You’re right about the Cylc 8 docs, which are still undergoing heavy revision, but the Cylc 7 docs have it (as I think you’ve noted in your follow-up post):

Cylc also supports custom Jinja2 globals, filters and tests. A custom global, filter or test is a single Python function in a source file with the same name as the function (plus “.py” extension) and stored in one of the following locations:

  • <cylc-dir>/lib/Jinja2[namespace]/
  • [suite configuration directory]/Jinja2[namespace]/
  • $HOME/.cylc/Jinja2[namespace]/

where [namespace]/ is one of Globals/ , Filters/ or Tests/ .

1 Like
6

Here’s an example integrated into a toy suite:

1 Like
7

Thanks very much Oliver for creating this - FWIW, as a user, toy suites like this / ones @wxtim has produced are super-useful for the noobier/dimmer amongst us for making link between description in documentation and how one can implement things.
And nicely runnable to boot - documplementation?

8

I doubt anyone here should be described as “dimmer”, and I dislike the term “noob” because everyone has to start somewhere.

But

Some of those giving customer support still need to write toy examples to be sure that they give sensible advice. :slight_smile:

2 Likes