Wall clock synchronization (real time scheduling)

Cylc Cylc Support
1

Hi All,

This could be a simple question for those who know.
Basically I want a suite where multiple tasks run periodically, and is linked/synced with real clock time (time on that computer).
After a few tries, I think the below works (or does it?)

  1 #title="spt_s21"
  2 [cylc]
  3    UTC mode = False
  4 [meta]
  5 # Suite metadata.
  6   title = SPT suite21
 11   description = """
 12   This is a long  description.
 13   Still continuing.
 14   """
 15   description = This is short suite21 description
 16
 17 [scheduling]
 18   initial cycle point = 20190605T04
 19   final cycle point =   20190605T05
 20   #[[special tasks]]
 21   #   clock-trigger = wave1(PT0H1M)
 22   [[dependencies]]
 27      [[[ T0405, T0410, T0415, T0420, T0425, T0430, T0435, T0440, T0445, T0450, T0455   ]]]
 28      graph = """
 29        @wall_clock => s21t1
 30        s21t1 => s21t2 => s21t3 => s21t4
 31              """
# .........  blah ..............

The suite is required to run from the start of suite-submission, every 5 minutes to the hour of wall clock time, until the final cycle point.

Specific questions.

  1. Apparently, I cannot remove/leave-out “initial cycle point” (ICP).
    Putting the ICP is not desirable. For example now is 4:23pm. The tasks need to re-run every 5 minutes.
    I would prefer not to specify ICP, so when I start suite at 4:23pm, the tasks will start running at 0425.
    If I take a guess and buffer, I state the ICP at 0400, then it will run historical tasks at 0405, 0410, 0415, 0420 - which is not my intention.
    What are the alternatives?

  2. [[[ T0405, T0410, T0415, T0420, T0425, T0430, T0435, T0440, T0445, T0450, T0455 ]]]
    Instead of above, can I replace by single [[[ PT0005 ]]]?
    I tried [[[ PT00H05M ]]] and it failed the cylc syntax check.
    Intention is to run every 5 minutes on the hour, eg 4:25, 4:30, etc.
    (Do not intend to run at 4:23, 4.28,…)

Thanks
Clinton

2

Hi Clinton,

After a lot of discussion between the development team, it looks like the best answer looks like this:

[scheduling]
    initial cycle point = next(T-00; T-05; T-10; T-15; T-20; T-25; T-30; T-35; T-40; T-45; T-50; T-55)
    [[dependencies]]
        [[[PT5M]]]
             graph = task1

Note:

  1. The - in T-00, etc is a placeholder for the hour, not a minus sign.
  2. This should work the way you expect. If current time is 16:23, the initial cycle point will become 16:25. The next(...) syntax is explained in the user guide: https://cylc.github.io/doc/built-sphinx-single/index.html#scheduling-initial-cycle-point-initial-cycle-point-relative-to-current-time
  3. This is most efficient for Cylc, because your graph will have a single recurrence, instead of 12 recurrences.

Matt

1 Like
3

I’ll add to Matt’s answer because I think our next syntax isn’t explained quite clearly enough in the documentation (at least it confused me too for a bit). To be explicit (I hope):

next(T-05) (for example) means the next (relative to “now”) point that is “5 mins past the hour” for any hour. So for now=16:04 (HH:mm) it will be 16:05; and for now=16:07 it will be 17:05.

Note also @chee_cylc that you will still need the wall_clock clock triggers to tie your date-time cycle points to real time (on the tasks that trigger off them, that is) - I see Matt didn’t use them in his example.

Hilary

4

(Good point on the efficiency @matthewrmshin of doing it this way versus mulitple recurrence relations in the graph).

5

Thanks Matt and Hilary,
I’ll test it out soon.

6

So the T-05 does have a literal ‘-’ minus symbol. But I’m confused about why @matthewrmshin said it was a placeholder:

The - in T-00 , etc is a placeholder for the hour, not a minus sign

7

Answering my own question after looking at Cylc doc:
So “T-00” the symbol “-” is both a placeholder for hour AND a minus sign on the keyboard (though it does not represent the mathematical meaning of minus).

8

[scheduling]
initial cycle point = next(T-00; T-05; T-10; T-15; T-20; T-25; T-30; T-35; T-40; T-45; T-50; T-55)
[[dependencies]]
[[[PT5M]]]

----- @matthewrmshin - in your example above, is the [[[ PT5M ]]] still necessary? From reading the doc, it seems to suggest the next(…) as specified above will make it run in 5mins multiples of every hour. Is the [[[ PT5M ]]] still necessary?

Thanks

Answered this by testing. It appears the [[[ PT5M ]]] is needed, otherwise cylc will complain that the [[Dependencies]] section is not complete or something. After adding this back in, the suite ran, at least the first section T-00 so far.

In general, would this be redundant and also cause unintended errors, if the user does not consistently specify next(…) AND [[[ PT… ]]] ?
What is the initial next(…) is like above (every 5mins), but the [[[ PT12M ]]]? I imagine it skips to multiple of 12 mins, but also need to wait for the ‘next’ closest T-xx?

9

Sorry. What I meant was that the usage of the - symbol here is as if it is a placeholder for the hour of the day, instead of denoting negativeness.

For the other question.

The initial cycle point gives you the first point of the cycling. Whereas the [[[PT5M]]] gives you the cycling steps.

To explain using a C-like for-loop:

for (i = icp; i <= fcp; i += step) {
    /* do stuff */
}
10

Thanks Matt, Sorry to push this a bit further.
In the C example, there is basically, (start, end, step)
whereas
initial cycle point = next(T-00; T-05; T-10; T-15; T-20; T-25; T-30; T-35; T-40; T-45; T-50; T-55)
… looks like it’s steps already in next(), then we need another step specification.
[[[ PT5M ]]]

Unless the idea is like this,
initial cycle point = next(T-00; T-10; T-20; T-30 ) # COURSE stepping
…
[[[ PT2M ]]] # finer-grain steps
… the result is 00m, 02m, 04m, 06m 08m, 10m, 12m, …30m,
will there be 32m, 34m,???

On the other hand, it is pointless to have
initial cycle point = next(T-00; T-10; T-20; T-30 ) # COURSE stepping
…
[[[ PT15M ]]] # finer-grain steps
… because this is step by 15 mins, but the initial point is every 10m up to 30m?

Or maybe I’m missing something deeper in the meaning of initial cycle point?

11

Hi Clinton,

The initial cycle point = next(...) is a function to return a single value for the start of the loop. It does not give you the step. The syntax may not be cleanest, but we can improve on that. In the mean time, the recommended syntax is probably the most efficient (and readable) as we can come up with.

Matt

12

Hi Clinton,

This is not quite as straightforward as you probably expected because you are trying to use Cylc for real time scheduling, as if it were cron.

In fact you could literally use cron to repeatedly run your (single-cycle) Cylc workflow at 5 min intervals. (Although if you have any dependence between cycles, or ever have to quickly run historical cycles “off the clock” to catch-up from delays or to process historical data, then forget about cron).

Cylc is primarily designed for cycling, which is a broader concept that can blend into real time scheduling at one end of the spectrum, if/when there is any connection to real time at all. I might attempt a clear explanation of the difference in a new thread, sometime soon, as it is a common source of confusion.

For the moment, you can easily force Cylc to demean itself (he heh :slight_smile: ) to plain old real time scheduling as we showed. It just requires (a) appropriate date-time clock-triggers to tie the cycle points of some tasks to the real time clock; and (b) understanding how to set the initial cycle point (at suite start-up) to be the next real time schedule point after “now”.

And note that (b) is not as simple as it sounds. In your case you always want to start from the next real time schedule point of a single cycle, which makes the next(...) arguments look superficially redundant with PT5M cycling interval - but you could want to start from some subset of the available schedule points, or you could have multiple different cycling intervals in the workflow, with or without clock-triggers … so I’m not convinced we could make initial cycle point definition any simpler for cron-like use without losing capability.

Hilary

13

Another way to understand this: to do real time scheduling with Cylc:

  • first make a cycle with the same interval as your desired real time schedule;
  • then tie the cycle down to real time, by attaching clock-triggers to one or more tasks;
  • then ensure that no matter when you push play, the first cycle point will be the same as the next real time schedule point (by using our initial cycle point = next(...) syntax - where “next” is relative to “now”).

Is this more difficult than using cron for real time scheduling? (Ignoring dependency graphs for the moment, which you can’t do with cron, of course). Yes it is more difficult! But that’s because Cylc is designed to do something much more flexible and general than this - i.e. date-time cycling - and you have to put some effort in to stop it from doing its usual magic.

(Note I’m trying to come up with the best way to explain this, because I’ve noticed that people who don’t have an atmospheric science background - where date-time cycling is drilled in to you at an early age - often assume that Cylc is supposed to do real time scheduling of workflows, and they struggle to understand what the difference is).

14

Raised https://github.com/cylc/cylc-flow/issues/3187

15

And: https://github.com/cylc/cylc-flow/issues/3188

1 Like
16

Following up with an explicit example that I hope will clearly illustrate how “cycling” relates to “real time scheduling”. We might add this to the User Guide in due course:

Firstly, the following suite runs a small workflow a => b & c on a PT10M (10 minute) non real-time cycle:

# (free cycling suite - no connection to real time)
[scheduling]
   initial cycle point = 20500101T0125
   [[dependencies]]
      [[[PT10M]]]
         graph = "a => b & c"
[runtime]
   [[root]]
      script = "sleep 10"

Note the initial cycle point (ICP) is at 01:25 on Jan 1 2050, to ram home the lack of connectedness to real time here - this suite will start running immediately, it won’t wait until 2050! Cylc will generate a sequence of date-time cycle points starting from 20500101T0125, on a 10 minute interval, and label the tasks with these cycle points (e.g. the first instance of a will be a.20500101T0125, then a.20500101T0135, and so on). Why is there no connection to real time? Well, a might be an atmospheric model (not just sleep 10 as here) that you want to simulate the climate in 2050 (or whatever) without actually waiting till 2050 to do it.

Now, we could replace the ICP definition with this:

initial cycle point = now

This generates the ICP according to the value of real time “now” (which is, whenever you start the suite running), but it’s still just a date-time label like “2050…” above - there’s still no connection to real time in terms of when the tasks submit their jobs to run.

To make tasks depend (i.e. wait) on the real time clock, do this:

graph = "@wall_clock => a => b & c"

This means, a is free to run when the real date-time is greater than or equal to a's complete date-time cycle point value (date and time, not just time of day). So with this “clock trigger” present, if the ICP is in 2050 nothing will happen until the (real) year 2050 arrives. But if the ICP is in the past (e.g. 1950) many cycles will run concurrently (as many as you and/or your compute resources allow) until the cycle point values catch up to the real date and time, at which time the clock trigger will start constraining the suite. This is a feature - it is what enables real time Cylc suites to catch up very quickly from delays without skipping over the intermediate processing, and then return to running “on the clock” as normal.

If you just want “simple real time scheduling” in the sense of “run a given workflow every 10 minutes, starting from NOW” (say) then you just have to combine clock triggers with initial cycle point = now. Then the first cycle point will be literally now (the time at which you start the suite running) and will run immediately; the second will be now + PT10M and will be delayed by 10 minutes in real time; and so on. The final optional step is to use next(...) instead of now to constrain the initial cycle point to be at the next clean multiple of 10 minutes past the hour, regardless of when you start the suite running, instead of literally “now”. This syntax is explained in a previous post above. Result:

# (cycling suite tied to clean real time intervals)
[scheduling]
   # start cleanly on the hour, or 10 min past, or 20 min past, etc.
   initial cycle point = next(T-00; T-10; T-20; T-30; T-40; T-50)
   [[dependencies]]
      [[[PT10M]]]
         graph = "@wall_clock => a => b & c"
[runtime]
   [[root]]
      script = "sleep 10"

Like cron, this suite will run your jobs repeatedly at the right interval, from the right point. Unlike cron, it can run a complex workflow, and handle dependence across cycles, and after downtime it will run all the missed cycles very quickly until it catches up to the clock again (if you don’t want to run the missed cycles just cold-start the suite again instead of continuing from where it left off with a restart).

17

Just getting back to this thread after being tied up with other things.
Matt / Hilary - thank you both for your responses on this thread.
In particular, Hilary, thanks for the very detailed example above. Yes, I think that would make a useful addition to the documentation.

1 Like