How use cylc show for tasks which haven't spawned yet?

Cylc Cylc Support
1

Hi,

If a task doesn’t appear yet, how do I do cylc show on it to check if things are displaying correctly and the expected task pre-requisites exist? When I try, it jsut says No matching n=0 tasks found: */<task name>

2

A task is “spawned” as soon as any of its prerequisites are satisfied. At this point you will be able to inspect it in cylc show.

If the task does not appear in cylc show, this means that none of its prerequisites have been met (i.e. its task prerequisites do not exist yet). Such a task may or may not be run by Cylc depending on graph branching.

3

Ok, my terminology may no be accurate. I want to do this:

  1. Install a workflow
  2. Check task prerequisites an other such items of a task which may or not be visible
a => b => c => d
install workflow
check prerequisites of d

Are there alternatives to cylc show which ignore if a task is in the pool yet? Is there a way to force Cylc to add all tasks to the pool?

4

I want to do this:

If the task is not found then all prerequisites are unsatisfied.

Are there alternatives to cylc show which ignore if a task is in the pool yet?

No

Is there a way to force Cylc to add all tasks to the pool?

Yes, you can trigger it, but I don’t think that’s what you want.

5

@TomC - why exactly would you want to inspect the prerequisites of task d in your example?

  • to see if any of its prerequisites are satisfied yet?

    • as @oliver.sanders notes, none are satisfied if the task is not yet visible to the scheduler
    • (we should perhaps amend the resulting message to the user, to be clear about this)

  • to see what its prerequisites are?

    • that is (at least arguably) not the job of the run-time scheduler, which can’t maintain an awareness of every task instance in a potentially infinite graph. Better to inspect or visualize the graph config.
    • (that said, the web GUI will grow static graph visualization capability in future releases, including, I hope, the ability to see an n-edge window surrounding selected tasks …; the timeline on this depends on priorities and resourcing of course)

Are there alternatives to cylc show which ignore if a task is in the pool yet? Is there a way to force Cylc to add all tasks to the pool?

In Cylc 7 and earlier, the exact content of the scheduler task pool depended on scheduler internals that were incomprehensible to most users, and which they really should not have had to care about. That was a side effect of the way Cylc first solved the problem of “unrolling the loop” to remove the barrier between cycles. Luckily, the algorithm required enough future “waiting” tasks that users could get by without worrying about the details most of the time, and using cylc insert during manual intervention if a task mysteriously did not “exist” (which, by the way, is a bizarre thing to have to do).

In Cylc 8, we are very close to eliminating the task pool as user-facing leakage of scheduler internals. Instead we have the more intuitive concept of a window of activity moving along the workflow graph, plus the ability to visualize the workflow by n-graph-edges out from the active tasks; and the ability to manually trigger tasks and flows anywhere in the graph regardless of where the current active window is. Tasks will not appear in the active window, however, unless they are actually “active” (albeit in a slightly broader sense than just submitted or running).

I appreciate this stuff may not be very clear to new Cylc 8 users yet, sorry. The good news is we are close to finishing work to make every conceivable intervention as easy as possible, for a near-future release. At that point we will also engage with users to make sure it’s clear how to handle all known intervention use cases.

6

@oliver.sanders correct, I do not want to trigger the task as I’m not wanting it to run, I just want it to be accessible to find out some information about it.

@hilary.j.oliver If you have a graph of thousands of tasks, viewing a graph image isn’t necessarily useful. I don’t know how to inspect it textually, or do you mean writing to a dot file? Parsing a large dot file to find what you need isn’t as easy as having it done for you via cylc show.

@hilary.j.oliver as you say, perhaps the error message could be improved. It currently says that No matching n=0 tasks found: 20200101T0000Z/b_024_gl which, I don’t really understand. What does n=0 mean? I know that task exists in the graph, it just isn’t in the pool. Can it distinguish by saying Task 20200101T0000Z/b_024_gl does not exist in the pool yet. vs Task 20200101T0000Z/b_024_gl does not exist in the graph. to highlight if its just yet to be placed into the pool vs it actually doesn’t exist at all and you may have a typo?

Why did this come up? Well, because of this bug and I wanted to see if the requirements for downstream tasks were correct. But hypothetically, say I’ve got a complex set of pre-req’s which involve conditionals and jinja2 loops to create them, and I want to do a sanity check to make sure they look right when resovled after I tweak the loops. I’m not sure a visual graph effectively shows a | b | c type triggers, but maybe I’m wrong and it does now? And, I dont’ think the image view of the graph is searchable (perhaps it is now in Cylc8?). Perhaps I want to visualise how the meta section is being displayed (side note: does the meta section allow html style rendering when displayed in the web UI - I don’t have access to the web UI yet to check that sort of thing, but it would be nice to have).

As you say though, I don’t know how the internals work, and don’t really want to as I don’t have time for that, but, I think it would be useful having something like cylc show which works for tasks which are not in the task pool yet.

7

Fair point on the difficulties of grokking a graph with thousands of tasks. I’ll post an issue for partial graphs around selected tasks (we’ve had this in mind for the web UI for some time, but it would be good for static graphs via the CLI too).

In lieu of graphing, “textual” inspection is generally best for understanding the structure of the workflow (as determined by task prerequisites and outputs). If the graph is constructed with complex Jinja2 logic you can process that out with cylc view -j or use cylc config -i "[scheduling]graph".

In Cylc 8 at run time we capture tasks (for UI view purposes) by number of graph edges (n) extending out from current active tasks. Thus, n=0 corresponds to just the active tasks.

https://cylc.github.io/cylc-doc/stable/html/user-guide/running-workflows/tasks-jobs-ui.html#n-window

However, it would be better for that error message to say the requested task is not currently active.

(Back later to respond to more points…)

8

Not yet.

Nope. We’re kind of beholden to the graph layout tech there, although there might be scope for that once we get static visualization in the browser UI.

Actually I don’t think we display task metadata yet in the web UIt.

That comment was aimed at Cylc 7, where you needed to understand the scheduling algorithm in order to understand exactly which tasks, out of all the possible tasks, get displayed in the Cylc 7 UI (because all it displayed was the scheduler task pool, which contained a mysteriously bounded lot of waiting and succeeded tasks as well as current active ones).

Cylc 8 does not have that problem, although we haven’t quite finished the follow-through on that yet. For your cylc show example, the Cylc 8 possibilities will be:

  • we return requested info not available, the task is not active yet
    or preferably
  • we compute and return the info even though the target task is not active yet