Cylc set slow to be able to run after workflow started?

TomC · November 29, 2024, 4:43am

Using Cylc 8.3.6 (but also in 8.3.5 at least, haven’t checked anything else).

I launched a workflow, then tried to do some cylc set operation, but it kept telling me the workflow was stopped.

$ cylc vip --pause $PWD
$ cylc validate /home/code/retry
Valid for cylc-8.3.6
$ cylc install /home/code/retry
INSTALLED retry/run4 from /home/retry
$ cylc play --pause retry/run4

 ▪ ■  Cylc Workflow Engine 8.3.6
 ██   Copyright (C) 2008-2024 NIWA
▝▘    & British Crown (Met Office) & Contributors

INFO - Extracting job.sh to /home/cylc-
    run/retry/run4/.service/etc/job.sh
retry/run4: dev-cylc-003.hpc.internal.bom.gov.au PID=348337
$ cylc set retry/run4 //20201010T0000Z/foo
WorkflowStopped: retry/run4 is not running
$ cylc set retry/run4 //20201010T0000Z/foo
WorkflowStopped: retry/run4 is not running
$ cylc set retry/run4 //20201010T0000Z/foo
WorkflowStopped: retry/run4 is not running
$ cylc set retry/run4 //20201010T0000Z/foo
WorkflowStopped: retry/run4 is not running
$ cylc set retry/run4 //20201010T0000Z/foo
WorkflowStopped: retry/run4 is not running
$ cylc set retry/run4 //20201010T0000Z/foo
WorkflowStopped: retry/run4 is not running
$ cylc set retry/run4 //20201010T0000Z/foo
WorkflowStopped: retry/run4 is not running
$ cylc set retry/run4 //20201010T0000Z/foo
Command queued

I tried very frequently, but it seemd to take 30 seconds before cylc set could be actioned. In the meantime, if I hadn’t paused it, tasks could run without issue (as they had in other runs I had tried this with).

2024-11-29T04:37:30Z INFO - Workflow: retry/run4
2024-11-29T04:37:30Z INFO - Scheduler: url=tcp://hostname:43032 pid=348337
2024-11-29T04:37:30Z INFO - Workflow publisher: url=tcp://hostname:43213
2024-11-29T04:37:30Z INFO - Run: (re)start number=1, log rollover=1
2024-11-29T04:37:30Z INFO - Cylc version: 8.3.6
2024-11-29T04:37:30Z INFO - Run mode: live
2024-11-29T04:37:30Z INFO - Initial point: 20201010T0000Z
2024-11-29T04:37:30Z INFO - Final point: 20201010T0000Z
2024-11-29T04:37:30Z INFO - Cold start from 20201010T0000Z
2024-11-29T04:37:30Z INFO - New flow: 1 (original flow from 20201010T0000Z) 2024-11-29T04:37:30+00:00
2024-11-29T04:37:30Z INFO - [20201010T0000Z/foo:waiting(runahead)] => waiting
2024-11-29T04:37:30Z INFO - [20201010T0000Z/foo:waiting] => waiting(queued)
2024-11-29T04:37:31Z INFO - Pausing the workflow: Paused on start up
2024-11-29T04:37:59Z INFO - Command "set" received. ID=87048f0d-1b61-4125-901c-655472530bec
    set(flow=[], flow_wait=False, outputs=[], prerequisites=[], tasks=['20201010T0000Z/foo'])
2024-11-29T04:38:00Z INFO - [20201010T0000Z/foo:waiting(queued)] setting implied output: submitted
2024-11-29T04:38:00Z INFO - [20201010T0000Z/foo:waiting(queued)] setting implied output: started
2024-11-29T04:38:00Z INFO - [20201010T0000Z/foo:waiting(queued)] => failed(queued)
2024-11-29T04:38:00Z INFO - [20201010T0000Z/bar:waiting(runahead)] => waiting
2024-11-29T04:38:00Z INFO - [20201010T0000Z/bar:waiting] => waiting(queued)
2024-11-29T04:38:00Z INFO - [20201010T0000Z/foo/00:failed(queued)] => failed
2024-11-29T04:38:00Z INFO - Command "set" actioned. ID=87048f0d-1b61-4125-901c-655472530bec

Is the delay for set being able to work expected?

dpmatthews · November 29, 2024, 8:14am

I can repeat this. It applies to lots of commands, e.g cylc ping. However, in my tests, if I run cylc scan first, that detects the workflow immediately and all other commands then work. So, something is clearly not right.

Note that this only happens when using a separate scheduler host. If you run with --host=localhost it doesn’t happen so I assume it is network filesystem related.

I’ve opened https://github.com/cylc/cylc-flow/issues/6504

oliver.sanders · November 29, 2024, 3:57pm

Is the delay for set being able to work expected?

Some delay might be expected, but not this much.

I can also repeat this.

I think Dave’s right, this is something to do with the network filesystem. When I try to cat the file, it doesn’t exist, but after I run cylc scan it magically appears???

$ cylc vip . -n foo
...

 ▪ ■  Cylc Workflow Engine 8.4.0.dev
 ██   Copyright (C) 2008-2024 NIWA
▝▘    & British Crown (Met Office) & Contributors
...

$ cat ~/cylc-run/foo/runN/.service/contact
cat ... No such file or directory
$ cylc scan
foo/run1
$ cat ~/cylc-run/foo/runN/.service/contact
CYLC_API=5
...

Questions:

Why is the contact file slow to sync on the network filesystem? Have we done anything to exacerbate this?
Why does cylc scan make the file magically appear?

oliver.sanders · November 29, 2024, 4:02pm

I’ve got an answer to question (2)!

The cylc scan command is performing filesystem listings, if I run ls on the workflow and .service directories first, the file magically appears:

$ cylc vip -n foo
...

$ cat ~/cylc-run/foo/runN/.service/contact
cat ... No such file or directory

$ ls /home/users/oliver.sanders/cylc-run/generic/runN/
...

$ ls /home/users/oliver.sanders/cylc-run/generic/runN/.service
...

$ cat ~/cylc-run/foo/runN/.service/contact
CYLC_API=5
...

This actually rings a bell, I’m sure we’ve performed directory listings before to force synchronization for something.

oliver.sanders · November 29, 2024, 4:21pm

It turns out that this is actually something we already defend against (it rang a bell for a reason)!

github.com/cylc/cylc-flow

contact file creation: fsync parent directory

cylc:master ← matthewrmshin:fsync-dot-service-after-creating-contact-file

opened 01:07PM - 05 Feb 19 UTC

matthewrmshin

+9 -0

On file system such as an NFS mounted file system, we may have a delay before t…he file becomes visible from other process running on other host. We already have a 1st fsync to ensure that the data of the contact file is written to disk. This change adds a 2nd fsync to ensure that the file metadata of the contact file is written as well - by doing an fsync on its directory. (The issue is causing `rose suite-run` to fail to launch `cylc gui` after calling `cylc run` successfully to start up a suite.) (Need to back port to 7.8.2.)

However, the addition of the run name/number added a new level to the directory hierarchy, as a result, it would appear that we also need to list the parent directory (hence the two ls calls required in my above example rather than one).

If you run the workflow with --no-run-name, the contact file appears straight away without the need for any directory listing.

Hierarchical installation (e.g. cylc vip -n a/b/c) does not appear to trigger this issue, so I think the cause of the issue is the runN symlink.

oliver.sanders · November 29, 2024, 4:31pm

Opened an issue, should be an easy fix running workflows may appear stopped on network filesystems · Issue #6506 · cylc/cylc-flow · GitHub

oliver.sanders · February 11, 2025, 4:50pm

I have opened a pull request. It would be helpful if you are able to confirm that it solves the issues you have been seeing.

github.com/cylc/cylc-flow

work around NFS caching issues

master ← oliver-sanders:6506

opened 03:13PM - 11 Feb 25 UTC

oliver-sanders

+51 -4

Closes https://github.com/cylc/cylc-flow/issues/6506 (hopefully!) NFS filesys…tems have a caching mechanism that can take a while to clear. This may cause workflows to appear stopped on the CLI, in the Tui & GUI when they are in fact running. Refresh NFS cache before: * Scanning workflows: * Fixes: Delay between startup and workflows appearing in `cylc scan`, `cylc tui` and `cylc gui` (on other servers). * Cost: We've traded a file access for a directory listing (`.service` dir only contains a few files, no biggie). * Loading the contact file. * Fixes: Delay between workflow startup and the CLI being able to operate on the workflow (on other servers). * Cost: An additional file access (always) and additional directory listings (only when needed). > Note these cashes are specific to the server, so you may get different results running Cylc commands on different servers. <details> <summary>To test, try running this script. It should say "no fix needed" for all cases.</summary> ```bash #!/bin/bash -l set -u WORKFLOW="$1" SERVER="$2" RUN_ROOT="$HOME/cylc-run/$WORKFLOW" reset () { cylc stop "*" >/dev/null rm -r "$HOME/cylc-run/$WORKFLOW" 2>/dev/null rm -r "$DATADIR/cylc-run/$WORKFLOW" 2>/dev/null rm -r "$SCRATCH/cylc-run/$WORKFLOW" 2>/dev/null } test_startup () { local run="$1" if cylc ping "$WORKFLOW/$run" >/dev/null 2>/dev/null; then echo " no fix needed" return fi local directory for directory in "$HOME/cylc-run" "$HOME/cylc-run/$WORKFLOW" "$HOME/cylc-run/$WORKFLOW/$run"; do ls "$directory" >/dev/null if cylc ping "$WORKFLOW/$run" >/dev/null 2>/dev/null; then echo " ls $directory needed" return fi done } run () { echo -e "\n$@" "$@" --pause "$WORKFLOW" >/dev/null 2>/dev/null } reset run cylc vip --host=localhost test_startup run1 reset run cylc vip --host=localhost --run-name=foo test_startup foo reset run cylc vip --host=localhost --no-run-name test_startup "" reset run cylc vip --host="$SERVER" test_startup runN reset run cylc vip --host="$SERVER" --run-name=foo test_startup foo reset run cylc vip --host="$SERVER" --no-run-name test_startup "" reset run ssh "$SERVER" CYLC_ENV_NAME="$CYLC_ENV_NAME" cylc vip --host=localhost test_startup runN reset run ssh "$SERVER" CYLC_ENV_NAME="$CYLC_ENV_NAME" cylc vip --host=localhost --run-name=foo test_startup foo reset run ssh "$SERVER" CYLC_ENV_NAME="$CYLC_ENV_NAME" cylc vip --host=localhost --no-run-name test_startup "" reset ``` </details> **Check List** - [x] I have read `CONTRIBUTING.md` and added my name as a Code Contributor. - [x] Contains logically grouped changes (else tidy your branch by rebase). - [x] Does not contain off-topic changes (use other PRs for other changes). - [x] Applied any dependency changes to both `setup.cfg` (and `conda-environment.yml` if present). - [ ] Tests are included (or explain why tests are not needed). - [x] Changelog entry included if this is a change that can affect users - [x] [Cylc-Doc](https://github.com/cylc/cylc-doc) pull request opened if required at cylc/cylc-doc/pull/XXXX. - [x] If this is a bug fix, PR should be raised against the relevant `?.?.x` branch.

Topic		Replies	Views
Can't restart workflow Cylc 8 Migration	5	237	April 20, 2023
Running Cylc in CI Cylc Support	2	303	July 26, 2022
Resume a workflow at a specific time? Cylc Support	3	12	January 20, 2025
Showing workflow state after run has finished Cylc Support	3	203	July 3, 2023
Workflow consistently stalling Cylc Support	9	282	April 18, 2023

Cylc set slow to be able to run after workflow started?

Related topics