Dear cylc community,
Sorry in advance for the long post but being a new user, I don’t know into how much detail I needed to go.
I’m trying to setup cylc-flow for groups in my institution. The documentation was sufficient so far but I’m now stuck with an rsync error I found in the log. So here is my set up so far:
- I’m running cylc version 8.0.1
- I define the workflow on my laptop
- I’d like to run the tasks on an HPC machine with host name daint equiped with the slurm scheduler.
- cylc is installed on daint
- My
global.cylcfile looks like this
[platforms]
[[daint]]
hosts = daint
job runner = slurm
install target = daint
retrieve job logs = True
communication method = ssh
cylc path = /users/leclairm/.local/bin
[[daint_bg]]
hosts = daint
job runner = background
install target = daint_bg
retrieve job logs = True
communication method = ssh
cylc path = /users/leclairm/.local/bin
[install]
source dirs = ~/cylc-src, /project/pr133/leclairm/cylc-src, /project/g110/cylc-src
[[symlink dirs]]
[[[daint]]]
run = /scratch/snx3000/leclairm/
[[[daint_bg]]]
run = /scratch/snx3000/leclairm/
- My flow.cylc, named ROMSOC, looks like this:
[scheduler]
UTC mode = True
allow implicit tasks = True
[scheduling]
initial cycle point = 2000-01-01T00Z
final cycle point = 2000-05-01T00Z
[[graph]]
R1 = """
Prepare_input_data_for_ROMS => ROMSOC
Extpar => INT2LM
"""
P1M = """
download_data_from_DKRZ => INT2LM
clean_COSMO_input_data[-P2M] => download_data_from_DKRZ
INT2LM => ROMSOC => Post_Process_COSMO & Post_Process_ROMS & clean_COSMO_input_data
ROMSOC[-P1M] => ROMSOC
"""
[runtime]
[[root]]
platform = daint
[[[directives]]]
--constraint = gpu
--account = g110
[[Prepare_input_data_for_ROMS]]
script = """
echo "running Prepare_input_data_for_ROMS"
sleep 1
"""
[[[directives]]]
--nodes = 1
--time = 00:00:01
[[ROMSOC]]
script = """
echo "running ROMSOC"
sleep 10
"""
[[[directives]]]
--nodes = 2
--time = 00:00:05
[[Extpar]]
script = """
echo "running Extpar"
sleep 2
"""
[[[directives]]]
--nodes = 1
--time = 00:00:01
[[INT2LM]]
script = """
echo "running INT2LM"
sleep 2
"""
[[[directives]]]
--nodes = 1
--time = 00:00:02
[[download_data_from_DKRZ]]
script = """
echo "running download_data_from_DKRZ"
sleep 2
"""
[[[directives]]]
--nodes = 1
--time = 00:00:02
[[clean_COSMO_input_data]]
script = """
echo "running clean_COSMO_input_data"
sleep 1
"""
[[[directives]]]
--nodes = 1
--time = 00:00:01
[[Post_Process_COSMO]]
script = """
echo "running Post_Process_COSMO"
sleep 2
"""
[[[directives]]]
--nodes = 1
--time = 00:00:02
[[Post_Process_ROMS]]
script = """
echo "running Post_Process_ROMS"
sleep 2
"""
[[[directives]]]
--nodes = 1
--time = 00:00:02
cylc install ROMSOC is able to install the workflow on my machine. Then cylc run ROMSOC starts and creates the folder hierarchy and symlinks on the daint platform. Then it fails with the following error I found on the daint platform in ~/cylc-run/ROMSOC/run1/log/scheduler:
2022-08-25T14:50:22Z ERROR - platform: daint - initialisation did not complete
COMMAND:
rsync --delete \
--rsh=ssh -oBatchMode=yes -oConnectTimeout=10 \
--include=/.service/ --include=/.service/server.key -a \
--checksum --out-format=%o %n%L --no-t --exclude=log \
--exclude=share --exclude=work --include=/app/*** \
--include=/bin/*** --include=/etc/*** --include=/lib/*** \
--exclude=* /home/matth/cylc-run/ROMSOC/run1/ \
daint:$HOME/cylc-run/ROMSOC/run1/
RETURN CODE:
11
STDERR:
rsync: mkdir "/users/leclairm/$HOME/cylc-run/ROMSOC/run1" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(664) [Receiver=3.1.3]
Obviously, the rsync command should not include $HOME. Am I missing a configuration step or is that a bug?