# Wrong rsync command

**URL:** <https://cylc.discourse.group/t/wrong-rsync-command/526>\
**Category:** Cylc Support\
**Created:** [August 26, 2022, 7:55am UTC](https://cylc.discourse.group/t/wrong-rsync-command/526 "2022-08-26T07:55:01Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Matthieu](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/matthieu/32/202_2.png) [@Matthieu](https://cylc.discourse.group/u/Matthieu)\
**Post date:** [August 26, 2022, 7:55am UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/1 "2022-08-26T07:55:01Z")

</div>

Dear cylc community,

Sorry in advance for the long post but being a new user, I don’t know into how much detail I needed to go.

I’m trying to setup cylc-flow for groups in my institution. The documentation was sufficient so far but I’m now stuck with an rsync error I found in the log. So here is my set up so far:

- I’m running cylc version 8.0.1
- I define the workflow on my laptop
- I’d like to run the tasks on an HPC machine with host name daint equiped with the slurm scheduler.
- cylc is installed on daint
- My `global.cylc` file looks like this

```nohighlight
[platforms]
    [[daint]]
        hosts = daint
        job runner = slurm
        install target = daint
        retrieve job logs = True
        communication method = ssh
        cylc path = /users/leclairm/.local/bin
    [[daint_bg]]
        hosts = daint
        job runner = background
        install target = daint_bg
        retrieve job logs = True
        communication method = ssh
        cylc path = /users/leclairm/.local/bin

[install]
    source dirs = ~/cylc-src, /project/pr133/leclairm/cylc-src, /project/g110/cylc-src
    [[symlink dirs]]
        [[[daint]]]
            run = /scratch/snx3000/leclairm/
        [[[daint_bg]]]
            run = /scratch/snx3000/leclairm/     

```

- My flow.cylc, named ROMSOC, looks like this:

```auto
[scheduler]
    UTC mode = True
    allow implicit tasks = True
[scheduling]
    initial cycle point = 2000-01-01T00Z
    final cycle point = 2000-05-01T00Z
    [[graph]]
        R1 = """
           Prepare_input_data_for_ROMS => ROMSOC
           Extpar => INT2LM
        """
        P1M = """
            download_data_from_DKRZ => INT2LM
            clean_COSMO_input_data[-P2M] => download_data_from_DKRZ
            INT2LM => ROMSOC => Post_Process_COSMO & Post_Process_ROMS & clean_COSMO_input_data
            ROMSOC[-P1M] => ROMSOC
        """
[runtime]
    [[root]]
        platform = daint
        [[[directives]]]
            --constraint = gpu
            --account = g110
    [[Prepare_input_data_for_ROMS]]
        script = """
            echo "running Prepare_input_data_for_ROMS"
            sleep 1
            """
        [[[directives]]]
            --nodes = 1
            --time = 00:00:01
    [[ROMSOC]]
        script = """
            echo "running ROMSOC"
            sleep 10
            """
        [[[directives]]]
            --nodes = 2
            --time = 00:00:05
    [[Extpar]]
        script = """
            echo "running Extpar"
            sleep 2
            """
        [[[directives]]]
            --nodes = 1
            --time = 00:00:01
    [[INT2LM]]
        script = """
            echo "running INT2LM"
            sleep 2
            """
        [[[directives]]]
            --nodes = 1
            --time = 00:00:02
    [[download_data_from_DKRZ]]
        script = """
            echo "running download_data_from_DKRZ"
            sleep 2
            """
        [[[directives]]]
            --nodes = 1
            --time = 00:00:02
    [[clean_COSMO_input_data]]
        script = """
            echo "running clean_COSMO_input_data"
            sleep 1
            """
        [[[directives]]]
            --nodes = 1
            --time = 00:00:01
    [[Post_Process_COSMO]]
        script = """
            echo "running Post_Process_COSMO"
            sleep 2
            """
        [[[directives]]]
            --nodes = 1
            --time = 00:00:02
    [[Post_Process_ROMS]]
        script = """
            echo "running Post_Process_ROMS"
            sleep 2
            """
        [[[directives]]]
            --nodes = 1
            --time = 00:00:02

```

`cylc install ROMSOC` is able to install the workflow on my machine. Then `cylc run ROMSOC` starts and creates the folder hierarchy and symlinks on the daint platform. Then it fails with the following error I found on the daint platform in `~/cylc-run/ROMSOC/run1/log/scheduler`:

```auto
2022-08-25T14:50:22Z ERROR - platform: daint - initialisation did not complete
    COMMAND:
        rsync --delete \
            --rsh=ssh -oBatchMode=yes -oConnectTimeout=10 \
            --include=/.service/ --include=/.service/server.key -a \
            --checksum --out-format=%o %n%L --no-t --exclude=log \
            --exclude=share --exclude=work --include=/app/*** \
            --include=/bin/ ***--include=/etc/*** --include=/lib/*** \
            --exclude=* /home/matth/cylc-run/ROMSOC/run1/ \
            daint:$HOME/cylc-run/ROMSOC/run1/
    RETURN CODE:
        11
    STDERR:
        rsync: mkdir "/users/leclairm/$HOME/cylc-run/ROMSOC/run1" failed: No such file or directory (2)
        rsync error: error in file IO (code 11) at main.c(664) [Receiver=3.1.3]

```

Obviously, the `rsync` command should not include `$HOME`. Am I missing a configuration step or is that a bug?

---

<div class="post-metadata">

**Author:** ![Matthieu](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/matthieu/32/202_2.png) [@Matthieu](https://cylc.discourse.group/u/Matthieu)\
**Post date:** [August 26, 2022, 8:32am UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/2 "2022-08-26T08:32:31Z")

</div>

I think I found the way how the `'$HOME'` string made its way into the `rsync` command. The 2 first lines of `cylc.flow.pathutil` read like this:

```python
# monkeypatching:
_CYLC_RUN_DIR = os.path.join('$HOME', 'cylc-run')

```

The `_CYLC_RUN_DIR` variable is then used to construct the string returned by `get_remote_workflow_run_dir` which is used by `cylc.flow.task_remote_mgr.TaskRemoteMgr.file_install` to construct the `dst_path` variable for the `rsync` command.

Somehow my `rsync` version doesn’t play well with this way of specifying the destination.

---

<div class="post-metadata">

**Author:** ![datamel](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/datamel/32/149_2.png) [@datamel](https://cylc.discourse.group/u/datamel)\
**Post date:** [August 26, 2022, 9:07am UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/3 "2022-08-26T09:07:25Z")

</div>

Hello @Matthieu

Welcome and thanks very much for your post. We are currently looking into this.

As a side note, there is a small error in your `global.cylc` which will need correcting to avoid problems later down the line. The `install target` for `daint` and `daint_bg` should be the same. So change required:

```auto
[platforms]
    [[daint]]
        hosts = daint
        job runner = slurm
        install target = daint
        retrieve job logs = True
        communication method = ssh
        cylc path = /users/leclairm/.local/bin
    [[daint_bg]]
        hosts = daint
        job runner = background
        install target = daint # this tells cylc that daint_bg shares a file system with daint
        retrieve job logs = True
        communication method = ssh
        cylc path = /users/leclairm/.local/bin

```

We will update the documentation to make this clearer.

I will keep you posted when a fix becomes available for you.

Best wishes

Mel

---

<div class="post-metadata">

**Author:** ![Matthieu](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/matthieu/32/202_2.png) [@Matthieu](https://cylc.discourse.group/u/Matthieu)\
**Post date:** [August 26, 2022, 9:09am UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/4 "2022-08-26T09:09:49Z")

</div>

Actually, I don’t really get why this whole machinery to get remote run dirs is necessary because the file hierarchy under `~/cylc-run` is replicated anyhow from the origin to the host under the same root `~/cylc-run`. So the resulting `rsync` command shoud just be

```bash
rsync [...] /absolute/user/home/cylc-run/workflow_name/run1 host:cylc-run/workflow_name/run1

```

Maybe I’m wrong but I also don’t see how the original `rsync` command could work on any system as the $HOME expansion can only happen on the origin side.

---

<div class="post-metadata">

**Author:** ![Matthieu](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/matthieu/32/202_2.png) [@Matthieu](https://cylc.discourse.group/u/Matthieu)\
**Post date:** [August 26, 2022, 9:12am UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/5 "2022-08-26T09:12:23Z")

</div>

Thanks a lot for this Mel!  
This makes a lot of sense now, I was a bit puzzled by this `install target` setting, mostly because of the word “file system” in the docs.

---

<div class="post-metadata">

**Author:** ![datamel](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/datamel/32/149_2.png) [@datamel](https://cylc.discourse.group/u/datamel)\
**Post date:** [August 26, 2022, 9:26am UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/6 "2022-08-26T09:26:03Z")

</div>

Yes, we are a little confused as to how it is working for us and hasn’t been spotted before!

---

<div class="post-metadata">

**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\
**Post date:** [August 26, 2022, 10:08am UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/7 "2022-08-26T10:08:26Z")

</div>

Note that sections can be repeated in the global config which is useful for simplifying the platforms config. I think the following should work:

```auto
[platforms]
    [[daint, daint_bg]]
        hosts = daint
        install target = daint
        retrieve job logs = True
        communication method = ssh
        cylc path = /users/leclairm/.local/bin
    [[daint]]
        job runner = slurm

```

Also, assuming the `cylc` command is in your `$PATH` when you login to `daint`, you shouldn’t need to configure `cylc path` (all remote commands use a login shell by default).

---

<div class="post-metadata">

**Author:** ![Matthieu](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/matthieu/32/202_2.png) [@Matthieu](https://cylc.discourse.group/u/Matthieu)\
**Post date:** [August 26, 2022, 10:10am UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/8 "2022-08-26T10:10:52Z")

</div>

Thanks a lot for the config tips.

And yes, I figured out for the `cylc path`. I was just trying a few things out before narrowing down the issue.

---

<div class="post-metadata">

**Author:** ![Matthieu](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/matthieu/32/202_2.png) [@Matthieu](https://cylc.discourse.group/u/Matthieu)\
**Post date:** [August 26, 2022, 11:19am UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/9 "2022-08-26T11:19:09Z")

</div>

By the way, can I also merge the `[install][symlink dirs][daint]` and `[install][symlink dirs][daint_bg]`? So that the whole thing would finally be:

```auto
[platforms]
    [[daint, daint_bg]]
        hosts = daint
        install target = daint
        retrieve job logs = True
        communication method = ssh
    [[daint]]
        job runner = slurm

[install]
    source dirs = ~/cylc-src, /project/pr133/leclairm/cylc-src, /project/g110/cylc-src
    [[symlink dirs]]
        [[[daint, daint_bg]]]
            run = /scratch/snx3000/leclairm/

```

---

<div class="post-metadata">

**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\
**Post date:** [August 26, 2022, 11:39am UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/10 "2022-08-26T11:39:52Z")

</div>

The `[symlink dirs]` section refers to install targets and both platforms are now sharing a common `install target` so all you need is:

```auto
    [[symlink dirs]]
        [[[daint]]]
            run = /scratch/snx3000/leclairm/

```

---

<div class="post-metadata">

**Author:** ![Matthieu](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/matthieu/32/202_2.png) [@Matthieu](https://cylc.discourse.group/u/Matthieu)\
**Post date:** [August 26, 2022, 11:41am UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/11 "2022-08-26T11:41:51Z")

</div>

Ah true, thank you.  
Obviously, still need to get used to the `install target` concept …

---

<div class="post-metadata">

**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\
**Post date:** [August 26, 2022, 2:46pm UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/12 "2022-08-26T14:46:29Z")

</div>

Looking at your rsync issue, it’s turns out we’ve been using this for a long time without anyone reporting a problem. I wonder what’s different about your environment?  
Can you confirm that it really is the `$HOME` causing the problem.  
For example, try something like:

```auto
$ touch test_file
$ rsync test_file 'daint:$HOME'

```

Does that fail in the same way?  
I’ve tested this on a variety of remote systems without any problem.

---

<div class="post-metadata">

**Author:** ![Matthieu](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/matthieu/32/202_2.png) [@Matthieu](https://cylc.discourse.group/u/Matthieu)\
**Post date:** [August 26, 2022, 5:30pm UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/13 "2022-08-26T17:30:58Z")

</div>

Yes it’s failing. If I don’t use quotes `$HOME` is getting expanded on the origin side:

```bash
❯ rsync test_file daint:$HOME                                                
rsync: mkstemp "/home/.matth.8OAzy0" failed: Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1327) [sender=v3.2.5pre2-3-gcff8f044]

```

On the origin system, my home is `/home/matth` while on destination system, it’s `/users/leclairm`, hence the permission error for `/home/...`.

If I use the quotes with `rsync test_file 'daint:$HOME'`, the transfer completes but still fails as the destination file is then named ‘$HOME’:

```bash
leclairm@daint102:~
> ls -lrt
...
-rw-r--r-- 1 leclairm s83 0 Aug 26 19:23 '$HOME'

```

Maybe you could test the other way round if omitting `$HOME` would work on your systems. I’m fairly confident a relative destination path with rsync is relative to the destination home directory (and of course an absolute path is just kept untouched). At least for me

```bash
rsync test_file daint:

```

transfers `test_file` to my destination home directory in the same way as

```auto
rsync test_file daint:/users/leclairm

```

---

<div class="post-metadata">

**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\
**Post date:** [August 26, 2022, 6:15pm UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/14 "2022-08-26T18:15:50Z")

</div>

> [@Matthieu](#):
>
> I’m fairly confident a relative destination path with rsync is relative to the destination home directory

Agreed. I’m just curious why your system is behaving differently to all the other systems we’ve encountered.

---

<div class="post-metadata">

**Author:** ![Matthieu](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/matthieu/32/202_2.png) [@Matthieu](https://cylc.discourse.group/u/Matthieu)\
**Post date:** [August 26, 2022, 7:11pm UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/15 "2022-08-26T19:11:50Z")

</div>

> [@dpmatthews](#):
>
> ```auto
> rsync test_file 'daint:$HOME'
> 
> ```

I found out running the test from another HPC system to daint works. they both share this version of rsync:

```bash
leclairm@daint105:~
> rsync --version
rsync version 3.1.3 protocol version 31
Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc, SLP

rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.

```

while my laptop runs this version:

```bash
rsync version v3.2.5pre2-3-gcff8f044 protocol version 31
Copyright (C) 1996-2022 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, symlinks, symtimes, hardlinks, hardlink-specials,
    hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, ACLs,
    xattrs, optional protect-args, iconv, prealloc, stop-at, no crtimes
Optimizations:
    SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5
Checksum list:
    xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
    zstd lz4 zlibx zlib none

rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details

```

My OS is Manjaro, so a rolling release distribution, with often newer software versions as, e.g., HPC systems. So I guess the issue will pop up on other systems as well sooner or later. Or there’s just a bug in my bleeding edge software which will be fixed later, that’s sometimes the down side of rolling releases …

As stated in [https://rsync.samba.org/](https://rsync.samba.org/), version 3.2.5 was released Aug 14th. The pre-release I’m using is from Aug 9th so I guess 3.2.5 will be installed soon and I’ll be able to check if the new behavior persists. Anyhow, it might still make sense to consider dropping the usage of `$HOME` in the `rsync` command.

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [August 26, 2022, 9:24pm UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/16 "2022-08-26T21:24:40Z")

</div>

> [@Matthieu](#):
>
> Obviously, still need to get used to the `install target` concept …

Perhaps we need to document that better.

If you define a job `platform` with multiple hosts and a `job runner` that Cylc can use to interact with jobs, the `install target` is the **one** host that Cylc should target to install files and symlink run directories. (One only, because platform hosts see the same shared filesystem).

[And note, multiple platforms can be defined for the same shared filesystem.]

---

<div class="post-metadata">

**Author:** ![ScottWales](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/scottwales/32/192_2.png) [@ScottWales](https://cylc.discourse.group/u/ScottWales)\
**Post date:** [August 29, 2022, 1:03am UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/17 "2022-08-29T01:03:00Z")

</div>

This same issue has just popped up for us at NCI. I wonder if a change in a system package has happened or something, it was working fine before

---

<div class="post-metadata">

**Author:** ![ScottWales](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/scottwales/32/192_2.png) [@ScottWales](https://cylc.discourse.group/u/ScottWales)\
**Post date:** [August 29, 2022, 1:34am UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/18 "2022-08-29T01:34:11Z")

</div>

Adding to the platform definition

```auto
rsync command = rsync --old-args

```

appears to fix the issue

---

<div class="post-metadata">

**Author:** ![Matthieu](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/matthieu/32/202_2.png) [@Matthieu](https://cylc.discourse.group/u/Matthieu)\
**Post date:** [August 29, 2022, 9:51am UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/19 "2022-08-29T09:51:06Z")

</div>

Thanks for the quick temporary fix!  
Works for me as well.

---

<div class="post-metadata">

**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\
**Post date:** [August 31, 2022, 10:07am UTC](https://cylc.discourse.group/t/wrong-rsync-command/526/20 "2022-08-31T10:07:50Z")

</div>

@ScottWales - thanks for finding a workaround.  
This appears to be related to a behaviour change in rsync 3.2.4:  
[https://download.samba.org/pub/rsync/NEWS#3.2.4](https://download.samba.org/pub/rsync/NEWS#3.2.4)  
We’ll look at removing the use of `$HOME/` in the `rsync` command.

[Next page](https://cylc.discourse.group/t/wrong-rsync-command/526.md?page=2)