# Rose 2: Retirement of \`rose\_prune\` job-log file rsync

**URL:** <https://cylc.discourse.group/t/rose-2-retirement-of-rose-prune-job-log-file-rsync/348>\
**Category:** Rose Announce\
**Created:** [June 10, 2021, 1:28pm UTC](https://cylc.discourse.group/t/rose-2-retirement-of-rose-prune-job-log-file-rsync/348 "2021-06-10T13:28:56Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\
**Post date:** [June 10, 2021, 1:28pm UTC](https://cylc.discourse.group/t/rose-2-retirement-of-rose-prune-job-log-file-rsync/348/1 "2021-06-10T13:28:56Z")

</div>

This relates to the synchronisation of job log files (e.g. `job.out`, `job.err`) between job hosts (places where jobs run) and the Scheduler host (the place where the Cylc workflow process runs).

The `rose_pune` built-in application currently performs an rsync of remote job log files when using the `prune-remote-logs-at=cycle` option.

We believe this functionality has become obsolete since Cylc developed job log retrieval functionality. Cylc has extra logic for delaying log file retrieval and skipping retrieval based on file size limits.

In some situations these two systems can interact with nasty side-effects so we are proposing to retire the rsync of remote job log files by `rose_prune` at Rose 2.

For more information see this issue on GitHub:

> <https://github.com/metomi/rose/issues/2469>
>
> The rose prune built-in app currently performs an rsync of remote job log files …when using the \`prune-remote-logs-at=cycle\` option.
> https://github.com/metomi/rose/blob/2.0b1/sphinx/api/built-in/rose\_prune.rst#configuration
> We propose removing this functionality.
> 
> I think this option dates back to the early days of Cylc when job log retrieval was done by Rose via hook scripts. Performing the log retrieval immediately after task completion sometimes resulted in incomplete log files so the extra rsync prior to pruning remote log files was useful. However, remote log retrieval is now done by Cylc and a configurable delay can be used which should ensure incomplete log files don't occur (or, at least, are very rare). Therefore, the rsync done by rose prune is not necessary and results in additional load on the server running the scheduler (which is where rose prune tasks typically run). Furthermore, Cylc allows you to configure the max size of the job log files to retrieve but this is not used by rose prune which can result in large log files being transferred unexpectedly. Cylc also allows you to not perform remote log retrieval but, again, rose prune will currently perform the rsync regardless.

We don’t think this change will cause any problems but please let us know if otherwise.