# Symlink dirs not working for slurm job request out of localhost **URL:** \ **Category:** Cylc Support\ **Created:** [April 11, 2025, 5:29am UTC](https://cylc.discourse.group/t/symlink-dirs-not-working-for-slurm-job-request-out-of-localhost/1149 "2025-04-11T05:29:23Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![rsoutelino](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/rsoutelino/32/449_2.png) [@rsoutelino](https://cylc.discourse.group/u/rsoutelino)\ **Post date:** [April 11, 2025, 5:29am UTC](https://cylc.discourse.group/t/symlink-dirs-not-working-for-slurm-job-request-out-of-localhost/1149/1 "2025-04-11T05:29:23Z")
Hi there everyone. We are new to cylc and trying to sort out how to submit tasks via slurm jobs and running into an issue when using --nodelist slurm directive. We have things working for slurm jobs that coincidentally run on the same host we are running `cylc install`, but not when we request a different node. Our platform config: ```auto [platforms] [[kapua]] hosts = localhost job runner = slurm install target = localhost [install] [[symlink dirs]] [[[localhost]]] run = /hub/scratch ``` Rationale and context: This is an experimental cluster with `kap-xeon02` configured as a slurm controller and `kap-xeon02` and `kap-epyctest` as slurm nodes. Cylc + Cylchub is installed in `kap-xeon02`, and we are installing cylc workflows directly there. We are using `kap-xeon02` as a “login node”). The reason to use symlink dirs is that these are not headless nodes, so home directories are not in a shared filesystem, but `/hub/scratch` is. The workflow file is: ```auto [scheduler] UTC mode = True [scheduling] initial cycle point = now [[xtriggers]] delay3h = wall_clock(offset=PT3H) delay8h = wall_clock(offset=PT8H) [[graph]] # repeat every 6 h starting at the initial cycle point PT12H = """ @delay8h => roms_saus2d_pre => roms_saus2d_run => roms_saus2d_post """ # Repeat every 12 h starting 12 h after the initial cycle point. +PT12H/PT12H = """ @delay8h => roms_saus2d_post[-PT12H] => roms_saus2d_pre => roms_saus2d_run => roms_saus2d_post """ [runtime] [[roms_saus2d_pre]] script = roms_preproc.sh [[[environment]]] MODEL_CONFIG = model.roms_gfs_saus2d.yml [[roms_saus2d_run]] script = roms_run.sh platform = kapua [[[environment]]] ROMSBIN = ROMS2D_NO-TIDE_NO-OBC SCRATCH_DIR = /hub/scratch/forecast/roms/gfs_saus2d SLURM_TASKS = 12 [[[directives]]] --mem = 8G --ntasks = 12 --cpus-per-task = 1 --propagate = STACK --partition = mainQ --output = /hub/scratch/forecast/roms/gfs_saus2d/roms.out --error = /hub/scratch/forecast/roms/gfs_saus2d/roms.err --nodelist = kap-epyctest [[roms_saus2d_post]] script = roms_postproc.sh [[[environment]]] MODEL_CONFIG = model.roms_gfs_saus2d.yml ``` Everything runs fine without the `--nodelist = kap-epyctest` directive. The slurm job is submitted to `kap-xeon02` as there are enough resources and the workflow succeeds. But when we add `--nodelist = kap-epyctest`, we get this error on job submission. ```auto cat /hub/scratch/forecast/roms/gfs_saus2d/roms.err /var/lib/slurm-llnl/slurmd/job00311/slurm_script: line 65: /home/metocean/cylc-run/ops-tiny/run1/.service/etc/job.sh: No such file or directory /var/lib/slurm-llnl/slurmd/job00311/slurm_script: line 66: cylc __job__ main: command not found ``` I expected the symlink dirs to fix that, but didn’t work. We have this `global.cylc` file in `/etc/cylc/flow/global.cylc` in both servers. Is this the correct way of configuring the platform? Maybe cylc expects home directories to be in a shared filesystem? Thanks in advance! 🙂 ---
**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\ **Post date:** [April 11, 2025, 7:11am UTC](https://cylc.discourse.group/t/symlink-dirs-not-working-for-slurm-job-request-out-of-localhost/1149/2 "2025-04-11T07:11:59Z")
Yes, Cylc expects all the nodes in a platform to share home directories. The problem is that, although Cylc supports setting up symlinks to refer to different filesystems, the locations are still referenced via `$HOME`. However, we have seen this kind of problem before so you should be able to workaround it by adding the following to your platform config: ```auto [platforms] [[kapua]] global init-script = "export HOME=/hub/scratch" ``` I’m rather surprised your job run at all - I would have expected Slurm to complain about the location of the stdout / stderr files. If you encounter this problem you can fix it using the Slurm directive `--chdir=/hub/scratch`. Note that, if you need to support multiple users, you’re going to need to use a user specific directory, e.g. `/hub/scratch/${USER}` ---
**Author:** ![wxtim](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/wxtim/32/151_2.png) [@wxtim](https://cylc.discourse.group/u/wxtim)\ **Post date:** [April 15, 2025, 7:37am UTC](https://cylc.discourse.group/t/symlink-dirs-not-working-for-slurm-job-request-out-of-localhost/1149/3 "2025-04-15T07:37:40Z")
Documentation link [Platform Configuration — Cylc 8.4.2 documentation](https://cylc.github.io/cylc-doc/stable/html/reference/config/writing-platform-configs.html#platform-with-no-home-directory) ---
**Author:** ![Rafael\_Soutelino](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/rafael_soutelino/32/482_2.png) [@Rafael\_Soutelino](https://cylc.discourse.group/u/Rafael_Soutelino)\ **Post date:** [April 15, 2025, 8:23am UTC](https://cylc.discourse.group/t/symlink-dirs-not-working-for-slurm-job-request-out-of-localhost/1149/4 "2025-04-15T08:23:04Z")
Thanks for your quick reply @dpmatthews. Yes, I believe we had it working in the localhost because we indeed had the --chdir set. Sounds like `global init-script = "export HOME=/hub/scratch"` is exactly what we need. We’ll try that. Many thanks!