# Running cylc tasks on compute nodes that cannot see /home **URL:** \ **Category:** Cylc Support\ **Created:** [October 22, 2020, 5:36pm UTC](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282 "2020-10-22T17:36:27Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Rosalyn\_Hatcher](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/rosalyn_hatcher/32/39_2.png) [@Rosalyn\_Hatcher](https://cylc.discourse.group/u/Rosalyn_Hatcher)\ **Post date:** [October 22, 2020, 5:36pm UTC](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282/1 "2020-10-22T17:36:27Z")
Hi, We’ve just got access to our new supercomputer and now trying to set up to submit jobs using Rose/Cylc and having problems. The problem with cylc referencing its directories via `~` rear its head again. 😬 In a similar setup to our current HPC, ARCHER, the compute nodes on the new HPC also cannot see the `/home` disk. With ARCHER, you may remember, we were able to get around the issue of cylc referencing its directories via `~` by setting `init-script = "export HOME=$DATADIR"` in the `suite.rc` file. This worked, I believe, because the job script commands prior to the `aprun` were executed on the job launcher (MOM) nodes which could see `/home`. On ARCHER2 (SLURM) the job script and all commands prior to the `srun` are run on pe0 (as there are no job launcher nodes) so cannot see `/home` at all. Any ideas on how we can get around this are much appreciated? Hacking around with the cylc generated job script and manually submitting on the HPC, things like the path to the stdout and error don’t work… ``` #SBATCH --output=cylc-run/u-al624/log/job/1/archer/08/job.out #SBATCH --error=cylc-run/u-al624/log/job/1/archer/08/job.err ``` It needs the full path to the /work disk, etc Hope that makes sense and happy to provide any more information. Regards Ros. ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [October 22, 2020, 11:25pm UTC](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282/2 "2020-10-22T23:25:34Z")
Hi Ros, I’m sure this is solvable, but it might be difficult to figure out without access to the system. Do you know if any of the UK Cylc team have, or can get, access to this new HPC of yours? For the moment, a few thoughts off the top of my head, after looking back at a cylc-7.8.6 task job script: - literal `~` isn’t used anywhere - `$HOME` is referenced twice in the job script, most importantly for the suite run directory location - and also once in the `lib/cylc/job.sh` file, which holds some boilerplate shell functions for job scripts `$SBATCH --output=cylc-run/ros/log/...` That path doesn’t reference `$HOME` (or `~`) and is presumably relative to the directory in which the `sbatch` command is executed. On a normal system that would happen to be the user’s home directory because that’s where `ssh` dumps you (and Cylc submits job scripts via `ssh`). The problem with setting `$HOME` manually in the job script might be that it isn’t consistent with the location used to “initialize” the standard run directory structure etc. on the remote host. That’s done over `ssh` by the `cylc remote-init` command, as I recall, so that should also be the default location you end up in when you `ssh` in to the machine. Regards, Hilary ---
**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\ **Post date:** [October 23, 2020, 7:09am UTC](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282/3 "2020-10-23T07:09:41Z")
I wonder whether using the sbatch option “–chdir=/my/data/dir” will help. (You can’t use environment variables so “–chdir=$DATADIR” won’t work. However, that should be easy to fix with a bit of Jinja2.) ---
**Author:** ![Rosalyn\_Hatcher](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/rosalyn_hatcher/32/39_2.png) [@Rosalyn\_Hatcher](https://cylc.discourse.group/u/Rosalyn_Hatcher)\ **Post date:** [October 23, 2020, 8:04am UTC](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282/4 "2020-10-23T08:04:20Z")
Hi Hilary, Dave, Thanks for the suggestions. I will take a look and let you know how I get on. Cheers, Ros. ---
**Author:** ![Rosalyn\_Hatcher](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/rosalyn_hatcher/32/39_2.png) [@Rosalyn\_Hatcher](https://cylc.discourse.group/u/Rosalyn_Hatcher)\ **Post date:** [October 27, 2020, 9:12am UTC](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282/5 "2020-10-27T09:12:03Z")
Hi Dave, Hilary, Typically HPC had to go offline for a bit 🙄 So the `--chdir` fixes the job stderr/stdout file problem, but I’m not getting very far with the other issues being unfamiliar with what’s going on inside cylc. If someone in the Cylc team would be able to take a look, it would be very much appreciated. There should be no problem in getting an account on the HPC for you. Best Regards, Ros. ---
**Author:** ![Rosalyn\_Hatcher](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/rosalyn_hatcher/32/39_2.png) [@Rosalyn\_Hatcher](https://cylc.discourse.group/u/Rosalyn_Hatcher)\ **Post date:** [October 27, 2020, 7:01pm UTC](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282/6 "2020-10-27T19:01:04Z")
I made some progress this afternoon and think I might have got around the problem with a combination of using sbatch option `--chdir` and then modifying the `lib/cylc/job_file.py` script to write an extra line in the task job script to redefine `HOME` only when host is archer2 (may need to tie this down to “and only when submitting to parallel queue” but we’ll see). ``` cylc __job__ inst__cylc_env() { .... export HOME="/work/n02/n02/$USER" export CYLC_SUITE_RUN_DIR="$HOME/cylc-run/u-al624" export CYLC_SUITE_DEF_PATH="${HOME}/cylc-run/u-al624" export CYLC_SUITE_DEF_PATH_ON_SUITE_HOST="/home/ros/cylc-run/u-al624" export CYLC_SUITE_UUID="c7231788-8004-49c2-82d6-f442a1618c6d" .... } ``` It’s not the most elegant solution and whether or not this is likely to have unintended consequences I have no idea - it seems to be working with the limited testing I’ve done so far… Cheers, Ros. ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [October 28, 2020, 9:42am UTC](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282/7 "2020-10-28T09:42:49Z")
Cylc jobs run in Bash login shells which means that Bash sources the `.bash_profile` file. So you should be able to “shim” that `export HOME` line into the `.bash_profile` on Archer2, or potentially if able into `/etc/bashrc`. Cylc uses login shells largely to allow this kind of environment meddling. ---
**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\ **Post date:** [October 28, 2020, 9:55am UTC](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282/8 "2020-10-28T09:55:26Z")
I assume you can’t do anything in `.bash_profile` because you have no home directory! I’d suggest trying to define `global-init-script` for the relevant host: > **[15.2. Global (Site, User) Config File Reference — The Cylc Suite Engine 7.8.6...](https://cylc.github.io/doc/built-sphinx/appendices/site-user-config-ref.html?highlight=init%20script#hosts-host-global-init-script)** Alternatively try `/etc/job-init-env.sh`: > **[3. Installation — The Cylc Suite Engine 7.8.6 documentation](https://cylc.github.io/doc/built-sphinx/installation.html#configure-site-environment-on-job-hosts)** However, note that support for this has been removed at Cylc 8 so let us know if you end up relying on this. > ---
**Author:** ![Rosalyn\_Hatcher](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/rosalyn_hatcher/32/39_2.png) [@Rosalyn\_Hatcher](https://cylc.discourse.group/u/Rosalyn_Hatcher)\ **Post date:** [October 28, 2020, 5:28pm UTC](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282/9 "2020-10-28T17:28:07Z")
Hi Dave, Oliver, Yes that is correct we can’t do anything in `.bash_profile` which is real pain. ☹ I’ve tried the two suggestions above but they both run too late the flow. I need something that will run before `cylc __job__ inst __cylc_env`. Looking in `lib/cylc/job.sh` it looks like this is the first thing run in `cylc__ job_main` … Thanks. ---
**Author:** ![Rosalyn\_Hatcher](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/rosalyn_hatcher/32/39_2.png) [@Rosalyn\_Hatcher](https://cylc.discourse.group/u/Rosalyn_Hatcher)\ **Post date:** [October 28, 2020, 7:42pm UTC](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282/10 "2020-10-28T19:42:17Z")
Defining `global-init-script` will work but only if I redefine the 2 problematic `CYLC_`variables. Not sure if that is a safer option or not: ``` # GLOBAL-INIT-SCRIPT: . /work/n02/n02/shared/umshared/bin/rose-um-env export HOME=$DATADIR export CYLC_SUITE_RUN_DIR=$HOME/cylc-run/$CYLC_SUITE_NAME export CYLC_SUITE_DEF_PATH=${HOME}/cylc-run/$CYLC_SUITE_NAME ``` ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [October 28, 2020, 8:48pm UTC](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282/11 "2020-10-28T20:48:04Z")
Hi @Rosalyn_Hatcher, Out of curiosity, what’s the rationale for no home-dir on this HPC? Is it that users should never need an interactive login (for “security reasons”?) and only submit jobs to it remotely? I suppose more of this might be coming our way, so we have should have a robust solution for it. What directory do you end up in if you do ssh to the host? > Defining `global-init-script` will work but only if I redefine the 2 problematic `CYLC_` variables. Not sure if that is a safer option or not: I’m a bit worried about Cylc scheduler access to the remote run-dir, which doesn’t happen inside of a job script. E.g. for remote init (set up the basic run-dir structure before the first remote job) and job poll and kill (which need to read the remote job status file). And job log retrieval. You should be able to test all this by playing with a small test suite that submits a single `sleep 30` job (or whatever) to the platform (if you aren’t already doing that!). Hilary ---
**Author:** ![Rosalyn\_Hatcher](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/rosalyn_hatcher/32/39_2.png) [@Rosalyn\_Hatcher](https://cylc.discourse.group/u/Rosalyn_Hatcher)\ **Post date:** [October 28, 2020, 10:41pm UTC](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282/12 "2020-10-28T22:41:36Z")
Hi Hilary, Sorry for the confusion, there is a home-dir on the HPC. So when you ssh in for interactive login you land as usual on home. It’s just the batch compute nodes that cannot see the /home disk at all (not even to source a `.bash_profile` to set up user environment). It’s a complete pain in the butt, not just for cylc. The reasoning we’ve been given for this restriction is again one of performance. So, consequently the cylc remote-init, job poll, retrigger, kill etc work ok in my test suite as the ssh lands on the /home disk and can work as normal. It’s just persuading cylc in a compute job not to access the suite-run dir via /home which I think is just needing to modify or redefine the setting of `CYLC_SUITE_RUN_DIR` & `CYLC_SUITE_DEF_PATH` in the job script. Hope that make sense. Cheers, Ros. ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [October 29, 2020, 3:01am UTC](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282/13 "2020-10-29T03:01:58Z")
Ah got it, thanks, I should have read more carefully! That makes the problem sound rather less horrifying, phew. Hilary ---
**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\ **Post date:** [October 29, 2020, 8:34am UTC](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282/14 "2020-10-29T08:34:19Z")
> [@Rosalyn\_Hatcher](#): > > Defining `global-init-script` will work but only if I redefine the 2 problematic `CYLC_` variables That looks safe to me. I don’t think you actually need $HOME so this should be enough: ```auto CYLC_SUITE_RUN_DIR=${DATADIR}/cylc-run/${CYLC_SUITE_NAME} CYLC_SUITE_DEF_PATH=${CYLC_SUITE_RUN_DIR} ``` ---
**Author:** ![Rosalyn\_Hatcher](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/rosalyn_hatcher/32/39_2.png) [@Rosalyn\_Hatcher](https://cylc.discourse.group/u/Rosalyn_Hatcher)\ **Post date:** [October 29, 2020, 10:09am UTC](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282/15 "2020-10-29T10:09:09Z")
Great. Thanks Dave. I did debate about the `HOME=$DATADIR` - I will check that out - don’t want more in there than is necessary.