# Setting up a Platform Config for machines that share a home directory **URL:** \ **Category:** Cylc 8 Migration\ **Created:** [March 4, 2022, 9:48pm UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446 "2022-03-04T21:48:15Z")\ **Posts on this page:** 16\ **Page:** 2
**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\ **Post date:** [March 8, 2022, 6:54pm UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446/21 "2022-03-08T18:54:57Z")
> [@swartn](#): > > If we could just (optionally) add a platform name between `$HOME` and `cylc-run` , it naively seems like it might be able to solve this issue, as they would be no conflicting links. It might be possible to support this in the future but it would be a non-trivial enhancement. It would also lead to some duplication of files and additional work (remote installation and log retrieval) so would have some downsides. > [@swartn](#): > > So I guess the idea is we call all platforms the same install target as they share home, then manually configure the work directories as you suggest. That’s right. > [@swartn](#): > > We will have to figure out how to get cylc to use those directories though, or how to make sure the relevant paths get to the scheduler. Cylc won’t know anything about those directories but it doesn’t need to. The `env-script` becomes part of every job - you just need to define the appropriate logic (e.g. creation of directories, changing directory, setting environment variables). ---
**Author:** ![Clint\_Seinen](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/clint_seinen/32/185_2.png) [@Clint\_Seinen](https://cylc.discourse.group/u/Clint_Seinen)\ **Post date:** [March 8, 2022, 8:09pm UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446/22 "2022-03-08T20:09:13Z")
Okay thanks @dpmatthews - I’ll look at creating a workflow implementation work around using [these user defined scripts](https://cylc.github.io/cylc-doc/latest/html/user-guide/task-implementation/job-scripts.html#jobscripts) As you say, I think I can easily come up with a way to clean the temporary scratch directories in the `err-script` or `exit-script`, but we may want to leave the temporary directories until a few cycle points ahead, so I’ll probably add some scripting to intelligently call `cylc clean` at the derised points. From _looking_ at `cylc clean` it doesn’t look there is a way to inject some special consideration into `cylc clean`? If not, I think we can add something to the scripting that will call `cylc clean` to delete the scratch directories. Obviously this would be a work around for now, but I’m thinking it should get us off to the races to test more aspects of our model pipeline. ---
**Author:** ![Clint\_Seinen](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/clint_seinen/32/185_2.png) [@Clint\_Seinen](https://cylc.discourse.group/u/Clint_Seinen)\ **Post date:** [March 8, 2022, 9:52pm UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446/23 "2022-03-08T21:52:32Z")
So I’ve setup the following script ```bash #!/bin/bash # setup-scratch-space export JOB_SCRATCH_SPACE=${WORK_SPACE}/tmp_${USER}_cylc_${CYLC_TASK_NAME}_${CYLC_TASK_CYCLE_POINT}_$(date +"%Y%m%d%H%M") mkdir $JOB_SCRATCH_SPACE ln -s $JOB_SCRATCH_SPACE cd $JOB_SCRATCH_SPACE ``` and then added this to my `flow.cylc`: ```nohighlight [runtime] [[root]] env-script = setup-scratch-space [[find_diamonds]] platform = ppp3 script = find_diamonds.sh ``` where `find_diamonds.sh` contains ```bash sleep 15 echo "I'm on $(hostname) at $(pwd)" >> out.txt ``` and the `ppp3` platform is defined as ```nohighlight [platforms] [[ppp3]] cylc path = /home/rcs001/miniconda3/envs/cylc-8.0rc1/bin job runner = jobctl install target = localhost global init-script = """ export WORK_SPACE=/space/hall3/sitestore/eccc/crd/ccrn_tmp """ ``` Then after getting the flow to run, when I look at the resulting `work` directory for `find_diamonds`, I see that the `JOB_SCRATCH_SPACE` directory is created, along with the softlink to it, but the `out.txt` file is still created in `runN/work/${CYLC_TASK_CYCLE_POINT}/find_diamonds` and the message in it says the `find_diamonds.sh` script was called in the same directory. So it seems that the `cd` command in the `env-script` isn’t applying to where the task script runs? Looking at the [documentation](https://cylc.github.io/cylc-doc/latest/html/user-guide/task-implementation/job-scripts.html#task-job-scripts) it looks like it _should_ be respected? Is there some other consideration I need here? Also, what is the difference in use case for the `init-script` vs `env-script`? It looks like the `init-script` kind of runs outside of the normal task environment? Beyond that I’m not super sure on the differences. _Update_: I’ve added a simple `echo $(pwd)` statement [here](https://github.com/cylc/cylc-flow/blob/master/cylc/flow/etc/job.sh#L179) to check the directory after each user script is executed, and it seems like the `cd` command is definitely not respected. Looking at the `job.sh` and the produced `job` file further, it looks like the individual user scripts are ran via them being on the `PATH` variable, which makes a lot of sense, however, this does track with my experience as `cd` in executable scripts don’t change the directory of the calling shell? This would be different if a `source` command would be used, but I’m not sure that is desirable as `sourc`ing a bunch of scripts in succession can lead to environment/namespace problems. It’d likely be better for me to just add the `cd $JOB_SCRATCH_SPACE` to the main script ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [March 8, 2022, 11:39pm UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446/24 "2022-03-08T23:39:48Z")
Here’s a small self-contained local-job example that shows the directory change is respected. So we just need to figure out what’s going on in your example. ```ini [scheduling] [[graph]] R1 = foo [runtime] [[root]] env-script = """ cylc __job__ wait_cylc_message_started WORKSPACE=/tmp/$USER/$CYLC_WORKFLOW_ID/$CYLC_TASK_JOB echo env-script 1...$PWD mkdir -p $WORKSPACE cd $WORKSPACE echo env-script 2...$PWD """ [[[environment]]] [[foo]] init-script = "echo init-script: $PWD" script = """ echo "script.........$PWD" echo "the quick brown fox" > out.txt cat ${WORKSPACE}/out.txt """ ``` Run it… ```nohighlight cylc-src/clint$ cylc install && cylc play -n clint ``` … and check the result: ```nohighlight cylc-src/clint$ cylc cat-log clint//1/foo init-script: /home/oliverh/cylc-src/clint Workflow : clint/run3 Task Job : 1/foo/01 (try 1) User@Host: oliverh@niwa-1007823l 2022-03-09T12:31:36+13:00 INFO - started env-script 1.../home/oliverh/cylc-run/clint/run3/work/1/foo env-script 2.../tmp/oliverh/clint/run3/1/foo/01 script........./tmp/oliverh/clint/run3/1/foo/01 the quick brown fox 2022-03-09T12:31:38+13:00 INFO - succeeded ``` ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [March 8, 2022, 11:46pm UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446/25 "2022-03-08T23:46:30Z")
> [@Clint\_Seinen](#): > > Also, what is the difference in use case for the `init-script` vs `env-script` ? As the diagram in the documentation shows, an `init-script` can (e.g.) set variables that affect the Cylc-defined environment; and an `env-script` can affect the user-defined environment, for the job. Most users don’t use either, but they’re available if needed. ---
**Author:** ![Clint\_Seinen](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/clint_seinen/32/185_2.png) [@Clint\_Seinen](https://cylc.discourse.group/u/Clint_Seinen)\ **Post date:** [March 9, 2022, 12:06am UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446/26 "2022-03-09T00:06:15Z")
Hey @hilary.j.oliver I think I nailed down why the `cd` wasn’t being respected - I was trying to just run an executable script as part of `env-script`, i.e. with ```nohighlight env-script = setup-scratch-space ``` This of course won’t work to set any environment considerations/directory moves because it will be ran as an executable. But when I change it to `env-script = source setup-scratch-space`, it works as intended 😄 ! Thanks for all the help - I think @swartn and I have a good path forward now ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [March 9, 2022, 12:07am UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446/27 "2022-03-09T00:07:51Z")
Haha, I was just about to say the same! As an executable, it runs in a subshell and does not affect the parent shell. Hopefully you’re on the right track now 🎉 ---
**Author:** ![MetRonnie](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/metronnie/32/125_2.png) [@MetRonnie](https://cylc.discourse.group/u/MetRonnie)\ **Post date:** [March 9, 2022, 12:31pm UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446/28 "2022-03-09T12:31:56Z")
> [@Clint\_Seinen](#): > > As you say, I think I can easily come up with a way to clean the temporary scratch directories in the `err-script` or `exit-script` , but we may want to leave the temporary directories until a few cycle points ahead, so I’ll probably add some scripting to intelligently call `cylc clean` at the derised points. For your use case I’m afraid it won’t be possible for `cylc clean` to remove your scratch dirs. `cylc clean` does not follow user-created symlinks, so it will remove the symlink itself but not touch the target (this is in contrast to the standard Cylc symlink dirs, which it does follow). ---
**Author:** ![Clint\_Seinen](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/clint_seinen/32/185_2.png) [@Clint\_Seinen](https://cylc.discourse.group/u/Clint_Seinen)\ **Post date:** [March 9, 2022, 4:54pm UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446/29 "2022-03-09T16:54:35Z")
Thanks @MetRonnie - our plan for now is to just add a temporary function _before_ we call `cylc clean` to get rid of the temporary directories. It should give us a work around for all the issues noted above until a more robust solution can be implemented ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [March 9, 2022, 10:50pm UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446/30 "2022-03-09T22:50:12Z")
@Clint_Seinen and @swartn - regarding a more robust solution in the longer term: It is tempting to suggest you have a strangely-configured platform (no offense 😁) but I can see why some might view shared-home-only as a good idea, and in any case I imagine you have no control over that. `$HOME/cylc-run` with underlying symlinks to data areas is very nice for several reasons: everyone (Cylc, users, and 3rd party applications) knows exactly where to find workflow files; and when interacting with remote platforms `ssh` automatically lands you in the home directory. But that aside, we think it would be feasible to build in support for a configurable top-level cylc-run location per `install target` (e.g. `/scratch/$USER/cylc-run` instead of `$HOME/cylc-run`). It wouldn’t be a trivial development, but this is an Open Source project so you’re welcome to propose a solution (via a new Cylc repository Issue) and, post discussion and agreement, work on implementation. Regards, Hilary ---
**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\ **Post date:** [March 10, 2022, 5:20pm UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446/31 "2022-03-10T17:20:24Z")
> [@hilary.j.oliver](#): > > But that aside, we think it would be feasible to build in support for a configurable top-level cylc-run location per `install target` (e.g. `/scratch/$USER/cylc-run` instead of `$HOME/cylc-run` ). If you really want to support this I’d suggest sticking with just being able to change the `cylc-run` bit (i.e. assume `$HOME`). Otherwise you hit the issue of not knowing `$USER` on the remote platform (or other environment variables defined on that platform). ---
**Author:** ![Clint\_Seinen](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/clint_seinen/32/185_2.png) [@Clint\_Seinen](https://cylc.discourse.group/u/Clint_Seinen)\ **Post date:** [March 10, 2022, 5:56pm UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446/32 "2022-03-10T17:56:19Z")
Thanks @hilary.j.oliver and @dpmatthews for all your help! @swartn and I are currently in a sprint to implement our entire model pipeline in `cylc` and making relatively fast progress now that we’ve figured out a way to handle the temporary directories. Once we get things in a good state we are happy to revisit things and we’ll see what we can do 😄 That said, now that we’re moving on this, we actually like having everything in one `cylc-run` directory, even if some of the underlying links are only valid on certain machines - makes it really easy to see what is going on. ---
**Author:** ![swartn](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/swartn/32/186_2.png) [@swartn](https://cylc.discourse.group/u/swartn)\ **Post date:** [March 17, 2022, 1:21am UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446/33 "2022-03-17T01:21:19Z")
Thanks to everyone for their help in this thread. We have made good progress and have a complex model/diagnostic string running across multiple machines sharing their home directories, and we are ambitious about porting to other platforms, and seeing a broader uptake in our organization. ---
**Author:** ![swartn](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/swartn/32/186_2.png) [@swartn](https://cylc.discourse.group/u/swartn)\ **Post date:** [March 17, 2022, 1:25am UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446/34 "2022-03-17T01:25:42Z")
One small issue we have been experiencing is in the tui/gui, the updating across hosts. This question might belong in another thread. But what we see is that we have a host named `daley`, and it’s login nodes have a name like `xc4elogin1`. If we `cylc play` on daley, and then try to monitor results from a different machine, we get an error like `ocket.gaierror: [Errno -2] Name or service not known: 'xc4elogin1'`. Indeed: `socket.gethostbyname_ex('xc4elogin1')` returns an error. However, `socket.gethostbyname_ex('daley')` functions as expected. Is there any configuration guidance that might help with this? I note if we `play` the workflow originally from the other machine, everything seems to work as expected. ---
**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\ **Post date:** [March 17, 2022, 8:33am UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446/35 "2022-03-17T08:33:33Z")
That implies that `hostname -f` is returning `xc4elogin1` but that name isn’t valid on the other hosts. (See [Https messaging fails, communication error between HPC and cylc hosts - #3 by oliver.sanders](https://cylc.discourse.group/t/https-messaging-fails-communication-error-between-hpc-and-cylc-hosts/273/3)) You can hardwire the name in the global config but that then limits you to always running on that host. You can also try using IP addresses instead of host names. See [Global Configuration — Cylc 8.0rc1 documentation](https://cylc.github.io/cylc-doc/8.0rc1/html/reference/config/global.html#global.cylc%5Bscheduler%5D%5Bhost%20self-identification%5D) Alternatively you can configure which hosts you want the scheduler to run on and only choose ones which don’t suffer this problem: [Global Configuration — Cylc 8.0rc1 documentation](https://cylc.github.io/cylc-doc/8.0rc1/html/reference/config/global.html#global.cylc%5Bscheduler%5D%5Brun%20hosts%5D) ---
**Author:** ![swartn](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/swartn/32/186_2.png) [@swartn](https://cylc.discourse.group/u/swartn)\ **Post date:** [March 17, 2022, 2:25pm UTC](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446/36 "2022-03-17T14:25:51Z")
Perfect -thank you @dpmatthews [Previous page](https://cylc.discourse.group/t/setting-up-a-platform-config-for-machines-that-share-a-home-directory/446.md?page=1)