# Cylc8 on archer2 **URL:** \ **Category:** Cylc Support\ **Created:** [November 9, 2021, 12:10pm UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396 "2021-11-09T12:10:49Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![mhagdorn](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/mhagdorn/32/174_2.png) [@mhagdorn](https://cylc.discourse.group/u/mhagdorn)\ **Post date:** [November 9, 2021, 12:10pm UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/1 "2021-11-09T12:10:49Z")
Hi there, I am trying to get cylc8 to work on archer2. I am very much struggling getting a simple hello world cylc8 workflow to run. The problem is that The home directory is not visible on the worker nodes, see also a similar [question about cylc7](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282) from last year. Following [the instructions](https://cylc.discourse.group/t/running-cylc-tasks-on-compute-nodes-that-cannot-see-home/282/14) I set the following variables in my global init script: > export DATADIR=/work/n02/n02/n02magi > export CYLC\_WORKFLOW\_RUN\_DIR=${DATADIR}/cylc-run/${CYLC\_WORKFLOW\_NAME} > export CYLC\_WORKFLOW\_DEF\_PATH=${CYLC\_WORKFLOW\_RUN\_DIR} Unfortunately, it is still looking for the script in the wrong place: > /var/spool/slurmd/job634423/slurm\_script: line 67: /home/n02/n02/n02magi/cylc-run/hello-archer/.service/etc/job.sh: No such file or directory > /var/spool/slurmd/job634423/slurm\_script: line 68: cylc\_\_job\_\_main: command not found Any suggestions? Thanks magnus ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [November 9, 2021, 12:29pm UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/2 "2021-11-09T12:29:30Z")
Hello, So long as your login and compute nodes share one common filesystem it is possible to make this set up work using the `symlink dirs` configuration. [https://cylc.github.io/cylc-doc/8.0b2/html/reference/config/global.html#global.cylc[install][symlink%20dirs]](https://cylc.github.io/cylc-doc/8.0b2/html/reference/config/global.html#global.cylc%5Binstall%5D%5Bsymlink%20dirs%5D) This is a platform configuration which allows you to move your `cylc-run` (or other workflow sub directories) to other locations. In your `global.cylc`, tell Cylc to move the workflow run dir onto the shared filesystem: ```ini [install] [[symlink dirs]] [[[archer2]]] # <= the archer2 install target run = /shared/fs/ ``` When the first task in a workflow runs on a platform, Cylc will set up the symlink and install onto the shared filesystem: ```auto $HOME/cylc-run/ my-workflow -> /shared/fs/cylc-run/my-workflow ``` Cylc commands will follow this symlink. I’ve added a cylc-doc issue to document this edge-case - [platforms: platform configuration for compute nodes with no $HOME · Issue #315 · cylc/cylc-doc · GitHub](https://github.com/cylc/cylc-doc/issues/315). Let us know how you get on, Oliver. ---
**Author:** ![mhagdorn](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/mhagdorn/32/174_2.png) [@mhagdorn](https://cylc.discourse.group/u/mhagdorn)\ **Post date:** [November 9, 2021, 12:56pm UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/3 "2021-11-09T12:56:47Z")
Thanks for the quick reply. Unfortunately it is not working yet. So I have: ```ini n02magi@uan01:~> cat ~/.cylc/flow/global.cylc [install] [[symlink dirs]] [[[archer2]]] run = /work/n02/n02/n02magi [platforms] [[archer2]] job runner = slurm hosts = localhost install target = archer2 ``` However, it still installs the workflow into $HOME ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [November 9, 2021, 3:53pm UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/4 "2021-11-09T15:53:38Z")
Ok, not sure why that isn’t working, ideas: - This `global.cylc` configuration needs to be on the host where you issue the `cylc install ; cylc play` commands. - Could try running `cylc config` on the host where you issue the `cylc install ; cylc play` commands to make sure the config is being picked up properly. - Symlink dirs changes take effect when the workflow is first installed. If it has already been installed you will need to remove it (in all locations) or create a fresh install to test the changes. ---
**Author:** ![mhagdorn](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/mhagdorn/32/174_2.png) [@mhagdorn](https://cylc.discourse.group/u/mhagdorn)\ **Post date:** [November 9, 2021, 5:29pm UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/5 "2021-11-09T17:29:58Z")
I got it to install into the correct place using ```auto [install] [[symlink dirs]] [[[localhost]]] run = /work/n02/n02/n02magi [[archer2]] job runner = slurm hosts = localhost install target = localhost cylc path = /work/n02/n02/n02magi/bin/ ``` However it is trying to run ```auto . "$HOME/cylc-run/hello-archer/.service/etc/job.sh" cylc __job__ main ``` The variable HOME exists put the file system is just not visible. So I guess I need to modify how that script is generated. ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [November 9, 2021, 5:43pm UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/6 "2021-11-09T17:43:23Z")
> I got it to install into the correct place using Great. Sorry, yes that should be `[install][symlink dirs]localhost` rather than `archer2` if > So I guess I need to modify how that script is generated. You will not be able to modify this without editing the Cylc source code as the `global init-script` (documented in the Cylc 7 issue) is run _after_ this job file has been sourced. I’ve taken a look in, Cylc 7 hardcoded the job file location whereas Cylc 8 uses the `$HOME` environment variable to locate it which breaks your setup. Should be a fairly simple fix, I’ll see if I can work something out. ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [November 9, 2021, 6:14pm UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/7 "2021-11-09T18:14:06Z")
> > > The archer2 machine does not have access to \`$HOME\` on the compute nodes. > > On …the whole we have handled this use case ok, however, one sticking point remains. > > Because Cylc 8 sources the job script from the \`cylc-run\` directory this is a problem for them: > > \`\`\`bash > . "$HOME/cylc-run/two/.service/etc/job.sh" > \`\`\` > > This PR uses \`$0\` to locate the job script rather than \`$HOME\` to avoid this issue. > > \*\*Requirements check-list\*\* > \- \[x\] I have read \`CONTRIBUTING.md\` and added my name as a Code Contributor. > \- \[x\] Contains logically grouped changes (else tidy your branch by rebase). > \- \[x\] Does not contain off-topic changes (use other PRs for other changes). > \- \[x\] Applied any dependency changes to both \`setup.py\` and > \`conda-environment.yml\`. > \- \[x\] Already covered by existing tests. > \- \[x\] Appropriate change log entry included. > \- \[x\] No documentation update required. ---
**Author:** ![mhagdorn](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/mhagdorn/32/174_2.png) [@mhagdorn](https://cylc.discourse.group/u/mhagdorn)\ **Post date:** [November 10, 2021, 9:28am UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/8 "2021-11-10T09:28:23Z")
excellent, thanks. I’ll have a go at trying this out ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [November 10, 2021, 9:40am UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/9 "2021-11-10T09:40:29Z")
Unfortunately we’ve found issues with this approach and are looking at alternatives, hopefully will have something soon. ---
**Author:** ![mhagdorn](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/mhagdorn/32/174_2.png) [@mhagdorn](https://cylc.discourse.group/u/mhagdorn)\ **Post date:** [November 10, 2021, 9:43am UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/10 "2021-11-10T09:43:04Z")
Let me know when there is something I can try out or if I can help in any other way ---
**Author:** ![mhagdorn](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/mhagdorn/32/174_2.png) [@mhagdorn](https://cylc.discourse.group/u/mhagdorn)\ **Post date:** [November 10, 2021, 9:45am UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/11 "2021-11-10T09:45:02Z")
I just noticed the comment in the issue tracker. The use of the home directory for cylc-run has puzzled me. Why is that not configurable and that way it would be a lot easier to find the correct location. ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [November 10, 2021, 10:31am UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/12 "2021-11-10T10:31:44Z")
The `cylc-run` directory used to be configurable at Cylc 7. On the surface, this seems like a simple feature, however, the distributed nature of Cylc makes this surprisingly hard to implement, the feature was never fully functional. Hardcoding the path of the `cylc-run` directory has greatly simplified Cylc’s internal logic and should work for all Unix systems. Except, unfortunately Archer2. Note that many Unix systems rely on paths relative to `$HOME` such as Bash and SSH. ---
**Author:** ![mhagdorn](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/mhagdorn/32/174_2.png) [@mhagdorn](https://cylc.discourse.group/u/mhagdorn)\ **Post date:** [November 10, 2021, 11:47am UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/13 "2021-11-10T11:47:08Z")
Although bash/ssh can take command line arguments (or I could source a script to setup an environment) and work without $HOME. I guess I will ask the archer2 team if they would consider setting HOME to the work directory on the workers. ---
**Author:** ![mhagdorn](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/mhagdorn/32/174_2.png) [@mhagdorn](https://cylc.discourse.group/u/mhagdorn)\ **Post date:** [November 10, 2021, 1:20pm UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/14 "2021-11-10T13:20:14Z")
Is there a way I can inject a line into the submission script so that I can set the HOME environment variable? ---
**Author:** ![mhagdorn](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/mhagdorn/32/174_2.png) [@mhagdorn](https://cylc.discourse.group/u/mhagdorn)\ **Post date:** [November 10, 2021, 1:28pm UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/15 "2021-11-10T13:28:56Z")
Thinking about it… I can set the HOME var using the slurm option `--export=HOME=/work/n02/n02/n02magi`. The problem is that cylc starts a login shell which I guess loses this setting. Will a non-login shell help? ---
**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\ **Post date:** [November 10, 2021, 4:05pm UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/16 "2021-11-10T16:05:05Z")
> [@mhagdorn](#): > > Is there a way I can inject a line into the submission script so that I can set the HOME environment variable? At present you can’t insert a line where you’d need to but we’re hoping to propose a change which will allow this to work for you … ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [November 10, 2021, 9:14pm UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/17 "2021-11-10T21:14:19Z")
> [@oliver.sanders](#): > > Hardcoding the path of the `cylc-run` directory has greatly simplified Cylc’s internal logic and should work for all Unix systems. Except, unfortunately Archer2. For completeness, I’ll note that while `~/cylc-run` is now hardcoded as the top level run directory, the new workflow installation process can be configured to automatically symlink sub-directories to other locations. (In case anyone reading this thinks their home disk quota has to be big enough to handle all workflow output). Having a standard location for the top level run directory not only simplifies Cylc internals, it’s also important for 3rd party tools that (for instance) need to view workflow outputs, or job logs, or databases. ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [November 26, 2021, 1:28pm UTC](https://cylc.discourse.group/t/cylc8-on-archer2/396/18 "2021-11-26T13:28:38Z")
Ok, we think we have a solution using a combination of `symlink dirs` and `global init-script`. If you’re able to try it out the proposed changes are here: - [job-script: run the global init-script before sourcing the job script by oliver-sanders · Pull Request #4534 · cylc/cylc-flow · GitHub](https://github.com/cylc/cylc-flow/pull/4534) - [platforms: document how to run jobs on a homeless platform. by oliver-sanders · Pull Request #344 · cylc/cylc-doc · GitHub](https://github.com/cylc/cylc-doc/pull/344) This should make it into the first Cylc 8 release candidate.