Running cylc tasks on compute nodes that cannot see /home

Cylc Cylc Support
1

Hi,

We’ve just got access to our new supercomputer and now trying to set up to submit jobs using Rose/Cylc and having problems. The problem with cylc referencing its directories via ~ rear its head again. :grimacing:

In a similar setup to our current HPC, ARCHER, the compute nodes on the new HPC also cannot see the /home disk. With ARCHER, you may remember, we were able to get around the issue of cylc referencing its directories via ~ by setting init-script = "export HOME=$DATADIR" in the suite.rc file. This worked, I believe, because the job script commands prior to the aprun were executed on the job launcher (MOM) nodes which could see /home. On ARCHER2 (SLURM) the job script and all commands prior to the srun are run on pe0 (as there are no job launcher nodes) so cannot see /home at all.

Any ideas on how we can get around this are much appreciated?

Hacking around with the cylc generated job script and manually submitting on the HPC, things like the path to the stdout and error don’t work…

#SBATCH --output=cylc-run/u-al624/log/job/1/archer/08/job.out
#SBATCH --error=cylc-run/u-al624/log/job/1/archer/08/job.err

It needs the full path to the /work disk, etc

Hope that makes sense and happy to provide any more information.

Regards
Ros.

2

Hi Ros,

I’m sure this is solvable, but it might be difficult to figure out without access to the system. Do you know if any of the UK Cylc team have, or can get, access to this new HPC of yours?

For the moment, a few thoughts off the top of my head, after looking back at a cylc-7.8.6 task job script:

  • literal ~ isn’t used anywhere
  • $HOME is referenced twice in the job script, most importantly for the suite run directory location
  • and also once in the lib/cylc/job.sh file, which holds some boilerplate shell functions for job scripts

$SBATCH --output=cylc-run/ros/log/...

That path doesn’t reference $HOME (or ~) and is presumably relative to the directory in which the sbatch command is executed. On a normal system that would happen to be the user’s home directory because that’s where ssh dumps you (and Cylc submits job scripts via ssh).

The problem with setting $HOME manually in the job script might be that it isn’t consistent with the location used to “initialize” the standard run directory structure etc. on the remote host. That’s done over ssh by the cylc remote-init command, as I recall, so that should also be the default location you end up in when you ssh in to the machine.

Regards,
Hilary

3

I wonder whether using the sbatch option “–chdir=/my/data/dir” will help.
(You can’t use environment variables so “–chdir=$DATADIR” won’t work.
However, that should be easy to fix with a bit of Jinja2.)

4

Hi Hilary, Dave,

Thanks for the suggestions. I will take a look and let you know how I get on.

Cheers,
Ros.

5

Hi Dave, Hilary,

Typically HPC had to go offline for a bit :roll_eyes: So the --chdir fixes the job stderr/stdout file problem, but I’m not getting very far with the other issues being unfamiliar with what’s going on inside cylc.

If someone in the Cylc team would be able to take a look, it would be very much appreciated. There should be no problem in getting an account on the HPC for you.

Best Regards,
Ros.

6

I made some progress this afternoon and think I might have got around the problem with a combination of using sbatch option --chdir and then modifying the lib/cylc/job_file.py script to write an extra line in the task job script to redefine HOME only when host is archer2 (may need to tie this down to “and only when submitting to parallel queue” but we’ll see).

cylc__job__inst__cylc_env() {
    ....
    export HOME="/work/n02/n02/$USER"
    export CYLC_SUITE_RUN_DIR="$HOME/cylc-run/u-al624"
    export CYLC_SUITE_DEF_PATH="${HOME}/cylc-run/u-al624"
    export CYLC_SUITE_DEF_PATH_ON_SUITE_HOST="/home/ros/cylc-run/u-al624"
    export CYLC_SUITE_UUID="c7231788-8004-49c2-82d6-f442a1618c6d"
    ....
}

It’s not the most elegant solution and whether or not this is likely to have unintended consequences I have no idea - it seems to be working with the limited testing I’ve done so far…

Cheers,
Ros.

1 Like
7

Cylc jobs run in Bash login shells which means that Bash sources the .bash_profile file.

So you should be able to “shim” that export HOME line into the .bash_profile on Archer2, or potentially if able into /etc/bashrc.

Cylc uses login shells largely to allow this kind of environment meddling.

8

I assume you can’t do anything in .bash_profile because you have no home directory!

I’d suggest trying to define global-init-script for the relevant host:

Alternatively try <cylc-dir>/etc/job-init-env.sh:


However, note that support for this has been removed at Cylc 8 so let us know if you end up relying on this.
9

Hi Dave, Oliver,

Yes that is correct we can’t do anything in .bash_profile which is real pain. :frowning:

I’ve tried the two suggestions above but they both run too late the flow.

I need something that will run before cylc__job__inst__cylc_env. Looking in lib/cylc/job.sh it looks like this is the first thing run in cylc__job_main …

Thanks.

10

Defining global-init-script will work but only if I redefine the 2 problematic CYLC_variables. Not sure if that is a safer option or not:

# GLOBAL-INIT-SCRIPT:
. /work/n02/n02/shared/umshared/bin/rose-um-env
export HOME=$DATADIR
export CYLC_SUITE_RUN_DIR=$HOME/cylc-run/$CYLC_SUITE_NAME
export CYLC_SUITE_DEF_PATH=${HOME}/cylc-run/$CYLC_SUITE_NAME
11

Hi @Rosalyn_Hatcher,

Out of curiosity, what’s the rationale for no home-dir on this HPC? Is it that users should never need an interactive login (for “security reasons”?) and only submit jobs to it remotely? I suppose more of this might be coming our way, so we have should have a robust solution for it.

What directory do you end up in if you do ssh to the host?

Defining global-init-script will work but only if I redefine the 2 problematic CYLC_ variables. Not sure if that is a safer option or not:

I’m a bit worried about Cylc scheduler access to the remote run-dir, which doesn’t happen inside of a job script. E.g. for remote init (set up the basic run-dir structure before the first remote job) and job poll and kill (which need to read the remote job status file). And job log retrieval.

You should be able to test all this by playing with a small test suite that submits a single sleep 30 job (or whatever) to the platform (if you aren’t already doing that!).

Hilary

12

Hi Hilary,

Sorry for the confusion, there is a home-dir on the HPC. So when you ssh in for interactive login you land as usual on home. It’s just the batch compute nodes that cannot see the /home disk at all (not even to source a .bash_profile to set up user environment). It’s a complete pain in the butt, not just for cylc. The reasoning we’ve been given for this restriction is again one of performance.

So, consequently the cylc remote-init, job poll, retrigger, kill etc work ok in my test suite as the ssh lands on the /home disk and can work as normal. It’s just persuading cylc in a compute job not to access the suite-run dir via /home which I think is just needing to modify or redefine the setting of CYLC_SUITE_RUN_DIR & CYLC_SUITE_DEF_PATH in the job script.

Hope that make sense.
Cheers,
Ros.

1 Like
13

Ah got it, thanks, I should have read more carefully!

That makes the problem sound rather less horrifying, phew.

Hilary

14

That looks safe to me. I don’t think you actually need $HOME so this should be enough:

CYLC_SUITE_RUN_DIR=${DATADIR}/cylc-run/${CYLC_SUITE_NAME}
CYLC_SUITE_DEF_PATH=${CYLC_SUITE_RUN_DIR}
1 Like
15

Great. Thanks Dave.
I did debate about the HOME=$DATADIR - I will check that out - don’t want more in there than is necessary.