So I’ve setup the following script
#!/bin/bash
# setup-scratch-space
export JOB_SCRATCH_SPACE=${WORK_SPACE}/tmp_${USER}_cylc_${CYLC_TASK_NAME}_${CYLC_TASK_CYCLE_POINT}_$(date +"%Y%m%d%H%M")
mkdir $JOB_SCRATCH_SPACE
ln -s $JOB_SCRATCH_SPACE
cd $JOB_SCRATCH_SPACE
and then added this to my flow.cylc
:
[runtime]
[[root]]
env-script = setup-scratch-space
[[find_diamonds]]
platform = ppp3
script = find_diamonds.sh
where find_diamonds.sh
contains
sleep 15
echo "I'm on $(hostname) at $(pwd)" >> out.txt
and the ppp3
platform is defined as
[platforms]
[[ppp3]]
cylc path = /home/rcs001/miniconda3/envs/cylc-8.0rc1/bin
job runner = jobctl
install target = localhost
global init-script = """
export WORK_SPACE=/space/hall3/sitestore/eccc/crd/ccrn_tmp
"""
Then after getting the flow to run, when I look at the resulting work
directory for find_diamonds
, I see that the JOB_SCRATCH_SPACE
directory is created, along with the softlink to it, but the out.txt
file is still created in runN/work/${CYLC_TASK_CYCLE_POINT}/find_diamonds
and the message in it says the find_diamonds.sh
script was called in the same directory. So it seems that the cd
command in the env-script
isn’t applying to where the task script runs?
Looking at the documentation it looks like it should be respected? Is there some other consideration I need here?
Also, what is the difference in use case for the init-script
vs env-script
? It looks like the init-script
kind of runs outside of the normal task environment? Beyond that I’m not super sure on the differences.
Update: I’ve added a simple echo $(pwd)
statement here to check the directory after each user script is executed, and it seems like the cd
command is definitely not respected. Looking at the job.sh
and the produced job
file further, it looks like the individual user scripts are ran via them being on the PATH
variable, which makes a lot of sense, however, this does track with my experience as cd
in executable scripts don’t change the directory of the calling shell? This would be different if a source
command would be used, but I’m not sure that is desirable as sourc
ing a bunch of scripts in succession can lead to environment/namespace problems. It’d likely be better for me to just add the cd $JOB_SCRATCH_SPACE
to the main script