(@bpabla50 - did you forget to write the body of your message? )
Thanks. Not sure, what happened, I did write the message in body.
I will try to remember, as I have cleaned my work directory.
I defined the name of machine “cedar” in global.cylc. What is wrong with my global.cylc and flow.cycl?
[bpabla68@cedar1 8.4.2]$ pwd
/home/bpabla68/.cylc/flow/8.4.2
[bpabla68@cedar1 8.4.2]$ more global.cylc
[platforms]
[[cedar]]
hosts = localhost
install target = localhost
job runner = slurm
$ more flow.cylc
scheduler]
allow implicit tasks = True
[scheduling]
[[graph]]
R1 = """
run_prep_10km
"""
[runtime]
[[run_prep_10km]]
platform = cedar
script = """ ${MY_DEFAULT_PATH}/runjob/run_prep_10km.sh """
[[[environment]]]
MY_DEFAULT_PATH = /home/bpabla68/projects/def-yorkaqrl/bpabla68/cylc_learning/ex3
Your global and workflow configs look fine.
What’s the error you’re seeing?
This section of the troubleshooting docs has some suggestions:
2025-05-27T08:13:26-07:00 INFO - New flow: 1 (original flow from 1) 2025-05-27T08:13:26
2025-05-27T08:13:26-07:00 INFO - [1/run_prep_10km:waiting(runahead)] => waiting
2025-05-27T08:13:26-07:00 INFO - [1/run_prep_10km:waiting] => waiting(queued)
2025-05-27T08:13:27-07:00 INFO - [1/run_prep_10km:waiting(queued)] => waiting
2025-05-27T08:13:27-07:00 INFO - [1/run_prep_10km:waiting] => preparing
2025-05-27T08:13:28-07:00 ERROR - [jobs-submit cmd] cylc jobs-submit --path=/bin --path=/usr/bin --path=/usr/local/bin --path=/sbin --path=/usr/sbin --path=/usr/local/sbin – ‘$HOME/cylc-run/ex3_workflow/ex3_runname/log/job’ 1/run_prep_10km/01
[jobs-submit ret_code] 1
[jobs-submit out] 2025-05-27T08:13:28-07:00|1/run_prep_10km/01|1|None
2025-05-27T08:13:28-07:00 ERROR - [1/run_prep_10km/01:preparing] submission failed
2025-05-27T08:13:28-07:00 INFO - [1/run_prep_10km/01:preparing] => submit-failed
2025-05-27T08:13:28-07:00 WARNING - [1/run_prep_10km/01:submit-failed] did not complete the required outputs:
⨯ ┆ succeeded
2025-05-27T08:13:28-07:00 ERROR - Incomplete tasks:
* 1/run_prep_10km did not complete the required outputs:
The job has failed to submit, this is usually a setup / configuration issue.
To find out why the job failed to submit, see the job-activity.log file.
Thanks Oliver. Yah, job-activity.log gave the reason. On our system, we can not run anything from /home. My all directories are on non-home directories(workflow/runname; ~/cylc-run is a link to scratch FS) except ~/.cylc/flow/8.4.2/global.cylc. I s the stuff from global.cylc is actually run or sourced?
Even I tried created ~/.cylc as a link to non-home FS, even that did not work.
I am issuing my cylc play command from non-home FS, where I installed my worlflow/event i.e. should submit job from non-home FS.
So not very clear, why it is trying to submit from /home, as all my directories are soft links, which point to non- home filesystems.
jobs-submit cmd] cylc jobs-submit --path=/bin --path=/usr/bin --path=/usr/local/bin --path=/sbin --path=/usr/sbin --path=/usr/local/sbin -- '$HOME/cylc-run/ex3_workflow/ex3_runname/log/job' 1/run_prep_10km/01
[jobs-submit ret_code] 1
[jobs-submit out] 2025-05-27T08:13:28-07:00|1/run_prep_10km/01|1|None
2025-05-27T08:13:28-07:00 [STDERR] sbatch: error: ---------------------------------------------------------------------
2025-05-27T08:13:28-07:00 [STDERR] sbatch: error: Submitting jobs from directories residing in /home is not permitted.
2025-05-27T08:13:28-07:00 [STDERR] sbatch: error: Transfer your files to a directory in /scratch or /project and submit
2025-05-27T08:13:28-07:00 [STDERR] sbatch: error: the job from there.
2025-05-27T08:13:28-07:00 [STDERR] sbatch: error: ---------------------------------------------------------------------
2025-05-27T08:13:28-07:00 [STDERR] sbatch: error: Batch job submission failed: Unspecified error
~Yikes, I’ve never seen this before - it is a horrendous restriction that does not make any sense!
It is normal for /home disk to be restricted:
- small quota
- not optimal for intensive IO
But that just means users should not write model output etc. to their home directories. Not allowing jobs to submit job FROM /home is a weird and ineffective way to enforce this, if that’s what the intention is. You can still have jobs write to /home if you want, regardless of where they are submitted from!
cylc jobs-submit --path=... -- '$HOME/cylc-run/ex3_workflow/ex3_runname/log/job' 1/run_prep_10km/01
Is the rejection based on where you submit the job from?, or where the job script is located?
If the latter, have you configured cylc install to symlink your workflow run directories to non-/home locations, so that your installed workflow files (and generated job scripts) are NOT actually located under /home?
If so, that is perfectly sufficient to avoid misusing your home directory.
But note that Cylc will still use the standard location (under /home) - which is just a symlink! - that’s how we always know how to access the files no matter where they are actually stored.
Your system should NOT be rejecting the job purely because /home appears in the job script path when it is only a symlink!
(I accidentally posted a response to this, now deleted, on the wrong topic, in which I suggested a Cylc code tweak to realpath the job script path in the job submission command. However team members reminded me that wouldn’t work for remote hosts with a different $HOME …better suggestion coming soon!)
Create a script named qsub-realpath which simply calls qsub realpath $1
Then configure
job runner command template = /path/to/qsub-realpath %(job)s
https://cylc.github.io/cylc-doc/stable/html/reference/config/global.html#global.cylc[platforms][%3Cplatform%20name%3E]job%20runner%20command%20template
That might work?
(Except for Slurm (sbatch) not PBS (qsbub)).
Here’s an example of Dave’s suggestion above:
# global.cylc
[platforms]
[[myhpc]]
hosts = login1, login2
job runner = slurm
# get Cylc to use the custom submission script
job runner command template = /path/to/sbatch-realpath %(job)s
[install]
[[symlink dirs]]
[[[myhpc]]]
# get Cylc to create the job script somewhere other than $HOME
log = $DATADIR