# Trouble in submitting job **URL:** \ **Category:** Cylc Support\ **Created:** [May 26, 2025, 12:55am UTC](https://cylc.discourse.group/t/trouble-in-submitting-job/1176 "2025-05-26T00:55:41Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\ **Post date:** [May 26, 2025, 12:55am UTC](https://cylc.discourse.group/t/trouble-in-submitting-job/1176/1 "2025-05-26T00:55:42Z")
---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [May 26, 2025, 3:27am UTC](https://cylc.discourse.group/t/trouble-in-submitting-job/1176/2 "2025-05-26T03:27:27Z")
(@bpabla50 - did you forget to write the body of your message? ) ---
**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\ **Post date:** [May 26, 2025, 12:43pm UTC](https://cylc.discourse.group/t/trouble-in-submitting-job/1176/3 "2025-05-26T12:43:58Z")
Thanks. Not sure, what happened, I did write the message in body. I will try to remember, as I have cleaned my work directory. I defined the name of machine “cedar” in global.cylc. What is wrong with my global.cylc and flow.cycl? ```console [bpabla68@cedar1 8.4.2]$ pwd /home/bpabla68/.cylc/flow/8.4.2 [bpabla68@cedar1 8.4.2]$ more global.cylc ``` ```ini [platforms] [[cedar]] hosts = localhost install target = localhost job runner = slurm ``` ```console $ more flow.cylc ``` ```ini scheduler] allow implicit tasks = True [scheduling] [[graph]] R1 = """ run_prep_10km """ [runtime] [[run_prep_10km]] platform = cedar script = """ ${MY_DEFAULT_PATH}/runjob/run_prep_10km.sh """ [[[environment]]] MY_DEFAULT_PATH = /home/bpabla68/projects/def-yorkaqrl/bpabla68/cylc_learning/ex3 ``` ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [May 26, 2025, 11:49pm UTC](https://cylc.discourse.group/t/trouble-in-submitting-job/1176/4 "2025-05-26T23:49:14Z")
Your global and workflow configs look fine. What’s the error you’re seeing? ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [May 27, 2025, 7:15am UTC](https://cylc.discourse.group/t/trouble-in-submitting-job/1176/5 "2025-05-27T07:15:53Z")
This section of the troubleshooting docs has some suggestions: > **[Troubleshooting — Cylc 8.4.2 documentation](https://cylc.github.io/cylc-doc/stable/html/user-guide/troubleshooting.html#my-job-submit-failed)** ---
**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\ **Post date:** [May 27, 2025, 3:16pm UTC](https://cylc.discourse.group/t/trouble-in-submitting-job/1176/6 "2025-05-27T15:16:51Z")
2025-05-27T08:13:26-07:00 INFO - New flow: 1 (original flow from 1) 2025-05-27T08:13:26 2025-05-27T08:13:26-07:00 INFO - [1/run\_prep\_10km:waiting(runahead)] =\> waiting 2025-05-27T08:13:26-07:00 INFO - [1/run\_prep\_10km:waiting] =\> waiting(queued) 2025-05-27T08:13:27-07:00 INFO - [1/run\_prep\_10km:waiting(queued)] =\> waiting 2025-05-27T08:13:27-07:00 INFO - [1/run\_prep\_10km:waiting] =\> preparing 2025-05-27T08:13:28-07:00 ERROR - [jobs-submit cmd] cylc jobs-submit --path=/bin --path=/usr/bin --path=/usr/local/bin --path=/sbin --path=/usr/sbin --path=/usr/local/sbin – ‘$HOME/cylc-run/ex3\_workflow/ex3\_runname/log/job’ 1/run\_prep\_10km/01 [jobs-submit ret\_code] 1 [jobs-submit out] 2025-05-27T08:13:28-07:00|1/run\_prep\_10km/01|1|None 2025-05-27T08:13:28-07:00 ERROR - [1/run\_prep\_10km/01:preparing] submission failed 2025-05-27T08:13:28-07:00 INFO - [1/run\_prep\_10km/01:preparing] =\> submit-failed 2025-05-27T08:13:28-07:00 WARNING - [1/run\_prep\_10km/01:submit-failed] did not complete the required outputs: ⨯ ┆ succeeded 2025-05-27T08:13:28-07:00 ERROR - Incomplete tasks: \* 1/run\_prep\_10km did not complete the required outputs: ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [May 27, 2025, 3:38pm UTC](https://cylc.discourse.group/t/trouble-in-submitting-job/1176/7 "2025-05-27T15:38:08Z")
The job has failed to submit, this is usually a setup / configuration issue. To find out why the job failed to submit, see the `job-activity.log` file. ---
**Author:** ![bpabla50](https://avatars.discourse-cdn.com/v4/letter/b/b9bd4f/32.png) [@bpabla50](https://cylc.discourse.group/u/bpabla50)\ **Post date:** [May 27, 2025, 4:09pm UTC](https://cylc.discourse.group/t/trouble-in-submitting-job/1176/8 "2025-05-27T16:09:07Z")
Thanks Oliver. Yah, job-activity.log gave the reason. On our system, we can not run anything from /home. My all directories are on non-home directories(workflow/runname; ~/cylc-run is a link to scratch FS) except ~/.cylc/flow/8.4.2/global.cylc. I s the stuff from global.cylc is actually run or sourced? Even I tried created ~/.cylc as a link to non-home FS, even that did not work. I am issuing my cylc play command from non-home FS, where I installed my worlflow/event i.e. should submit job from non-home FS. So not very clear, why it is trying to submit from /home, as all my directories are soft links, which point to non- home filesystems. ```auto jobs-submit cmd] cylc jobs-submit --path=/bin --path=/usr/bin --path=/usr/local/bin --path=/sbin --path=/usr/sbin --path=/usr/local/sbin -- '$HOME/cylc-run/ex3_workflow/ex3_runname/log/job' 1/run_prep_10km/01 [jobs-submit ret_code] 1 [jobs-submit out] 2025-05-27T08:13:28-07:00|1/run_prep_10km/01|1|None 2025-05-27T08:13:28-07:00 [STDERR] sbatch: error: --------------------------------------------------------------------- 2025-05-27T08:13:28-07:00 [STDERR] sbatch: error: Submitting jobs from directories residing in /home is not permitted. 2025-05-27T08:13:28-07:00 [STDERR] sbatch: error: Transfer your files to a directory in /scratch or /project and submit 2025-05-27T08:13:28-07:00 [STDERR] sbatch: error: the job from there. 2025-05-27T08:13:28-07:00 [STDERR] sbatch: error: --------------------------------------------------------------------- 2025-05-27T08:13:28-07:00 [STDERR] sbatch: error: Batch job submission failed: Unspecified error ~ ``` ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [May 27, 2025, 11:44pm UTC](https://cylc.discourse.group/t/trouble-in-submitting-job/1176/9 "2025-05-27T23:44:20Z")
> [@bpabla50](#): > > ```auto > jobs-submit cmd] cylc jobs-submit --path=/bin --path=/usr/bin --path=/usr/local/bin --path=/sbin --path=/usr/sbin --path=/usr/local/sbin -- '$HOME/cylc-run/ex3_workflow/ex3_runname/log/job' 1/run_prep_10km/01 > [jobs-submit ret_code] 1 > [jobs-submit out] 2025-05-27T08:13:28-07:00|1/run_prep_10km/01|1|None > sbatch: error: --------------------------------------------------------------------- > sbatch: error: Submitting jobs from directories residing in /home is not permitted. > sbatch: error: Transfer your files to a directory in /scratch or /project and submit > sbatch: error: the job from there. > sbatch: error: --------------------------------------------------------------------- > sbatch: error: Batch job submission failed: Unspecified error > > ``` Yikes, I’ve never seen this before - it is a horrendous restriction that does not make any sense! It is normal for `/home` disk to be restricted: - small quota - not optimal for intensive IO But that just means users should not write model output etc. to their home directories. Not allowing jobs to submit job FROM `/home` is a weird and ineffective way to enforce this, if that’s what the intention is. You can still have jobs write to `/home` if you want, regardless of where they are submitted from! ```auto cylc jobs-submit --path=... -- '$HOME/cylc-run/ex3_workflow/ex3_runname/log/job' 1/run_prep_10km/01 ``` Is the rejection based on **where you submit the job from**?, or **where the job script is located**? If the latter, have you configured `cylc install` to symlink your workflow run directories to non-`/home` locations, so that your installed workflow files (and generated job scripts) are NOT actually located under `/home`? If so, that is perfectly sufficient to avoid misusing your home directory. But note that Cylc will still use the standard location (under `/home`) - which is just a symlink! - that’s how we always know how to access the files no matter where they are actually stored. Your system should NOT be rejecting the job purely because `/home` appears in the job script path when it is only a symlink! ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [May 30, 2025, 8:04am UTC](https://cylc.discourse.group/t/trouble-in-submitting-job/1176/10 "2025-05-30T08:04:58Z")
(I accidentally posted a response to this, now deleted, on the wrong topic, in which I suggested a Cylc code tweak to `realpath` the job script path in the job submission command. However team members reminded me that wouldn’t work for remote hosts with a different `$HOME` …better suggestion coming soon!) ---
**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\ **Post date:** [May 30, 2025, 11:05am UTC](https://cylc.discourse.group/t/trouble-in-submitting-job/1176/11 "2025-05-30T11:05:54Z")
Create a script named `qsub-realpath` which simply calls `qsub realpath $1` Then configure `job runner command template = /path/to/qsub-realpath %(job)s` [https://cylc.github.io/cylc-doc/stable/html/reference/config/global.html#global.cylc[platforms][%3Cplatform%20name%3E]job%20runner%20command%20template](https://cylc.github.io/cylc-doc/stable/html/reference/config/global.html#global.cylc%5Bplatforms%5D%5B%3Cplatform%20name%3E%5Djob%20runner%20command%20template) That might work? ---
**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\ **Post date:** [May 30, 2025, 11:09am UTC](https://cylc.discourse.group/t/trouble-in-submitting-job/1176/12 "2025-05-30T11:09:09Z")
(Except for Slurm (sbatch) not PBS (qsbub)). ---
**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\ **Post date:** [May 30, 2025, 11:51am UTC](https://cylc.discourse.group/t/trouble-in-submitting-job/1176/13 "2025-05-30T11:51:39Z")
Here’s an example of Dave’s suggestion above: ```auto # global.cylc [platforms] [[myhpc]] hosts = login1, login2 job runner = slurm # get Cylc to use the custom submission script job runner command template = /path/to/sbatch-realpath %(job)s [install] [[symlink dirs]] [[[myhpc]]] # get Cylc to create the job script somewhere other than $HOME log = $DATADIR ``` > **[Global Configuration — Cylc 8.4.2 documentation](https://cylc.github.io/cylc-doc/stable/html/reference/config/global.html#global.cylc%5Bplatforms%5D%5B%3Cplatform%20name%3E%5Djob%20runner%20command%20template)** > **[Global Configuration — Cylc 8.4.2 documentation](https://cylc.github.io/cylc-doc/stable/html/reference/config/global.html#global.cylc%5Binstall%5D%5Bsymlink%20dirs%5D%5B%3Cinstall%20target%3E%5Dlog)**