BASH_XTRACEFD error in batch job

Cylc Cylc Support
1

Hi
I have a task running in batch mode, which is suddenly dying. I don’t think, it is timing out. Only error I found in job.err is :

/usr/bin/bash: BASH_XTRACEFD: 19: invalid value for trace file descriptor

what causes it? Is it can force to terminate job?

$ cylc vip --debug .

# flow.cylc:
...
   [[GEMMACH]]
      script = "$TASK_RUN_SCRIPT $TASK_CONFIG_FILE"
   [[get_emissions]]
      inherit = GEMMACH
      platform = my_frontend
      execution time limit = PT15M
      [[[directives]]]
         -l = select=1:ncpus=1:mem=2gb
         -q = development
      [[[environment ]]]
         TASK_CONFIG_FILE = ${CYLC_WORKFLOW_RUN_DIR}/scripts/${CYLC_TASK_NAME}.config
         TASK_RUN_SCRIPT = ${CYLC_WORKFLOW_RUN_DIR}/scripts/${ CYLC_TASK_NAME}.scr

$ more job.status:
CYLC_JOB_INIT_TIME=2026-07-14T17:19:47Z
CYLC_JOB_EXIT=ERR
CYLC_JOB_EXIT_TIME=2026-07-14T17:24:35Z

So job died after 5 minutes, but wall clock given was 15 minutes.

2

That’s to do with redirecting bash set -x output to job.xtrace in Cylc’s debug mode. There’s a 5 year old issue on the cylc-flow repository that suggests (a) it’s caused by a bug in bash version 4 (maybe 4.1 only); and (b) it does not actually cause Cylc jobs to fail. What is your bash version? (type bash --version)

If the executable you run in a Cylc task dies silently - i.e. it does not print anything useful to stdout or stderr before exiting - then there is nothing Cylc can do to help you with that; you need to fix the executable.

Perhaps your executable can be configured to run in its own verbose or debug mode (e.g. script = "my-model.exe --debug" or whatever), so you can see what it’s doing before it dies?

3

Thanks @hilary.j.oliver

bash-5.1$ bash --version
GNU bash, version 5.1.8(1)-release (x86_64-redhat-linux-gnu)

I removed set -x from my script, hence I am no longer getting BASH_XTRACEFD error. Not sure, it makes sense or not, by removing set -x, it helped me in finding, my script was dying due to syntax  error in my script.
Generally set -x helps in debugging your script code as it spits the output line by line to look at the values of your variables.
4

FYI: Running a Cylc workflow with the --debug option (e.g, cylc vip --debug <workflow>) causes all job scripts to be invoked with “xtrace” for debugging. The output is written to job.xtrace and is separated from job.out and job.err (unless using an older version of Bash which does not support this separation).

See also the cylc verbosity command for toggling debug whilst a workflow is running.