A-ha!!! @russbnavy - you’re not running in Cylc 7 back-compat mode are you? (i.e., with a suite.rc
config file, not flow.cylc
). The 8.2.3 release announcement reminded me that back-compat mode had a more severe version of the submit-failed handling bug.
Running my test case in back-compat mode does indeed result in this:
CRITICAL - [1/D submitted job:01 flows:none] submission failed
INFO - [1/D submitted job:01 flows:none] => submit-failed
DEBUG - [1/D submit-failed job:01 flows:none] task proxy removed (finished) # !!!
Hopefully that explains it. If so, there’s nothing wrong with your system. You just need to upgrade to 8.2.3, or upgrade your workflow config to Cylc 8.