Suite shutting down with "name 'archive_bool' is not defined" error

Cylc Cylc Support
1

Hi,
I have a suite (running under cylc 7.9.1) which shuts down within a few seconds of launching. The task logs show that a few tasks have completed, while a couple active at the time show error messages like:

    2020-06-28T11:32:13Z WARNING - Message send failed, try 1 of 7: Cannot connect: https://w-cylc03.maui.niwa.co.nz:43012/put_messages: HTTPSConne
ctionPool(host='w-cylc03.maui.niwa.co.nz', port=43012): Max retries exceeded with url: /put_messages (Caused by NewConnectionError('<requests.p
ackages.urllib3.connection.VerifiedHTTPSConnection object at 0xcea250>: Failed to establish a new connection: [Errno 111] Connection refused',)
)

The suite log reports (skipping all the INFO statements)

...

2020-06-28T11:32:13Z ERROR - name ‘archive_bool’ is not defined
Traceback (most recent call last):
File “/scale_wlg_persistent/filesets/opt_nesi/share/cylc/7.9.1/lib/cylc/scheduler.py”, line 247, in start
self.run()
File “/scale_wlg_persistent/filesets/opt_nesi/share/cylc/7.9.1/lib/cylc/scheduler.py”, line 1596, in run
self.process_task_pool()
File “/scale_wlg_persistent/filesets/opt_nesi/share/cylc/7.9.1/lib/cylc/scheduler.py”, line 1226, in process_task_pool
self.pool.match_dependencies()
File “/scale_wlg_persistent/filesets/opt_nesi/share/cylc/7.9.1/lib/cylc/task_pool.py”, line 950, in match_dependencies
itask.state.satisfy_me(all_task_outputs)
File “/scale_wlg_persistent/filesets/opt_nesi/share/cylc/7.9.1/lib/cylc/task_state.py”, line 289, in satisfy_me
if prereq.satisfy_me(all_task_outputs):
File “/scale_wlg_persistent/filesets/opt_nesi/share/cylc/7.9.1/lib/cylc/prerequisite.py”, line 204, in satisfy_me
self._all_satisfied = self._conditional_is_satisfied()
File “/scale_wlg_persistent/filesets/opt_nesi/share/cylc/7.9.1/lib/cylc/prerequisite.py”, line 182, in conditional_is_satisfied
res = eval(self.conditional_expression)
File “”, line 1, in
NameError: name ‘archive_bool’ is not defined
2020-06-28T11:32:13Z CRITICAL - Suite shutting down - name ‘archive_bool’ is not defined
2020-06-28T11:32:13Z WARNING - interp_tide_cons.20200623T00: orphaned task (running)
2020-06-28T11:32:13Z WARNING - merge_nc_wind.20200623T00: orphaned task (running)
2020-06-28T11:32:13Z WARNING - fetch_nc_ss_bc.20200623T12: orphaned task (submitted)
2020-06-28T11:32:13Z WARNING - fetch_restart_cold.20200623T00: orphaned task (submitted)
2020-06-28T11:32:13Z WARNING - wave_bc_ok.20200623T12: orphaned task (submitted)
2020-06-28T11:32:20Z INFO - DONE
Traceback (most recent call last):
File "/scale_wlg_persistent/filesets/opt_nesi/share/cylc/7.9.1/bin/cylc-run", line 25, in
main(is_restart=False)
File “/scale_wlg_persistent/filesets/opt_nesi/share/cylc/7.9.1/lib/cylc/scheduler_cli.py”, line 134, in main
scheduler.start()
File “/scale_wlg_persistent/filesets/opt_nesi/share/cylc/7.9.1/lib/cylc/scheduler.py”, line 276, in start
raise exc
NameError: name ‘archive_bool’ is not defined

The strange thing is that I have several other suites with virtually the same structure (but run with different data) that don’t give this problem, so it’s hard to track down any problem with any of the tasks…

2

Hi Richard,

I think something about the form of your task names, or perhaps similarity between several of them, has triggered a bug in the system.

Since I have access to the same HPC :grinning: I tracked down your suite definition, and I can reproduce the error by running it in dummy mode. Investigating now …

Hilary

3

Problem diagnosed. It’s actually somewhat surprising that no one else has run into this bug.

Cylc evaluates conditional prerequisites by constructing a Python expression that reflects the trigger in the graph string, and that can go wrong when a task name gets repeated at the end of another task name in the same expression: NAME | prefixNAME => blah. In your case, Richard, these lines trigger the problem:

wave_bc_ok | archive_wave_bc_ok => rose_prune
wave_bc_ok | archive_wave_bc_ok => !poll_wave_bc

We’ll get this fixed for next release. In the meantime, the workaround is to change the name of your archive task slightly, e.g. to archive_wave_bc-ok or archive_wave_bc__ok.

Hilary

4

Thanks Hilary, much appreciated!
The suite in question is intended to form the “wave” part of a coupled NWP-wave forecasting suite, so in starting from an existing wave forecast suite I did change some of these task names to be more obviously “wave-related” to avoid confusion when we bolt on the NWP bits. Those problematic lines previously looked like

wave_bc_ok | archive_bc_ok => rose_prune
wave_bc_ok | archive_bc_ok => !poll_wave_bc

which doesn’t trigger the problem. So it’s a case of being caught out making what I thought would be purely cosmetic changes!

1 Like