KeyboardInterrupt error from cylc 7.8.11

Rose Rose Support
1

The task worked correctly when using version intel mpi 2019.9.304 in the Cylc workflow, but after switching to version intel mpi 2021.3.0, the following error occurred.

We are using Cylc 7.8.11 and Python 2.7.14 libraries. There are no plans to update or change the libraries.

Could you provide some advice on whether there are any parts within the Cylc library that I can debug?

job.err

File “/opt/lib/apps/rose/2019.01.7/lib/python/rose/app_run.py”, line 459, in _command
task = get()
File “/opt/lib/apps/python/2.7.14/lib/python2.7/multiprocessing/queues.py”, line 374, in get
stdin=sys.stdin)
File “/opt/lib/apps/rose/2019.01.7/lib/python/rose/popen.py”, line 226, in run_ok
racquire()
racquire()
return recv()
racquire()
racquire()
racquire()
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
ret_code, stdout, stderr = self.run(*args, **kwargs)
File “/opt/lib/apps/rose/2019.01.7/lib/python/rose/popen.py”, line 159, in run
stdout, stderr = proc.communicate(stdin)
File “/opt/lib/apps/python/2.7.14/lib/python2.7/subprocess.py”, line 476, in communicate
self.wait()
File “/opt/lib/apps/python/2.7.14/lib/python2.7/subprocess.py”, line 1077, in wait
pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
File “/opt/lib/apps/python/2.7.14/lib/python2.7/subprocess.py”, line 121, in _eintr_retry_call
return func(*args)
KeyboardInterrupt
forrtl: error (69): process interrupted (SIGINT)
Image PC Routine Line Source
um-atmos.exe 000000000297112B Unknown Unknown Unknown
libpthread-2.28.s 00001468A91C8B20 Unknown Unknown Unknown
libpthread-2.28.s 00001468A91C78B0 _read Unknown Unknown
libmpi.so.12.0.0 00001468A9FB4FDF Unknown Unknown Unknown
libmpi.so.12.0.0 00001468A9FB12A3 Unknown Unknown Unknown
libmpi.so.12.0.0 00001468A9E141AC Unknown Unknown Unknown
libmpi.so.12.0.0 00001468A9D8DA9C Unknown Unknown Unknown
libmpi.so.12.0.0 00001468A9D07097 Unknown Unknown Unknown
libmpi.so.12.0.0 00001468A9EA7DDB Unknown Unknown Unknown
libmpi.so.12.0.0 00001468A9FB0A76 Unknown Unknown Unknown
libmpi.so.12.0.0 00001468A99E7A06 Unknown Unknown Unknown
libmpi.so.12.0.0 00001468A9CECD26 Unknown Unknown Unknown
libmpi.so.12.0.0 00001468A9CEC17B MPI_Init Unknown Unknown
libmpifort.so.12. 00001468AB270D9B MPI_INIT Unknown Unknown
um-atmos.exe 00000000027C50AB mod_oasis_method 97 mod_oasis_method.F90
um-atmos.exe 00000000026CFDE0 Unknown Unknown Unknown
um-atmos.exe 0000000002662A17 Unknown Unknown Unknown
um-atmos.exe 00000000004191EF Unknown Unknown Unknown
um-atmos.exe 0000000000418C3A Unknown Unknown Unknown
um-atmos.exe 0000000000418BE2 Unknown Unknown Unknown
libc-2.28.so 00001468A867D7B3 __libc_start_main Unknown Unknown
um-atmos.exe 0000000000418AEE Unknown Unknown Unknown

2

Hi,

These lines are important:

KeyboardInterrupt

forrtl: error (69): process interrupted (SIGINT)

Note KeyboardInterrupt is Python syntax for the SIGINT kill signal.

This suggests that something was killing process(es) within this job.

These lines appear to come from the MPI job, not Cylc or Rose (note, rose/popen is the bit of Rose that runs your job), so this is probably an issue with MPI / the system, not Rose itself.

What happens when you invoke the MPI job manually (outside of Cylc / Rose)?

We are using Cylc 7.8.11 and Python 2.7.14 libraries. There are no plans to update or change the libraries.

Cylc 7, Rose 2019 and Python 2.7 are all deprecated and will need to be updated in due course.

3

Note:

  • Python 2 is typically not available on next-gen platforms, and is being removed from some current systems because it has not received security updates since 2020
  • Cylc 8 installs easily into a conda virtual environment, so you don’t have to rely on the system Python library
  • We won’t be able to support Cylc 7 indefinitely
  • Cylc 8 is much more advanced, modern, workflow manager

… in other words, you should upgrade to Cylc 8!