I have encountered an unusual issue that seems to only be impacting me on a shared system. I’ve installed a completely new cylc version 7.8.4 and am using a virtual python environment but still get this error. The error occurs when invoking cylc monitor for an actively running job.
Traceback (most recent call last):
File “/software/depot/cylc/cylc-flow-7.8.4/lib/cherrypy/_cprequest.py”, line 676, in respond
self.hooks.run(‘before_handler’)
File “/software/depot/cylc/cylc-flow-7.8.4/lib/cherrypy/_cprequest.py”, line 123, in run
raise exc
ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
ERROR Bad return code: http://calval3.nrlmry.navy.mil:43041/get_suite_state_summary: 500 Server Error: Internal Server Error
What is unusual is that Cylc seems to be referencing cherrypy in a completely different directory than where my cylc build is located. I can’t seem to find how or why Cylc is attempting to use /software/depot/cylc/cylc-flow-7.8.4/lib/ when this pathway isn’t defined in my PATH or LD_LIBRARY_PATH. (For reference, cylc-flow-7.8.4 is built in my $HOME/.local and that directory is indeed in my PATH)
Cylc 7.8.4 was released in 2019! It would take some effort to count the number of fixes and enhancements that have gone in since then.
The state of the art right now is 8.2.4, and if you have good reason not to migrate to Cylc 8 yet, the latest Cylc 7 release was 7.9.9 (or 7.8.14 if, god forbid, you’re stuck with Python 2.6).
So, it could be a waste of effort chasing down bugs in versions that ancient. Cylc is designed to work with multiple releases installed at once, so you should be able to use 7.9.9 at least…
That said, your symptom does suggest a local Python environment, rather than Cylc, problem.
this pathway isn’t defined in my PATH or LD_LIBRARY_PATH.
You don’t mention PYTHONPATH - are you aware of that variable? Perhaps your /software/depot is set via that, in the default shell environment on your system?
Thanks for the response. I chose cylc 7.8.4 because that is the version a coworker is using without any problems, though I have also tried using versions 7.8.0,4,7,11 and 7.9.6, all with the same exact error. Yes, my PYTHONPATH has no value currently and even before, I would explicitly define it in bashrc but that directory was never assigned value.
It turns out the suite I was attempting to monitor was referencing an old run that had a different PATH specification which contained that /software/depot/cylc pathway. I’ve killed the old run and started a new one but the error is the same (but this time with the correct pathway to cherrypy):
ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
ERROR Bad return code: http://calval3.nrlmry.navy.mil:43095/get_suite_state_summary: 500 Server Error: Internal Server Error
Traceback (most recent call last):
File “/users/tsu/.local/external/cylc/cylc-flow-7.8.4/lib/cherrypy/_cprequest.py”, line 676, in respond
self.hooks.run(‘before_handler’)
File “/users/tsu/.local/external/cylc/cylc-flow-7.8.4/lib/cherrypy/_cprequest.py”, line 123, in run
raise exc
Ok, taking a closer look at the top of your traceback, I see “disabled for fips” so maybe this is something to do with your local security environment:
Option flags for the communication method. Currently only ‘SHA1’ is supported for HTTPS, which alters HTTP Digest Auth to use the SHA1 hash algorithm rather than the standard MD5. This is more secure but is also less well supported by third party web clients including web browsers. You may need to add the ‘SHA1’ option if you are running on platforms where MD5 is discouraged (e.g. under FIPS).
I’ve added options to my [communications] block to be SHA1. So now my global.rc located in /users/tsu/.cylc/7.8.4 reads
process pool size = 4
enable run directory housekeeping = False
run directory rolling archive length = 4
[communication]
method = http
options = SHA1
[monitor]
sort order = alphanumeric
[cylc]
UTC mode = True
Can you check and confirm that that global config is bring picked up in the environment in which you’re running cylc monitor and the scheduler (cylc run)?
Also, are there other Cylc users on this system who do not see this problem?
You can use the cylc get-global-config command in Cylc 7 (or cylc config at Cylc 8) to print the parsed global config file, or selected items from it, to confirm that the right config file is being picked up in your environment.
OK, if I can take it from that statement that there are other Cylc users on the same platform who do not see this error, then perhaps you can compare notes with them to see what the difference is. Do they install their own Cylc releases like you have done, or are they using a central installation?
Finally, if your error traceback is to be taken seriously (which I presume it should be) it does appear to say that something critical has been disabled “for FIPS” on your system:
ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
Maybe there are other U.S. users reading this who know what that means exactly, and whether or not the aforementioned global config setting should get around the restrictions.