cylc-7.9.1
recently after a maintenance upgrade our suites began failing with:
.service/ssl.pem does not exist.
.service/ssl.cert does not exist.
Previously upon install and initiation of a suite.rc
in response to
2023-10-18T19:21:05Z ERROR - no HTTPS/OpenSSL support. Aborting...
2023-10-18T19:21:05Z ERROR - "No HTTPS support.
Configure user's global.rc to use HTTP."
we changed (in global.rc)
method = http -> method =https
This change (prior to the ‘upgrade’) would
permit suites to run.
a recent workaround in response to the ‘missing’ files (post upgraded)
consisted of editing global.rc:
< host = `hostname -f`
< method = hardwired
we were then informed that ‘http’ would no longer be permitted and
that ‘https’ is required.
my questions:
where do I begin?
will a patch to cylc be required?
it would be best if the global.rc file generated by
cylc get-global-config > global.rc
could be used.
thank you for your time.
If you want to use HTTPS (in any context, not just with Cylc) you need the right SSL libraries on the system, and an SSL certificate.
For SSL use with Cylc 7, the installation requirements are here: 3. Installation — The Cylc Suite Engine 7.9.3 documentation
If you have those packages installed, Cylc will automatically generate the certificate ssl.cert and private key ssl.pem files. Presumably they have not been installed?
The only alternative is plain HTTP, which it seems you’re not allowed to use anymore.
OR migrate to Cylc 8! (Which doesn’t use HTTP(S) for job comms, and manages software dependencies automatically, via pip or conda).
thank you H:
I/ll review the installation requirements.
Run cylc check-software and it will tell you whether your installation meets the requirements for the https comms.
So, interestingly, I’ve had similar issues on our RHEL 8 upgrade system to where check-software on Cylc 7.9.7 reports no issues but connections to the suite instances hang on SSL handshake and we haven’t had any luck solving it. We’ve tried turning off SE linux, opening up ip tables all the way, generating keys with different cyphers, etc and just can’t get past the SSL handshake. Has anyone else reported that kind of behavior? We gave up and disabled HTTPS for now.
Edit:
The specific test we used was given a suite host running on HOST:PORT:
openssl s_client -msg -debug -state -connect HOST:PORT
After connecting and getting some handshake hex, it hangs on:
SSL_connect:SSLv3/TLS write client hello
We’re seeing this behavior on two seperate clusters now…
@russbnavy - thanks for the info.
Has anyone else reported that kind of behavior?
This is the first I’ve heard, but maybe your move to RHEL 8 on HPC is ahead of others still using Cylc 7.
check-software on Cylc 7.9.7 reports no issues
check-software only checks that the right packages are installed and available in your environment.
The specific test we used was given a suite host running on HOST:PORT:…
Just to confirm, you are seeing this behaviour with openssl itself, independent of Cylc? If so, that’s (arguably!) good, and I would hope that it has been reported elsewhere. Have you tried asking at the openssl project?
Unfortunately we haven’t seen this issue with any other tools that utilize OpenSSL but good point, I’ll try reaching out to the OpenSSL project to see if there’s a way to glean more information besides hanging at the first stage of the handshake.
Edit: we’re seeing it on SUSE Enterprise 15 SP4 as well as RHEL 8… So perhaps it’s a Python 2.7 bug with the latest OpenSSL…
check-software shows (my edit):
*OPTIONAL SOFTWARE for the HTTPS communications layer*
Python:requests (2.4.2+).FOUND& min. version MET (2.9.1)
Python:urllib3 (any)..........FOUND (1.13.1)
Python:OpenSSL (any).....NOT FOUND (-)
looking at the NOT FOUND.
edit:
Python:OpenSSL (any)....................FOUND (21.0.0)
yahoo
1 Like
testing russbnavy's openssl debug (above)
...
edit:
SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:332:
I have https working on Centos 8 with pyOpenSSL 19.0 installed via
sudo pip2 install "pyOpenSSL<19.1"
I can’t remember why I needed “<19.1” but it seems to work.
Hey! That was it for us, downgrading pyOpenSSL to <19.1 from 20.0.1 did the trick. Thanks!
2 Likes
modifying the definition file of the container referenced
Cylc and Singularity container
a container running cylc-7.9.8 ran a simple suite using the HTTPS layer.
key changes (including pyOpenSSl<19.0) are listed at the referenced URL.