How to enable FIPS mode for libcrypto and libssl packaged with Python?
Question:
I have a python application which comes packaged with Python and Libcrypto and LibSSL shared objects. The application was built with Openssl Fips Module 2.0. These shared objects are used by Python’s request module and urllib3 under the hood to make TLS requests.
I enabled the OPENSSL_FIPS flag in the environment where i was building the application. Now if want to check whether the shared objects have the fips mode enabled when i take them out of the development environment and put them in another machine, how can i do that?
How can i check whether the fips mode is enabled or not? And if it isn’t, how can i enable the fips mode for these shared objects?
Additional Details that might help:
OpenSSL Version: 1.0.2h (built from source)
Fips Module: 2.0.12 (built from source)
Python: 3.6
OS: Ubuntu 16.04 LTS
Please let me know if any additional details are required.
Thanks!
Answers:
I’ve built the OpenSSL-FIPS module using regular flags (e.g.: no-asm, shared, some ancient ciphers disabled):
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q049320993]> ~/sopr.sh
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###
[064bit-prompt]> ls ssl/build/bin ssl/build/lib
ssl/build/bin:
c_rehash openssl
ssl/build/lib:
engines libcrypto.a libcrypto.so libcrypto.so.1.0.0 libssl.a libssl.so libssl.so.1.0.0 pkgconfig
And started playing a little bit with it:
[064bit-prompt]> ssl/build/bin/openssl version
OpenSSL 1.0.2h-fips 3 May 2016 (Library: OpenSSL 1.0.2g 1 Mar 2016)
Note the "(Library: OpenSSL 1.0.2g 1 Mar 2016)" part. That (being present) states that the openssl executable is OK (expected version), but it’s using a wrong libcrypto (it’s the one that comes installed by default on the system – under /lib – and typically that one isn’t built with FIPS support).
It must load our libraries, and that is done by setting LD_LIBRARY_PATH (the same behavior could have also been achieved by setting an env var when building OpenSSL that would have set the rpath in the openssl executable, but I forgot, and I didn’t want to build it again):
[064bit-prompt]> LD_LIBRARY_PATH=ssl/build/lib ssl/build/bin/openssl version
OpenSSL 1.0.2h-fips 3 May 2016
Now, that the setup is successful, let’s dive into OPENSSL_FIPS env var:
[064bit-prompt]> LD_LIBRARY_PATH=ssl/build/lib ssl/build/bin/openssl md5 ./code00.py
MD5(./code00.py)= 47fb26ec5d1ca16d3537fe7fd12ea529
[064bit-prompt]> LD_LIBRARY_PATH=ssl/build/lib ssl/build/bin/openssl sha1 ./code00.py
SHA1(./code00.py)= 5188a221ba61309e78e70004285bc6fd148701b6
[064bit-prompt]> OPENSSL_FIPS=1 LD_LIBRARY_PATH=ssl/build/lib ssl/build/bin/openssl sha1 ./code00.py
SHA1(./code00.py)= 5188a221ba61309e78e70004285bc6fd148701b6
[064bit-prompt]> OPENSSL_FIPS=1 LD_LIBRARY_PATH=ssl/build/lib ssl/build/bin/openssl md5 ./code00.py
Error setting digest md5
139778679649944:error:060A80A3:digital envelope routines:FIPS_DIGESTINIT:disabled for fips:fips_md.c:180:
As seen from above, the md5 hash behavior is influenced by the OPENSSL_FIPS env var (when FIPS mode is on, its usage is not allowed).
Notes:
-
Most likely, newer OpenSSL-FIPS versions will also have sha1 disabled since it’s considered weak, so the invariant should be switched to one of the sha2 hash functions family (e.g. sha256) or even better, sha3 (older OpenSSL versions might not have it)
-
From my PoV this is a little too restrictive, as there might be cases when a hashing algorithm is needed for purposes that don’t care about security, and more complex (and also time consuming) allowed algorithms still have to be used
Since OPENSSL_FIPS env var is handled at openssl executable level, which will be bypassed (as libcrypto will be used directly), it’s no use for the current situation, so we have to go deeper. These are the functions that control FIPS mode in a loaded libcrypto instance:
They will be used to read / write FIPS mode. In order to test whether FIPS mode is really set, md5 hash (from the example above) will be used.
code00.py:
#!/usr/bin/env python
import ctypes as cts
import ssl
import sys
libcrypto = cts.CDLL("libcrypto.so.1.0.0")
#ssleay = libcrypto.SSLeay
#ssleay.argtypes = ()
#ssleay.restype = cts.c_ulong
fips_mode = libcrypto.FIPS_mode
fips_mode.argtypes = ()
fips_mode.restype = cts.c_int
fips_mode_set = libcrypto.FIPS_mode_set
fips_mode_set.argtypes = (cts.c_int,)
fips_mode_set.restype = cts.c_int
def main(*argv):
text = b""
print("OPENSSL_VERSION: {:s}".format(ssl.OPENSSL_VERSION))
enable_fips = len(sys.argv) > 1
print("FIPS_mode(): {:d}".format(fips_mode()))
if enable_fips:
print("FIPS_mode_set(1): {:d}".format(fips_mode_set(1)))
print("FIPS_mode(): {:d}".format(fips_mode()))
#print("SSLeay: {:X}".format(ssleay()))
import hashlib
print("SHA1: {:s}".format(hashlib.sha1(text).hexdigest()))
print("MD5: {:s}".format(hashlib.md5(text).hexdigest()))
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}n".format(" ".join(elem.strip() for elem in sys.version.split("n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("nDone.")
sys.exit(rc)
Notes:
-
Set argtypes and restype for the 2 functions. Check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati’s answer) for more details
-
The md5 hashing algorithm is provided at Python level by [Python.Docs]: hashlib – Secure hashes and message digests
-
Important: the import hashlib statement is located after setting the FIPS mode (and not at the file beginning, as it should be), because hashlib does some caching at import time, so it captures the FIPS value at import time, and doesn’t care if it changes afterwards
Output:
[064bit-prompt]> LD_LIBRARY_PATH=ssl/build/lib ./code00.py
Python 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609] 064bit on linux
OPENSSL_VERSION: OpenSSL 1.0.2h-fips 3 May 2016
FIPS_mode(): 0
FIPS_mode(): 0
SHA1: da39a3ee5e6b4b0d3255bfef95601890afd80709
MD5: d41d8cd98f00b204e9800998ecf8427e
Done.
[064bit-prompt]> LD_LIBRARY_PATH=ssl/build/lib ./code00.py 1
Python 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609] 064bit on linux
OPENSSL_VERSION: OpenSSL 1.0.2h-fips 3 May 2016
FIPS_mode(): 0
FIPS_mode_set(1): 1
FIPS_mode(): 1
SHA1: da39a3ee5e6b4b0d3255bfef95601890afd80709
fips_md.c(149): OpenSSL internal error, assertion failed: Digest Final previous FIPS forbidden algorithm error ignored
Aborted (core dumped)
As seen, setting FIPS mode via CTypes, really sets it.
I don’t know why it SegFaults, but the md5 related code is there only for testing purposes, so it’s not needed in production.
I remember that on some Nix version (might be RH based), FIPS mode could also be set (globally for the system), by editing some entry (under /proc ?), but I can’t remember it.
A more elegant approach would be to expose Python wrappers for the 2 functions.
Check [Python.Bugs]: FIPS_mode() and FIPS_mode_set() functions in Python (ssl), I’ve also submitted a patch for Python 3.4 (where they were exposed by the ssl module), but it was rejected based on the following arguments (out of which the 1st 2 are relevant):
-
FIPS is a bad standard
-
OpenSSL will drop support for it
-
It breaks up generality
You can apply it to Python 3.6 (I don’t think it will work OOTB, since line numbers most likely changed), and (obviously) you’ll have to build Python from sources.
Bottom line:
-
There’s a big difference between FIPS working and FIPS validated, as I’m sure you’ve read on [OpenSSL]: User Guide for the OpenSSL FIPS Object Module v2.0
-
[AskUbuntu]: Enable FIPS 140-2 in ubuntu might also contain some useful info
Some references that might be useful (although last ones could be a bit too "deep"):
-
[SO]: How to compile python3 on RHEL with SSL? SSL cannot be imported (@CristiFati’s answer)
-
-
[SO]: OpenSSL FIPS_mode_set not working in Python cryptography library (@CristiFati’s answer)
-
Update #0
It just stroke me, the behavior that you’re encountering on [SO]: Not able to call FIPS_mode_set() of libcrypto.so with Python ctypes [duplicate] might also be related to the wrong libcrypto being loaded (check the openssl version tests w / wo LD_LIBRARY_PATH from the beginning).
A non FIPS capable OpenSSL will still export the 2 functions, but they both simply return 0.
[064bit-prompt]> ./code00.py 1
Python 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609] 064bit on linux
OPENSSL_VERSION: OpenSSL 1.0.2g 1 Mar 2016
FIPS_mode(): 0
FIPS_mode_set(1): 0
FIPS_mode(): 0
SHA1: da39a3ee5e6b4b0d3255bfef95601890afd80709
MD5: d41d8cd98f00b204e9800998ecf8427e
Done.
So, make sure to load the correct libraries by specifying LD_LIBRARY_PATH ! (there are other ways, but this is the most straightforward one).
I have a python application which comes packaged with Python and Libcrypto and LibSSL shared objects. The application was built with Openssl Fips Module 2.0. These shared objects are used by Python’s request module and urllib3 under the hood to make TLS requests.
I enabled the OPENSSL_FIPS flag in the environment where i was building the application. Now if want to check whether the shared objects have the fips mode enabled when i take them out of the development environment and put them in another machine, how can i do that?
How can i check whether the fips mode is enabled or not? And if it isn’t, how can i enable the fips mode for these shared objects?
Additional Details that might help:
OpenSSL Version: 1.0.2h (built from source)
Fips Module: 2.0.12 (built from source)
Python: 3.6
OS: Ubuntu 16.04 LTS
Please let me know if any additional details are required.
Thanks!
I’ve built the OpenSSL-FIPS module using regular flags (e.g.: no-asm, shared, some ancient ciphers disabled):
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q049320993]> ~/sopr.sh ### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ### [064bit-prompt]> ls ssl/build/bin ssl/build/lib ssl/build/bin: c_rehash openssl ssl/build/lib: engines libcrypto.a libcrypto.so libcrypto.so.1.0.0 libssl.a libssl.so libssl.so.1.0.0 pkgconfig
And started playing a little bit with it:
[064bit-prompt]> ssl/build/bin/openssl version OpenSSL 1.0.2h-fips 3 May 2016 (Library: OpenSSL 1.0.2g 1 Mar 2016)
Note the "(Library: OpenSSL 1.0.2g 1 Mar 2016)" part. That (being present) states that the openssl executable is OK (expected version), but it’s using a wrong libcrypto (it’s the one that comes installed by default on the system – under /lib – and typically that one isn’t built with FIPS support).
It must load our libraries, and that is done by setting LD_LIBRARY_PATH (the same behavior could have also been achieved by setting an env var when building OpenSSL that would have set the rpath in the openssl executable, but I forgot, and I didn’t want to build it again):
[064bit-prompt]> LD_LIBRARY_PATH=ssl/build/lib ssl/build/bin/openssl version OpenSSL 1.0.2h-fips 3 May 2016
Now, that the setup is successful, let’s dive into OPENSSL_FIPS env var:
[064bit-prompt]> LD_LIBRARY_PATH=ssl/build/lib ssl/build/bin/openssl md5 ./code00.py MD5(./code00.py)= 47fb26ec5d1ca16d3537fe7fd12ea529 [064bit-prompt]> LD_LIBRARY_PATH=ssl/build/lib ssl/build/bin/openssl sha1 ./code00.py SHA1(./code00.py)= 5188a221ba61309e78e70004285bc6fd148701b6 [064bit-prompt]> OPENSSL_FIPS=1 LD_LIBRARY_PATH=ssl/build/lib ssl/build/bin/openssl sha1 ./code00.py SHA1(./code00.py)= 5188a221ba61309e78e70004285bc6fd148701b6 [064bit-prompt]> OPENSSL_FIPS=1 LD_LIBRARY_PATH=ssl/build/lib ssl/build/bin/openssl md5 ./code00.py Error setting digest md5 139778679649944:error:060A80A3:digital envelope routines:FIPS_DIGESTINIT:disabled for fips:fips_md.c:180:
As seen from above, the md5 hash behavior is influenced by the OPENSSL_FIPS env var (when FIPS mode is on, its usage is not allowed).
Notes:
-
Most likely, newer OpenSSL-FIPS versions will also have sha1 disabled since it’s considered weak, so the invariant should be switched to one of the sha2 hash functions family (e.g. sha256) or even better, sha3 (older OpenSSL versions might not have it)
-
From my PoV this is a little too restrictive, as there might be cases when a hashing algorithm is needed for purposes that don’t care about security, and more complex (and also time consuming) allowed algorithms still have to be used
Since OPENSSL_FIPS env var is handled at openssl executable level, which will be bypassed (as libcrypto will be used directly), it’s no use for the current situation, so we have to go deeper. These are the functions that control FIPS mode in a loaded libcrypto instance:
They will be used to read / write FIPS mode. In order to test whether FIPS mode is really set, md5 hash (from the example above) will be used.
code00.py:
#!/usr/bin/env python
import ctypes as cts
import ssl
import sys
libcrypto = cts.CDLL("libcrypto.so.1.0.0")
#ssleay = libcrypto.SSLeay
#ssleay.argtypes = ()
#ssleay.restype = cts.c_ulong
fips_mode = libcrypto.FIPS_mode
fips_mode.argtypes = ()
fips_mode.restype = cts.c_int
fips_mode_set = libcrypto.FIPS_mode_set
fips_mode_set.argtypes = (cts.c_int,)
fips_mode_set.restype = cts.c_int
def main(*argv):
text = b""
print("OPENSSL_VERSION: {:s}".format(ssl.OPENSSL_VERSION))
enable_fips = len(sys.argv) > 1
print("FIPS_mode(): {:d}".format(fips_mode()))
if enable_fips:
print("FIPS_mode_set(1): {:d}".format(fips_mode_set(1)))
print("FIPS_mode(): {:d}".format(fips_mode()))
#print("SSLeay: {:X}".format(ssleay()))
import hashlib
print("SHA1: {:s}".format(hashlib.sha1(text).hexdigest()))
print("MD5: {:s}".format(hashlib.md5(text).hexdigest()))
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}n".format(" ".join(elem.strip() for elem in sys.version.split("n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("nDone.")
sys.exit(rc)
Notes:
-
Set argtypes and restype for the 2 functions. Check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati’s answer) for more details
-
The md5 hashing algorithm is provided at Python level by [Python.Docs]: hashlib – Secure hashes and message digests
-
Important: the
import hashlibstatement is located after setting the FIPS mode (and not at the file beginning, as it should be), because hashlib does some caching at import time, so it captures the FIPS value at import time, and doesn’t care if it changes afterwards
Output:
[064bit-prompt]> LD_LIBRARY_PATH=ssl/build/lib ./code00.py Python 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609] 064bit on linux OPENSSL_VERSION: OpenSSL 1.0.2h-fips 3 May 2016 FIPS_mode(): 0 FIPS_mode(): 0 SHA1: da39a3ee5e6b4b0d3255bfef95601890afd80709 MD5: d41d8cd98f00b204e9800998ecf8427e Done. [064bit-prompt]> LD_LIBRARY_PATH=ssl/build/lib ./code00.py 1 Python 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609] 064bit on linux OPENSSL_VERSION: OpenSSL 1.0.2h-fips 3 May 2016 FIPS_mode(): 0 FIPS_mode_set(1): 1 FIPS_mode(): 1 SHA1: da39a3ee5e6b4b0d3255bfef95601890afd80709 fips_md.c(149): OpenSSL internal error, assertion failed: Digest Final previous FIPS forbidden algorithm error ignored Aborted (core dumped)
As seen, setting FIPS mode via CTypes, really sets it.
I don’t know why it SegFaults, but the md5 related code is there only for testing purposes, so it’s not needed in production.
I remember that on some Nix version (might be RH based), FIPS mode could also be set (globally for the system), by editing some entry (under /proc ?), but I can’t remember it.
A more elegant approach would be to expose Python wrappers for the 2 functions.
Check [Python.Bugs]: FIPS_mode() and FIPS_mode_set() functions in Python (ssl), I’ve also submitted a patch for Python 3.4 (where they were exposed by the ssl module), but it was rejected based on the following arguments (out of which the 1st 2 are relevant):
-
FIPS is a bad standard
-
OpenSSL will drop support for it
-
It breaks up generality
You can apply it to Python 3.6 (I don’t think it will work OOTB, since line numbers most likely changed), and (obviously) you’ll have to build Python from sources.
Bottom line:
-
There’s a big difference between FIPS working and FIPS validated, as I’m sure you’ve read on [OpenSSL]: User Guide for the OpenSSL FIPS Object Module v2.0
-
[AskUbuntu]: Enable FIPS 140-2 in ubuntu might also contain some useful info
Some references that might be useful (although last ones could be a bit too "deep"):
-
[SO]: How to compile python3 on RHEL with SSL? SSL cannot be imported (@CristiFati’s answer)
-
[SO]: OpenSSL FIPS_mode_set not working in Python cryptography library (@CristiFati’s answer)
Update #0
It just stroke me, the behavior that you’re encountering on [SO]: Not able to call FIPS_mode_set() of libcrypto.so with Python ctypes [duplicate] might also be related to the wrong libcrypto being loaded (check the openssl version tests w / wo LD_LIBRARY_PATH from the beginning).
A non FIPS capable OpenSSL will still export the 2 functions, but they both simply return 0.
[064bit-prompt]> ./code00.py 1 Python 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609] 064bit on linux OPENSSL_VERSION: OpenSSL 1.0.2g 1 Mar 2016 FIPS_mode(): 0 FIPS_mode_set(1): 0 FIPS_mode(): 0 SHA1: da39a3ee5e6b4b0d3255bfef95601890afd80709 MD5: d41d8cd98f00b204e9800998ecf8427e Done.
So, make sure to load the correct libraries by specifying LD_LIBRARY_PATH ! (there are other ways, but this is the most straightforward one).