PyInstaller FATAL FIPS SELFTEST FAILURE
Question:
I have used PyInstaller to package up a python application. When I run the application I receive this error:
crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE
Abort (core dumped)
If I run the script via "python3 scriptname.py" it runs fine. I do not use OpenSSL in my python scripts. I believe PyInstaller is using something that is not FIPS compliant.
With FIPS disabled I am able to successfully run the PyInstaller packaged application.
How can I debug this error? Is there a way to patch PyInstaller to make it FIPS compliant?
Answers:
This usually happens, if you installed python with FIPS enabled/disabled or if Python is consuming a module/library/something, where Python/ or that particular library/python pyInstaller may be consuming an artifact (that was built on some machine previously, with FIPS Disabled); then, using such executables (built with FIPS disabled) on another machine(where FIPS is actually enabled) i.e. where FIPS setting doesn’t match with the machine, where you built the app in python or that particular library, then, you’ll get such errors.
Ex: Let’s say a package manager ex: yum / zypper installed python without openssl (which is a pre-install/pre-requisite for python app, which needs Openssl functionality). If python is installed without Openssl first already installed, then any Python app, which needs openssl processing, Python won’t work, even if you install openssl later on the machine; Python will still spit out bunch of gnarly openssl errors :–) big grin.
Similarly, if something was installed/packaged/built using FIPS disabled (on some machine), it can be an app or any library that you are consuming… then later, if you try to run an app or an app(consuming that library/package) on a machine, where FIPS = 1 (enabled) then you’ll see this error.
In my case, cmake was failing for running simple cmake --version and all I did was used a cmake .tar bundle which may have been compiled on a machine, where source and target machine’s FIPS settings were same i.e. in my case, cmake tar bundle came from a FIPS enabled machine, and when I used it on this other new machine where FIPS was enabled, it worked!
MISC Info on OpenSSL errors:
The effects of self-test failures in the Module differ depending on the type of self-test that failed.
The FIPS_mode_set() function verifies the integrity of the runtime executable using a HMAC SHA-256 digest,
which is computed at build time. If this computed HMAC SHA-256 digest matches the stored, known digest, then
the power-up self-test (consisting of the algorithm-specific Pairwise Consistency and Known Answer tests) is
performed.
Non-fatal self-test errors transition the module into an error state. The application must be restarted to recover
from these errors. The non-fatal self-test errors are:
FIPS_R_FINGERPRINT_DOES_NOT_MATCH – The integrity verification check failed
FIPS_R_FIPS_SELFTEST_FAILED – a known answer test failed
FIPS_R_SELFTEST_FAILED – a known answer test failed
FIPS_R_TEST_FAILURE – a known answer test failed (RSA); pairwise consistency test failed (DSA)
FIPS_R_PAIRWISE_TEST_FAILED – a pairwise consistency test during DSA or RSA key generation failed
FIPS_R_FIPS_MODE_ALREADY_SET – the application initializes the FIPS mode when it is already initialized
These errors are reported through the regular ERR interface of the shared libraries and can be queried by
functions such as ERR_get_error(). See the OpenSSL Module manual page for the function description.
A fatal error occurs only when the module is in the error state (a self-test has failed) and the application calls a
crypto function of the module that cannot return an error in normal circumstances (void return functions). The
error message: ‘FATAL FIPS SELFTEST FAILURE‘ is printed to stderr and the application is terminated with the
abort() call.
The only way to recover from a fatal error is to restart the application. If failures persist, you must reinstall the Module. If you downloaded the software, verify the package hash to confirm a proper download.
IMPORTANT NOTE:
If you installed a newer version of OpenSSL x.y.z (rpm/etc) and didn’t reboot/restart your machine, some of the libraries which consume openssl will be vulnerable / error out too with similar errors.
You can test this by running:
uptime command to see how long your server/machine has been UP n running.sudo lsof | grep libssl.so or to Find processes running with deleted OpenSSL libraries, run:
sudo lsof | grep DEL.*libssl
So, try rebooting your machine and see if the errors goes away and check the output of the above commands.
For more info, refer this link: https://raymii.org/s/snippets/Find_all_services_using_libssl_to_restart_after_an_OpenSSL_update.html
This seems to be an old post but just in case…
I use PyInstaller quite often, but after upgrading from CentOS 7.x to RHEL 8.x I began experiencing the same behavior. Here is what I found… PyInstaller pulls in the correct openssl shared libraries but does not bundle in the the associated hmac’s ( in our case /usr/lib64/.libcrypto.so.1.1.hmac and /usr/lib64/.libssl.so.1.1.hmac ).
I chose to just bundle the hmac files generated by the system. I did this by editing the associated .spec file that pyinstaller uses to create the bundled executable.
Changed:
binaries=[],
to:
binaries=[('/usr/lib64/.libcrypto.so.1.1.hmac','.'),'/usr/lib64/.libssl.so.1.1.hmac','.')],
No other modifications were required. If you don’t use a .spec file then I suppose this could also be accomplished via the cmdline.
Piggybacking off of gcgold’s answer (11766308) since I don’t have enough reputation to comment yet.
That solution worked for us on RHEL8 as well, with minor tweaks. In our case, the libraries on the build server were version so.1.1.1k, but when bundled by PyInstaller they were renamed to version so.1.1. Thus, we had to copy and rename the HMAC files to match. E.g.: .libcrypto.so.1.1.1k.hmac to .libcrypto.so.1.1.hmac.
If not creating a onefile build then renaming the HMAC files to match, in the resulting dist/program/_internal directory, after the build works. But for a onefile build we had to copy and rename the HMAC files to a temporary directory ahead of time and then provide the renamed HMAC files via the --add-binary option or spec file.
When executing on Ubuntu 20.04 LTS with
$ pro enable fips-updates
A python3 script which contains os.spawnvp(os.PWAIT, ["ssh"], args) or executes a bash script that contains ssh commands is compiled with PyInstaller, execution results in the FATAL FIPS SELFTEST FAILURE error.
I have found to change the environment variable (env) like
mod_env = os.environment
mod_env.pop('LD_LIBRARY_PATH', None)
os.spawnvpe(os.PWAIT, ["ssh"], args, mod_env)
or inside the bash script to unset LD_LIBRARY_PATH.
I know I saw this fix presented somewhere before and I’ll look for the previous answer. I’ll try to include a working generic example here soon as well.
I have used PyInstaller to package up a python application. When I run the application I receive this error:
crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE
Abort (core dumped)
If I run the script via "python3 scriptname.py" it runs fine. I do not use OpenSSL in my python scripts. I believe PyInstaller is using something that is not FIPS compliant.
With FIPS disabled I am able to successfully run the PyInstaller packaged application.
How can I debug this error? Is there a way to patch PyInstaller to make it FIPS compliant?
This usually happens, if you installed python with FIPS enabled/disabled or if Python is consuming a module/library/something, where Python/ or that particular library/python pyInstaller may be consuming an artifact (that was built on some machine previously, with FIPS Disabled); then, using such executables (built with FIPS disabled) on another machine(where FIPS is actually enabled) i.e. where FIPS setting doesn’t match with the machine, where you built the app in python or that particular library, then, you’ll get such errors.
Ex: Let’s say a package manager ex: yum / zypper installed python without openssl (which is a pre-install/pre-requisite for python app, which needs Openssl functionality). If python is installed without Openssl first already installed, then any Python app, which needs openssl processing, Python won’t work, even if you install openssl later on the machine; Python will still spit out bunch of gnarly openssl errors :–) big grin.
Similarly, if something was installed/packaged/built using FIPS disabled (on some machine), it can be an app or any library that you are consuming… then later, if you try to run an app or an app(consuming that library/package) on a machine, where FIPS = 1 (enabled) then you’ll see this error.
In my case, cmake was failing for running simple cmake --version and all I did was used a cmake .tar bundle which may have been compiled on a machine, where source and target machine’s FIPS settings were same i.e. in my case, cmake tar bundle came from a FIPS enabled machine, and when I used it on this other new machine where FIPS was enabled, it worked!
MISC Info on OpenSSL errors:
The effects of self-test failures in the Module differ depending on the type of self-test that failed.
The FIPS_mode_set() function verifies the integrity of the runtime executable using a HMAC SHA-256 digest,
which is computed at build time. If this computed HMAC SHA-256 digest matches the stored, known digest, then
the power-up self-test (consisting of the algorithm-specific Pairwise Consistency and Known Answer tests) is
performed.
Non-fatal self-test errors transition the module into an error state. The application must be restarted to recover
from these errors. The non-fatal self-test errors are:
FIPS_R_FINGERPRINT_DOES_NOT_MATCH – The integrity verification check failed
FIPS_R_FIPS_SELFTEST_FAILED – a known answer test failed
FIPS_R_SELFTEST_FAILED – a known answer test failed
FIPS_R_TEST_FAILURE – a known answer test failed (RSA); pairwise consistency test failed (DSA)
FIPS_R_PAIRWISE_TEST_FAILED – a pairwise consistency test during DSA or RSA key generation failed
FIPS_R_FIPS_MODE_ALREADY_SET – the application initializes the FIPS mode when it is already initialized
These errors are reported through the regular ERR interface of the shared libraries and can be queried by
functions such as ERR_get_error(). See the OpenSSL Module manual page for the function description.
A fatal error occurs only when the module is in the error state (a self-test has failed) and the application calls a
crypto function of the module that cannot return an error in normal circumstances (void return functions). The
error message: ‘FATAL FIPS SELFTEST FAILURE‘ is printed to stderr and the application is terminated with the
abort() call.
The only way to recover from a fatal error is to restart the application. If failures persist, you must reinstall the Module. If you downloaded the software, verify the package hash to confirm a proper download.
IMPORTANT NOTE:
If you installed a newer version of OpenSSL x.y.z (rpm/etc) and didn’t reboot/restart your machine, some of the libraries which consume openssl will be vulnerable / error out too with similar errors.
You can test this by running:
uptimecommand to see how long your server/machine has been UP n running.sudo lsof | grep libssl.soor to Find processes running with deleted OpenSSL libraries, run:
sudo lsof | grep DEL.*libssl
So, try rebooting your machine and see if the errors goes away and check the output of the above commands.
For more info, refer this link: https://raymii.org/s/snippets/Find_all_services_using_libssl_to_restart_after_an_OpenSSL_update.html
This seems to be an old post but just in case…
I use PyInstaller quite often, but after upgrading from CentOS 7.x to RHEL 8.x I began experiencing the same behavior. Here is what I found… PyInstaller pulls in the correct openssl shared libraries but does not bundle in the the associated hmac’s ( in our case /usr/lib64/.libcrypto.so.1.1.hmac and /usr/lib64/.libssl.so.1.1.hmac ).
I chose to just bundle the hmac files generated by the system. I did this by editing the associated .spec file that pyinstaller uses to create the bundled executable.
Changed:
binaries=[],
to:
binaries=[('/usr/lib64/.libcrypto.so.1.1.hmac','.'),'/usr/lib64/.libssl.so.1.1.hmac','.')],
No other modifications were required. If you don’t use a .spec file then I suppose this could also be accomplished via the cmdline.
Piggybacking off of gcgold’s answer (11766308) since I don’t have enough reputation to comment yet.
That solution worked for us on RHEL8 as well, with minor tweaks. In our case, the libraries on the build server were version so.1.1.1k, but when bundled by PyInstaller they were renamed to version so.1.1. Thus, we had to copy and rename the HMAC files to match. E.g.: .libcrypto.so.1.1.1k.hmac to .libcrypto.so.1.1.hmac.
If not creating a onefile build then renaming the HMAC files to match, in the resulting dist/program/_internal directory, after the build works. But for a onefile build we had to copy and rename the HMAC files to a temporary directory ahead of time and then provide the renamed HMAC files via the --add-binary option or spec file.
When executing on Ubuntu 20.04 LTS with
$ pro enable fips-updates
A python3 script which contains os.spawnvp(os.PWAIT, ["ssh"], args) or executes a bash script that contains ssh commands is compiled with PyInstaller, execution results in the FATAL FIPS SELFTEST FAILURE error.
I have found to change the environment variable (env) like
mod_env = os.environment
mod_env.pop('LD_LIBRARY_PATH', None)
os.spawnvpe(os.PWAIT, ["ssh"], args, mod_env)
or inside the bash script to unset LD_LIBRARY_PATH.
I know I saw this fix presented somewhere before and I’ll look for the previous answer. I’ll try to include a working generic example here soon as well.