- Similar issue in Apache Tomcat Bugzilla archives?
- What does ProcessExplorer say?
- pedump gem
- –image-base argument to ld
- What about 64-bit, then?
- ASLR (Address Space Layout Randomization)
- A hack that kind of works?
- Back to ASLR and the Dynamic Base option
- Leaving in the option for image base
- OpenSSL 2.0 FIPS module is relatively old?
- Conclusion
I recently started debugging a problem with a FIPS-compatible version of OpenSSL linked into a build of Ruby 3.1.2 where we started getting fingerprint does not match (OpenSSL::OpenSSLError)
when setting OpenSSL.fips_mode = true
in Ruby.
Similar issue in Apache Tomcat Bugzilla archives?
There was a similar-looking bug in the Apache Tomcat Bugzilla archives, referencing the source of the error (FIPS_check_incore_fingerprint:fingerprint does not match)
This was determined by using Process Explorer to examine the tomcat7.exe process following a successful startup of Tomcat with FIPSMode set to “off”. The libeay32.dll library in that process displayed an “Image Base” address of 0xFB00000, indicating its desired base memory address, an a “Base” address of 0x63E20000 (or something else on other servers), indicating the actual base memory address being used for the library.
Bug 55113 – FIPS-compatible OpenSSL fails fingerprint test in TCNative with FIPS mode on
This research aligned with my digging into an OpenSSL FIPS version, especially the overlap with libeay
:
❯ ag FIPS_check_incore_fingerprint openssl-fips-2.0.16/crypto/fips_err.h 88:{ERR_FUNC(FIPS_F_FIPS_CHECK_INCORE_FINGERPRINT), "FIPS_check_incore_fingerprint"}, openssl-fips-2.0.16/util/libeay.num 4363:FIPS_check_incore_fingerprint 4733 EXIST:OPENSSL_FIPS:FUNCTION: openssl-fips-2.0.16/fips/tools/declarations.dat 4342: 'FIPS_check_incore_fingerprint' => { 4347: 'sym' => 'FIPS_check_incore_fingerprint', 4349: 'oldsym' => 'FIPS_check_incore_fingerprint' openssl-fips-2.0.16/fips/fips_post.c 161: if(!FIPS_check_incore_fingerprint()) openssl-fips-2.0.16/fips/fips.h 109:int FIPS_check_incore_fingerprint(void); openssl-fips-2.0.16/fips/fips.c 198:int FIPS_check_incore_fingerprint(void)
What does ProcessExplorer say?
Using Process Explorer with ruby.exe launched, select View -> Show Lower Pane and View -> Lower Pane View -> DLLs (Ctrl+L and Ctrl+D) and then right-click the header for the lower pane and check “Base Address” and “Mapped Size”

If libeay32.dll
is the problem, it doesn’t seem to show an explicit overlap:

pedump gem
I decided to hook in the pedump gem into a local copy of the
–image-base argument to ld
–image-base value Use value as the base address of your program or dll. This is the lowest memory location that will be used when your program or dll is loaded. To reduce the need to relocate and improve performance of your dlls, each should have a unique base address and not overlap any other dlls. The default is 0x400000 for executables, and 0x10000000 for dlls. [This option is specific to the i386 PE targeted port of the linker]
From ld(1) Linux man page
What about 64-bit, then?
But what’s that about i386 PE targeted port? SO: linker – Ensure program loads below 4GB using –image-base in Linux makes me think that 64-bit won’t work. Of course this is Windows… so… What about 64-bit… maybe I need the –-dynamicbase option for ld.exe (after reading The Case of the DLL that couldn’t be relocated), but it seems like it should be a default?
ASLR (Address Space Layout Randomization)
Microsoft’s Windows Vista (released January 2007) and later have ASLR enabled only for executables and dynamic link libraries that are specifically linked to be ASLR-enabled.[29] For compatibility, it is not enabled by default for other applications. Typically, only older software is incompatible and ASLR can be fully enabled by editing a registry entry
Address space layout randomization on WikipediaHKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\MoveImages
,[30] or by installing Microsoft’s Enhanced Mitigation Experience Toolkit.
So I would suggest that any Windows file containing the FIPS module (be
32-bit Windows “rebasing” of OpenSSL FIPS library
it a DLL or an EXE), needs to be linked with “/FIXED” to make it
loadable only at the address specified with “/BASE” and neither
relocatable nor rebasable.
A hack that kind of works?
One answer on StackOverflow that was suggested on StackOverflow is the following:
Open util\mk1mf.pl and add
https://stackoverflow.com/questions/45805955/how-do-i-compile-fips-capable-openssl-on-windows-x64$cflags.= " -DOPENSSL_FIPS_DEBUGGER";
after line 311
The problem is how this interacts with fips.c
:
if (len!=sizeof(FIPS_signature) ||
memcmp(FIPS_signature,sig,sizeof(FIPS_signature)))
{
if (FIPS_signature>=FIPS_rodata_start && FIPS_signature<FIPS_rodata_end)
FIPSerr(FIPS_F_FIPS_CHECK_INCORE_FINGERPRINT,FIPS_R_FINGERPRINT_DOES_NOT_MATCH_SEGMENT_ALIASING);
#if defined(__sgi) && (defined(__mips) || defined(mips))
else if (__dso_displacement!=NULL)
#else
else if (OPENSSL_NONPIC_relocated)
#endif
FIPSerr(FIPS_F_FIPS_CHECK_INCORE_FINGERPRINT,FIPS_R_FINGERPRINT_DOES_NOT_MATCH_NONPIC_RELOCATED);
else
FIPSerr(FIPS_F_FIPS_CHECK_INCORE_FINGERPRINT,FIPS_R_FINGERPRINT_DOES_NOT_MATCH);
#ifdef OPENSSL_FIPS_DEBUGGER
rv = 1;
#endif
goto err;
}
rv = 1;
err:
if (rv == 0)
fips_post_failed(FIPS_TEST_INTEGRITY, 0, NULL);
else
if (!fips_post_success(FIPS_TEST_INTEGRITY, 0, NULL))
return 0;
return rv;
If you set the -DOPENSSL_FIPS_DEBUGGER
option on the compiler, you will include the #ifdef OPENSSL_FIPS_DEBUGGER
code, which sets the rv
(return value?) to 1
… This invalidate all of the signature checks–but doesn’t necessarily force this function to pass… fips_post_success
still possibly executes a callback that might fail, and did about half the time I tested enabled FIPS in tests across different Windows environments.
Back to ASLR and the Dynamic Base option
In my searching around by the name of the function that the above fingerprint check snippet is from (FIPS_incore_fingerprint
), I found the following blurb:
You forgot to run the special "FIPS" linker script on your application, which sets the value of that fingerprint based on the load address and relocations of your application. Note, that this means that the design of the FIPS module security policy is incompatible with ASLR on almost every operating system having that feature.https://www.spinics.net/lists/openssl-users/msg00849.html
This is consistent with my seeing that --disable-dynamicbase
(/DYNAMICBASE:NO
) seems to reduce the frequency of FIPS fingerprint errors.
Leaving in the option for image base
The original patch for relocation of the DLL included setting --image-base
to configure where the library should be located in memory, and in testing, it still seemed to control the location of the library:
case $(LIBNAME) in \
crypto) SHLIB=libeay; base=-Wl,--disable-dynamicbase,--image-base,0x64000000;; \
ssl) SHLIB=ssleay; base=-Wl,--disable-dynamicbase,--image-base,0x65000000;; \
esac; \
SHLIB_SOVER=32; \
extras="$(LIBNAME).def"; \
$(PERL) util/mkdef.pl 32 $$SHLIB > $$extras; \
base=; [ $(LIBNAME) = "crypto" -a -n "$(FIPSCANLIB)" ] && base=-Wl,--disable-dynamicbase,--image-base,0x63000000; \
OpenSSL 2.0 FIPS module is relatively old?
I’m still new to looking at FIPS-related things but the OpenSSL 2.0 FIPS module (the latest of which is 2.0.16) sticks you with the 1.0.x versions of OpenSSL
The OpenSSL FIPS Object Module 2.0 was first validated with FIPS 140-2 certificate #1747 in mid-2012. This 2.0 FIPS module is compatible with OpenSSL releases 1.0.1 and 1.0.2, and not with any other releases.
https://wiki.openssl.org/index.php/FIPS_module_2.0
It appears that OpenSSL 3.0 has a built in FIPS module?
Conclusion
You probably want --disable-dynamicbase
(gcc
) or /dynamicbase:no
(MSVC) to disable ASLR if you’re using FIPS 2.0 still, at least on Windows. Setting a static --image-base
seems to also improve things. You might also want to jump through the hoops to get OpenSSL 3.0 working, but I have not crossed that bridge yet.
One response to “Turning on FIPS mode failed: fingerprint does not match”
[…] negative result is a fairly high risk, such as in a case where the broken code fails silently. In my case, I’ve identified an environment variable in the build that would be set to "true" if the […]