I recently started debugging a problem with a FIPS-compatible version of OpenSSL linked into a build of Ruby 3.1.2 where we started getting fingerprint does not match (OpenSSL::OpenSSLError) when setting OpenSSL.fips_mode = true in Ruby.

Similar issue in Apache Tomcat Bugzilla archives?

There was a similar-looking bug in the Apache Tomcat Bugzilla archives, referencing the source of the error (FIPS_check_incore_fingerprint:fingerprint does not match)

This was determined by using Process Explorer to examine the tomcat7.exe process following a successful startup of Tomcat with FIPSMode set to “off”. The libeay32.dll library in that process displayed an “Image Base” address of 0xFB00000, indicating its desired base memory address, an a “Base” address of 0x63E20000 (or something else on other servers), indicating the actual base memory address being used for the library.

Bug 55113 – FIPS-compatible OpenSSL fails fingerprint test in TCNative with FIPS mode on

This research aligned with my digging into an OpenSSL FIPS version, especially the overlap with libeay:

❯ ag FIPS_check_incore_fingerprint
openssl-fips-2.0.16/crypto/fips_err.h
88:{ERR_FUNC(FIPS_F_FIPS_CHECK_INCORE_FINGERPRINT),	"FIPS_check_incore_fingerprint"},

openssl-fips-2.0.16/util/libeay.num
4363:FIPS_check_incore_fingerprint           4733	EXIST:OPENSSL_FIPS:FUNCTION:

openssl-fips-2.0.16/fips/tools/declarations.dat
4342:                    'FIPS_check_incore_fingerprint' => {
4347:                                                         'sym' => 'FIPS_check_incore_fingerprint',
4349:                                                         'oldsym' => 'FIPS_check_incore_fingerprint'

openssl-fips-2.0.16/fips/fips_post.c
161:	if(!FIPS_check_incore_fingerprint())

openssl-fips-2.0.16/fips/fips.h
109:int FIPS_check_incore_fingerprint(void);

openssl-fips-2.0.16/fips/fips.c
198:int FIPS_check_incore_fingerprint(void) 

What does ProcessExplorer say?

Using Process Explorer with ruby.exe launched, select View -> Show Lower Pane and View -> Lower Pane View -> DLLs (Ctrl+L and Ctrl+D) and then right-click the header for the lower pane and check “Base Address” and “Mapped Size”

If libeay32.dll is the problem, it doesn’t seem to show an explicit overlap:

pedump gem

I decided to hook in the pedump gem into a local copy of the

–image-base argument to ld

–image-base value Use value as the base address of your program or dll. This is the lowest memory location that will be used when your program or dll is loaded. To reduce the need to relocate and improve performance of your dlls, each should have a unique base address and not overlap any other dlls. The default is 0x400000 for executables, and 0x10000000 for dlls. [This option is specific to the i386 PE targeted port of the linker]

From ld(1) Linux man page

What about 64-bit, then?

But what’s that about i386 PE targeted port? SO: linker – Ensure program loads below 4GB using –image-base in Linux makes me think that 64-bit won’t work. Of course this is Windows… so… What about 64-bit… maybe I need the –-dynamicbase option for ld.exe (after reading The Case of the DLL that couldn’t be relocated), but it seems like it should be a default?

ASLR (Address Space Layout Randomization)

Microsoft’s Windows Vista (released January 2007) and later have ASLR enabled only for executables and dynamic link libraries that are specifically linked to be ASLR-enabled.^[29] For compatibility, it is not enabled by default for other applications. Typically, only older software is incompatible and ASLR can be fully enabled by editing a registry entry HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\MoveImages,^[30] or by installing Microsoft’s Enhanced Mitigation Experience Toolkit.

Address space layout randomization on Wikipedia

So I would suggest that any Windows file containing the FIPS module (be
it a DLL or an EXE), needs to be linked with “/FIXED” to make it
loadable only at the address specified with “/BASE” and neither
relocatable nor rebasable. 

32-bit Windows “rebasing” of OpenSSL FIPS library

A hack that kind of works?

One answer on StackOverflow that was suggested on StackOverflow is the following:

Open util\mk1mf.pl and add
$cflags.= " -DOPENSSL_FIPS_DEBUGGER";
after line 311

https://stackoverflow.com/questions/45805955/how-do-i-compile-fips-capable-openssl-on-windows-x64

The problem is how this interacts with fips.c:

   if (len!=sizeof(FIPS_signature) ||
      memcmp(FIPS_signature,sig,sizeof(FIPS_signature)))
  {
    if (FIPS_signature>=FIPS_rodata_start && FIPS_signature<FIPS_rodata_end)
      FIPSerr(FIPS_F_FIPS_CHECK_INCORE_FINGERPRINT,FIPS_R_FINGERPRINT_DOES_NOT_MATCH_SEGMENT_ALIASING);
#if defined(__sgi) && (defined(__mips) || defined(mips))
    else if (__dso_displacement!=NULL)
#else
    else if (OPENSSL_NONPIC_relocated)
#endif
      FIPSerr(FIPS_F_FIPS_CHECK_INCORE_FINGERPRINT,FIPS_R_FINGERPRINT_DOES_NOT_MATCH_NONPIC_RELOCATED);
    else
      FIPSerr(FIPS_F_FIPS_CHECK_INCORE_FINGERPRINT,FIPS_R_FINGERPRINT_DOES_NOT_MATCH);
#ifdef OPENSSL_FIPS_DEBUGGER
    rv = 1;
#endif
    goto err;
  }
  rv = 1;
err:
  if (rv == 0)
    fips_post_failed(FIPS_TEST_INTEGRITY, 0, NULL);
  else
    if (!fips_post_success(FIPS_TEST_INTEGRITY, 0, NULL))
      return 0;
  return rv;

If you set the -DOPENSSL_FIPS_DEBUGGER option on the compiler, you will include the #ifdef OPENSSL_FIPS_DEBUGGER code, which sets the rv (return value?) to 1… This invalidate all of the signature checks–but doesn’t necessarily force this function to pass… fips_post_success still possibly executes a callback that might fail, and did about half the time I tested enabled FIPS in tests across different Windows environments.

Back to ASLR and the Dynamic Base option

In my searching around by the name of the function that the above fingerprint check snippet is from (FIPS_incore_fingerprint), I found the following blurb:

You forgot to run the special "FIPS" linker script on your 
application, which sets the value of that fingerprint based on the load address and relocations of your application.

Note, that this means that the design of the FIPS module 
security policy is incompatible with ASLR on almost every 
operating system having that feature.

https://www.spinics.net/lists/openssl-users/msg00849.html

This is consistent with my seeing that --disable-dynamicbase (/DYNAMICBASE:NO) seems to reduce the frequency of FIPS fingerprint errors.

Leaving in the option for image base

The original patch for relocation of the DLL included setting --image-base to configure where the library should be located in memory, and in testing, it still seemed to control the location of the library:

		case $(LIBNAME) in \
			crypto) SHLIB=libeay; base=-Wl,--disable-dynamicbase,--image-base,0x64000000;; \
			ssl) SHLIB=ssleay; base=-Wl,--disable-dynamicbase,--image-base,0x65000000;; \
		esac; \
		SHLIB_SOVER=32; \
		extras="$(LIBNAME).def"; \
		$(PERL) util/mkdef.pl 32 $$SHLIB > $$extras; \
		base=; [ $(LIBNAME) = "crypto" -a -n "$(FIPSCANLIB)" ] && base=-Wl,--disable-dynamicbase,--image-base,0x63000000; \

OpenSSL 2.0 FIPS module is relatively old?

I’m still new to looking at FIPS-related things but the OpenSSL 2.0 FIPS module (the latest of which is 2.0.16) sticks you with the 1.0.x versions of OpenSSL

The OpenSSL FIPS Object Module 2.0 was first validated with FIPS 140-2 certificate #1747 in mid-2012. This 2.0 FIPS module is compatible with OpenSSL releases 1.0.1 and 1.0.2, and not with any other releases.

https://wiki.openssl.org/index.php/FIPS_module_2.0

It appears that OpenSSL 3.0 has a built in FIPS module?

Conclusion

You probably want --disable-dynamicbase (gcc) or /dynamicbase:no (MSVC) to disable ASLR if you’re using FIPS 2.0 still, at least on Windows. Setting a static --image-base seems to also improve things. You might also want to jump through the hoops to get OpenSSL 3.0 working, but I have not crossed that bridge yet.