Starting point for your rspec test

Say you want to test both positive and negative results of a rspec test, because a false negative result is a fairly high risk, such as in a case where the broken code fails silently. In my case, I’ve identified an environment variable in the build that would be set to "true" if the functionality was enabled, but could be set to "false" or unset if the functionality was disabled.

if … in one test

  example "Error if fips_mode not available" do
    if ENV["OMNIBUS_FIPS_MODE"].to_s.downcase == "true"
      expect { enable_fips }.not_to raise_error
    else
      expect { enable_fips }.to raise_error
    end
  end

Well, that’s quick and dirty, but now… how to word the example description to account for a given scenario? I guess we could have a different description depending on the flag, but then the spec becomes unreadable?

if in two tests but with a skip

  example "Error enabling fips_mode if FIPS not linked" do
    skip unless ENV["OMNIBUS_FIPS_MODE"].to_s.downcase == "true"
    expect { enable_fips }.to raise_error(OpenSSL::OpenSSLError)
  end

  example "Do not error enabling fips_mode if FIPS linked" do
    skip if ENV["OMNIBUS_FIPS_MODE"].to_s.downcase == "true"
    expect { enable_fips }.not_to raise_error
  end

Ok, we’ve solved for the descriptions but now we have other issues:

• We’re repeated the magic ENV["OMNIBUS_FIPS_MODE"].to_s.downcase == "true" check… it was bad enough once, but twice? It was never 100% clear what the logic was.
• skip generates a “pending” test if it’s called.

if with separate tags

We could always tag the two examples with :fips_mode and :non_fips_mode to skip those examples, but then you have to call a config.filter_run_excluding for each tag. And you’ll have to repeat the check for each.

Setting helper, true/false on the tag, two tests

First create a helper to include your specs to wrap the slightly unsightly env check:

def omnibus_fips_mode_build?
  ENV["OMNIBUS_FIPS_MODE"].to_s.downcase == "true"
end

This can be in a file that gets required in your spec_helper.rb

Then you can configure the specs to be skipped in your spec config block:

config.filter_run_excluding fips_mode: !omnibus_fips_mode_build?

The ! on the helper is because this is an exclusion filter, so that the tag values are consistent with the meaning and the helper itself is consistent with the meaning of the flag.

Finally, you can use the value of the tag and keep a unified name for the positive and negative cases of the environment setting:

  # For non-FIPS OSes/builds of Ruby, enabling FIPS should error
  example "Error enabling fips_mode if FIPS not linked", fips_mode: false do
    expect { enable_fips }.to raise_error(OpenSSL::OpenSSLError)
  end

  # For FIPS OSes/builds of Ruby, enabling FIPS should not error
  example "Do not error enabling fips_mode if FIPS linked", fips_mode: true do
    expect { enable_fips }.not_to raise_error
  end

Conclusion

I’m always reluctant to extract logic from a single use, but once the positive/negative case was needed, the unsightliness of the ENV["OMNIBUS_FIPS_MODE"].to_s.downcase == "true" became really obvious. By taking care to establish this tag with a true/false value, future expansion of test cases around this functionality can easily be turned off for certain scenarios.

One big assumption behind this strategy, of course, is the assumption that no environment setting means that the FIPS functionality is disabled. If the FIPS functionality is enabled and the tests are run again an environment that doesn’t set the environment variable, then the tests will break. As the suite expands, this may be a legitimate concern, and may require a different strategy for both tagging the tests and filtering them.

Postscript

Indeed there was an environment that was under test that had fips linked in separately. There is a constant in the OpenSSL library, OpenSSL::OPENSSL_FIPS that is true or false, depending on how FIPS has been built.