Starting point for your rspec test
Say you want to test both positive and negative results of a rspec test, because a false negative result is a fairly high risk, such as in a case where the broken code fails silently. In my case, I’ve identified an environment variable in the build that would be set to "true"
if the functionality was enabled, but could be set to "false"
or unset if the functionality was disabled.
if … in one test
example "Error if fips_mode not available" do
if ENV["OMNIBUS_FIPS_MODE"].to_s.downcase == "true"
expect { enable_fips }.not_to raise_error
else
expect { enable_fips }.to raise_error
end
end
Well, that’s quick and dirty, but now… how to word the example description to account for a given scenario? I guess we could have a different description depending on the flag, but then the spec becomes unreadable?
if in two tests but with a skip
example "Error enabling fips_mode if FIPS not linked" do
skip unless ENV["OMNIBUS_FIPS_MODE"].to_s.downcase == "true"
expect { enable_fips }.to raise_error(OpenSSL::OpenSSLError)
end
example "Do not error enabling fips_mode if FIPS linked" do
skip if ENV["OMNIBUS_FIPS_MODE"].to_s.downcase == "true"
expect { enable_fips }.not_to raise_error
end
Ok, we’ve solved for the descriptions but now we have other issues:
- We’re repeated the magic
ENV["OMNIBUS_FIPS_MODE"].to_s.downcase == "true"
check… it was bad enough once, but twice? It was never 100% clear what the logic was. skip
generates a “pending” test if it’s called.
if with separate tags
We could always tag the two examples with :fips_mode
and :non_fips_mode
to skip those examples, but then you have to call a config.filter_run_excluding
for each tag. And you’ll have to repeat the check for each.
Setting helper, true/false on the tag, two tests
First create a helper to include your specs to wrap the slightly unsightly env check:
def omnibus_fips_mode_build?
ENV["OMNIBUS_FIPS_MODE"].to_s.downcase == "true"
end
This can be in a file that gets required in your spec_helper.rb
Then you can configure the specs to be skipped in your spec config block:
config.filter_run_excluding fips_mode: !omnibus_fips_mode_build?
The !
on the helper is because this is an exclusion filter, so that the tag values are consistent with the meaning and the helper itself is consistent with the meaning of the flag.
Finally, you can use the value of the tag and keep a unified name for the positive and negative cases of the environment setting:
# For non-FIPS OSes/builds of Ruby, enabling FIPS should error
example "Error enabling fips_mode if FIPS not linked", fips_mode: false do
expect { enable_fips }.to raise_error(OpenSSL::OpenSSLError)
end
# For FIPS OSes/builds of Ruby, enabling FIPS should not error
example "Do not error enabling fips_mode if FIPS linked", fips_mode: true do
expect { enable_fips }.not_to raise_error
end
Conclusion
I’m always reluctant to extract logic from a single use, but once the positive/negative case was needed, the unsightliness of the ENV["OMNIBUS_FIPS_MODE"].to_s.downcase == "true"
became really obvious. By taking care to establish this tag with a true/false value, future expansion of test cases around this functionality can easily be turned off for certain scenarios.
One big assumption behind this strategy, of course, is the assumption that no environment setting means that the FIPS functionality is disabled. If the FIPS functionality is enabled and the tests are run again an environment that doesn’t set the environment variable, then the tests will break. As the suite expands, this may be a legitimate concern, and may require a different strategy for both tagging the tests and filtering them.
Postscript
Indeed there was an environment that was under test that had fips linked in separately. There is a constant in the OpenSSL
library, OpenSSL::OPENSSL_FIPS
that is true or false, depending on how FIPS has been built.