When should you use 2-way? When should you use a higher strength? Learn the strategies behind using different coverage settings in DesignWise
Isn’t coffee great?
But there are many ways to order this beloved drink. A barista, for instance, would need to be familiar with these combinations. Let’s look at what happens when the coverage strength is set to “1-way interactions.”
Although the quantity of tests has increased, the quality and thoroughness of this set has drastically improved as well. By testing every possible pair of test conditions, you have guaranteed that every value has been tested with all other parameters’ values.
A hot, 20-ounce drink is guaranteed to appear in your set of tests. ‘Drink Type = Hot’ and ‘Drink Size = Short’ has been tested for. ‘Hot’ and any of the sizes listed have been tested for. Likewise, ‘Hot’ and any number of espresso shots has been tested. But this concept doesn’t just apply to ‘Hot’. It applies to ‘Frozen,’ ‘Decaf,’ ‘Grande,’ and any other parameter value in your set of inputs.
These examples go on, but there’s a catch: a hot, 20-ounce drink with espresso may not have been included. In 2-way interactions, only pairs have been tested. Enter the 3-way interactions.
215 scenarios achieve the goal of testing every possible “triplet” of test conditions
We’re sure you’ve noticed the concerning increase in the number of tests. Thoroughness comes with a price. The more parameter values that you want to be tested together, the more tests there will be.
By increasing the coverage level to “3-way interactions,”, you have guaranteed that a hot, 20-ounce drink with any number of espresso shots has been included. The same goes for every other possible 3-way interaction in our set of tests.
Let’s take this a step further; what if you absolutely needed a hot, 20-ounce drink with a specific number of espresso shots and decaf (or other caffeine type)? Although espresso shots with decaf in the same drink is unlikely in reality, people can have unusual tastes.
You may have also noticed the increased wait time to generate more-thorough scenarios. Not only is the number of tests getting larger, but any margin of error in your System under Test decreases. As you may have guessed, this tradeoff between test quantity and quality continues all the way up to 4-, 5-, and even 6-way interactions. This is because tests will include every quadruplet, quintuplet, and sextuplet of test conditions, respectively.
2-way (pairwise) coverage strength is often a good default strength for many types of functional tests
3-way coverage strength overall could be considered in higher-risk / more “mission-critical” testing. If you are executing tests by hand, rather than execute a complete set of 3-way scenarios, it is often preferable to execute a smaller set of “Mixed-Strength” tests (with 3-way coverage focused on the most important interactions and 2-way coverage for the other parts).
4-way and higher coverage strengths are rarely used except in situations where (a) all the tests are automated (therefore cheap and quick to execute) and/or (b) the potential risk for error could be extremely disastrous or expensive. For instance, safely sending people into space requires that no errors be made.
If you’re considering the pros and cons of, say, executing a smaller number of 2-way tests vs. a larger number of 3-way tests, it is extremely important for you to understand additional test design strategies that you can use to decrease (or increase) the number of tests DesignWise generates.