How can I best identify variation in the system I’m testing?

Learn how to identify & evaluate Parameter candidates in your systems under test and how to optimally include them in your DesignWise models.

When test designers struggle to use DesignWise, the most common reason is identifying what factors & ideas should be included in the “Parameters” screen. To help address this challenge, this article describes several categories of variables that are often useful to include in software tests.

At the risk of stating the obvious, you’ll need to use your judgment here; not all of these categories will be necessary (or even useful!) to include in all of your sets of tests.

Variables for software tests can be described in these general categories. It is important to remember that when you’re entering variables into the “Parameters” screen, you should include test inputs only. You should not include outcomes or expected results (as a general rule; we will talk about exceptions in the “Test Design Principles” category of articles).

Environmental variations relate to different potential combinations of hardware and software that people might use, as well as the different locations of data that might be pulled from when a given transaction is executed. Imagine you’re creating some end-to-end tests for a flight reservation system, like https://www.expedia.com/Flights. As a web application, you would probably want to include multiple combinations of hardware and software configurations.

Hardware configuration inputs could include:

Manufacturer: Dell, Lenovo, Apple

Processor: Intel, AMD

Age: manufactured within last 2 years, manufactured 2 or more years ago

Software configuration inputs could include:

Operating System: Windows 7, Windows 8, Windows 10, Mac OS

Browser: IE10, IE11, Firefox, Chrome, Safari

Java settings: JavaScript enabled, JavaScript disabled

Cookies: cookies enabled, cookies disabled

Channel inputs could include:

Channel: typical web transaction, typical web transaction, phone call center (Note: double-weighting ‘typical web transaction’ will make it show up twice as often as ‘phone call center’)

Additional Environmental Variations involve file types and storage locations

Flight schedule information for flights is stored: Database Huey, Database Dooey, Database Louie

Frequent Flier Mileage data stored in: United database, (Frontier Airlines database; not yet transferred into post-merger database), N/A

User Variations relate to how different people might navigate through the System Under Test based on their particular habits as computer users. They can also include, as in the case of an “Admin User” with special rights, the particular features of the system that they have available to them based on the kind of user that they are.

For example, User Variations that may be considered in end-to-end tests of https://www.expedia.com/Flights, might include:

User type: normal user (without prior account), normal user (with established account), call center agent

User persona: power user, clueless newbie user, typical user

Account characteristics: Gold Frequent Flier, Silver Frequent Flier, Bronze Frequent Flier, not a Frequent Flier

Other related ideas:

User navigates primarily using: keyboard (hot-keys wherever possible), keyboard (no hot-keys), mouse

User rapidly clicks multiple times on submit buttons and next buttons: no, yes, no (Note: double-weighting ‘no’ will make it appear twice as often as ‘yes’)

User enters information from top of screen to bottom of screen: always, usually, never

User enters all required information the first time they submit information: always, usually, never

Usage Variations relate to what different features of a system people might use, as well as how they might use those features, and the different types of data that might be used in different scenarios.

Actions that could vary from test to test in our https://www.expedia.com/Flights example could include:

• One Way or Round Trip: One Way, Round Trip
• Destination: South America, Europe, Asia, Africa, Australia, North America
• Class of Travel: Economy, Business, First
• Special Meal Requested: None, None, Vegetarian, Kosher (Note: double-weighting ‘None’ will make it show up twice as often as ‘Vegetarian’ or ‘Kosher’)
• Saturday Night Stay-over: Yes, No

Data that might vary will often be inextricably interrelated to usage examples (directly above). Each of the user selections above, for example, would create data that should be stored by the system. That’s one of the reasons we categorize “Actions” and “Data” together as “Usage Variations.” Even so, “Data” can be useful as a trigger for additional testing ideas such as quantities, data ranges, and specific data formats. Here are some examples of test inputs that could be included from this example:

• Date format: mm/dd/yy, dd/mm/yy, mm/dd/yyy, dd/mm/yyyy
• Number of passengers: 1, 2-4, more than 4
• Special characters included in customer name?: no, yes, no (Note: recall double-weighting)

Elisabeth Hendrickson published “Explore It!” in 2013. In our opinion, it is one of the best software testing books ever written. We highly recommend it to new and experienced software testers alike without reservations. Chapter 4, titled “Finding Interesting Things to Vary,” is particularly outstanding. The chapter does a superb job of articulating not only what kinds of variables you can change from test to test but also why it is such a good idea to vary them. These “Hendrickson Variables” come from Chapter 4 of “Explore It!”

Hendrickson lays-out her list of Interesting Things to Vary as follows:

Countable Things –

Relative Positioning –

Files and Storage-

Formats –

Size –

Geographic Locations –

Depth –

Timing, Frequency, and Duration –

Input and Navigation –

Deciding whether to include a specific test idea often comes down to a judgment call by the test designer using DesignWise.

INCLUDE – If you have reason to believe that including a test input is likely to matter and it would not take much extra time to vary that idea in each of the tests in your test set, then include it. An example: if you’re testing a transactional web application and prior releases have had quite a few defects associated with the IE10 browser, include “Browser Type” as a Parameter and “IE10” as one of your Values.

EXCLUDE – Conversely, if you suspect that a particular test idea is not going to make any difference and that variation would take considerable time to include in each of your tests, then do not enter those into the “Inputs” screen of DesignWise. An example: as pointed out by James Bach, defects have been triggered by blocked cooling vents above servers on extremely rare occasions . This caused a server to overheat. Even so, it would rarely be sensible to invest the time necessary to test out multiple test scenarios with overheated servers on a given set of tests.

Including test ideas using the Value Expansion feature allows for quick testing without wasting time (over testing).

An example: imagine that you’re testing a new feature of a financial transaction application. It will be used for foreign or domestic transactions. Setting up the testing environment for each foreign transaction requires dramatically more time than for domestic transactions. If you were to include “Type of Transaction” as a Parameter and use “Domestic” and “International” as the 2 Values for that Parameter, approximately 50% of your tests would be international transactions which would each require a great deal of extra setup time.

In such situations, you might instead consider using “Type of Transaction” as a Value and then adding “Domestic, International, Domestic, Domestic, Domestic, Domestic, Domestic” as sub-values when you create your Value Expansion. Using this test design technique would result in at least one “International” transaction (for coverage of this potentially important scenario) but you would avoid spending too much time “over-testing” the idea.

When you’re first getting started, feel free to experiment!

Your ability to make judgment calls will increase as you gain experience. When you’re experimenting, don’t be afraid to add a few more test ideas into the scope of your test sets than you’re used to. You will notice that when you design tests with DesignWise, you can include more test ideas in the scope of your tests than you’re used to including. That’s because:

• You will often find surprising defects when you add variation to your tests, yet adding the variation to your tests often hardly takes any time at all.
• Adding a Parameter with a few Values takes only a few seconds on the “Inputs” screen of DesignWise.
• Finally, unlike when you select and document tests by hand, generating the tests with the newly-added test ideas won’t add noticeably more test documentation time; the tests generated by DesignWise will all automatically include that test idea.
• Provided you keep the number of Values per Parameter small, you usually won’t increase the number of tests that DesignWise will generate.