Every test run in Ranorex Studio ends with a report. After all, how else would you know whether the test run was successful or not? The report details the entire test run from start to finish, for example, how many test cases succeeded, failed, or were blocked; what errors and warnings were raised; and the iterations of test containers.
You can customize reports to change what information is displayed, add your own logos, and export reports for further analysis.
This article explains how report levels work and how you can use them to control which messages are included in a report.
Basic characteristics
- The report starts collecting data as soon as a test run starts.
- Report levels control what data is included in the report.
- The report is generated as the test run progresses, and you can view it at any time during the run. This is especially useful for large test suites that take a long time to execute.
- Reports can be customized.
Further reading: These basic characteristics are explored in Ranorex standard reporting, Report levels, and Customization basics.
The explanations in this chapter are based on a sample solution that you can download below.
Theme: Build a test
Time: 15 minutes
Install the sample solution:
- Unzip to any folder on your computer.
- Start Ranorex Studio and open the solution file
Introduction.rxsln
The default report levels
Ranorex Studio comes with six default report levels, each with a corresponding color and integer value.
- The higher the integer value of a report level, the more important it is. For example, a Debug notification is less important than an Info notification.
- Recording modules use these default report levels.
- In code modules, you can define your report levels.
The purpose of report levels
In a test report, one executed action usually produces at least one report message.
In large test suites with thousands of actions, the report can quickly become cluttered. Report levels help you control which messages are included: they categorize report messages by importance and act as a filter when combined with a report level threshold.
- Actions 1 through 4 and 6 through 7 have the report level Info (value = 20).
-
A successful test validation (action 5) has the report level Success (value = 110). A failed validation has the report level Failure (value = 120).
Report level threshold
The report level threshold is the integer value that defines the minimum report level a message must have to be included in the report. If a message’s report level is below this value, the message will not appear in the report.
The report level threshold is set in the test suite for structure items, such as the test suite item, test cases, and smart folders. By default, each item inherits its threshold from its direct parent.
The images below illustrate this behavior.
Report threshold level of 20 (Info)
- The threshold is set to Info (value = 20).
- In the report, all messages with a minimum report level of Info are included.
Report level threshold of 110 (Success)
- The threshold is set to Success (value = 110).
- In the report, all messages with a minimum report level of Success are included. In this case, that’s only the message notifying you of a successful validation.
Set report level threshold
You set the report level threshold in the test suite for structure items, i.e. the test suite item, test cases, and smart folders. By default, each structure item inherits the setting from its direct parent.
To set the threshold:
- Switch to the test suite view.
- Right-click the test suite, a test case, or a smart folder.
- Click Report level and select the desired value.
Inherit: When this option is selected, the setting is inherited from the direct-parent structuring item.
Example
Let’s apply the above explanations to an example. We’ll insert a log message and set its report level.
Initial situation
The recording module in the sample solution contains seven recorded actions, one of them a validation action.
The report level threshold of the test case is set to Success. This means that, currently, only the successful or failed validation would be reported during a test run.
We now want to add a log message that tells us that the Submit button has been clicked. We want this message to show up in the report, while all other actions (except for the validation) remain hidden.
- Add a log message after action 4.
- Enter a message and set its report level to Success.
Result
The report levels for the actions in this recording and the report level threshold of the test case now look like this:
- Log message with report level Success in the report.
- Successful test validation with report level Success in the report.