Welcome to the new Ranorex Support Portal

Ranorex
Try DesignWise with Ranorex
Help Center › User Guide › Ranorex Studio fundamentals › Reporting
User Forum

Ranorex standard reporting

In this chapter, you’ll learn where reports are stored, what they contain, how to read them, and where you can change report settings.

In this chapter

    tipp icon

    Screencast

    The screencast “Standard reporting” walks you through the information found in this chapter.

    Watch the screencast now

    File name and location

    File name

    The file name of a report is generated automatically. It changes depending on whether the test is executed from a recording module or from a test suite.

    Automated report name
    • For a run from a test suite, the file name starts with the test suite’s name.
    • For a run from a recording, the file name starts with the recording’s name.
    • The second part of the report name is a combination of the date and time the report is generated.
      • Test date = 21 February 2018 = 20180221
      • Test time = 07:08:36 PM = 190836
    • Date and time are separated by an underscore.
    • The file ending is .rxlog, short for Ranorex Log

    Further reading

    You can customize how report file names are generated. This is explained in Ranorex Studio system details > Settings & configuration > ⇢ Report settings.

    Report file location

    In the projects view, reports are listed in the Reports folder of your project.
    Test report storage location
    In Windows, you can access reports from two different folders. The original reports along with the layout files and the raw data files of the report (explained later as part of customizations) are stored in the Reports folder in the output directory.

    The Reports folder in the project’s folder only contains a shortcut to the original reports, and does not contain any of the stylesheet or raw data files.

    See here for an illustration:

    Report files in default storage location
    Reports folder in the project’s folder.

    Reports folder in the output directory of the project.

    Layout files for the report.

    Shortcuts to the original report files.

    Original report files with raw data files.

    Report deletion

    To ease the report file maintenance, right-click Reports in the Project View panel.
    The following options are listed for report deletion which is based on the difference between the creation date and the current date:

    • Delete all reports
    • Delete reports older than 7 days
    • Delete reports older than 14 days
    • Delete reports older than 28 days
    Report files in default storage location

    The reports are removed from the current configured directory. 

    Ranorex does not track configured report directories that are changed. If the user changes the configured report directory, the deletion does not affect any report files in previous directories.

    The files with the following extensions that are not deleted:

    • .css
    • .png
    • .xsl

    A confirmation dialog displays to proceed with the removal.

     

    Note icon

    Note

    For solutions you use in Ranorex Studio 9 but created in an earlier version, the behavior is slightly different:

    The original reports along with the layout and raw data files are stored directly in the output directory of your project, and not collected in a Reports folder.

    Report header

    The report header displays a range of useful data and summarizes the test results.

    Basic report data

    Basic report data

    Test suite name/recording module name and description

    The name of the test suite or recording module from which the test was run. Under the name, find the description that you added in the Report properties.

    System and test data
    Error/warning counters

    If there’s at least one warning in the report, an orange notification appears, as shown below.

    Error/warning counter in test report

    Error/warning counter in test report

    Test results

    A pie chart summarizing the test results.

    Note icon

    Note

    Only test cases are counted for this chart. Smart folders are ignored; you can see their results in the report details further below.
    Types of test report summary
    Successful test

    A green pie chart means all test cases were successful.

    Failed test

    A red pie chart means all test cases failed.

    Test summary

    • Pie chart with successful, failed, and blocked test cases
    • In the example, the run configuration contained 8 test cases. The first two test cases passed successfully. The third failed, after which the test was aborted, blocking the remaining 5 test cases.

    Viewing report details

    By default, the details for test cases, smart folders, etc. are collapsed. Simply click the arrow next to an item’s name to show the details.

    Expanding/collapsing report details

    Basic report structure

    The report is structured in the same way as the test suite or recording module on which the test run was based. Each structure element in the test suite appears the report, as well as each test action item. Report messages appear for all actions in the order of execution provided that they met the report level threshold.

    Simple report structure
    The structure is the same as in the test suite of this test run.
    The Recording1 item contains all the report messages for the actions in it.

    Detailed report contents

    Detailed report content structure
    Time

    By default, this column displays the relative execution time measured from the beginning of the test run. This can be changed in the ⇢ report settings.

    Level

    The second column shows the report level of the action performed.

    Category

    This column shows the action type performed.

    Message

    The report message contains more detailed information as to what happened during the execution of the action.

    Example for a report message content

    Report message stating that the key sequence ‘Harry’ was performed on a specific repository item.

    In case of a failure, the report will also include two screenshots, one at the moment of failure and one right before the failure occurred.
    Screenshots in report
    Screenshot immediately before the failure.
    Screenshot during failure.

    Data iterations in reports

    In data-driven tests, a test case or smart folder bound to data goes through a number of iterations depending on the data source. In the report, each of these iterations is shown separately.

    Data iterations in reports

    Data iterations in reports

    Data iterations

    • Data iterations are indicated by the Rows: # labels next to test containers.
    • Rows # indicates how many times the data container will be iterated – one time for each row.
    Data rows in test report

    • Each data row/iteration and its details are shown in the report.
    Variable and value

    • Variables and values used in this iteration.
    Note icon

    Note

    • Secret test data can be masked in the data source dialog. This will also apply to the report.
    • If, for example, age and gender are masked, the data appears as shown below:
    Masked report data

    Masked report data

    Reference

    Data iterations and masking data are explained in Ranorex Studio advanced > ⇢ Data-driven testing.

    Run iterations in test reports

    If a test case or a smart folder is run iteratively, each run iteration appears in the report with its details.

    Run iterations in test reports

    Run iterations in test reports

    • Run iterations are indicated by the Iterations: # and Run: # labels next to test containers.
    • Iterations: # tells you how many times the test container is run.

    Reference

    Run iterations are explained in Ranorex Studio fundamentals > Test suite > ⇢ Running tests.

    Filter messages

    There are three filters available in the report: 

    • One for filtering test containers
    • One for filtering messages for actions
    • One for filtering the action category

    Filter test containers

    Use the checkboxes in the upper part of the report to filter test containers by their success status.

    Filtering report messages

    Filtering report messages

    Success: displays report messages with report level ‘Success’ if checked, and if these messages are included in the report based on report levels. Hides these messages if not checked.
    Failed: displays report messages with report level ‘Failure’. Hides these messages if not checked.
    Blocked: displays blocked test cases of a test run when checked. Hides blocked test cases if not checked.
    Note icon

    Note

    Blocked test cases or smart folders could not be executed during a test run, either because the test was aborted beforehand, a condition wasn’t met, or they were deselected in the run configuration.

    Filter action messages

    You can also filter action messages in modules by their report level.

    Filter category messages

    You can also filter the action messages in the modules by their action type using the Category filter.

    Jump to item

    You can directly jump from the report message to the corresponding action or test suite item in Ranorex Studio.

    Mouse over a report entry, i.e. a test container or a report message for an action.
    Click the Jump to item button that appears in the right upper corner of the report entry.
    Jump to item functionality
    Jump to item button.
    Corresponding action with linked repository item.

    Open in Spy

    When you open a report outside of Ranorex Studio, the Open in Spy button becomes available. Click the button to open the corresponding report item in Ranorex Spy.

    Open in Spy functionality
    Open in Spy report functionality.

    Video reporting

    Video reporting lets you record a test run as a collection of videos. This also works on Ranorex Agents.

    Enable video reporting

    Video reporting is disabled by default. You can enable and configure it in the ⇢ Report settings.

    Play video reports from the report

    In the report, click Play video next to a test case to play the respective video.

    Video directory

    The videos are saved in the output directory in the Reports folder. For each test run, a separate folder is created.

    Reports folder in the output directory

    Three video folders

    Progressive report preview

    Ranorex generates the report as the test run progresses. You can view it at any time during the test run. This is especially useful for very long test runs.

    • Ranorex starts generating the report as soon as the test run begins.
    • The report file is saved once the defined auto-save time elapses (default = 30s).
    • At this point, you can open the in-progress report simply by double-clicking the report file.
    Progressive reporting

    Progressive reporting with two notifications indicating that the test is still in progress.

    Note icon

    Note

    Naturally, if you try to open the report on the machine the test is running on, you will very likely cause a test failure because you will be interacting with the UI.

    Report settings and configuration

    To access the report settings:

    In the test suite view, right-click the test suite.
    Click Properties.
    Click the Report tab.
    Report settings

    Report settings

    Further reading

    Report settings are explained in Ranorex Studio system details > Settings & configuration > ⇢ Report settings.