This guide explains how to set up an Azure DevOps Pipeline for running Ranorex automated tests using a YAML pipeline definition.
The YAML file is the single source of truth for the pipeline configuration and can be used in two ways:
- Automatic setup: Create a new pipeline in Azure DevOps that runs directly from the YAML file.
- Manual setup: Use the YAML file as a reference to configure an existing pipeline (stages, jobs, variables, and integrations).
Download the sample pipeline YAML
Download the following file and commit it to your repository:
ado-testplans=execution-pipeline.yml
Setup steps
Add the YAML file to your repository
Add the provided .yml file to your repo (common locations):
- Repository root (e.g.,
/azure-pipelines.yml) - A pipelines folder (e.g.,
/.azuredevops/ranorex-pipeline.yml)
Create a pipeline from the YAML (new pipeline)
In Azure DevOps:
- Go to Pipelines
- Select New pipeline
- Choose your repository
- Select Existing Azure Pipelines YAML file
- Browse and select the YAML file path
- Select Save and run.
Configure variables (environment-specific)
This YAML expects environment values such as:
ADO_BASE_URLADO_ORGADO_PROJECT-
RANOREX_TIMEOUT(timeout used when executing tests)
Also configure any of the following as needed:
- Service connections
- Secrets (store these in Azure DevOps Variable Groups or secret variables)
- Agent pool selection
Run the pipeline
When the pipeline runs, it:
- Reads ado_test_map.json
- Discovers test suites and executable(s)
- Executes tests and generates reports
- Publish results and artifacts
Test results
After the pipeline run completes, you can review results in three places: the pipeline run (Tests tab), Test Plans (Test Points), and individual Test Case comments.
Pipeline test results (JUnit)
- Open the pipeline run in Pipelines.
- Go to the Tests tab.
Azure DevOps displays the overall status and individual test outcomes (published via PublishTestResults@2).
Note: Test results appear here only if the pipeline successfully publishes JUnit XML files.
Test Plan results (Test Points)
- Go to Test Plans.
- Open your test plan and navigate to the suite where your test cases are located.
- Open Test Points.
The Outcome column shows the latest status for each mapped test point (for example, Passed or Failed).
Test Case comments (History/Discussion)
- Open a mapped Test Case work item (for example, from the suite).
-
Check the Discussion / History section.
When the pipeline updates test cases, Azure DevOps adds a comment from the build identity with an execution summary (test name, pass/fail, execution time, and error details when available).
Ranorex reports and raw result files (Artifacts)
- Open the pipeline run in Pipelines.
- Go to Artifacts.
- Open/download RanorexTestResults_$(Build.BuildId).
This artifact contains:
- JUnit XML result files (
*.xml) - Ranorex report logs (
*.rxlog)