Pipeline Setup with YAML (Azure Pipelines)

This guide explains how to set up an Azure DevOps Pipeline for running Ranorex automated tests using a YAML pipeline definition.

The YAML file is the single source of truth for the pipeline configuration and can be used in two ways:

• Automatic setup: Create a new pipeline in Azure DevOps that runs directly from the YAML file.
• Manual setup: Use the YAML file as a reference to configure an existing pipeline (stages, jobs, variables, and integrations).

Download the sample pipeline YAML

Download the following file and commit it to your repository:

ado-testplans=execution-pipeline.yml

Important: This is a template. Update the variables (and move secrets to Azure DevOps secret variables/variable groups) before running it.

Setup steps

Add the YAML file to your repository

Add the provided .yml file to your repo (common locations):

• Repository root (e.g., /azure-pipelines.yml)
• A pipelines folder (e.g., /.azuredevops/ranorex-pipeline.yml)

Create a pipeline from the YAML (new pipeline)

In Azure DevOps:

1. Go to Pipelines
2. Select New pipeline
3. Choose your repository
4. Select Existing Azure Pipelines YAML file
5. Browse and select the YAML file path
6. Select Save and run.

Configure variables (environment-specific)

This YAML expects environment values such as:

• ADO_BASE_URL
• ADO_ORG
• ADO_PROJECT
• RANOREX_TIMEOUT (timeout used when executing tests)

Also configure any of the following as needed:

• Service connections
• Secrets (store these in Azure DevOps Variable Groups or secret variables)
• Agent pool selection

Tip: Keep secrets out of YAML. Use secret variables or variable groups.

Run the pipeline

When the pipeline runs, it:

• Reads ado_test_map.json
• Discovers test suites and executable(s)
• Executes tests and generates reports
• Publish results and artifacts

Test results

After the pipeline run completes, you can review results in three places: the pipeline run (Tests tab), Test Plans (Test Points), and individual Test Case comments.

Pipeline test results (JUnit)

1. Open the pipeline run in Pipelines.
2. Go to the Tests tab.

Azure DevOps displays the overall status and individual test outcomes (published via PublishTestResults@2).

Note: Test results appear here only if the pipeline successfully publishes JUnit XML files.

Test Plan results (Test Points)

1. Go to Test Plans.
2. Open your test plan and navigate to the suite where your test cases are located.
3. Open Test Points.

The Outcome column shows the latest status for each mapped test point (for example, Passed or Failed).

Test Case comments (History/Discussion)

1. Open a mapped Test Case work item (for example, from the suite).

2. Check the Discussion / History section.
   

   When the pipeline updates test cases, Azure DevOps adds a comment from the build identity with an execution summary (test name, pass/fail, execution time, and error details when available).

Ranorex reports and raw result files (Artifacts)

1. Open the pipeline run in Pipelines.
2. Go to Artifacts.
3. Open/download RanorexTestResults_$(Build.BuildId).

This artifact contains:

• JUnit XML result files (*.xml)
• Ranorex report logs (*.rxlog)