Create and execute an ADO pipeline

To create a new pipeline for your Azure DevOps project:

1. In your opened project, click Pipelines in the register on the left.

2. Click New pipeline.

3. Depending on whether you want to specify a pipeline using YAML or on your own:
   a. Click Azure Repos Git (YAML) (recommended for experts, not covered here).
   b. Click Use the classic editor.
   
   

4. Configure as shown below and click Continue.
   

5. In the next step, click Apply for .NET Desktop. This is the framework Ranorex Studio solutions are based.

 

The pipeline overview shows the various components a pipeline consists of and offers many possibilities for configuration. Here we only explain what’s relevant for the Ranorex Studio integration.

1. Configuration categories of a pipeline. In the image, Tasks is selected.

2. Basic pipeline configuration. Here you can give the pipeline a name, set the agent pool, and set optional parameters.

3. Here you can configure the sources (i.e. repositories) the pipeline uses. This corresponds to what we did in Create a pipeline above.

4. A pipeline always contains at least one agent job. This is the logical set of tasks that the pipeline carries out.

5. An agent job always contains at least one task. Tasks are carried out in the order they appear under the agent job.

6. The configuration window. Configuring one of the above items brings up this window. In the image, the window for configuring the agent job is shown.

 

1. Under Tasks, click Pipeline to bring up the basic pipeline configuration, where you can make the following settings:

1. 1. Here you can change the pipeline name.

   2. Here you select the agent pool for the pipeline. You can use the default local pool or choose from a range of hosted pools that have various images already set up.

   3. Here you can set optional pipeline parameters. The preset parameter in the image builds the executable from the Visual Studio solution file (.sln file ending).

 

For Ranorex Studio tests to be executed correctly in an Azure DevOps pipeline, the agent job must contain the tasks listed below.

Note
The .NET pipeline already contains various default tasks. Do not delete them.

Hint
Instead of the following tasks, you can also use the official Ranorex Studio build task, explained below after this topic.

Build the Ranorex Studio solution

The task Visual Studio build builds the Ranorex Studio test solution. It is already included by default in the agent job and is functional.

Configuration

1. Change the display name for easier identification.

2. Change the platform to x86, if required.

Execute the build

To execute the build of the Ranorex Studio solution, you need to add the Command line task to the agent job.

To do so:

1. Click the + symbol next to the agent job.

2. In the search bar, enter command line.

3. Click Add.
   
   

4. Configuration

   Change the Display name for easier identification.
   Replace the default script with the following:

Note
Make sure the Command line task comes after the Visual Studio build task. Otherwise, execution will fail.

1. 1. System variable for the current working directory of the Azure DevOps project.

   2. Folder for the Ranorex Studio solution.

   3. Folder for the Ranorex Studio project in the solution.

   4. Default output folder for Visual Studio projects.

   5. File name of the executable build. You can add further optional arguments to the .exe.

Reference
For a full list of command line arguments and how to use them, refer to Ranorex Studio expert > Runtime and remote execution > Command line execution

Reporting

When running a Ranorex Studio test through an Azure DevOps pipeline, the associated Ranorex Studio report is not saved to the working directory of the build by default. To include it, you have to add a Publish build artifacts task in the agent job. Then:

1. Click the Publish build artifacts task.

2. Change the display name, if desired.

3. Change the directory as shown in the image.

4. Change the artifact name, if desired.

Result

Once a pipeline has finished, you can see the report in the summary and open it from there.

1. Click Pipelines > Builds.

2. Click Summary.

3. Under Build artifacts published, click the item. It will have the name you specified in the task.

Show test results in the build results

By default, the results of a Ranorex Studio test are not displayed in Azure DevOps. You can however have them displayed by using the JUnit format of Ranorex Studio reports.

To do so:

1. To the agent job, add a Publish test results task.

2. Change the task’s display name, if desired.

3. Set the test result format to JUnit.

4. As test results files, specify **/*.xml.

5. As search folder, specify the current working directory.
   

6. Switch to the Command line task and extend the script as in the image:
   

1. 1. The executable build of the Ranorex Studio solution.

   2. Argument to create the report in the JUnit format.

   3. Argument to zip the report.

   4. Argument to specify a directory and file name for the zipped report.

Reference
For a full list of command line arguments and how to use them, refer to Ranorex Studio expert > Runtime and remote execution > Command line execution

Result

Under Pipelines > Builds > Tests, you can see various test details after a build has been executed.

The official Ranorex Studio build task collects all the tasks described above under Pipeline tasks for Ranorex Studio tests and more in one simple task with a straightforward UI to configure it. You can download it as a free extension from the Azure DevOps Marketplace here.

Simply add the task to a job to get started.

Note
The build task creates a set of command line arguments based on how you configure it. Therefore, it behaves in the same way as running a Ranorex Studio test from a command line interface, and the arguments themselves also work the same way.

All command line arguments are explained here.

1. Display the name of the task.

2. Task settings controlling the integration with Ranorex Studio. We explain these below.

General settings

1. The relative path from the repository folder to the test executable in the output folder of your Ranorex Studio solution. Required.
2. The relative path from the repository folder to a specific test suite file (.rxtst) in the output folder of your Ranorex Studio solution. Useful if your test contains several test suites and you only want to run one of them.
3. The run configuration that will be used for the test run. Enter exactly as it is named in the run configuration manager in Ranorex Studio. Useful if you have more than one run configuration.

   Reporting
   

4. Enables reporting, i.e. a Ranorex Studio report will be attached to the build.
5. Allows you to enter a custom report name. You can use report-specific placeholders, explained  here under Reporting. 
6. The minimum report level that events must have to be included in the report. (Default) uses the report level setting that was active when the solution was built in Ranorex Studio. Report levels are explained here.
   

   TestRail integration
   
   These options allow you to pass test results to a TestRail instance.

7. Enables reporting test results to TestRail.
8. Enter the service connection for your TestRail instance. If you don’t have one yet, you need to configure it in Azure DevOps under Project Settings > Service connections > New service connection > Generic. There, enter your TestRail credentials. You can also click the Manage link next to the field to get to the Service connections menu.
9. The identifier of an existing test run in TestRail to report test results to. If you don’t specify an ID or it doesn’t exist, results will be reported to a new test run.
10. Creates a new test run in TestRail with the specified name to report test results to. Without this argument the default name is used.

Advanced

Here you can specify additional command line arguments to customize the test run. All available command line arguments are explained here.

 

To execute a pipeline, you need a configured Azure Pipelines agent.

1. Click Pipelines > Builds

2. Click Run
   
   

3. In the dialog that appears, review the pipeline settings and click Run

Pipeline progress view

The pipeline will carry out all tasks in the specified order. You can view the progress in Azure DevOps.

1. Indicates that a pipeline has been started

2. Start time and elapsed time

3. Completed tasks

4. Current task, including debug info

Agent window

Once you’ve started a pipeline, the agent job is passed to the agent, which then runs it. The agent window will show that a job has been started.

Ranorex Studio test run

The agent runs the Ranorex Studio test as if it were started from within Ranorex Studio.

When a pipeline has been completed, it will be indicated in the pipeline overview.

1. List of all created pipelines for the current project

2. List of all builds in the selected pipeline, including execution stats (success/failure)

1. Click the build to display the results view for it

After completing a job, the agent shows the result and then returns to idling.

You can also trigger pipelines automatically. To do so, you need to enable the option Enable continuous integration. Any changes to the Ranorex Studio solution committed and pushed to Git will then trigger the respective pipeline. To enable this option:

1. Click Pipelines, select the pipeline, and click Edit.

2. Click Triggers.

3. Check Enable continuous integration.