Create and Run a Classic Azure DevOps Pipeline

To create a new pipeline for your Azure DevOps project:

1. In your Azure DevOps project, select Pipelines in the left navigation.
2. Select New pipeline. 
3. Choose Use the classic editor.
   
4. Under Select a source, choose Azure Repos Git, select your repository, and keep the default branch as needed. Select Continue.
   
5. Choose the template .NET Desktop, then select Apply.

The pipeline overview shows the components that make up the pipeline.

1. Configuration categories of a pipeline. In the image, Tasks is selected.
2. Basic pipeline configuration. Here you can give the pipeline a name, set the agent pool, and set optional parameters.
3. Here you can configure the sources (i.e. repositories) the pipeline uses. This corresponds to what we did in Create a pipeline above.
4. A pipeline always contains at least one agent job. This is the logical set of tasks that the pipeline carries out.
5. An agent job always contains at least one task. Tasks are carried out in the order they appear under the agent job.
6. The configuration window. Configuring one of the above items brings up this window. In the image, the window for configuring the agent job is shown.

Under Tasks, select Pipeline (top node) and review:

• Name (optional)
• Agent pool (must match the pool where your configured agent is available)
• Template parameters such as the solution pattern (often **/*.sln)

1. Change the pipeline name.
2. Select the agent pool for the pipeline. You can use the default local pool or choose from a range of hosted pools that have various images already set up.
3. Set optional pipeline parameters. The preset parameter in the image builds the executable from the Visual Studio solution file (.sln file ending).

Build the Ranorex Studio solution

The Visual Studio build task builds your solution and creates the test executable in the project output folder.

Recommended configuration:

• Keep the Visual Studio build task from the .NET Desktop template.
• Set Platform to x86, if required by your Ranorex Studio solution.
• Confirm that the solution pattern points to your Ranorex Studio solution.

Run the Test Executable

To run the built Ranorex Studio test executable, add a command-line task to the agent job.

1. Click the + symbol next to the agent job.
2. In the search bar, enter command line.
3. Click Add.
   

In Script, enter the path to your test executable (example below).

Example script:

1. System variable for the current working directory of the Azure DevOps project.
2. Folder for the Ranorex Studio solution.
3. Folder for the Ranorex Studio project in the solution.
4. Default output folder for Visual Studio projects.
5. File name of the executable build. You can add further optional arguments to the .exe.

Reference: For a full list of command line arguments and how to use them, refer to Ranorex Studio expert > Runtime and remote execution > Command line execution

Reporting

By default, the Ranorex report isn’t easy to access from the run summary unless you publish it as an artifact. 

1. Add Publish build artifacts task
2. Type a Display name
3. Path to publish: $(System.DefaultWorkingDirectory)\Reports\
4. Artifact name: for example, Ranorex Test Report

After a run completes, you can download/open the artifact from the run summary. 

Once a pipeline has finished, you can see the report in the summary and open it from there.

1. Click Pipelines > Builds.
2. Click Summary.
3. Under Build artifacts published, click the item. It will have the name you specified in the task.
   

Show test results in the build results

To display results under the Tests view, generate a JUnit-compatible report and publish it:

1. Add a Publish test results task.
2. Type a Display name
3. Test result format: JUnit
4. Test results files: **/*.junit.xml
5. Search folder: $(System.DefaultWorkingDirectory)
   

Update your Command line script to include JUnit + zipped report output, for example:

1.  
   1. The executable build of the Ranorex Studio solution.
   2. /junit creates a JUnit-compatible report (.junit.xml).
   3. /zr creates a zipped report.
   4. /zrf:... sets the output path/name for the zipped report.

Tip: If you want fully predictable locations, output the report/zip to  $(System.DefaultWorkingDirectory)\Reports\ (as shown above) and publish that directory as the artifact.

Reference: For a full list of command line arguments and how to use them, refer to Ranorex Studio expert > Runtime and remote execution > Command line execution

Result

Under Pipelines > Builds > Tests, you can see various test details after a build has been executed.

Instead of manually adding the tasks above, you can install the Ranorex Azure DevOps build task extension from the Azure DevOps Marketplace and configure Ranorex execution through a single task.

This task generates the required Ranorex command-line arguments based on your settings, so it behaves like running the test executable from the command line.

Note: The build task creates a set of command line arguments based on how you configure it. Therefore, it behaves in the same way as running a Ranorex Studio test from a command line interface, and the arguments themselves also work the same way.

1. Display the name of the task.
2. Task settings controlling the integration with Ranorex Studio. We explain these below.

General settings

1. The relative path from the repository folder to the test executable in the output folder of your Ranorex Studio solution. Required.
2. The relative path from the repository folder to a specific test suite file (.rxtst) in the output folder of your Ranorex Studio solution. Useful if your test contains several test suites and you only want to run one of them.

3. The run configuration that will be used for the test run. Enter exactly as it is named in the run configuration manager in Ranorex Studio. Useful if you have more than one run configuration.

   Reporting

4. Enables reporting, i.e., a Ranorex Studio report will be attached to the build.
5. Allows you to enter a custom report name. You can use report-specific placeholders, explained here under Reporting. 

6. The minimum report level that events must have to be included in the report. (Default) uses the report level setting that was active when the solution was built in Ranorex Studio. Report levels are explained here.

   TestRail integration
   
   These options allow you to pass test results to a TestRail instance.

7. Enables reporting test results to TestRail.
8. Enter the service connection for your TestRail instance. If you don’t have one yet, you need to configure it in Azure DevOps under Project Settings > Service connections > New service connection > Generic. There, enter your TestRail credentials. You can also click the Manage link next to the field to get to the Service connections menu.
9. The identifier of an existing test run in TestRail to report test results to. If you don’t specify an ID or it doesn’t exist, results will be reported to a new test run.
10. Creates a new test run in TestRail with the specified name to report test results to. Without this argument the default name is used.

Advanced

Here you can specify additional command line arguments to customize the test run. All available command line arguments are explained here.

To execute a pipeline, you need a configured Azure Pipelines agent.

1. Go to Pipelines, select your pipeline, and select Run.
   
2. Review settings and select Run.
   While the run is in progress, Azure DevOps shows task-by-task progress. On the agent machine, the agent window also shows the job being executed.

Pipeline progress view

The pipeline will carry out all tasks in the specified order. You can view the progress in Azure DevOps.

1. Indicates that a pipeline has been started
2. Start time and elapsed time
3. Completed tasks
4. Current task, including debug info

Agent window

Once you’ve started a pipeline, the agent job is passed to the agent, which then runs it. The agent window will show that a job has been started.

Ranorex Studio test run

The agent runs the Ranorex Studio test as if it were started from within Ranorex Studio.

When a pipeline has been completed, it displays in the pipeline overview.

1. List of all created pipelines for the current project
2. List of all builds in the selected pipeline, including execution stats (success/failure)

1. Click the build to display the results view for it

After completing a job, the agent shows the result and then returns to idling.

You can also trigger pipelines automatically. To do so, you need to enable the option Enable continuous integration. Any changes to the Ranorex Studio solution committed and pushed to Git will then trigger the respective pipeline. To enable this option:

1. Open your pipeline and select Edit.
2. Select Triggers.
3. Enable continuous integration.