How to Generate Allure Reports in Azure Pipeline with Playwright

This article on How to Generate Allure reports in Azure Pipeline with Playwright assumes some things beforehand. That you have been able to successfully create reports in Playwright or whatever tool you are using, but you are facing trouble with its CI integration on Azure Pipeline. But don’t worry we will also see how to install allure reporting in your local as well.

Things were indeed much easier with other CI/CD tools, but with Azure Pipeline, you walk in with many limitations. The fact that you are left at the mercy of plugins when you have access issues is, to name a few.

When the marketplace has only limited stuff to offer you, it becomes really frustrating. But I have found a gem of a plugin that does the job.

The plugin we are going to use in this tutorial is Allure Report Viewer. Thank you, Michael Clay!

UPDATE: We have the official plugin for Allure Report now. Just called Allure Report, it is by Qameta Software. Here’s a link for it:
https://marketplace.visualstudio.com/items?itemName=qameta.allure-azure-pipelines

Don’t fret looking at the limited information provided on the page for this plugin. That nobody has ever asked anything or rated it, beats me.

What I have learned about this plugin is that it simply looks for your report folder to build the Allure report in a new tab. Now since you can’t edit this plugin, you can do things with your Command-Line Agent job. So the real trick is to help Azure Pipeline understand what allure is, then generate allure reports, and then allow this plugin to access the folder where your report is. Tada!

NOTE: Make sure you have gitignored both ‘/allure-report’ and /allure-results’ from your code, or your pipeline would simply read what is available. If you haven’t then it would be the same result and report pushed from your code that gets read.

Confused? Don’t worry I have got you covered with baby steps in the following session.

Here is a step-by-step tutorial on how to generate allure reports in Azure Pipeline with Playwright.

Steps on How to Generate Allure Reports in Azure Pipeline with Playwright

The first step would be to of course to make sure to install allure in your code. You can use the existing ‘allure-playwright’ npm package to do that.

Installation of Allure in Playwright

Step 1: Navigate to the following URL:

https://www.npmjs.com/package/allure-playwright

You can follow the steps mentioned in the above URL to get things going as well.

Step 2: Next step would be to go to your terminal and install the above package. Type:

npm install -g allure-playwright --save-dev 

or just

npm i -D allure-playwright

and press enter.

–save-dev or -D flag will ensure that the file saves in your devDependencies section in package.json

You have to install allure-commandline to run allure reports from Azure pipeline. The same can be used on Local as well.

The command to do that will be:

npm i -D allure-commandline

Doing so will install the latest version of allure-commandline in your devDependencies folder.

Generating Reports on Local

Step 3: Once allure-playwright is installed, you have to execute your code, but remember to specify the reporter towards the end. Just type the following in the terminal:

npx playwright test --reporter=line,allure-playwright

Doing so will let playwright know that your reporter isn’t the default one configured in your playwright.config.js file i.e. HTML, and is actually something called line reporting which generates your reports in text format. The input to that is Allure Playwright which basically lets playwright know that you are using Allure for line reporting.

NOTE: Of course, you can update the above code to reflect the specific files you want to run, but make sure to add the –reporter=line,allure-playwright in the end.

Step 4: Running the above command will not only run your code but will also create a folder called ‘allure-results’ in your explorer. This folder will have tons of json and png files in it, everything caused by the –reporter=line command.

But it is not a readable format, is it? So we need to generate a report using the above.

Step 5: So naturally our next command is going to be:

allure generate ./allure-results --clean

What it does is that it consolidates all the above results into an HTML file. A separate allure-report folder gets created with its own HTML, which uses its own plugins, CSS, data etc. to be able to open a visually aesthetic report.

NOTE: Clean at the end basically cleans whatever existing data the results folder has to do a fresh generation.

Now the time has come to open the report. So this much is clear that we need to access the allure report folder. But how do we access our report?

Step 6: The next command would be exactly that:

allure open ./allure-report

Type the above command in the terminal and press enter.

You will notice the command will flare open a webpage which might look something like this:

Go ahead and explore the different sections of the report to get a look and feel of how allure reports really look like.

Now until this point, everything was on your local. How do you make sure it runs on your Azure pipeline?

How to Generate Allure Reports in Azure Pipeline

Step 7: Push your code branch upstream.

Step 8: Open Azure Pipelines.

Make sure you have the following agent pools from your marketplace:

1. Node.js
2. Command line
3. Allure Report Viewer

If not add them.

Update the Node.js version to 16.x as shown in the image:

Step 9: Add a command line as your next agent job. Update the following in its script section:

npm install && npx playwright install && set CI=true && npx playwright test –reporter=line,allure-playwright

where ‘npm install‘ will first install all the dependencies from your code first.
‘npx playwright install’ will install playwright on the server
‘set CI=true’ will take care of the Ctrl+C user input wait message that comes after every failure
‘npx playwright test –reporter=line,allure-playwright’ will basically do the execution along with letting playwright know that it is line reporting with allure that we intend to use and will eventually cause the creation of allure-results folder.

NOTE: For the last line, if you are using package.json scripts, you can simply replace the last line with npm run <scriptName>. Make sure you use –reporter=line,allure-playwright in your script though.

Step 10: Add another command line agent job and put the following in its script:

npm run <Generate Report Script Name>

where <Generate Report Script Name> is the name of the script that you have used in package.json for the following command:

allure generate -c

In my case I am using genReport to define the above, so my script would be:

npm run genReport

The reason we are separating this agent job from the above command line is that on failure playwright tends to stall and it will cause the next command to not run. Hence a separate command line ensures that our job continues the execution after the previous command line exits.

NOTE: The -c flag means cleans. So it will clean any existing allure-report folders and generate the report.

Step 11: The last step would be to open the allure report which it does using the generated allure-report folder. To host that we are using the plugin ‘Allure Report Viewer’. In this plugin, we just have to specify the path of the reports folder. That’s it!

Here in the Azure file Directory section, just mention the name of the folder which would be:

allure-report

That’s it! Save & queue and run the pipeline from the branch you have pushed.

Step 12: When the whole job finishes, navigate to your job. You will notice a separate Allure Reporting section, just click on it and wait.

There you go! Explore the report by clicking on random stuff.

Let me know if our How to Generate Allure Reports in Azure Pipeline with Playwright tutorial was helpful to you in the comments section.

Check out more Playwright related tutorials on our website.