[Question] Up-to-date overview: How to use ReportGenerator with Azure DevOps? #646
Comments
|
Thanks for your detailed issue and the research behind it. You are right, there are several options. PublishCodeCoverageResults@1 uses an (outdated) version of ReportGenerator internally. PublishCodeCoverageResults@2 does not use ReportGenerator internally. If you have different information, I would be interested. Regarding your questions:
Correct. You'll have to combine it with PublishCodeCoverageResults@1.
You could generate two reports in separate directories. Then you would probably have to rename the - task: reportgenerator@5
displayName: ReportGenerator
inputs:
reports: 'src\target\reports\coverage\coverage.xml'
targetdir: '$(Build.SourcesDirectory)/coveragereport'
reporttypes: 'HtmlInline_AzurePipelines;HtmlSummary;Cobertura'
customSettings: 'settings:createSubdirectoryForAllReportTypes=true'
# Now rename file `$(Build.SourcesDirectory)/coveragereport/HtmlSummary/summary.html` to `$(Build.SourcesDirectory)/coveragereport/HtmlSummary/index.html`
# Create artifact of directory $(Build.SourcesDirectory)/coveragereport/HtmlInline_AzurePipelines
- task: PublishCodeCoverageResults@1
displayName: 'Publish code coverage results'
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(Build.SourcesDirectory)/coveragereport/Cobertura/Cobertura.xml'
reportDirectory: '$(Build.SourcesDirectory)/coveragereport/HtmlSummary'
env:
DISABLE_COVERAGE_AUTOGENERATE: 'true'
As mentioned above, I'm pretty sure that you have to use PublishCodeCoverageResults@1.
I guess that's not possible. See also your next question.
This is also my experience. See my answers above. I always use PublishCodeCoverageResults@1 instead of PublishCodeCoverageResults@2. |
|
Thanks for your detailed response!
My example pipeline above with - task: DotNetCoreCLI@2
inputs:
command: test
projects: '$(solution)'
arguments: '--no-restore --configuration Release -s CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
testRunTitle: 'release'
publishTestResults: true
displayName: dotnet test Release
- task: PublishCodeCoverageResults@2
displayName: 'Publish code coverage report'
inputs:
summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'
failIfCoverageEmpty: true- and notably without any
=> Thus, a V2
Not sure how this might affect the behaviour. I changed my pipeline now like this and it works as exected. The Code Coverage tab does contain the ReportGenerator report even after the end of the job (and also still accounts for - task: DotNetCoreCLI@2
inputs:
command: test
projects: '$(solution)'
arguments: '--no-restore --configuration Release -s CodeCoverage.runsettings --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
testRunTitle: 'release'
publishTestResults: true
displayName: dotnet test Release
# https://github.com/danielpalme/ReportGenerator/wiki/Integration#azure-devops-extension
- task: reportgenerator@5
displayName: '[Coverage] ReportGenerator'
inputs:
reports: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'
targetdir: '$(Agent.TempDirectory)/coveragereport'
reporttypes: 'HtmlInline_AzurePipelines;Cobertura;Badges'
assemblyfilters: '-xunit*'
- task: PublishCodeCoverageResults@1
displayName: '[Coverage] Publish results'
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(Agent.TempDirectory)/coveragereport/Cobertura.xml'
reportDirectory: '$(Agent.TempDirectory)/coveragereport'
env:
DISABLE_COVERAGE_AUTOGENERATE: 'true'Thanks for pointing me to https://github.com/danielpalme/ReportGenerator/wiki/Integration#azure-devops-extension. I actually overlooked this first (even though it's linked in the README 🙄 - I guess there are just many places providing pieces of information, some of which are more up to date while others aren't). |
There is no such option. I looked at the source code: https://github.com/microsoft/azure-pipelines-tasks/tree/master/Tasks/PublishCodeCoverageResultsV2 Internally |
|
FWIW I would have benefitted from such a guide. It took me some time to understand that the PublishCodeCoverageResults@2 task was flawed, and the older version is preferred in my scenario (which was similar to the one described above). The main difficulty I faced in researching the production of a Devops Pipeline code coverage report, was that I didn't realise that a lot of the examples out in the wild (using the V2 task) are quite simplistic and don't cover what I'd consider a "real-world" scenario. In addition I thought any examples using the older V1 task were out-of-date :-/ |
|
@rikrak But the overall process is documented here: The FAQs also answer the most common question: |
@georg-jung I come cross the same behaviour. When I test steps like:
I can see that PublishCodeCoverageResults@2 internally use ReportGenerator, it shows a footer in the report saying that. But later, the whole report is override when pipeline finished. Feel like a bug for that task. Also, I try the So the options I have tested and working for me:
Option2:
Option3:
Opeion 2 and 1 have no much different. But Option1 give your more customization option in ReportGenerator task. To me, Option 2 is easier to setup then option 1 and do the job, if you don't need too much customization on ReportGenerator task and dont mind to use ReportGenerator 5.1.14.0. One thing is missing and I believe is useful is that the 'collapose all' and 'expand all' link. If I generate the report locally on my laptop, I can use them to quickly collapose the list. But they are not in ADO's coverage tab, meaning that I have to use scrol bar. |
|
Thanks @jchenathub24 this clears my doubts about PublishCodeCoverageResults@2 ,I faced the same issue when using V2 task as it seems to use reportgenerator internally; First it generates the HtmlInline_AzurePipelines report and publishes that to the Code Coverage tab then after a couple of seconds it repopulates the tab with the default ADO layout. I'm having to resort to using PublishCodeCoverageResults@1 as mentioned in your Option1 cause the ReportGenerator report is just so much better. Also wish there was |
That's because Azure DevOps blocks all JavaScript. |
|
@danielpalme I think this issue should be reopened because the ReportGenerator documentation regarding Azure DevOps tasks should be updated. A few weeks ago Azure DevOps pipeline builds that use V1 of the PublishCodeCoverageResults tasks started showing deprecation warnings. See also the announcement here: https://devblogs.microsoft.com/devops/new-pccr-task/ Important part in that announcement:
So there should be a documentation that shows how to correctly use ReportGenerator with PublishCodeCoverageResults V2. I think if you don't care how the HTML report looks and if you don't need to set any additional ReportGenerator settings, then you can simply use |
|
@cremor
The new version has several disadvantages regarding the report in the
|
|
I just released version 5.3.0 of ReportGenerator. The Azure DevOps task now has a new setting Example: - task: reportgenerator@5
displayName: ReportGenerator
inputs:
reports: 'coverage.xml'
targetdir: 'coveragereport'
publishCodeCoverageResults: true |
|
When publishing the report "your way" is then still the html file regenerated from azure devops (which does not contain the branch and method coverage and details per class) or is your richer report uploaded? |
|
@MTomBosch |
Thanks! Documentation |
I don't know. I have not tested this. Would like to hear if this also appears in PRs. |
Interesting.
They say it only supports |
That's wrong or outdated. The screenshot I posted above is from a project that doesn't do anything with a .coverage file. At least not explicitly. - task: DotNetCoreCLI@2
displayName: 'Run tests'
inputs:
command: 'test'
# Collect code coverage in OpenCover format because this is what SonarCloud supports.
arguments: '--no-restore --configuration $(buildConfiguration) --collect:"XPlat Code Coverage;Format=opencover"'
publishTestResults: true
- task: PublishCodeCoverageResults@2
displayName: 'Publish code coverage report'
inputs:
summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.opencover.xml'I'm planning to replace |
I see, this might be an undisclosed feature they're adding the |
|
|
I can now answer this. Sadly PR line coverage does not show up with At least I know that it is not related to the files which are published as artifacts. Because currently |
Still, cheers for sharing that something like code coverage for pull requests was a thing. That would be a very useful feature indeed. Until then, thanks to @danielpalme we can now get rid of the deprecated warning that came with the V1 task. 🙌 |
|
OK, I just got caught up on this thread after finally getting the time to address the MS Publish code coverage results@v1 deprecation. @danielpalme Been using this tool for year, such a great tool. TY for adding the simple publish option in the latest of your ADO task, this solves the immediate issue of MS's deprecated publish coverage results task as well as the long-standing complicated config to publish the reports from your tool instead of the ADO-native reports. Nothing like a forcing function to push progress! To summarize this thread, this post shows how to publish this tool's reports directly to the Tests/Coverage tab in ADO without another "publish" task, MS or otherwise. TY again! I still get line coverage after only changing the publish option and removing MS's publish task entirely. If it is missing, you may have your test step configured wrong. I use this package to generate the raw coverlet data, which this tool can then consume to create the reports. Include it in all unit-test projects like: The step to run tests in my pipeline: |
@jasells My tests and coverage options are configured correctly. When I tested this a month ago, I simply created a pull request with a few changes and checked the PR for line coverage indicators. There were none when I used Are you sure that we are talking about the same thing? I explicitly mean the line coverage indicators in pull requests. I do not mean the coverage tab on the pipeline build result page. Documentation |
Ah, I think we are talking about different things? I get the line-coverage in the coverage reports, not inline in the PR file-compare. I've never had that, didn't know it was an option. I'll look into that. What I have done is save the coverage history to a Universal Package at end of pipeline, and download the latest before running the generator to get a coverage history graph. That might help you a little? It would be nice to have to coverage inline in the PR, for sure. |
Thanks for creating ReportGenerator! I think it provides a great overview of code coverage and I really like all the integration options.
I got however a bit confused about all the options to use it and the best way to use ReportGenerator with Azure DevOps as of now, 2024. I've seen many issues discussing this topic including #398, but many of them are outdated as of 2024. Maybe it could be helpful for new users to have one central up-to-date page with guidance (which the other places including the Azure DevOps Marketplace page could link to). I'd be happy to contribute in that direction if that makes sense, but I guess at this point the best thing I can do is asking the questions I thought about when getting started with Code Coverage in Azure DevOps.
My Goals (decreasing prio)
Options I came across
What I think I understood
.netconfig fileto configure ReportGenerator while using PublishCodeCoverageResults@2 (maybe this should be documented somewhere, because I think it's a great feature). Mentioning how to use a ReportGenerator license with PublishCodeCoverageResults@2 on https://reportgenerator.io/pro would probably be easy and could even make some people buy a license (I tried this and it works).Questions
Whats the minimal, optimal but full example to use ReportGenerator with Azure DevOps today? I've seen https://reportgenerator.io/usage but in my testing using only the
reportgeneratortask didn't produce e.g. the Code Coverage tab.Report type HtmlSummary sounds like it could help with PublishCodeCoverageResults is slow with many files in the generated report microsoft/azure-pipelines-tasks#4945, if it's acceptable to e.g. have just the summary in the "Code Coverage" tab but the full report in the zip artifact. Is there an easy way to set this up?
disable.coverage.autogenerate: 'true'doesn't seem to work for me withPublishCodeCoverageResults@2. Is there a workaround for V2 too? Is it better to use V1 withreportgenerator@5?I have a DotNetCoreCLI@2 test task like the following. The PublishCodeCoverageResults@2 seems to first publish ReportGenerator's report, which is later overwritten. How to fix this while still publishing the trx test results?
I hope I didn't miss any obvious places to research this myself. Thanks again for creating ReportGenerator!
The text was updated successfully, but these errors were encountered: