Update plugin: display list of unit tests

[tijana88]

Currently once I pass path to unit test report to MSBuils sonar scanner (I'm using latest c# plugin) it gets imported in SonarQube but only as static number. If I click on it does not list any file.

Is there a way to display all affected files? Especially important for failed tests.

[Evangelink]

Hi @tijana88,

There is no built-in way yet for C#. I will raise the point to our Product Managers. In the meantime, you could setup some script to convert the unit test result report (probably .trx in your case) to our generic xml format (more info here).

[tijana88]

Hi @Evangelink sure thanks for info. I'll try with suggested workaround while waiting fro your feedback.

[tijana88]

@Evangelink we also have nunit reports that we pass using /d:sonar.cs.nunit.reportsPaths="Nunit\Result.xml" -> this is xml why the results are same - just static number ? In c++ there is https://github.com/SonarOpenCommunity/sonar-cxx/wiki/Get-test-execution-metrics

[Evangelink]

Hi @tijana88,

We have decided that we should provide this feature to align with the other plugins. Regarding your last comment I am not 100% sure what you meant. The list will look exactly like for other languages (for example for Java: https://sonarcloud.io/component_measures?id=com.softserve.academy%3ASpaced.Repetition&metric=tests).

[awesley]

I created a workaround for this. Instead of passing SonarQube a trx directly, I converted to SonarQube's Generic Test Data format [1]. You can then provide the argument and path to the Scanner to the transformed XML, and you'll get the standard level of results. A link to the .XSLT transform is at [2].

[1] https://docs.sonarqube.org/display/SONAR/Generic+Test+Data
[2] https://github.com/awesley/trxToSonarQubeXslt

[PulsarFX]

@awesley can you provide a tool name or example how to do the transformation?

[awesley]

@maulwuff

I used Saxonica. Here's a gist of a Groovy script I wrote to call it: https://gist.github.com/awesley/8993eb0a2646203261df4a35f251328d

In there you can see the line with transform.exe - that's Saxonica.

Consuming that script from a Jenkins pipeline step:

def sonarQube = new SonarQube({ x -> bat(x) }, { x -> echo(x) }, { x -> findFiles(x) }, true)
def cwd = /c:\workspace\workspace\${env.JOB_NAME}/
sonarQube.WorkspaceFolder = cwd.toString()
sonarQube.SolutionFolder = /${cwd}\Source/.toString()
sonarQube.SolutionName = "Redacted"
sonarQube.Solution = "${sonarQube.SolutionName}.sln".toString()
sonarQube.BaseNamespace = "LLamasoft.Redacted"
sonarQube.TestProjectPattern = "Tests"
sonarQube.SonarQubeServer = "redacted"
sonarQube.SonarQubeToken = "redacted"
sonarQube.SonarQubeKey = "redacted"            

sonarQube.Delete()
sonarQube.InitializeScanner()
sonarQube.BuildSolution()
sonarQube.TestAndAnalyzeCoverage()
sonarQube.ConvertTrxToSonarQubeGenericFormat()
sonarQube.EndScanner()

[uprasenjit]

Can you guys tell if this issue is fixed and is available in the latest SonarQube version 7.1 ?

[Evangelink]

Hi @uprasenjit,

As you can see the ticket is still open which means it wasn't yet fixed.

[fromorbonia]

Please add this in as a feature! It feels as though you've done all the hard work (i.e. mechanism to import results), but the easy bit of making it clearly visible isn't there!
I've just been going through a process with a business and SonarQube seems the natural place to have that "dashboard view" of code health. Even if all the Unit Tests don't pass (and it's not deployed) the developers need to see progress on the code base. It also helps track down issues between local dev builds and the build server.

[alexshikov]

+1 here interested in this feature

[bdominguez]

Any news?

[andrei-epure-sonarsource]

An update on this topic - this year we are planning to improve the user experience regarding test code, including showing the unit test results in SQ.

At this moment, I don't have a more specific ETA to share.

[andrei-epure-sonarsource]

The workaround from this community thread

I wrote a small ps script to convert the trx sml to the Generic Test Data xml requested by sonarqube.

[andrei-epure-sonarsource]

Currently, we seem to generate (see UnitTestResultsTest )

• tests
• test_errors
• test_failures
• skipped_tests

However, we don't push them to SQ.

More info in #2764

[andrei-epure-sonarsource]

I have an update on this issue, after a discussion with Product Management.

We are closing this issue as Won't fix because:

• SonarQube / SonarCloud analysis should only run if all tests pass - if tests are failing, the code quality of the project is misleading; if tests are failing, the "code coverage" metric is not relevant either (a failed test does not mean that code is not covered, it means that code is covered and it does not do what it should do)

• we believe skipped unit tests should be tracked inside SQ/SC, and we have a rule for it - RSPEC-1607

Our products should be used on a working build. Unit tests / integration tests should be used as functional regression tests, and regressions should not be allowed in the code base.

If unit / integration tests are failing, we believe that the build should fail.

[pascalberger]

@andrei-epure-sonarsource I don't see how your reason for closing this is related to the original report. The original report of this issue was that if test results were reported in a specific format (NUnit, XUnit, ...) it will show the number of test cases, but not the individual tests. If the same data is reported through generic XML format it will also show the individual tests. I don't think this has anything to do with failing tests.

[andrei-epure-sonarsource]

thanks for your reply, @pascalberger. To understand better, what is the use case of having the list of executed unit tests in SonarQube?

[pascalberger]

@andrei-epure-sonarsource First of all it would not be a new feature, as it is already there, just not working. So having a number of tests shown, but while clicking on it showing an empty list is confusing and inconsistent at best, but I would classify it as a bug, especially since it works for other formats.

For the use case of the feature, independent of the format in which the tests were reported: It gives the ability to have SonarQube as central place for everything quality related of a build. Of course I can report test cases somewhere else (e.g. to build server), but for developers this would mean they need to check just another tool: One which shows them test results and another one which shows code coverage.

So the reverse questions would be: What is the use case that SonarQube can show code coverage, numbers of tests, and has an UI for listing tests which is empty if tests are reported in a certain format?

[raulplanas]

@andrei-epure-sonarsource First of all it would not be a new feature, as it is already there, just not working. So having a number of tests shown, but while clicking on it showing an empty list is confusing and inconsistent at best, but I would classify it as a bug, especially since it works for other formats.

For the use case of the feature, independent of the format in which the tests were reported: It gives the ability to have SonarQube as central place for everything quality related of a build. Of course I can report test cases somewhere else (e.g. to build server), but for developers this would mean they need to check just another tool: One which shows them test results and another one which shows code coverage.

So the reverse questions would be: What is the use case that SonarQube can show code coverage, numbers of tests, and has an UI for listing tests which is empty if tests are reported in a certain format?

Completely agree. Would b great to have a central location to look up all quality metrics (Static Analysis, Test Results and Code Coverage) rather than having to go lookup in various different tools depending on scenarios and have to aggregate manually.

[fromorbonia]

@andrei-epure-sonarsource - very disappointed that SonarQube have decided they know better than the community. Not really clear why #2764 is a duplicate of this, I guess you are saying it has the same root cause and is therefore the same? (Still not really a duplicate though, as it is addressing two separate issues).
I'm not saying that this ticket and the related one are the perfect answer, I'm sure there are better solutions.
However, tools like SonarQube are there to make life easier, not to enforce "conceptual boundaries". The fact that it took 2 1/2 years to close the issue clearly shows that there has been plenty of internal debate. Life is messy, development is messy.
The way the CI/CD pipeline (I was working on) got built out resulted in SonarQube potentially being a key hub point, saving the implementation of even more tools on an already complex process.
I hope this is re-considered.

[maulemon]

Hi @andrei-epure-sonarsource

Perhaps some of the drivers to pull in reporting on unit test could be SonarQube as a centre for build quality and security. We wanted to use a single system as our current CI (GitHub Actions) doesn't have good functionality to report on Unit Test failures (Azure Pipelines, TeamCity, etc. have much richer functionality in this area). SonarQube (we actually evaluate SonarCloud) would be a prefect solution giving us flexibility to use different CI systems (for different projects) while keeping one system that guards the quality of development.

We would love for SonarQube to re-evaluate the position - for the time being we are forced to look at alternate solutions.

Thanks,
Mau

[rjgotten]

@andrei-epure-sonarsource

This sounds suspiciously a lot like

It's not a bug, it's (no longer) a feature

It's a very poor excuse when there is clear demand, to the point where the community at large is engineering their own workarounds to make it happen.

I was also looking to centralize code quality checks, including test execution results, in SonarQube. Because it's just that more convenient than having to keep eyes on multiple tools.

Our products should be used on a working build. Unit tests / integration tests should be used as functional regression tests, and regressions should not be allowed in the code base.

If unit / integration tests are failing, we believe that the build should fail.

That's Sonar's philosophy. And frankly, it's heavily opinionated and to the point: quite flawed. Operating from the flawed point of perception of a build being an atomic thing that runs through to live code. In reality, even within CI environments a 'build' consists of several steps that each fall into build; test; or deploy phases. It's the deploy phase which shouldn't go ahead when a test fails.

So, you gate your deployments -- or rather: your automatic deployments. Even if a test is failing, you may still want to push a deployment ahead manually if it fixes important production issues and only breaks a largely non-consequential unit-test.

It is however nonsense to gate your analysis tools. Infact; using the output of your analysis tools to drive the greenlighting of an automatic deployment is a good thing. That's why you have quality gates -- I mean, it's pretty much in the damn name, isn't it? A level of quality you want to uphold and gate actions such as publishing a release behind.

---

For anyone stuck wanting to get decent source-code associations linked to trx reports:
Technically you should be able to cobble together a tool that can interface with the VSTest adapter in use and query its ITestDiscoverer to get the information you need. It will give you TestCase instances that contain both the fully qualified test name as well as the source file location and even the line number of the method signature.
https://github.com/microsoft/vstest/blob/master/src/Microsoft.TestPlatform.ObjectModel/TestCase.cs

[pavel-mikula-sonarsource]

Hi @rjgotten,

https://community.sonarsource.com/ is the correct place to suggest new features, raise traction from community and to get attention of our product managers.

Please feel free to describe intended scenarios and use cases there.

[gmarokov]

Hi guys. I've created a small tool to convert trx files to Generic Test Data expected by SonaCloud - https://github.com/gmarokov/dotnet-trx2sonar
Just remember to use sonar.testExecutionReportPaths when passing the report path.

[flobernd]

@gmarokov Is that solution still valid? I'm trying to use your tool like:

.\.sonar\scanner\dotnet-sonarscanner begin /k:"AUTHORED" /o:"AUTHORED" /d:sonar.login="${{ secrets.SONAR_TOKEN }}" /d:sonar.host.url="https://sonarcloud.io" /d:sonar.cs.vscoveragexml.reportsPaths=coverage.xml /d:sonar.testExecutionReportPaths=SonarTestResults.xml
          dotnet build --no-incremental
          dotnet-coverage collect "dotnet test -l:trx;LogFileName=TestResult.trx" -f xml -o "coverage.xml"
          dotnet-trx2sonar -d ./ -o ./SonarTestResults.xml
          .\.sonar\scanner\dotnet-sonarscanner end /d:sonar.login="${{ secrets.SONAR_TOKEN }}"

But it does not seem to have any effect, No test results are shown.

[yannic-hamann-abb]

Seems to be still an issue....

[andrei-epure-sonarsource]

We got green light from Product to fix this problem.