[REGRESSION]: VSTest: Change from TestCaseFilter:FullyQualifiedName to TestCaseFilter:ClassName&Name breaks Rerun Failed Tests option

[aaron-luptak]

New issue checklist

• I searched for existing GitHub issues
• I read pipeline troubleshooting guide
• I checked how to collect logs

Task name

VSTest

Breaking task version

3.246.1

Last working task version

3.231.0

Regression Description

Getting

System.Exception: No tests were discovered while attempting to rerun failed tests. This is likely to happen if these tests are xunit or nunit datadriven/parameterized tests or tests with custom display names with spaces or special characters which are not supported with rerun.

on a pipeline where test rerun was previously working.

Tests are MSTest, not datadriven or parameterized, no custom display names.

Classic Pipeline, Windows 7 agent.

Seems to be a change in the TestCaseFilter arugment:
/TestCaseFilter:"FullyQualifiedName=TestCxEpjSerializer.CxEpjSerializerTests.GetEvents_MeetsPerformanceExpectations" - works
/TestCaseFilter:"(ClassName=TestSECSConnect.CxSMSTests&Name=SECSI_NormalMessaging_Works)" - does not work

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows 7

Relevant log output

2024-10-11T20:32:08.3832790Z **************** Rerunning failed tests for Test run 1095563 *********************
2024-10-11T20:32:08.3832790Z Max attempts: 3; Current attempt: 1; Failed test cases threshold: 30; Failed test cases max limit: 0
2024-10-11T20:32:08.3832790Z [command]C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "@C:\Temp\buildagent_work_temp\1k2kqbofe5a.tmp"
2024-10-11T20:32:08.4929018Z Microsoft (R) Test Execution Command Line Tool Version 16.11.0
2024-10-11T20:32:08.4929018Z Copyright (c) Microsoft Corporation. All rights reserved.
2024-10-11T20:32:08.4929018Z vstest.console.exe /TestCaseFilter:"(ClassName=TestSECSConnect.CxSMSTests&Name=SECSI_NormalMessaging_Works)"
2024-10-11T20:32:08.4929018Z "C:\Temp\buildagent_work\4\s\Source\Shared\TestSECSConnect.dll"
2024-10-11T20:32:08.4929018Z /Settings:"C:\Temp\buildagent_work\4\s\Source\dcbmssrluio.tmp.runsettings"
2024-10-11T20:32:08.4929018Z /Logger:"trx"
2024-10-11T20:32:08.4929018Z /TestAdapterPath:"C:\Temp\buildagent_work\4\s"
2024-10-11T20:32:08.4929018Z /Diag:"C:\Temp\buildagent_work_temp\Attempt-3_drsndv.diag"
2024-10-11T20:32:08.4929018Z /logger:trx /platform:x64
2024-10-11T20:32:08.8687514Z Starting test execution, please wait...
2024-10-11T20:32:08.8687514Z Logging Vstest Diagnostics in file: C:\Temp\buildagent_work_temp\Attempt-3_drsndv.diag
2024-10-11T20:32:09.0096950Z A total of 1 test files matched the specified pattern.
2024-10-11T20:32:09.9806398Z The test execution was delegated to legacy TestPlatform runner. You are seeing this message most likely because you are running MSTest v1 tests with a testsettings file, Generic tests, Ordered tests or CodedUI tests. All of those test types, and the legacy TestPlatform runner are legacy. It is recommended that you migrate to MSTest v2.
2024-10-11T20:32:10.2312062Z TestSettings are legacy. It is recommended that you use RunSettings to configure test runs. To learn more, see http://aka.ms/runsettings
2024-10-11T20:32:10.5600746Z No test matches the given testcase filter (ClassName=TestSECSConnect.CxSMSTests&Name=SECSI_NormalMessaging_Works) in C:\Temp\buildagent_work\4\s\Source\Shared\TestSECSConnect.dll
2024-10-11T20:32:10.6853578Z Results File: C:\Temp\buildagent_work\4\s\TestResults\QEBuildService_QEBACOMMON_2024-10-11_14_32_10.trx
2024-10-11T20:32:12.9874366Z ##[debug]Exited vstest.console.exe with code 0.
2024-10-11T20:32:12.9874366Z ##[debug]PERF: ExecuteVsTestPhase.InvokeVSTest: took 4577.1487 ms
2024-10-11T20:32:12.9874366Z Vstest.console.exe exited with code 0.
2024-10-11T20:32:12.9874366Z **************** Completed test execution *********************
2024-10-11T20:32:12.9874366Z ##[debug]Current Phase: MS.VS.TestService.VstestConsoleAdapter.ExecuteVsTestPhase Phase Result: MS.VS.TestService.VstestConsoleAdapter.ExecutionVsTestPhaseResults
2024-10-11T20:32:12.9874366Z ##[debug]PERF WARNING: Running the phase MS.VS.TestService.VstestConsoleAdapter.ExecuteVsTestPhase: took 4577.2488 ms
2024-10-11T20:32:12.9874366Z ##[debug]Current phase: MS.VS.TestService.VstestConsoleAdapter.PublishTestResultPhase
2024-10-11T20:32:12.9874366Z ##[debug]Starting Publish Test Results: TIA: False
2024-10-11T20:32:12.9874366Z ##[debug]Starting PublishTestResult from Result Files
2024-10-11T20:32:12.9874366Z Test results files: C:\Temp\buildagent_work\4\s\TestResults\QEBuildService_QEBACOMMON_2024-10-11_14_32_10.trx
2024-10-11T20:32:12.9874366Z ##[debug]Updating Test results: 1095563
2024-10-11T20:32:12.9874366Z ##[debug]Entering ParseTestResultFiles
2024-10-11T20:32:12.9874366Z ##[debug]runContext.ReleaseURI is null
2024-10-11T20:32:12.9874366Z ##[debug]runContext.ReleaseEnvironmentUri is null
2024-10-11T20:32:12.9874366Z ##[debug]Reading test results from file 'C:\Temp\buildagent_work\4\s\TestResults\QEBuildService_QEBACOMMON_2024-10-11_14_32_10.trx'.
2024-10-11T20:32:12.9874366Z ##[debug]Setting run start and finish times.
2024-10-11T20:32:12.9874366Z ##[debug]Attachment location: C:\Temp\buildagent_work\4\s\TestResults\QEBuildService_QEBACOMMON_2024-10-11_14_32_10\In.
2024-10-11T20:32:12.9874366Z No Result Found to Publish 'C:\Temp\buildagent_work\4\s\TestResults\QEBuildService_QEBACOMMON_2024-10-11_14_32_10.trx'.
2024-10-11T20:32:12.9874366Z ##[debug]Total test results: 0.
2024-10-11T20:32:12.9874366Z ##[debug]Leaving ParseTestResultFiles
2024-10-11T20:32:12.9874366Z Updating test results: 0
2024-10-11T20:32:12.9874366Z ##[debug]Entering UpdateTestResultsAsync
2024-10-11T20:32:13.0500782Z Publishing test results to test run '1095563'.
2024-10-11T20:32:13.6139822Z ##[debug]PERF: GetTestCaseResults: took 544.585 ms
2024-10-11T20:32:13.6139822Z ##[debug]PERF WARNING: GetTestCaseResults: took 544.585 ms
2024-10-11T20:32:13.6139822Z TestResults To Publish 0, Test run id:1095563
2024-10-11T20:32:13.6139822Z ##[debug]Leaving UpdateTestResultsAsync
2024-10-11T20:32:13.6139822Z Updated test results: 0
2024-10-11T20:32:13.6139822Z Publishing Attachments: 1
2024-10-11T20:32:13.6139822Z ##[debug]Entering CreateTestRunAttachmentsAsync
2024-10-11T20:32:13.6766430Z ##[debug]Leaving CreateTestRunAttachmentsAsync
2024-10-11T20:32:13.6766430Z ##[debug]Uploading test run attachements individually
2024-10-11T20:32:13.6766430Z ##[debug]Entering UploadTestRunLogAsync
2024-10-11T20:32:13.6766430Z ##[debug]Blob name: GeneralAttachment/QEBuildService_QEBACOMMON_2024-10-11_14_32_10.trx .
2024-10-11T20:32:13.8646254Z ##[debug]File uploaded successsfully on LogStore QEBuildService_QEBACOMMON_2024-10-11_14_32_10.trx
2024-10-11T20:32:13.8646254Z ##[debug]Completed PublishTestResult
2024-10-11T20:32:13.8646254Z ##[debug]PERF: PublishTestResultPhase.Publish: took 890.0333 ms
2024-10-11T20:32:13.8646254Z ##[debug]Completed Publish Test Results.
2024-10-11T20:32:13.8646254Z ##[debug]Current Phase: MS.VS.TestService.VstestConsoleAdapter.PublishTestResultPhase Phase Result: MS.VS.TestService.VstestConsoleAdapter.PublishTestResultPhaseResults
2024-10-11T20:32:13.8646254Z ##[error]Error occurred while publishing test results : System.Exception: No tests were discovered while attempting to rerun failed tests. This is likely to happen if these tests are xunit or nunit datadriven/parameterized tests or tests with custom display names with spaces or special characters which are not supported with rerun. Please reach out to us at https://github.com/microsoft/azure-pipelines-tasks/issues if you are hitting this for tests other than the ones mentioned before.
2024-10-11T20:32:13.8646254Z ##[debug]Processed: ##vso[task.logissue type=error;]Error occurred while publishing test results : System.Exception: No tests were discovered while attempting to rerun failed tests. This is likely to happen if these tests are xunit or nunit datadriven/parameterized tests or tests with custom display names with spaces or special characters which are not supported with rerun. Please reach out to us at https://github.com/microsoft/azure-pipelines-tasks/issues if you are hitting this for tests other than the ones mentioned before.

Full task logs with system.debug enabled

unsuccess_log.txt
success_log.txt

Repro steps

No response

[aaron-luptak]

A few things we've tried that have not resolved the issue:

• Removing Legacy settings/.testrunconfig so that MSTest v2 can be used
• Using VS2022 Test Platform

I have been able to work around the issue by changing the "Batch tests" option under "Advanced execution options" to "Based on test assemblies" - in this case, it is able to rerun failed tests.