Changes are relative to v1.0.0
This is a big release that adds some features to make some common use cases easier to work with. Most notably is the ability to directly use a dll as a test source, which allows Test Explorer to pick up on changes to the tests and re-discover tests without the need to jump through hoops.
- Add direct support for dll based tests
- Add support for test source specific configuration overrides
- Remove DiscoverCommandline requirements check
- Change default setting of
<ExecutionMode>toCombine(i.e., run tests faster by default) - Prepend tag with process ID to log lines
- Issue #54 Recognize also catch output xml without a name
- Issue #70 Fix potential endless loop
This version introduces support for Catch2 v3, which has a small change in xml-reporting format and has improved support for test discovery.
- Add support for Catch2 v3 (issue #52)
- Enable support for environmental variables when running in debug mode.
- Add support for Visual Studio 2022.
- Add support for environmental variables (inspired by issue #37). Environmental variables to be set for the Catch2 executable process can now be configured with the
<Environment>option (see Settings documentation for details).
- Issue #46: A problem deleting temporary files, caused the test adapter to fail.
This version introduces some major changes in the way the backend works. Where before for each test case an instance of the test executable was started (still the default behavior), now it is possible to run multiple test cases in a single instance of the test executable, improving test execution time significantly. Related to this there is also a mechanism introduced to force a test case to be run in its own test executable instance, when using the new way of executing test cases.
Furthermore, the test results are no longer written to memory, but to a temporary file before being processed. Though needed to make the above change possible, the positive side effect is that the test explorer now no longer has problems with tests that write information to the terminal in a way that is not intercepted by Catch2. In addition, the backend is better able to handle partial xml test result output, which typically occurs when the test executable instance needs to be killed due to a timeout, or when the test executable is prematurely terminated.
This version fixes a problem installing the Test Adapter for Catch2 on VS2019 RC.
This version contains significant improvements in testcase discovery. No longer do you need a custom discovery mechanism to enable source links to the testcase. Also, the discovery mechanism has been made more robust.
- Issue #18: Added support for verbose default discovery. This enables creating a source file link to the position of a testcase for easy navigation via the Test Explorer, by using default discovery. This removes the need to add a custom Xml-based discovery mechanism to enable this testcase navigation feature in the Test Explorer.
- The default value for
<DiscoverCommandLine>has been set to--verbosity high --list-tests *, this enables creating a source file link to the position of a testcase for easy navigation via the Test Explorer.
- Issue #17: Testcases and tags with long names are not detected correctly using default discovery settings. Mostly fixed, there are a few corner cases that cannot be fixed.
- Updated the walkthrough to reflect the availability of verbose default discovery.
- Added a page on the discovery mechanism. This includes an updated and improved custom discovery example.
This version adds support for VS2019. It also updates the Microsoft.TestPlatform.ObjectModel to version 15.9.0.
- Bug: When tests are run via "Debug Selected Tests" the working directory is set incorrectly. Fixed.
- Improve Additional info output with respect to section. Is now more like standard Catch2 output.
- Improve how is dealt with problematic test case names. Tests with such names are now marked as skipped, with an error message that suggests a possible solution.
- Bug: Test durations in Test Explorer are wrong for systems setup with a decimal symbol that is not '.'. Fixed.
-
Add option
<StackTracePointReplacement>to allow replacement of decimal points in StackTrace description with a custom string. This is related to a bug fix, where decimal points in the StackTrace description interferes with the displayed StackTrace link. -
Handle test cases with names differing only in case more gracefully (e.g., "TestName", "TESTNAME", and "testname"). Test cases with names that only differ by case are always run together. Now the correct test result and failures are shown for those tests. Shown assertion statistics are of all tests that were run, and a note is added to the test result message to indicate this.
- Bug: StackTrace link does not display correctly when decimal points are part of the description (e.g., when displaying floating point numbers). Fixed.
- Bug: Test cases with names ending in '\' generate invalid output. Fixed.
This version contains an important fix that enables stack trace links to source code. This is a significant usability improvement. Please update to this version and stop using the older ones. Documentation has been adapted to assume you are using this version.
- Added
<MessageFormat>option to configure what information is shown in the message part of the test case detail view. The default is to only show the assertion statistics for the test. It is also possible to choose to show additional info in the message, or to not have a message at all.
- Improve error handling, specifically when a discovery timeout occurs it is now logged as a warning at logging level normal.
- Enable stack trace links in the Test Explorer detail view. With the help of Microsoft I was able to figure out the correct string format to use for the Stacktrace info in order to turn it into a source link. As a result the
<StackTraceFormat>option was also altered and now has the optionsNoneandShortInfo, the latter being the default or fall-back value.
- The default value for
<DiscoverCommandLine>has been set to--list-tests *, as having two settings with invalid defaults is not really useful. - The default value for
<DiscoverTimeout>has been set to 1000 ms. There were situations where 500 ms turned out to be too short, doubling that hopefully will give less problems.
- Bug: CHECK_FALSE and REQUIRE_FALSE failed output expansion shows '!' in front of value results. E.g., "CHECK_FALSE( !true )", should be "CHECK_FALSE( true )". Fixed.
- Added
disabledattribute to<Catch2Adapter>node for use in the .runsettings file. - Added
<DebugBreak>setting that turns on or off Catch2's break on test failure feature (--break), when running tests viaDebug Selected Tests.
- Added
Debuglevel option to<Logging>setting and updated what is logged in theVerboselevel. - Improve error handling, specifically when a duplicate test name is used, the potential resulting error is logged.
- Add support for reporting Fatal Error Conditions
- Bug: Warnings in Sections are not displayed when there are no failures. Fixed.
- Bug: Get invalid test runner output error when Catch2 xml output produces additional text after xml report. Additional text is now ignored.