TestAdapter does not run when using built in CMake

[iwubcode]

I was not able to get the test adapter to be recognized when running a project that was built using the native Visual Studio CMake (File > Open > CMake...). I did have CTest tests added but even after removing all of those, it still was not recognized.

When I say 'not recognized', I mean that there was no log entry for it in the 'Tests' log:

Started Catch2Adapter test discovery

The same project/runsettings uses Catch2Adapter correctly when running CMake by hand (cmake -G "Visual Studio 15 2017 Win64 ...). And I'm able to use it on the reference project.

After slogging through diagnostic logs under the 'Test' section without any luck, I figured I'd come ask here in case there were any suggestions.

Thank you very much for this adapter, it's quite helpful.

[JohnnyHendriks]

Thank you for the feedback. I haven't had the chance yet to look into this particular use case (actually I hadn't even thought of it yet). But it is going to the top of the list now. I'm not sure if I'll be able to fix it, it might be a limitation of the Test Explorer itself. So I can't promise a quick fix.

For now I can only recommend to open a (generated) solution file and use the solution explorer in folder view mode. This should make the experience similar to opening a project in folder view directly.

[iwubcode]

The cmake by hand is a usable workaround, so no rush on my end. I appreciate you taking the time to look into it!

[iwubcode]

With no Visual Studio 2019 cmake support, I'm back to trying to get Visual Studio's cmake to be supported. I haven't opened a visual studio bug yet because I am wondering if there's something this extension needs to do.

I found by looking at NUnit that VSIX test adapters will be deprecated in VS 2019, the following link specifically mentions managed implementations but might still be relevant. See: nunit/nunit3-vs-adapter#518 and https://github.com/Microsoft/vstest-docs/blob/master/RFCs/0022-User-Specified-TestAdapter-Lookup.md.

[JohnnyHendriks]

As far as I've read the deprecation of vsix adapters only goes for .NET unit test framework adapters. Test Adapters for C++ unit test frameworks can keep using vsix (see: http://hermit.no/vsix-based-test-adapters-to-be-deprecated-in-visual-studio-for-c-and-visual-basic-to-speed-up-testing/)

Also, I'm currently setting up a cmake-project with Catch2 unit tests to test/reproduce your problems with the open folder mode of Visual Studio. Hopefully I will know more soon.

[JohnnyHendriks]

Finally got a CMake test project set up. Looks like the CTest integration into Test Explorer only supports Google and Boost unit testing frameworks. All other frameworks get the generic CTest Test Explorer experience (which is very limited). See this blog post.

I will contact Microsoft and see if it is possible to add the Test Adapter for Catch2 to the CTest Test Explorer integration experience.

[iwubcode]

Wow, excellent investigation @JohnnyHendriks ! That was my thought as well. However, I only see a small handful of the tests using the CTest integration, there might be a bug there.

Please let me know what Microsoft says. I would hope the other two frameworks are not hard-coded into the system. Also, please let me know if I can chime in any where to add some numbers when filing the complaint!

[JohnnyHendriks]

What you are seeing is just the name of each executable marked as test in CMake. The individual tests inside the executable are not extracted, for that you need a Test Adapter. Hence my remark that the generic CTest Test Explorer experience is limited.

If you are not seeing all executables that are marked as test in CMake, then there might be a bug. Though I understand the initial reaction of "where are the rest of my tests?", I had the same reaction initially before I realized all I was seeing were the names of the test executables and not the many tests that are inside the test executables.

[jdumas]

Hi @JohnnyHendriks. I was wondering about support for Catch2/CTest in a CMake project in Visual Studio. Did you get an answer from Microsoft on this? Is there an open feature request somewhere so we can track progress? I would love to have my Catch2 unit tests working out of the box in my CMake project. I was thinking that CMake could write the .runsettings file automatically, but support through CTest would be preferable in the long run.

[JohnnyHendriks]

I'm in contact with one of the project managers at Microsoft but I didn't get any feedback yet from the team responsible for the CTest integration. I pinged them again yesterday. All I can do is wait in this case. I'm not aware of an open feature request, but I haven't really looked for one yet.

[carycodes]

Any progress on this?

I did some digging myself and I think it's worth pointing out that if you use the cmake integration provided by catch2 itself (i.e. the catch_discover_tests cmake function) the test explorer will discover all of the catch2 tests. In this case it's still only one executable, but the cmake function is asking the executable to list tests, then dynamically generating an include-able cmake file with add_test commands for all the results. Presumably the default ctest adapter is set up to recognize add_test commands.

This is still unsatisfying, because (unlike the --list-tests output) add_test commands aren't linked to the source location, so double clicking them doesn't navigate to the test, and also debugging from test explorer doesn't properly attach in a way that breakpoints will work.

My current workaround is to set the "Debug and Launch settings" to manually filter which tests I want to run using catch2 command line arguments, and to just debug the entire executable. When a test fails, I have to manually search/navigate to the source for it. This works, but it's a nuisance compared to having that done automatically through the test explorer.

[flooooooooKe]

After 2 full days of research with no success, I considered to ask.

I tried the workaround proposed by @grandopener, but I was not able to get this running. Could you maybe provide a little pet project showing how this is working?

As @grandopener pointed out it would be very usefull to have it included to the test explorer. If you need any support on this, I could give a hand.

Thank you very much for this amazing project.

[ScottHutchinson]

FYI. In lieu of a test adapter, we have found this other extension to be really helpful in choosing which Catch2 tests to run: https://marketplace.visualstudio.com/items?itemName=MBulli.SmartCommandlineArguments

[JohnnyHendriks]

I created a feature suggestion on the Microsoft Developer Community, to request support for custom test adapters in the CTest test adapter (which is used when using the CMake experience in Visuual Studio).

Feel free to upvote the request to let the Visual Studio team know that you would like to have this feature.

https://developercommunity.visualstudio.com/idea/640938/ctest-test-adapter-for-test-explorer-add-custom-te-1.html