One of the most important jobs of the Test Adapter for Catch2 is to discover Catch2 test cases. Of course the default discovery mechanisms that come out of the box with Catch2 are supported. However, there is also support for a custom discovery mechanism. Note, that use of duplicate testcase names results in discovery failure, and an appropriate warning will be logged. In Catch2 v3 it is possible to have identical testcase names as long as the tags are different. However, the test explorer in Visual Studio is not able to deal with this. In this case the name of the duplicate testcase names will be modified (e.g., prepended with [[DUPLICATE 1>>), so it is clear which testcases have this problem. When those tests are run they will be skipped/fail.
The discovery process works as follows. The Test Explorer provides the Test Adapter for Catch2 with a list of executables that may contain Catch2 testcases. In Visual Studio this basically is a list of all projects in a solution that generate an executable with a ".exe" file extension. On each file in the provided list the following actions are performed.
- Check if filename with extension removed, matches the configured filename Regex filter.
- Run executable with configured discovery command line parameters.
- Extract testcases from the output received from the executable.
- Send the testcase information back to the Test Explorer.
Before going into the details in each step, note that typically a collection of Catch2 unit tests are contained in a command line application. It is possible to have Catch2 tests collected in a dynamic link library (dll), however you then need to provide an accessor command line application to run them. The Test Adapter for Catch2 does not have direct support for Catch2 unit tests collected in a dll.
This is a very important step. Typically the Test Adapter for Catch2 is provided by a list of executables based on the contents of a Visual Studio Solution. As such there is no guarantee that all the provided executables actually contain tests. A hypothetical worst case scenario: the name of an executable is provided that if run will format your system drive regardless of the command line parameters it is passed. This is the reason that by default the <FilenameFilter> setting will reject all filenames provided to it. It is the only setting that must be set explicitly.
The Test Adapter for Catch2 supports three different discovery mechanisms:
--list-testsbased (default setting)--list-test-names-onlybased (not recommended, and removed in Catch2 v3)- custom (xml based)
As of version 1.5.0 of the Test Adapter for Catch2, support for the Catch2 discovery command line options in combination with the --verbosity high option was added. Adding the latter option allows discovery of information about the source file and line number the testcase can be found at. The --verbosity high option was added to the default setting as of version 1.5.0 of the Test Adapter for Catch2 as it is supported by Catch2 version 2.0.1 and up.
Before version 1.5.0 the default was not able to retrieve information about the source file and line number the testcase could be found at. This is basically the reason that the Test Adapter for Catch2 has support for a custom discovery mechanism, which is explained below.
As of version 1.8.0 of the Test Adapter for Catch2, support for the Catch2 v3 capability to set a reporter for discovery is supported. More specifically the output generated by the xml reporter in combination with --list-tests. This effectively removes the need for the custom discovery option, when Catch2 v3 is used.
As discovery of testcases requires the provided executable to be run, there is a chance that for whatever reason the executable does not stop automatically after it is run. Maybe it accidentally passed the filename filter in step 1, and it is actually a process that runs forever. For this reason you can configure a <DiscoverTimeout> that will kill the process if the discovery process takes longer than the set timeout. By default, this timeout is set to 1 second, which is typically more than enough.
The output generated in step 2 is processed in this step. The actual processing algorithm used depends on the discovery mechanism. Apart from that it is possible to filter out hidden testcases based on the tags associated with a testcase. This filter can be configured with the <IncludeHidden> setting.
After the testcases are extracted they are provided back to the Test Explorer in a format that it understands. The Test Explorer can then use this information to request the Test Adapter for Catch2 to execute specific test cases.
This mechanism is used by default. It can discover both testcase names and tag names. Each tag-name is added as a Test Explorer testcase trait. It can discover most testcase names, with the exception of some edge cases.
This mechanism is only able to discover testcase names. It is however more robust in its ability to discover testcase names. Note however, that this discovery mechanism has been put on the Catch2 deprecation list and has been removed in Catch2 v3.
When custom discovery is used the output to be processed is expected to be Catch2 Xml (i.e., the same Xml used by the Catch2 xml reporter). Initially this was introduced to enable discovery of information about the source file and line number the testcase can be found at. I was unaware of the possibility to use the --verbosity high option to get this information and came up with this solution. As of version 1.5.0 of the Test Adapter for Catch2 awareness was raised and the need for custom discovery is diminished. However, there may still be use cases for it, so the feature remains. Note, if you make use of Catch2 v3 there should be no need to make use of this feature.
To make use of the custom discovery mechanism you need to add a new command line option to the Catch2 executable. This means you must make use of a custom main implementation. If you then configure the <DiscoverCommandLine> setting to make use of this newly introduced option, the custom discovery mechanism will be used. More precisely, the custom discovery mechanism will be used whenever none of the default discovery options (i.e., --list-tests, -l, or --list-test-names-only) are used in the <DiscoverCommandLine> setting.
Based on the requirement for custom discovery you can also successfully trigger custom discovery by just running the tests using the xml-reporter (_e.g., using -r xml * as a discover command line). There are however some issues with this approach, apart from the obvious that you probably do not want to run the actual tests just to discover them. The default xml reporter that comes with Catch2 trims the names of testcases. As a result, testcases for which the name was modified by trimming the name cannot be executed by the Test Adapter for Catch2.
So, to demostrate the use of the custom discovery mechanism we will solve that problem, by making use of a custom reporter specially geared towards testcase discovery. Note that this custom discoverer is also used by the reference tests used to test the Test Adapter for Catch2.
This is an example of a minimal .runsettings file that makes use of the custom discovery algorithm shown in this example.
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Adapter Specific sections -->
<Catch2Adapter disabled="false">
<DiscoverCommandLine>--discover *</DiscoverCommandLine>
<FilenameFilter>.*</FilenameFilter><!-- Regex filter -->
</Catch2Adapter>
</RunSettings>#define CATCH_CONFIG_RUNNER
#include <catch.hpp>
#include "catch_discover.hpp"
int main(int argc, char* argv[])
{
Catch::Session session;
bool doDiscover = false;
Catch::addDiscoverOption(session, doDiscover);
int returnCode = session.applyCommandLine(argc, argv);
if (returnCode != 0) return returnCode;
return Catch::runDiscoverSession(session, doDiscover);
}Note, that for convenience the actual discover mechanism is contained inside the "catch_discover.hpp" header file. There the Catch::addDiscoverOption and Catch::runDiscoverSession functions are defined. This header should only be included after #include <catch.hpp> and only used in the place main is defined. This basically is the way custom reporters are defined for Catch2.
This basically follows the example of how to add you own command line options in the Catch2 documentation.
void addDiscoverOption(Session& session, bool& doDiscover)
{
using namespace Catch::clara;
auto cli = session.cli()
| Opt(doDiscover)
["--discover"]
("Perform VS Test Adaptor discovery");
session.cli(cli);
}Again, this basically follows the example of how to add you own command line options in the Catch2 documentation. The actual interesting bit is contained in the discoverTests function.
int runDiscoverSession(Session& session, bool& doDiscover)
{
if(doDiscover)
{
try
{
discoverTests(session);
return 0;
}
catch( std::exception& ex )
{
cerr() << ex.what() << std::endl;
return 255;
}
}
return session.run();
}Note, that use is made of a custom reporter named "discover". This reporter is then used to report the testcases that were retrieved from the Catch2 session.
void discoverTests(Catch::Session& session)
{
// Retrieve testcases
const auto& config = session.config();
auto testspec = config.testSpec();
auto testcases = filterTests( Catch::getAllTestCasesSorted(config)
, testspec
, config );
// Setup reporter
TestRunInfo runInfo(config.name());
auto pConfig = std::make_shared<Config const>(session.configData());
auto reporter = getRegistryHub().getReporterRegistry()
.create("discover", pConfig);
// Start report
Catch::Totals totals;
reporter->testRunStarting(runInfo);
reporter->testGroupStarting(GroupInfo(config.name(), 1, 1));
// Report test cases
for (const auto& testcase : testcases)
{
Catch::TestCaseInfo caseinfo( testcase.name
, testcase.className
, testcase.description
, testcase.tags
, testcase.lineInfo );
reporter->testCaseStarting(caseinfo);
}
// Close report
reporter->testGroupEnded(Catch::GroupInfo(config.name(), 1, 1));
TestRunStats testrunstats(runInfo, totals, false);
reporter->testRunEnded(testrunstats);
}Note that this reporter is basically a modified version of the xml reporter that comes with Catch2. The differences are that Section and Assertion info are not reported, and that the testcase names are not trimmed. You could even use this reporter for running tests. Although any test result information would not be reported, which means it is pretty much useless for that use case.
class DiscoverReporter : public StreamingReporterBase<DiscoverReporter>
{
public:
DiscoverReporter(ReporterConfig const& _config);
~DiscoverReporter() override;
static std::string getDescription();
virtual std::string getStylesheetRef() const;
void writeSourceInfo(SourceLineInfo const& sourceInfo);
public: // StreamingReporterBase
void testRunStarting(TestRunInfo const& testInfo) override;
void testGroupStarting(GroupInfo const& groupInfo) override;
void testCaseStarting(TestCaseInfo const& testInfo) override;
void assertionStarting(AssertionInfo const&) override;
bool assertionEnded(AssertionStats const& assertionStats) override;
private:
XmlWriter m_xml;
};
DiscoverReporter::DiscoverReporter( ReporterConfig const& _config )
: StreamingReporterBase( _config ),
m_xml(_config.stream())
{ }
DiscoverReporter::~DiscoverReporter() = default;
std::string DiscoverReporter::getDescription()
{
return "Reports testcase information as an XML document";
}
std::string DiscoverReporter::getStylesheetRef() const
{
return std::string();
}
void DiscoverReporter::writeSourceInfo( SourceLineInfo const& sourceInfo )
{
m_xml.writeAttribute( "filename", sourceInfo.file )
.writeAttribute( "line", sourceInfo.line );
}
void DiscoverReporter::testRunStarting( TestRunInfo const& testInfo )
{
StreamingReporterBase::testRunStarting( testInfo );
std::string stylesheetRef = getStylesheetRef();
if( !stylesheetRef.empty() )
m_xml.writeStylesheetRef( stylesheetRef );
m_xml.startElement( "Catch" );
if( !m_config->name().empty() )
m_xml.writeAttribute( "name", m_config->name() );
}
void DiscoverReporter::testGroupStarting( GroupInfo const& groupInfo )
{
StreamingReporterBase::testGroupStarting( groupInfo );
m_xml.startElement( "Group" )
.writeAttribute( "name", groupInfo.name );
}
void DiscoverReporter::testCaseStarting( TestCaseInfo const& testInfo )
{
StreamingReporterBase::testCaseStarting(testInfo);
m_xml.startElement( "TestCase" )
.writeAttribute( "name", testInfo.name )
.writeAttribute( "description", testInfo.description )
.writeAttribute( "tags", testInfo.tagsAsString() );
writeSourceInfo( testInfo.lineInfo );
m_xml.endElement();
}
void DiscoverReporter::assertionStarting( AssertionInfo const& ) { }
bool DiscoverReporter::assertionEnded( AssertionStats const& )
{
return true;
}
CATCH_REGISTER_REPORTER( "discover", DiscoverReporter )