Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

<DataCollector> Tag makes tests fail with error message "Invalid test runner output" #62

Open
GlassOfJuice opened this issue Sep 15, 2022 · 2 comments

Comments

@GlassOfJuice
Copy link

Some background, I started to use the test adapter for catch2 for my own project and it worked wonderfully. One day I was fed up with the abundance of random std/catch code that was shown to me in the test coverage results so I sought to remove them.

After some time I managed to get them deleted using this .runsettings file (Only showing the important part):

<DataCollecttionRunSettings>
    <DataCollector friendlyName="Code Coverage">
        <Configuration>
            <CodeCoverage>
                <Functions>
                    <Exclude>
                        <Function>.*Catch.*</Function>
                        <Function>.*std.*</Function>
                    </Exclude>
                </Functions>
            </CodeCoverage>
        </Configuration>
    </DataCollector>
</DataCollecttionRunSettings>

The problem is that if I use this .runsettings file the results actually get filtered as expected, the problem is that the tests fail with Invalid test runner output. -------------------------..., but if I remove friendlyName=Code Coverage the tests resume to run correctly but the filtering fails.

I was wondering if it is a problem with the adapter or not, I wasn't able to find any explanation to this in the web nor in the documentation of the project sadly :(

@JohnnyHendriks
Copy link
Owner

I have no experience with <DataCollecttionRunSettings>. But if it somehow interferes with the output that is generated for the Catch2 test, that might explain the problem. This was more of a problem before version 1.6.0 of the test adapter, so if you're on an older version by any chance, updating to the latest version of the test adapter, might solve the problem.
If you already are on the latest version of the test adapter, then setting the test adapter <Logging> option to debug could help. When logging is set to debug, the temporary test result files are not automatically deleted, so you can analyse them for the presence of unexpected lines. The temporary test result files should be valid xml. They will be located next to the test executables.

@GlassOfJuice
Copy link
Author

GlassOfJuice commented Sep 19, 2022

( Btw the tag is <DataCollectionRunSettings>, I had a typo while copying that ;) )

I managed to find those temp test results you've mentioned (Ran in debug mode of course) but it seems that they are fine, I even compared them to the output I get when I right click a test -> Profile, which runs the test again.

The temp test result file is the same whenever I run code coverage or when I use Profile, but for some reason profile succeeds and shows the expected output in the Test Explorer whilst the code coverage fails. Are there any more stuff that I can do on my end to hopefully deduce what the problem is?

Thanks for the fast response as well! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants