[Feature]: Label screenshots made in playwright with test file name

[mastrzyz]

🚀 Feature Request

Append the sanitized test name to the automatic screenshot mechanism of Playwright.

Why?

Currently the names are very generic and cause conflicts in CI/CD Systems like Azure Devops when we have multiple jobs failing with the same generic name:

It's also not very clear which number responds to which testcase when we are able to upload:

Example

Fix!

Added the sanitized file name, a good example of how this looks like are the fixed unit tests :

This will allow us to find the artifact easier and not conflict in CI/CD systems like in Azure Devops

Motivation

This is purely a CI/CD Improvement for Playwright integration for us, we have a massively parallel system running hundreds of tests across multiple shards

[dgozman]

@mastrzyz Thank you for filing the issue!

Don't you have the same problem with traces, all named trace.zip? It seems like the better solution is to update your reporter to include the test name in any attachments it generates, since these are presented as a plain list, not grouped by test.

[mastrzyz]

Sorry @dgozman , I'm not using any custom reporter and relying on the existing Playwright reporters which are outputting trace.zip and screenshot1.png.

[dgozman]

@mastrzyz In this case, why do you only focus on the screenshots? I'd think that trace.zip would be equally important. Do I miss something?

If I understand correctly, you are using junit reporter that is uploaded to Azure DevOps through this task, don't you? Perhaps we can improve that part instead? Why does it not show the full attachment path? Reading this post, I would think the full attachment path that already includes test path could be shown. Not to mentioning that it's a part of the specific test output.

[mastrzyz]

You know to tell you the truth, why don't I see this failure for trace.zip!

Could it be that trace.zip is only one? we have retries enabled so I wonder if we take screenshots in first attempt and then seccond also takes and collides