support to execlude somes 'test' in option of command line

[bobshie]

TestNG Version

Note: only the latest version is supported

Expected behavior

three suite file

test.xml

<suite name="Suite">
    <suite-files>
        <suite-file path="./test1.xml" />
        <suite-file path="./test2.xml" />
    </suite-files>
</suite>

test1.xml

<suite name="Suite1" verbose="1">
    <test name="Demo1" preserve-order="true">
        <parameter name="name" value="name1" />
        <classes>
            <class name="com.demo.Demo" />
        </classes>
    </test>
    <test name="Demo2" preserve-order="true">
        <parameter name="name" value="name2" />
        <classes>
            <class name="com.demo.Demo" />
        </classes>
    </test>
    <test name="Demo3" preserve-order="true">
        <parameter name="name" value="name3" />
        <classes>
            <class name="com.demo.Demo" />
        </classes>
    </test>
</suite>

test2.xml

<suite name="Suite2" verbose="1">
    <test name="Demo1" preserve-order="true">
        <parameter name="name" value="name1" />
        <classes>
            <class name="com.demo.Demo" />
        </classes>
    </test>
    <test name="Demo2" preserve-order="true">
        <parameter name="name" value="name2" />
        <classes>
            <class name="com.demo.Demo" />
        </classes>
    </test>
    <test name="Demo3" preserve-order="true">
        <parameter name="name" value="name3" />
        <classes>
            <class name="com.demo.Demo" />
        </classes>
    </test>
</suite>

using existing parameter: 'testnames'

-testnames	A comma separated list of test names.	Only tests defined in a tag matching one of these names will be run.

if we execute the command with --testnames' '/Demo1/ the test result will include
only test: Suite1.Demo1 and Suite2.Demo1
Suite1.Demo1
Suite2.Demo1

if we execute the command with --testnames' '/^(?!Demo1-).*$/ the test result will include
exclude test: Suite1.Demo1 and Suite2.Demo1

includes: 
Suite1.Demo2
Suite1.Demo3
Suite2.Demo2
Suite2.Demo3

Actual behavior

NA.

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue
NA.

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.

[krmahadevan]

@bobshie - Can you please share what is the expectations for the following scenarios?

Order of precedence when this is used in conjunction with the below command line arguments

• -groups
• -excludegroups
• -methodselectors
• -testnames
• -methods

Order of precedence when this is used in conjunction with the below listeners

• org.testng.IAlterSuiteListener
• org.testng.IAnnotationTransformer
• org.testng.IMethodInterceptor
• org.testng.IMethodSelector

Order of precedence when this is used in conjunction with a combo of the above two

What should the behaviour be, when there are one or more names of <test> provided via --excludetests are not present in the <suite> ?

Should TestNG resort to an exact match or support pattern matching for filtering <test>

I will add more points for clarification if something additional pops up at my end.

[YutingZhang-A]

Hi @krmahadevan,
Is it possible to make '-testnames' support regular expression matching? In this way, exclude Test can be implemented through the '^', and we don't need to consider the issue of the order of precedence when they are used in combination.

[krmahadevan]

Hi @krmahadevan, Is it possible to make '-testnames' support regular expression matching? In this way, exclude Test can be implemented through the '^', and we don't need to consider the issue of the order of precedence when they are used in combination.

@YutingZhang-A - I think that would be a good option too. That change would be less invasive, and we can perhaps evaluate if that solves the use case.

@bobshie - WDYT ?

[YutingZhang-A]

Hi @krmahadevan, Is it possible to make '-testnames' support regular expression matching? In this way, exclude Test can be implemented through the '^', and we don't need to consider the issue of the order of precedence when they are used in combination.

@YutingZhang-A - I think that would be a good option too. That change would be less invasive, and we can perhaps evaluate if that solves the use case.

@krmahadevan But we need to think about how to distinguish between a string and a regular expression that the user specifies.

[bobshie]

@bobshie - WDYT ?

that's a good idea. if '-testnames' can be used to exclude the tests, I think it's better than add new options.

if we need to distinguish between a string and a regular expression, we can use '/regex/' (as regex in javascript, and vim) and 'normal string' in -testnames.

@krmahadevan
if you think the solution is acceptable, I can change the description of the issue.

[krmahadevan]

@bobshie - Sure. That would work too. Can you share a sample of how an actual input is going to look like for the regex part ?
Just trying to ensure that we are being explicit in terms of understanding and expectations

[bobshie]

I updated the comments. do you think it's ok?
@krmahadevan , if you have no comments, we will try to implement it and you can review.

[krmahadevan]

@bobshie - So if I use -testnames Demo1 then the existing behaviour should be retained and if I use -testnames /^(?!Demo1-).*$/ then the regex pattern matching kicks in. Is that a fair understanding? So basically one need to wrap the value of -testnames within a pair of delimiter (| in this case I guess). Did I get it right?

Please do proceed if this understanding is in sync with this issue's expectations.

[bobshie]

Yes, I use '/' as delimiter.