added feature only fetch the tests #112

omkarkhatavkar · 2020-11-02T16:30:05Z

Fixed #93

- now we can fetch only modified and new tests written from the files
- It uses git diff to know what files and tests are updated and it run those tests only.
- writing the unit tests

New Command ?

pytest --picked --mode=onlychanged

How to use this ?

If you wanted to run modified or newly added tests here is the example of that

(env) /home/okhatavk/Satellite/robottelo (fixed_failures_ldap_1) $ git diff 
diff --git a/tests/foreman/ui/test_ldap_authentication.py b/tests/foreman/ui/test_ldap_authentication.py
index 2ce07c4..ebd406e 100644
--- a/tests/foreman/ui/test_ldap_authentication.py
+++ b/tests/foreman/ui/test_ldap_authentication.py
@@ -181,6 +181,7 @@ def test_positive_end_to_end_ad(session, ldap_tear_down, ad_data):
        assert session.ldapauthentication.read_table_row(ldap_auth_name)['Server'] == new_server
        session.ldapauthentication.delete(ldap_auth_name)
        assert not session.ldapauthentication.read_table_row(ldap_auth_name)
+        assert not session.ldapauthentication.read_table_row(ldap_auth_name)
+
+@tier2
+@run_in_one_thread
+def test_verify_ldap_filters_ipa(session, ipa_data, ldap_tear_down):
+    """Verifying ldap filters in authsource to restrict access

With this change or PR we can easily run those tests instead whole file

(env) /home/okhatavk/Satellite/robottelo (fixed_failures_ldap_1) $ pytest --picked --mode=onlychanged --collect-only

Changed test files... 2. ['tests/foreman/ui/test_ldap_authentication.py::test_positive_end_to_end_ad', 'tests/foreman/ui/test_ldap_authentication.py::test_verify_ldap_filters_ipa',]
Changed test folders... 0. []
========================================================= test session starts ==========================================================
platform linux -- Python 3.6.8, pytest-5.4.3, py-1.8.1, pluggy-0.13.1
rootdir: /home/okhatavk/Satellite/robottelo
plugins: forked-1.1.3, cov-2.8.1, cagoule-0.4.0, mock-1.10.4, xdist-1.34.0, lazy-fixture-0.6.3, services-1.3.1, instafail-0.4.2, ibutsu-1.1.1, picked-0.4.5
collecting ... 2020-11-02 16:23:28 - conftest - DEBUG - Collected 6 test cases
collected 2 items                                                                                                                      
<Package /home/okhatavk/Satellite/robottelo/tests/foreman/ui>
  <Module test_ldap_authentication.py>
    <Function test_positive_end_to_end_ad>
    <Function test_verify_ldap_filters_ipa>

omkarkhatavkar · 2020-11-02T16:43:50Z

@anapaulagomes how locally I can test this PR to certify for CI?

anapaulagomes · 2020-11-03T08:39:04Z

Hi @omkarkhatavkar! Thanks for your contribution. Could you please add an entry to our CHANGELOG?
You can test it locally by installing the package using python setup.py develop and running pytest - that's it.

PS.: this week I'm quite busy but I'll get back to your PR most likely next week.

anapaulagomes · 2020-11-17T07:59:47Z

Hi @omkarkhatavkar! Just to let you know, I'll review your PR as soon as it has tests on it. This way we avoid back and forths. Thanks!

omkarkhatavkar · 2020-11-17T13:22:37Z

@anapaulagomes got busy with Diwali Festival will be updating soon

anapaulagomes · 2020-11-17T13:32:08Z

No worries, @omkarkhatavkar! Hope you had a good time. :)

omkarkhatavkar · 2020-11-18T18:57:21Z

No worries, @omkarkhatavkar! Hope you had a good time. :)

Yeah had nice time ! Now the tests are added. Let me know if there anything need to be added.

omkarkhatavkar · 2020-11-19T17:19:54Z

@anapaulagomes I made some more logical change, added more validation with tests and supporting the test class as well.

omkarkhatavkar · 2020-12-05T07:00:33Z

@anapaulagomes any update on this?

anapaulagomes · 2020-12-05T15:09:39Z

Hi @omkarkhatavkar! I'll review it next week. Sorry for the delay.

anapaulagomes

Thanks for your contribution, @omkarkhatavkar! I'm really looking forward to having this feature merged. 👯 I left a few questions and raised some concerns.

CHANGELOG.md

pytest_picked/modes.py

tests/test_modes.py

pytest_picked/modes.py

README.rst

omkarkhatavkar · 2020-12-12T18:37:44Z

@anapaulagomes there was a linting issue but not because of my PR also resolved that.

anapaulagomes · 2020-12-17T16:09:41Z

Hi @omkarkhatavkar! Just so you know, I'll take a look on it next week. 🙏🏽

omkarkhatavkar · 2020-12-28T05:28:05Z

Any update on this @anapaulagomes

anapaulagomes

Hi @omkarkhatavkar, sorry for not getting back to you when I promised. The truth is: I've been involved in different open source projects in my free time and I try to do my best in each of them. This means that it takes much of my free time. That said: I'll try to review the next changes as soon as I can - please don't expect it to be on the next day. Hope you understand this.

As I said before, I'm really looking forward to releasing this feature. But we still need to do some work here. I left a few comments on how the code can be improved. I also would like to suggest adding more tests for all different scenarios: test files, classes, and test methods. After breaking the method only_tests down is going to be easier to write tests for each of them. Also, I think we should invest time in adding different diff files for these scenarios. This will bring the security we need that the feature is working fine and it's change proof.

Thanks for the work you've done! 🥇

anapaulagomes · 2021-01-08T09:44:57Z

tests/test_modes.py

@@ -53,7 +53,8 @@ def test_parser_should_ignore_no_paths_characteres(self, line, expected_line):
        assert parsed_line == expected_line

    def test_should_list_unstaged_changed_files_as_affected_tests(self):
-        test_file_convention = ["test_*.py", "*_test.py"]
+        test_conventions = dict()
+        test_conventions["file"] = ["test_*.py", "*_test.py"]


You can directly create the dict with the file values:

Suggested change

test_conventions["file"] = ["test_*.py", "*_test.py"]

test_conventions = {

"file": ["test_*.py", "*_test.py"]

}

anapaulagomes · 2021-01-08T09:45:06Z

tests/test_modes.py

@@ -135,11 +136,12 @@ def test_should_list_branch_changed_files_as_affected_tests(self):
            b"tests/test_new_pytest_picked.py\n"
            b"M       tests/test_other_module.py"
        )
-        test_file_convention = ["test_*.py", "*_test.py"]
+        test_conventions = dict()


anapaulagomes · 2021-01-08T09:45:33Z

tests/test_modes.py

+            b"\n+++ b/tests/pytestpicked/test_modes6.py"
+            b"\n@@ -191,29 +191,8 @@ class InvalidClass:"
+        )
+        test_conventions = dict()


anapaulagomes · 2021-01-08T09:47:48Z

tests/test_modes.py

+        test_conventions = dict()
+        test_conventions["file"] = ["test_*.py", "*_test.py"]
+        test_conventions["class"] = ["Test", "Check"]
+        test_conventions["function"] = ["test", "check_*"]


Different conventions are passed but the input doesn't include all of them (tests with Check in the beginning and *_test.py are missing, for example).

anapaulagomes · 2021-01-08T09:48:11Z

pytest_picked/plugin.py

-        "unstaged": Unstaged(test_file_convention),
+        "branch": Branch(test_conventions),
+        "unstaged": Unstaged(test_conventions),
+        "onlychanged": OnlyChanged(test_conventions, only_modified_tests=True),


Nice 👍🏽

anapaulagomes · 2021-01-08T10:24:39Z

pytest_picked/modes.py

+                if re.search(class_regex, file_or_test):
+                    match = re.findall(class_regex, file_or_test)
+                    class_name = match[0]
+                    file_dict[last_file_name].append(class_name)


Suggested change

file_dict[last_file_name].append(class_name)

file_dict[last_file_name].append(match[0])

class_name won't be used anywhere again so we can append it directly.

anapaulagomes · 2021-01-08T10:28:38Z

pytest_picked/modes.py

+                    if not was_it_class:
+                        match = re.findall(test_regex, file_or_test)
+                        test_name = match[0]
+                        file_dict[last_file_name].append(class_name + "::" + test_name)


Suggested change

file_dict[last_file_name].append(class_name + "::" + test_name)

file_dict[last_file_name].append(class_name + "::" + match[0])

anapaulagomes · 2021-01-08T10:30:15Z

pytest_picked/modes.py

+
+                elif re.search(test_regex, file_or_test):
+                    if not was_it_class:
+                        match = re.findall(test_regex, file_or_test)


Sorry, I couldn't understand why the class validation is needed. Would care to explain it to me a bit more?

Here you're doing double regex parsing again.

anapaulagomes · 2021-01-08T10:34:04Z

pytest_picked/modes.py

+                        test_name = match[0]
+                        file_dict[last_file_name].append(class_name + "::" + test_name)
+            class_name = ""
+        for file_name in file_dict.keys():


Suggested change

for file_name in file_dict.keys():

for file_name, test_names in file_dict.items():

for test_name in test_names :

anapaulagomes · 2021-01-08T10:40:05Z

tests/test_modes.py

+            b"\n+++ b/tests/pytestpicked/test_modes5.py"
+            b"\n+    def invalid_test(self):"
+            b"\n+++ b/tests/pytestpicked/test_modes6.py"
+            b"\n@@ -191,29 +191,8 @@ class InvalidClass:"


All lines are about the things we want to extract. But it isn't how real diffs looks like. So would be nice to have some real diff here. Feel free to add a file in a directory in our test folder if you think it would be better.

anapaulagomes · 2023-07-18T06:35:46Z

Closing stale PR.

omkarkhatavkar force-pushed the dev branch 2 times, most recently from f7999ce to 10f768b Compare November 3, 2020 05:06

omkarkhatavkar force-pushed the dev branch from 10f768b to 951304c Compare November 3, 2020 09:47

omkarkhatavkar force-pushed the dev branch 3 times, most recently from 2b6f477 to e1eefca Compare November 18, 2020 18:53

omkarkhatavkar force-pushed the dev branch from e1eefca to bcb528c Compare November 19, 2020 17:08

added feature only fetch the tests

90587c1

omkarkhatavkar force-pushed the dev branch from bcb528c to 90587c1 Compare November 19, 2020 17:16

anapaulagomes requested changes Dec 9, 2020

View reviewed changes

anapaulagomes reviewed Dec 10, 2020

View reviewed changes

README.rst Show resolved Hide resolved

updated based on review comments, logic change

ef6acd3

omkarkhatavkar requested a review from anapaulagomes December 12, 2020 18:12

omkarkhatavkar force-pushed the dev branch from c4c4bce to 80d712c Compare December 12, 2020 18:31

resolved all conflicts

e02a45a

omkarkhatavkar force-pushed the dev branch from 80d712c to e02a45a Compare December 12, 2020 18:35

resolved conflicts from the changelog

5def377

anapaulagomes requested changes Jan 8, 2021

View reviewed changes

anapaulagomes closed this Jul 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added feature only fetch the tests #112

added feature only fetch the tests #112

omkarkhatavkar commented Nov 2, 2020 •

edited

Loading

omkarkhatavkar commented Nov 2, 2020

anapaulagomes commented Nov 3, 2020

anapaulagomes commented Nov 17, 2020

omkarkhatavkar commented Nov 17, 2020

anapaulagomes commented Nov 17, 2020

omkarkhatavkar commented Nov 18, 2020

omkarkhatavkar commented Nov 19, 2020

omkarkhatavkar commented Dec 5, 2020

anapaulagomes commented Dec 5, 2020

anapaulagomes left a comment

omkarkhatavkar commented Dec 12, 2020 •

edited

Loading

anapaulagomes commented Dec 17, 2020

omkarkhatavkar commented Dec 28, 2020

anapaulagomes left a comment

anapaulagomes Jan 8, 2021

anapaulagomes Jan 8, 2021

anapaulagomes Jan 8, 2021

anapaulagomes Jan 8, 2021

anapaulagomes Jan 8, 2021

anapaulagomes Jan 8, 2021

anapaulagomes Jan 8, 2021

anapaulagomes Jan 8, 2021

anapaulagomes Jan 8, 2021

anapaulagomes Jan 8, 2021

anapaulagomes Jan 8, 2021

anapaulagomes commented Jul 18, 2023

-        test_conventions["file"] = ["test_*.py", "*_test.py"]
+        test_conventions = {
+            "file": ["test_*.py", "*_test.py"]
+        }

	file_dict[last_file_name].append(class_name)
	file_dict[last_file_name].append(match[0])

	for file_name in file_dict.keys():
	for file_name, test_names in file_dict.items():
	for test_name in test_names :

added feature only fetch the tests #112

added feature only fetch the tests #112

Conversation

omkarkhatavkar commented Nov 2, 2020 • edited Loading

New Command ?

How to use this ?

If you wanted to run modified or newly added tests here is the example of that

With this change or PR we can easily run those tests instead whole file

omkarkhatavkar commented Nov 2, 2020

anapaulagomes commented Nov 3, 2020

anapaulagomes commented Nov 17, 2020

omkarkhatavkar commented Nov 17, 2020

anapaulagomes commented Nov 17, 2020

omkarkhatavkar commented Nov 18, 2020

omkarkhatavkar commented Nov 19, 2020

omkarkhatavkar commented Dec 5, 2020

anapaulagomes commented Dec 5, 2020

anapaulagomes left a comment

Choose a reason for hiding this comment

omkarkhatavkar commented Dec 12, 2020 • edited Loading

anapaulagomes commented Dec 17, 2020

omkarkhatavkar commented Dec 28, 2020

anapaulagomes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anapaulagomes commented Jul 18, 2023

omkarkhatavkar commented Nov 2, 2020 •

edited

Loading

omkarkhatavkar commented Dec 12, 2020 •

edited

Loading