Add --single-process-tag option to run specific Gherkin scenarios sequentially #810

andrii-rymar · 2021-03-30T12:50:38Z

This PR adds new --single-process-tag option to run all Gherkin scenarios marked by some tag within single process.
It's aimed to deal with the situation when we have some scenarios sharing the same data thus required to run sequentially.

Currently to achieve that we have two options:

move all that cases into a separate file, use --exclude-pattern and run them outside parallel_tests
tag that cases somehow, exclude that tag from run with -t Cucumber option (-t "@tag_to_run and not @tag_not_to_run") and run them separately as well.
Both ways require either changes to features which can be undesirable, for example if features are used as a documentation and need to have a specific structure, or to execution logic to have multiple runs performed sequentially instead of one.

Proposed change simplify this process down to one change required - adding a specific tag to all scenarios needed to be run sequentially. Also it's possible to combine it with --isolate to have a single process only for selected cases.

As for the code - it includes some workarounds to store scenario tags both with scenario location in Grouper. Probably it would be helpful in the future to add other scenario-related features.

Would like to know your opinion whether or not we can proceed with that as these changes are pretty major.

Checklist

Feature branch is up-to-date with master (if not - rebase it).
Added tests.
Added an entry to the Changelog if the new
code introduces user-observable changes.
Update Readme.md when cli options are changed

grosser · 2021-04-08T15:36:12Z

lib/parallel_tests/cli.rb

@@ -197,7 +197,11 @@ def parse_options!(argv)
          (options[:single_process] ||= []) << /#{pattern}/
        end

-        opts.on("-i", "--isolate", "Do not run any other tests in the group used by --single(-s)") do
+        opts.on("--single-process-tag [TAG]", "Run all scenarios with specified tag in the same process") do |tag|


Suggested change

opts.on("--single-process-tag [TAG]", "Run all scenarios with specified tag in the same process") do |tag|

opts.on("--single-tag [TAG]", "Run all scenarios with specified tag in the same process") do |tag|

this would be more consistent ?

also say cucumber/gherkin or so since scenarios makes no sense for rspec/test

grosser

this is a refactor and a new feature ... hard to know what's going on and if all edge-cases are covered :(

I'd prefer if we do:

@scenarios << [[uri, feature_element.source_line].join(":"), scenario_tags]

and keep the old structure so it's clear what is new
... or do the refactor first without new behavior

…arios sequentially

grosser · 2021-04-14T16:05:19Z

lib/parallel_tests/grouper.rb

@@ -9,20 +11,16 @@ def by_steps(tests, num_groups, options)

      def by_scenarios(tests, num_groups, options = {})
        scenarios = group_by_scenarios(tests, options)
-        in_even_groups_by_size(scenarios, num_groups)
+        in_even_groups_by_size(scenarios, num_groups, options.slice(*BY_SCENARIOS_SUPPORTED_OPTIONS))


could just pass in all options and let it pick what it wants

grosser · 2021-04-14T16:05:31Z

lib/parallel_tests/grouper.rb

-          single_process_patterns.any? { |pattern| item =~ pattern }
-        end
+        # add all files/scenarios that should run in a single process to one group
+        single_items, items = separate_single_items(items, options)


nice refactor!

grosser · 2021-04-14T16:06:19Z

lib/parallel_tests/grouper.rb

+          item = item_with_tags?(item) || item_with_size?(item) ? item[0] : item
+          options[:single_process].any? { |pattern| item =~ pattern }
+        elsif options[:single_process_tag]
+          raise "--single-tag option can only be used with '--group-by scenarios'" unless item_with_tags?(item)


put the raise into the option parser so it fails early ?

grosser · 2021-04-14T16:07:17Z

lib/parallel_tests/grouper.rb

+        items.partition { |item| to_single_items?(item, options) }
+      end
+
+      def to_single_items?(item, options)


Suggested change

def to_single_items?(item, options)

def single_item?(item, options)

this whole "item is item or item and size or item and tags" is kinda funky ... if you have any good ideas how to clean that up lmk or leave a TODO comment on how that can be improved

grosser · 2021-04-14T16:10:04Z

lib/parallel_tests/grouper.rb

@@ -138,7 +153,30 @@ def group_features_by_size(items, groups_to_fill)
      end

      def items_to_group(items)


some rough comment on what this method is used/does for would be nice

grosser · 2021-04-14T16:15:12Z

lib/parallel_tests/grouper.rb

+
+      def to_single_items?(item, options)
+        if options[:single_process]
+          item = item_with_tags?(item) || item_with_size?(item) ? item[0] : item


Suggested change

item = item_with_tags?(item) || item_with_size?(item) ? item[0] : item

item = item[0] if item_with_tags?(item) || item_with_size?(item)

... maybe just if item.is_a?(Array) ... but not sure

grosser

needs some tests that use this feature so I don't break it down the line :)

andrii-rymar · 2021-04-15T07:00:04Z

@grosser thanks for the comments
will polish it and add tests soon

grosser reviewed Apr 8, 2021

View reviewed changes

Add --single-process-tag option to run specific Cucumber/Gherkin scen…

a5a01bd

…arios sequentially

andrii-rymar force-pushed the single-process-tag branch from 5aa32e4 to a5a01bd Compare April 10, 2021 18:22

Add --single-process-tag option to run specific Cucumber/Gherkin scen…

d4bc8e7

…arios sequentially

andrii-rymar force-pushed the single-process-tag branch from 339e4ed to d4bc8e7 Compare April 14, 2021 10:53

grosser reviewed Apr 14, 2021

View reviewed changes

Merge branch 'grosser:master' into single-process-tag

d672b32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add --single-process-tag option to run specific Gherkin scenarios sequentially #810

Add --single-process-tag option to run specific Gherkin scenarios sequentially #810

andrii-rymar commented Mar 30, 2021 •

edited

Loading

grosser Apr 8, 2021

grosser Apr 8, 2021

grosser Apr 8, 2021

grosser left a comment

grosser Apr 14, 2021

grosser Apr 14, 2021

grosser Apr 14, 2021

grosser Apr 14, 2021

grosser Apr 14, 2021

grosser Apr 14, 2021

grosser Apr 14, 2021

grosser Apr 14, 2021

grosser left a comment

andrii-rymar commented Apr 15, 2021

	opts.on("--single-process-tag [TAG]", "Run all scenarios with specified tag in the same process") do \|tag\|
	opts.on("--single-tag [TAG]", "Run all scenarios with specified tag in the same process") do \|tag\|

	def to_single_items?(item, options)
	def single_item?(item, options)

		@@ -138,7 +153,30 @@ def group_features_by_size(items, groups_to_fill)
		end

		def items_to_group(items)

	item = item_with_tags?(item) \|\| item_with_size?(item) ? item[0] : item
	item = item[0] if item_with_tags?(item) \|\| item_with_size?(item)

Add --single-process-tag option to run specific Gherkin scenarios sequentially #810

Are you sure you want to change the base?

Add --single-process-tag option to run specific Gherkin scenarios sequentially #810

Conversation

andrii-rymar commented Mar 30, 2021 • edited Loading

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grosser left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grosser left a comment

Choose a reason for hiding this comment

andrii-rymar commented Apr 15, 2021

andrii-rymar commented Mar 30, 2021 •

edited

Loading