Calculation of parallel_tests chunks does not take into account CLI submitted args

[luke-hill]

When running a CLI command where you pass in a profile, i.e. lets say the profile omits all tests that have the @foo tag.

When you get to the calculation phase for how to "chunk" the tests up, those features with the tag are included in the calculation. They won't be executed, but it means the chunking can be very uneven.

This may need fixing downstream @enkessler

[enkessler]

@luke-hill Do you have an example? Having one would help determine where in the stack the problem is.

[grosser]

this is tricky because parallel_tests tries to avoid going too deep into the test-framework
it would require parsing the profile and understanding what it means and then deselecting files based on that
it's doable, but might be brittle and will always have to be updated in lock-step with the test-framwork
... so not fun to write/maintain

[luke-hill]

bundle exec cucumber --format ParallelTests::Gherkin::RuntimeLogger --out tmp/parallel_runtime_cucumber.log -p reports features/advice/foo/bar.feature features/advice/bar/baz.feature features/advice/bay/abc.feature  --retry 2
bundle exec cucumber --format ParallelTests::Gherkin::RuntimeLogger --out tmp/parallel_runtime_cucumber.log -p reports features/advice/bar/abc.feature features/advice/baz/def.feature features/advice/foo/zyx.feature --retry 2
bundle exec cucumber --format ParallelTests::Gherkin::RuntimeLogger --out tmp/parallel_runtime_cucumber.log -p reports features/advice/baz/prq.feature features/advice/baz/tuv.feature features/advice/bar/ghi.feature features/advice/foo/foo.feature features/business_rules/baz.feature features/business_rules/abc.feature --retry 2
bundle exec cucumber --format ParallelTests::Gherkin::RuntimeLogger --out tmp/parallel_runtime_cucumber.log -p reports features/advice/baz/123.feature features/advice/baz/456.feature features/business_rules/ghi.feature features/business_rules/xyz.feature --retry 2

Every single feature inside features/advice has scenarios. Every single feature inside business rules has 0 scenarios and/or is tagged as @future_release and will be excluded

[enkessler]

Oh, this is a feature request? Sorry, my mind was in Bug Mode when I was reading the initial issue.

As far as the "it's difficult but it could be done" angle goes, I can say that the cuke_modeler family of gems won't be able to provide any additional capability that would help figure out what to do with a Cucumber profile. Their scope ends at the .feature file level.

Regarding the necessity of implementing something like this, it strikes me as a bit self inflicted if a user is both requesting a profile that will filter out certain features/tests while also requesting that those things are run. It might end up being a lot of work to implement this kind of thing just to squeeze a bit more performance out of a large test suite when the problem could be avoided by having better test organization/planning by the user if they really need every last drop of speed.

[luke-hill]

I'm not requesting they are run. So normally (Without parallelism), we say run all features except these features.

When we run in parallel, we don't get the chance to say exclude these features at any point. So at the point the space delimited list of features comes in, we don't have any chance to filter. Maybe if we had a manual way to filter that would help.

i.e. if we could run parallel cucumber only on one subdirectory/subdirectories @grosser ?

[grosser]

you can run parallel_cucumber folder_a folder_b if that helps ... or use the --pattern for a regex approach

[luke-hill]

I'll try this, maybe this is functionality that isn't needed.