Allow for random spec ordering when running tests

[joelzimmer]

Current behavior:

When running specs in cypress, the files run in alphabetical order, and tests within each spec run in order. See #2901.

Desired behavior:

It'd be great to pass a flag into cypress (or for it to do some sort of optimization like Jest does) to randomize the order of test runs. This would allow tests that rely on other state to fail and could help increase confidence in how the app works.

Versions

3.1.3, Alpine Linux, Electron

[Strajk]

Just noticed there is a library for randomizing tests in Mocha by the almighty @bahmutov, would be great to integrate it to Cypress :)

https://github.com/bahmutov/rocha

[bahmutov]

One thought: it would be nice to have a callback from Cypress into userspace code with the list of specs and tests in each. The user code then can return the list of tests - in any desired order. I kind of do this in https://github.com/bahmutov/cypress-select-tests projects where I get a callback of all tests found in the spec file and can remove all tests I don't want to run

…

On Tue, Apr 16, 2019 at 11:20 AM Strajk ***@***.***> wrote: Just noticed there is a library for randomizing tests in Mocha by the almighty @bahmutov <https://github.com/bahmutov>, would be great to integrate it to Cypress :) https://github.com/bahmutov/rocha — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2908 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACHApsM-jXyR5MgkhhuybvFWTHANHjBJks5vhepFgaJpZM4ZHKIm> .

-- Dr. Gleb Bahmutov, PhD Schedule video chat / phone call / meeting with me via https://calendly.com/bahmutov [email protected] @bahmutov <https://twitter.com/@bahmutov> https://glebbahmutov.com/ https://glebbahmutov.com/blog https://github.com/bahmutov

[terencechow]

+1

If you share where the location is in this repo that grabs spec files I can take a stab at this. Would it be enough to:

1. randomize the spec files order
2. use https://github.com/bahmutov/cypress-select-tests to randomize the tests within the spec files?

Or is there another reason that randomizing spec file order might break things? (Does something depend on spec file ordering?)

[mncharlton]

I've had a go at randomising the order of the tests (it blocks) within a spec, it's not perfect yet, but is a start - https://www.npmjs.com/package/cypress-random-test-order

[jennifer-shehane]

@mncharlton Could you open a pull request to add this plugin to our documentation?

Instructions here as well as a PR Template Checklist. Thanks!

[mncharlton]

Thanks for the prompt @jennifer-shehane! PR is here: cypress-io/cypress-documentation#3024

[duraz0rz]

Any other movement on this? Seems like the cypress-random-test-order plugin is deprecated by the owner as they don't have time to work on issues.

[AlexandreRozier]

Hi, we'd also love to have randomized test execution (at least at spec file level), is there an alternative to brewing our own execution ordering ( - based on @bahmutov's plugin for instance) ? Cheers :)

EDIT : I achieved random test execution using the config.specPattern property : #390 , this is heavily inspired from @bahmutov's @cypress/grep plugin.
This is very hacky but works for the time being...

// cypress/plugins/randomizeSpecs.ts
import * as fs from 'node:fs/promises';
/**
 * @param {Cypress.ConfigOptions} config
 */
export default async function randomizeSpecsPlugin(config) {
  if (!config?.env?.randomizedTestsSeed) {
    return config;
  }

  const specPattern = /.*.cy.ts/;
  const integrationFolder = process.cwd();

  const allFiles = await fs.readdir(integrationFolder, { recursive: true });
  const specFiles = allFiles.filter(s => specPattern.exec(s)).map(s => 'cypress/' + s);
 
  // Run your  custom shuffling function here - for me it's modifying specFiles in-place 
  shuffle(specFiles);

  // Warning : we're overriding existing specPattern to ensure tests are run with this new order
  // Any existing specPattern will be OVERWRITTEN
  // See https://github.com/cypress-io/cypress/issues/390  as to why we're doing things this way
  config.specPattern = specFiles;
  return config;
}

// cypress/cypress.local.config.ts

import randomizeSpecsPlugin from './plugins/randomizeSpecs';
...
async setupNodeEvents(on, config) {
    
      await randomizeSpecsPlugin(config);
      return config
}