Road to 100% Code Coverage

[TsayAdobe]

Unit testing stands as the primary line of defense against defective code. Achieving 100% code coverage is akin to setting up a robust safety net, ensuring that every line of the code is scrutinized and verified.

DC team started this journey with the Web Test Runner (WTR) framework based on Milo team’s recommendation. A standout feature of WTR is its ability to run code within an actual browser environment, providing a realistic testing ground that closely mimics real-world usage.

However, this approach is not without its limitations. One significant challenge is the difficulty in mocking certain environments and conditions within the browser. This limitation can hinder our ability to comprehensively test all lines of the code.

Further complicating matters is the emphasis on performance optimization. Our functions are meticulously crafted for peak performance, but this often comes at the expense of testability. The intricate structure that boosts efficiency can also create formidable obstacles in achieving thorough test coverage.

To navigate these challenges, a solution was proposed. Jest is the most popular framework for this purpose. Jest offers a versatile environment for JavaScript testing, allowing us to simulate browser-like conditions without the limitations we encountered with WTR.
By leveraging both WTR and Jest, we have crafted a more robust testing infrastructure. This dual approach enables us to enjoy the benefits of real browser testing while also utilizing the flexibility and control provided by Jest.

This journey is not just about reaching a numerical target; it's about fostering a culture of quality and reliability in our software development process.

Steps for setting up dual unit test frameworks

1. Add jest and jest-environment-jsdom to your dependencies.

2. Add Jest configuration to package.json. With the configuration, Jest and WTR tests co-exist in the /test folder. Jest test files have the .jest.js extension and WTR .test.js.

  "jest": {
    "testRegex": "\\.jest\\.(js|jsx)$",
    "collectCoverageFrom": [
      "acrobat/**/*.js"
    ],
    "coverageProvider": "v8"
  }

3. Make two sets of npm run commands: wtr and jest. Change npm test to run the two commands.

  "scripts": {
    ...
    "wtr": "wtr \"./test/**/*.test.(js|html)\" --node-resolve --port=2000 --coverage --concurrent-browsers 4",
    "wtr:watch": "npm run wtr -- --watch",
    "jest": "jest --coverage --coverageDirectory=coverage/jest",
    "jest:watch": "npm run jest -- --watchAll",
    "test": "npm run wtr && npm run jest",
    ...
  }

4. Update the GitHub Actions workflow for unit tests. Upload both coverage reports to Codecov, which will show a combined report.

    - name: Upload coverage to Codecov
      uses: codecov/codecov-action@v3
      with:
        token: ${{ secrets.CODECOV_TOKEN }}
        files: coverage/wtr/lcov.info,coverage/jest/lcov.info

5. (Optional) To see the combined coverage locally, Use the following npm run command.

    "lcov": "lcov -a coverage/jest/lcov.info -a coverage/wtr/lcov.info -o coverage/lcov.info",

[mokimo]

I'd love to hear what actual limitations you are running into that would be solvable with jest? @JasonHowellSlavin had a similar question in the previous thread:

That makes sense. @TsayAdobe maybe we could look at the code which led you to want to use JSDOM?

I agree with @auniverseaway point in the previous thread that this increases complexity & setup without a clear benefit. Or at least the problems jest solves are not clear to me.

@JasonHowellSlavin @TsayAdobe to be crystal clear: we (I) do not want two unit test frameworks. We are trying to reduce how much or engineers have to learn, not make it bigger. Having two ways to do the same thing adds a level of training and complexity we try to avoid at all costs.

As you've mentioned not being able to easily mock window.location is the only limitation I've personally ran into so far - but with this alone you'd still be able to reach >90% test coverage. Most of the currently tested milo blocks have 100% using WTR if you check codecov & the only reason some aren't, are some hard to test codepaths - they would be equally hard to test with jest though.
On top of that, we still have nala which in theory should be able to catch use-cases that can't (and likely should not) be covered by unit tests

Awesome to see that coverage on DC though! Great job 🚀 🎉