Add Test command

[AngelMunoz]

Is your feature request related to a problem? Please describe.
One thing perla doesn't have right now is a way to create/run tests, normally this would be complicated and messy but with the help of web test runner we can re-use those libraries to enable in-browser tests, which are the best kind since our code and tests would be running in the same environment it is going to run.

Describe the solution you'd like
As usual, the Perla CLI has to be quite simple and should be convention based

perla test

perla test --files ./tests/file-a.test.js ./tests/feature/file-b.test.js

perla test --groups featureA featureB

perla test --browsers edge chrome firefox

Arguments I have considered right now are:

• files - a list of individual files to test

• Groups - all the files under a specific path inside the testing directory

  Asuming tests as the testing directory, example:
  perla --groups featureA featureB featureC/subFeatureA

  testing
     featureA // grabs all inside here
       **/*.test.js
     featureB // grabs all inside here
       **/*.test.js
     featureC
       subFeatureA // only grabs everything inside here
         **/*.test.js
       feature-c1.js // ignored
       subFeatureB // ignored
  

• Browsers - a list of specific browsers to test with

• All_Browsers - test in every browser available

  • Chromium
  • Chrome
  • Edge
  • Firefox
  • Webkit

Describe alternatives you've considered
I have not seen any alternatives for web test runner servers in F#/.NET

Additional context
I already have a PoC in a project called Nacre as a standalone project I think it could also take off, not sure if it's worth getting this inside perla (as we did with the scaffolding features: Clam -> Perla)

Technical quirks
Inside Nacre we had to use import maps to mock the file that calls into the test runner's script that handles socket connections to the server, we can use/re-use the Http/SSE mechanisms we're using inside Perla instead of sockets.

Inside Nacre we had to use import maps to mock the file that...

Following this point, we could also use import maps to let the user mock any other files and dependencies, this requires some way to specify in the mocks (either with code or a configuration property)

Considerations

• Support for files other than .js

  Since these tests run on the browser they must be served somehow, so we should be able to re-use existing code inside perla like ASP.NET middleware and esbuild on-the fly compilation.

• Fable projects

  Fable produces JS by default so, we're covered out of the box here we just need to run fable first to have the .test.js files in place, we might be able to use Fable.Expect out of the box but submitting a few PR's could also get us there in case there are some issues

• Another field in the JSON configuration file

  As with other configuration properties, we will provide sensible defaults out of the box, but we should be able to expose at least a few options like browsers, testing directory, test all and similar options

• Run After/Before build

  we can add a flag that allows to run tests before we can actually run the build command, not sure if this is actually a good thing, the users can always call the commands as required in their scripts we just need to be sure to not return 0 if a test fails

• Testing Dependencies
  We might to also maintain a separate set of dependencies for development, for example inside the tests of the sample project we are importing 3 dependencies:

  • @web/test-runner-mocha
  • @esm-bundle/chai
  • @open-wc/testing-helpers

  While we could just hardcode/provide these out of the box (which could be simpler) users might have other test only dependencies that we could merge with normal dependencies

Disadvantages

• playwright must be set up outside of perla, which I guess it's fine if you're a .NET dev there's a global tool, if you're a python dev you can use pip, for node there's an npx command.

• CI systems must be able to run headless browsers otherwise testing in CI won't be possible. Fortunately it seems playwright has good docs we can point to and can run on major CI providers and also provides docker images

Non Goals

• Full testing suite with coverage reports
• Complexity Analysis

[AngelMunoz]

I wouldn't mind some thoughts if you're able to check this out @nojaf, @kaeedo 🙏🏼

[nojaf]

Hello, it has been a while since I've done anything with unit tests in JavaScript.
I've never worked with @web/test-runner, so I can't really say anything clever.
If you use @web/test-runner can you combine this with any test framework of the day?
Is Jest still cool these days?

I have been thinking about checking out PlayWright and just staying in .NET land to run some UI tests.
But that doesn't really give an answer to run plain JS unit tests I guess.

[AngelMunoz]

The gist of @web/test-runner is basically setting up some basic infrastructure to run tests in an environment browser the npm package brings more to the table as how to run playwright and other few set of settings you can configure for it.

In our case, using @web/test-runner + playwright just makes it easier to add tests since we don't have to write any code for setting up tests, teardown, run them and such
We can still use things like mocha/jasmine assuming they provide ways to integrate with it. The advantage of this approach is that your tests either UI or Unit ones run in the browser in the environment they are going to run on production as well

Is Jest still cool these days?

No Idea haha, I think Jest is node only and runs tests in node rather than in the browser

I have been thinking about checking out PlayWright and just staying in .NET land to run some UI tests.
But that doesn't really give an answer to run plain JS unit tests I guess.

That's actually what the PoC does https://github.com/AngelMunoz/Nacre it simply runs in headless mode, For fable users tests could look like this https://github.com/fable-compiler/Fable.Lit/blob/main/test/LitElementTest.fs#L91 and at that point one would be able to also code unit tests besides UI tests

[kaeedo]

Is Jest still cool these days

I believe Jest can run tests in either node or the browser, but the bigger problem is that it doesn't support ES modules at all. So another tool is required to transpile es modules down to commonJS modules.

Other than that, Fable.Lit uses @web/test-runner for its testing. https://fable.io/Fable.Lit/docs/testing.html. This could be a good place to get a feel for "modern" fable testing

@web/test-runner also relies on the same import-map feature that Perla does, so I think it would be nice since both runtime and testing (test-time?) use the same way of importing dependencies. Additionally, it runs tests inside a headless browser instead of node, which fits with perla's usage without needing node installed at all

[DejanMilicic]

I am attempting to customize the existing Fable.Mocha tests to run with Perla.
What I have is a Fake script with "test" target.
The part I am missing here is ability to point Perla to serve files from a specific folder and for this to be available as a command line parameter.

So, something like perla serve -source ... would do the job.
Unfortunately, if I am not wrong, the only way to specify source folder for file is via devServer section of perla.jsonc

Is there any way to circumvent this? Ability to pass source folder via command line would do.

[AngelMunoz]

I am currently working on this in the #89 branch
the current perla implementation isn't flexible enough to implement a test runner

The PoC is done there but there are a few missing pieces

• Ensure live reload is working for tests as well source code
• Ensure parallel browser testing works and doesn't overlap (E.g. running unit tests in firefox/chromium/webkit at the same time if desired by the user)
• Enable Fable projects for tests (i.e. using a fable project for testing) as it only works with js files in the PoC
• Expose CLI/Options for testing

[AngelMunoz]

I tried to get preview builds a couple of weeks ago, but I never got to it.

No promises but maybe in a couple of weeks I can finally get it in a well enough state to start making a few preview releases for vnext