Lighthouse on non-docker sitespeed

[yashgarde]

Your question

I am currently in the process of deploying the Onlinetest product within sitespeed and we are not using the dockerized version of the sitespeed.io tool to run the tests from the testrunner. We are directly running the sitespeed.io command after installing the npm package globally on the server and running it through that.

My question is whether there is a way to have Lighthouse metrics be generated when using sitespeed.io in this way (not using the -plus1 docker image) and if so, are there steps anywhere for this as well?

[soulgalore]

Hi @yashgarde yes you should install https://github.com/sitespeedio/plugin-lighthouse on the server globally (npm install @sitespeed.io/plugin-lighthouse -g). Then you need to enable Lighthouse in your yaml file for the server in the HTML block:
useLighthouse: true

Please let me know if it works or if something is missing. I can update the documentation tomorrow so its included.

[yashgarde]

I tried this on my local deployment for testing purposes and it does not seem to work. I have installed the lighthouse plugin globally and changed that config in the onlinetest/server/config/default.yaml file to say useLighthouse: true.

Do I need to modify anything on the testrunner side? Are there any sitespeed configs I need to add in the JSON config file?

[soulgalore]

Server: https://github.com/sitespeedio/onlinetest/blob/main/server/config/default.yaml#L68

Then before you start your test you need to enable Lighthouse:

That will pass on information to the test runner to add/run Lighthouse plugin.

[yashgarde]

Unfortunately this didn't work either. Earlier I was triggering tests using the API so I thought perhaps the GUI checkbox was needed to trigger lighthouse but this didn't seem to change anything either.

Any other suggestions?

[soulgalore]

Hmm when I just tried it works for me?

1. In the server configuration, set useLighthouse: true
2. On the test runner server install the plugin npm install @sitespeed.io/plugin-lighthouse -g
3. Add a new test through the GUI, make sure to chose Lighthouse in the extra tab.
4. When the test runs I get (and then the test runs):

[2024-07-20 15:05:50] INFO: Versions OS: darwin 23.5.0 nodejs: v20.9.0 sitespeed.io: 34.7.0 browsertime: 22.6.0 coach: 8.0.2
[2024-07-20 15:05:52] INFO: Using default Lighthouse configuration for desktop
[2024-07-20 15:05:52] INFO: Will run Lighthouse tests after Browsertime has finished

Is those the steps you tried? Do you get the change in the GUI but no Lighthouse tests run?

[yashgarde]

This is exactly what I did, I'll give it another shot today and follow up here. The Lighthouse checkbox does appear and I selected it but I did not see the Lighthouse tab in the results.

Additionally, is there a way to trigger lighthouse from the API side of things without the UI element?

[soulgalore]

If you check the log for the test runner do you see anything there? I think you could add --useLighthouse true but the thing is if it do not work, it must be something on the server. Have you enabled/disabled other plugins?

[yashgarde]

I have gone through a few scenarios regarding Lighthouse, here are the findings:

1. When executing from the Onlinetest GUI, testing a single URL and using the Lighthouse checkbox: This works as expected and there are Lighthouse results in the HTML results
2. When executing from the Onlinetest GUI, using a script (that accesses the same page as the single URL example) and using the Lighthouse checkbox: This does not work and there are no Lighthouse results generated in the final results
3. When executing using the Onlinetest API and adding the --useLighthouse true option as recommended, both a single URL test and a script based test do not generate the Lighthouse results.

NOTE: When I am running the API tests, instead of using the CLI API, I am making use of REST requests as this is the final use case for our application of sitespeed.io. All other command line options (ex: --viewPort, -n, --slug etc.) all work as expected but the --useLighthouse flag does not. I wanted to make this distinction as this is not a code path or method of test execution that is necessarily covered in the documentation but has worked for the most part in my testing.

[soulgalore]

Ah ok, I see, I didn't first get that it was user scenarios that do not work. Lighthouse does not work for user scenarios, only for single URLs. The reason is that Lighthouse use their own Chrome instance instead of the Browsertime version (maybe there could be a way to work around that) and Lighthouse use the Chromium only Puppeteer to drive the browser instead of the W3C standard webdriver.

[yashgarde]

Ah I see, thanks for the information and the update. Will close this issue for now then!