perf: use swc or esbuild toolchain to transpile code locally

[varunsrin]

What is the feature you would like to implement?

All local actions (yarn start, yarn test) should use a fast transpiler like esbuild or swc, and remote CI actions should compile the code at most once before running tests.

Why is this feature important?

Faster build times makes it easier to work with the Hub codebase

Will the protocol spec need to be updated??

How should this feature be built? (optional)

Currently, we use tsx for yarn start which uses esbuild under the hood, and nothing special for yarn test. In CI, we run yarn tsc and then yarn test with some CI flags. Ideally, we'd do a full typecheck in CI but when running locally we'd just do a fast transpile and execute. We may also be double compiling in CI adding to our run times, needs investigation.

Additional context

There is some complexity due to ESM where additional configuration is needed to get certain packages to work. For example, swc is a drop in replacement in jest, but esbuild requires some fiddling with configurations. Similarly, tsx was the only drop in replacement for yarn start that actually worked.

[deodad]

Looked into this, here's my recommendation:

• setup swc to replace ts-jest
• leave tsx for running dev server

This solution achieves the best performance w/ minimal configuration.

Ideally we wouldn't need two different transpilers but going with either one for both test and start involves adding undesirable configuration complexity and slight performance warts:

• swc is easy to configure w/ jest and ts-node (ts-node is used by jest to transpile `jest.config.ts) but setting up a dev server requires a workaround that involves maintaining a custom node loader until this issue regarding esm + path aliases is resolved
• esbuild is already working via tsx for start, but I wasn't able to get jest transpiling the project w/ esbuild using any of the existing transformers. getting it working for this project would likely require writing a custom transformer. it also doesn't integrate w/ ts-node, so there's the tiny wart of taking 1+ second just to load jest.config.ts

lmk if this sounds good and I'll open up the PR

[varunsrin]

How much faster does this hybrid approach make us when running tests from a cold boot locally? If the savings are meaningful, I'm supportive otherwise I don't think the additional complexity of having multiple build setups is worthwhile.

I would also document this rationale in the code in the config files or somewhere obvious, so that it's clear to future maintainers why we chose this hybrid approach. You can just link to this ticket or copy paste the info there if its short enough.

[deodad]

yarn test goes from ~43s to ~20s on my machine
yarn test ./src/utils/crypto.test.ts from ~5.8s to ~2.4s

I opened up a PR to see how it performed in GH Actions:
shard 1/2 went from averaging ~45s to ~40s
shard 2/2 went from averaging ~70s to ~53s

Github hosted runners are only 2-core by default so the slowest tests end up being a bigger factor.

[varunsrin]

Let’s do it then, the speed up is worth it

…

On Tue, Dec 27, 2022 at 9:00 AM adaddeo ***@***.***> wrote: yarn test goes from ~43s to ~20s on my machine yarn test ./src/utils/crypto.test.ts from ~5.8s to ~2.4s I opened up a PR <#330> to see how it performed in GH Actions: shard 1/2 went from averaging ~45s to ~40s shard 2/2 went from averaging ~70s to ~53s Github hosted runners are only 2-core by default so the slowest tests end up being a bigger factor. — Reply to this email directly, view it on GitHub <#314 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEBD4RGLOKW3IWGB6WI5W3WPMODTANCNFSM6AAAAAATF2POV4> . You are receiving this because you authored the thread.Message ID: ***@***.***>