[WIP] 🚀🚀🚀 Transformers.js V3 🚀🚀🚀

[xenova]

In preparation for Transformers.js v3, I'm compiling a list of issues/features which will be fixed/included in the release.

• WebGPU support (upgrade onnxruntime-web to 1.17.0).
  • closes WebGPU Support #20
  • closes [Feature request] WebGPU support #73
  • closes [Bug] image segmentation very slow #79
  • closes Whisper on webGPU? #100
  • closes [Question] A WebGPU-accelerated ONNX inference run-time #119
  • closes [Feature request] Update ONNX runtime to version 1.15.1 #298
  • closes GPU Acceleration to increase performance #377
  • closes [Feature request] Real time whisper transcription #405
  • closes How do I use WebGL as executionProvider? #505
  • closes YOLOS model extremely slow #533
• Fix logging/disable warnings (onnxruntime-web → 1.17.0). Closes:
  • closes [Question] How to stop warning log #270
  • closes env.backends.onnx.logLevel = "fatal"; does not work #529
• Fix WASM backend for large models (onnxruntime-web → 1.17.0). Closes:
  • closes An error occurred during model execution: "RangeError: offset is out of bounds". #499
• Deno support (upgrade sharp.js to 0.33.x). Closes:
  • closes [Feature request] Deno Support #78
  • closes Sharpe Linux-x86 #541
• CommonJS compatibility. Closes [Feature request] CommonJS compatability #152
• Skip local model check when running in-browser, unless explicitly set by user. This is an issue experienced by many beginners, where requests made to localhost redirect to error page, but dev server incorrectly returns status code 200. Closes
  • closes [Question] React.js serve sentence bert in browser keep reporting models not found. #328
  • closes getModelJSON fails with a bundler (see reproduction) #366
• Improve unit test suite and allow local testing. Closes Running tests locally fail #491
• Create model conversion space. Closes
  • closes [Visual Model Conversion] Do-It-Yourself: A visual tool that allows users to convert their own models without sweats and tears #222
  • closes [Question] How to convert to ONNX a fine-tuned model #274
• Upgrade conversion script dependency versions (+fixes sentence-transformers conversions). Closes
  • closes [Question] distiluse-base-multilingual-cased-v2 - wrong vector dimension (768 vs 512) in onnx version? #230
• Versioned documentation, so that users still on v2 will be able to access the correct documentation.
• Improve PretrainedModel, PretrainedTokenizer, and Processor types. In a similar way to how the pipeline API has conditional types, we'll add the same for the other classes accessible by users.
• Consistency issues:
  • topk -> top_k parameter.
  • Tensor transpose -> permute
  • Like the python library, encapsulate multimodal preprocessing in the AutoProcessor class, which uses image processor and tokenizer
• Improved model caching of revisions, and purge cache if model files are out-of-date. Ideally, this should be enabled by a setting to avoid unnecessary network calls to check versions if the files are present in cache.
• Improve pipeline fallback errors
  • closes Unsupported model type: whisper and CORS error #314
• WebNN support
  • reference: [DO NOT MERGE] Code example for testing ORT-Web WebNN EP #608

How to use WebGPU

First, install the development branch

npm install xenova/transformers.js#v3

Then specify the device parameter when loading the model. Here's example code to get started. Please note that this is still a WORK IN PROGRESS, so the following usage may change before release.

import { pipeline } from '@xenova/transformers';

// Create feature extraction pipeline
const extractor = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2', {
    device: 'webgpu',
    dtype: 'fp32', // or 'fp16'
});

// Generate embeddings
const sentences = ['That is a happy person', 'That is a very happy person'];
const output = await extractor(sentences, { pooling: 'mean', normalize: true });
console.log(output.tolist());

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Huguet57]

Hey! This is great. Is this already in alpha?

[kishorekaruppusamy]

Team, is there any tentative time to release this v3 alpha ???

[jhpassion0621]

I can't wait anymore :) Please update me when it will be released!

[NawarA]

@xenova it looks like #596 is part of this release?! I think that means onnx_data files will be supported?

If true, I'm stoked!

Beyond upgrading ort to 1.17, are there other changes needed to support models with onnx_data files? Happy to try to lend a hand if possible

[xenova]

Hi everyone! Today we released our first WebGPU x Transformers.js demo: The WebGPU Embedding Benchmark (online demo). If you'd like to help with testing, please run the benchmark and share your results! Thanks!

[khmyznikov]

@xenova can this bench pick the GPU 1 instead of 0? For the laptops with dGPU

[xenova]

@xenova can this bench pick the GPU 1 instead of 0? For the laptops with dGPU

Not currently, but this is being worked on here: microsoft/onnxruntime#19857. We will add support here once ready.

[xenova]

@beaufortfrancois - I've added the source code for the video background removal demo. On my device, I get ~20fps w/ WebGPU support (w/ fp32 since fp16 is broken). Here's a screen recording (which drops my fps to ~14):

webgpu-modnet.mp4

• Model used: https://huggingface.co/Xenova/modnet (~4 years old, and it clearly struggles on hands moving quickly). I will try on more up-to-date models soon.
• Video tested: https://www.youtube.com/watch?v=NXpdyAWLDas
• Online demo: https://huggingface.co/spaces/Xenova/webgpu-video-background-removal

[beaufortfrancois]

@beaufortfrancois - I've added the source code for the video background removal demo. On my device, I get ~20fps w/ WebGPU support (w/ fp32 since fp16 is broken). Here's a screen recording (which drops my fps to ~14):

You rock. Thanks! It's a cool demo! 👍

I've been wondering how we could improve it:

• I've noticed you read the current frame of the video on the main thread. Would it help to move the entire demo to a web worker?
• output[0].mul(255).to('uint8') takes some non negligible time to run. Is there a faster path?
• How much you expect fp16 to improve perf? In https://developer.chrome.com/blog/new-in-webgpu-120#support_for_16-bit_floating-point_values_in_wgsl, we've noticed on an Apple M1 Pro device that the f16 implementation of Llama2 7B models used in the WebLLM chat demo is significantly faster than the f32 implementation, with a 28% improvement in prefill speed and a 41% improvement in decoding speed.
• A way to feed a GPUExternalTexture to the model as an input could also come handy.

[beaufortfrancois]

Out of curiosity, what is 'gpu'?

[xenova]

It's meant to be a "catch-all" for the different ways that the library can be used with GPU support (not just in the browser with WebGPU). The idea is that it will simplify documentation, as transformers.js will select the best execution provider depending on the environment. For example, DML/CUDA support in onnxruntime-node (see microsoft/onnxruntime#16050 (comment))

Of course, this is still a work in progress, so it can definitely change!