[Feature request] Deno Support #78

omar2205 · 2023-04-08T18:25:20Z

Name of the feature
Support running in the Deno runtime.

Additional context
I tried all the tricks, esm.sh, unpkg, esm.run, and npm:@xenova/transformers, but nothing worked.

Using unpkg import I get

This browser lacks typed array (Uint8Array) support which is required 
by `buffer` v5.x. Use `buffer` v4.x if you require old browser support.

Using esm.sh

Uncaught SyntaxError: The requested module '/v114/[email protected]/deno/pngjs.mjs' 
does not provide an export named 'PNG'

The text was updated successfully, but these errors were encountered:

xenova · 2023-04-08T18:33:50Z

This seems to be an problem with onnxruntime itself: microsoft/onnxruntime#10913

The person who opened the issue did seem to find a workaround, so, maybe that will help too?

@josephrocca is quite active in the community, and since he opened up that issue, he might have some more insight.

omar2205 · 2023-04-08T19:25:29Z

So @nestarz created a Deno onnx_runtime here https://github.com/nestarz/onnx_runtime

xenova · 2023-04-08T20:08:39Z

So @nestarz created a Deno onnx_runtime here https://github.com/nestarz/onnx_runtime

That's good to know - thanks! Have you tried overriding the dependency in your project?

omar2205 · 2023-04-08T20:12:56Z

Not sure how. For example, this is my script file

  import * as t from 'npm:@xenova/transformers'
  
  const pipeline = t.default.pipeline
  
  let pipe = await pipeline('sentiment-analysis')
  
  let out = await pipe('I love transformers!')

This gets a new error, error: Uncaught Error: Not implemented: Worker from onnxruntime-web/1.14.0/dist/ort-web.node.js

xenova · 2023-04-08T20:25:18Z

Maybe this might help: https://www.reddit.com/r/Deno/comments/x9q9vp/how_do_you_patch_a_module_like_patchpackage_in/ ? I haven't used Deno before, so, I probably won't be able to provide much more help (other than googling or "chatgpt-ing" 🤣 )

xenova · 2023-05-15T17:37:10Z

Have you been able to get it working in Deno yet? If not, feel free to try with the latest version 2.0.0-alpha.0. This may fix it.

omar2205 · 2023-05-15T20:10:52Z

Hey, I just tried it, using npm: and esm.sh and both got an error.
esm.sh:

Uncaught Error: Dynamic require of "../bin/napi-v3/linux/x64/onnxruntime_binding.node" is not supported

npm:

Uncaught Error:                                                                 
Something went wrong installing the "sharp" module                              
                                                                                
Cannot find module '../build/Release/sharp-linux-x64.node'
Require stack:

takoyaro · 2023-05-16T11:40:10Z

it looks like sharp doesn't support deno at the moment

xenova · 2023-05-16T16:15:02Z

@omar2205 Have you tried the steps outlined here: lovell/sharp#2583 (comment) ?

This is an apparent workaround until denoland/deno#16164 is made available.

omar2205 · 2023-05-16T17:24:06Z

Thanks for the ping @xenova, I followed that and it worked, however, I'm met with Invalid URL: error.

import { pipeline } from 'npm:@xenova/transformers';
/* warning
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
*/

let pipe = await pipeline('sentiment-analysis');

/* Error
No model specified. Using default model: "Xenova/distilbert-base-uncased-finetuned-sst-2-english".
Uncaught TypeError: Invalid URL: 'Xenova/distilbert-base-uncased-finetuned-sst-2-english/tokenizer.json'
*/

xenova · 2023-05-16T17:51:33Z

Okay great! That error is probably deno specific, because the check to see whether something is a valid URL is surrounded by a try-catch block (and works in other environments).

transformers.js/src/utils/hub.js

Lines 171 to 179 in 0b403ce

    
           function isValidHttpUrl(string) { 
        
               // https://stackoverflow.com/a/43467144 
        
               let url; 
        
               try { 
        
                   url = new URL(string); 
        
               } catch (_) { 
        
                   return false; 
        
               } 
        
               return url.protocol === "http:" || url.protocol === "https:";

While I look into it, you can get around it temporarily by disabling local models:

import { pipeline, env } from 'npm:@xenova/transformers';

env.allowLocalModels=false;

let pipe = await pipeline('sentiment-analysis');

omar2205 · 2023-05-16T18:09:25Z

I tried to do that before, but it fails:

No model specified. Using default model: "Xenova/distilbert-base-uncased-finetuned-sst-2-english".
Uncaught TypeError: Invalid URL: 'Xenova/distilbert-base-uncased-finetuned-sst-2-english/tokenizer.json'

xenova · 2023-05-16T18:15:08Z

Hmm, it must be failing elsewhere then. Are you able to send the full stack trace?

omar2205 · 2023-05-17T07:18:49Z

Here you go

No model specified. Using default model: "Xenova/distilbert-base-uncased-finetuned-sst-2-english".
Uncaught TypeError: Invalid URL: 'Xenova/distilbert-base-uncased-finetuned-sst-2-english/tokenizer.json'
    at getSerialization (ext:deno_url/00_url.js:88:11)
    at new URL (ext:deno_url/00_url.js:383:27)
    at new Request (ext:deno_fetch/23_request.js:297:25)
    at Cache.[[[matchAll]]] (ext:deno_cache/01_cache.js:230:11)
    at Cache.match (ext:deno_cache/01_cache.js:173:36)
    at getModelFile (file:///home/ubuntu/.cache/deno/npm/registry.npmjs.org/@xenova/transformers/2.0.0-alpha.2/src/utils/hub.js:347:32)
    at eventLoopTick (ext:core/01_core.js:166:11)
    at async getModelJSON (file:///home/ubuntu/.cache/deno/npm/registry.npmjs.org/@xenova/transformers/2.0.0-alpha.2/src/utils/hub.js:456:18)
    at async Promise.all (index 0)
    at async loadTokenizer (file:///home/ubuntu/.cache/deno/npm/registry.npmjs.org/@xenova/transformers/2.0.0-alpha.2/src/tokenizers.js:49:16)

xenova · 2023-05-17T12:15:51Z

Thanks! It looks like deno has it's own caching API, which only accepts HTTP URLs. I'll see what I can do.

xenova · 2023-05-17T12:59:03Z

In the meantime, can you try disabling the cache system? You can do this by setting:
env.useBrowserCache=false;

omar2205 · 2023-05-17T14:38:44Z

That fixed it. But we got another error 😅

> let out = await pipe('I love transformers!');

An error occurred during model execution: "TypeError: Tensor.data must be a typed array (9) for int64 tensors, but got typed array (-1).".
Inputs given to model: {
  input_ids: Tensor {
    dims: [ 1, 6 ],
    type: "int64",
    data: BigInt64Array(6) [ 101n, 1045n, 2293n, 19081n, 999n, 102n ],
    size: 6
  },
  attention_mask: Tensor {
    dims: [ 1, 6 ],
    type: "int64",
    data: BigInt64Array(6) [ 1n, 1n, 1n, 1n, 1n, 1n ],
    size: 6
  }
}
Uncaught TypeError: Tensor.data must be a typed array (9) for int64 tensors, but got typed array (-1).
    at file:///home/ubuntu/.cache/deno/npm/registry.npmjs.org/onnxruntime-node/1.14.0/dist/backend.js:45:108
    at Array.processTicksAndRejections (ext:deno_node/_next_tick.ts:23:21)
    at eventLoopTick (ext:core/01_core.js:178:29)

xenova · 2023-05-17T14:42:39Z

BigInt64Array is a typed array, so, I'm not too sure what the error message means... 👀 This is most likely a limitation of onnxruntime-web/onnxruntime-node :/

omar2205 · 2023-05-17T15:30:19Z

Yeah looks like it. Is it possible to use this https://github.com/nestarz/onnx_runtime?

xenova · 2023-05-17T15:33:26Z

If you'd like, you can fork this repo and replace the onnxruntime import with that?

omar2205 · 2023-05-17T15:53:31Z

After second thought, I don't know how that would even work. It's written for Deno.

xenova · 2023-05-17T16:04:55Z

They seem to provide an import syntax in their README:

import * as ort from "https://deno.land/x/onnx_runtime/mod.ts";

You can edit the ONNX imports in https://github.com/xenova/transformers.js/blob/main/src/backends/onnx.js

josephrocca · 2023-05-17T17:21:35Z

Maybe I'm missing something, but: Wouldn't it make more sense to use the browser version of onnxruntime for Deno? Deno is aimed at being strongly web-compatible, so running transformers.js/onnxruntime with node.js emulation (i.e. importing with npm: rather than e.g. jsdelivr) seems like a very round-about way that would result in more problems? Again, might be missing something as I have only skimmed this issue.

omar2205 · 2023-05-17T18:13:11Z

Trying using jsdelivr gives Uncaught DOMException error

xenova · 2023-05-17T19:02:45Z

I set up a codespace and got it working:

import { pipeline, env } from 'https://cdn.jsdelivr.net/npm/@xenova/transformers';
// import {pipeline, env} from '@xenova/transformers';

env.useBrowserCache=false;
env.allowLocalModels=false;

let pipe = await pipeline('text-classification')

let output = await pipe('I love Transformers.js')

console.log(output)

console output:

$ cat main.js | deno run --allow-all --unstable -
No model specified. Using default model: "Xenova/distilbert-base-uncased-finetuned-sst-2-english".
[ { label: "POSITIVE", score: 0.99961256980896 } ]

The only drawback with this approach is that you might experience a performance hit since it's running with the WASM backend (and not native CPU). I'll look into why the tensor types are giving errors when running the native version.

omar2205 · 2023-05-18T07:29:25Z

I got Uncaught DOMException when I opened this issue and tried it yesterday
But it looks like it worked fine with you, and in this replit
https://replit.com/@omar22051/IncompleteMenacingPattern#index.ts

When I tried it in a deno deploy, I got hit with the memory limit.

xenova · 2023-05-18T08:57:29Z

I got Uncaught DOMException when I opened this issue and tried it yesterday
But it looks like it worked fine with you, and in this replit
https://replit.com/@omar22051/IncompleteMenacingPattern#index.ts

I don't quite know where Transformers.js would be throwing a DOMException 👀 Is it maybe an issue with your environment (since it works in that replit)?

When I tried it in a deno deploy, I got hit with the memory limit.

Do you know what the memory limit is for deno deploy? Some sources suggest either 256MB or 512MB. The model you're testing with isn't that large.

omar2205 · 2023-05-18T11:17:41Z

I believe it's 512 MB. I managed to get a QA bot going, deploy to Deno Deploy here, which was the demo I was going for.

I don't know if you want to close this issue or wait for onnx to support Deno.

xenova · 2023-05-18T11:29:17Z

Cool! I'll try look into the previous issue and see if there's a workaround since it might take a while for them to support Deno properly. I think it will also be good to create a Deno tutorial/example project, so, I'll close the issue after I make that.

birkskyum · 2023-08-31T10:44:56Z

Related to

Unable to find napi_register_module_v1 on mac m1 denoland/deno#20071

kivlor · 2023-09-22T21:25:05Z

@xenova I gave your codespace example a try locally with Deno 1.37.0 and I get the following:

No model specified. Using default model: "Xenova/distilbert-base-uncased-finetuned-sst-2-english".
error: Uncaught (in promise) NotSupported: Classic workers are not supported.
    at createWorker (ext:runtime/11_workers.js:41:14)
    at new Worker (ext:runtime/11_workers.js:110:16)
    at Object.yc (https://cdn.jsdelivr.net/npm/@xenova/transformers:34:7988)
    at Object.Cc (https://cdn.jsdelivr.net/npm/@xenova/transformers:34:8046)
    at de (https://cdn.jsdelivr.net/npm/@xenova/transformers:34:5797)
    at Ae (https://cdn.jsdelivr.net/npm/@xenova/transformers:34:9771)
    at <anonymous> (https://cdn.jsdelivr.net/npm/@xenova/[email protected]/dist/ort-wasm-simd-threaded.wasm:1:8324360)
    at <anonymous> (https://cdn.jsdelivr.net/npm/@xenova/[email protected]/dist/ort-wasm-simd-threaded.wasm:1:999116)
    at <anonymous> (https://cdn.jsdelivr.net/npm/@xenova/[email protected]/dist/ort-wasm-simd-threaded.wasm:1:1913327)
    at <anonymous> (https://cdn.jsdelivr.net/npm/@xenova/[email protected]/dist/ort-wasm-simd-threaded.wasm:1:4813118)

Any quick ideas? I'll happily dig in the mean time :)

xenova · 2023-09-22T21:39:31Z

You could try just extract the worker.js logic out into the main script. If the error is happening within onnxruntime, it might be fixable with env.backends.onnx.wasm.numThreads=1 (see here/docs)

kivlor · 2023-09-27T22:48:49Z

Thanks @xenova, I've got it running now with the following code:

import { pipeline, env } from "https://cdn.jsdelivr.net/npm/@xenova/transformers";

env.backends.onnx.wasm.numThreads = 1;

const classifier = await pipeline("text-classification", "Xenova/distilbert-base-uncased-finetuned-sst-2-english");

const input = ["I love Transformers.js", "I hate Transformers.js", "I not sure about Transformers.js"];
const result = await classifier(input);

console.log(result);

then running in 1.37.1 with:

$ deno run -A main.ts

devoutdrawai · 2023-10-26T14:13:35Z

I am running into
TypeError: c is not a function at Function.fromBlob (https://cdn.jsdelivr.net/npm/@xenova/transformers:100:1417) when trying to use RawImage.fromBlob, is there a work around for this?

xenova · 2023-10-26T14:22:43Z

@devoutdrawai can you provide the code snippet you are using? Also, if you import the unminified version of the script (https://cdn.jsdelivr.net/npm/@xenova/[email protected]/dist/transformers.js), you'll see a more helpful error message.

Regardless, I believe you'll be running into issues since deno doesn't yet support offscreen canvas (denoland/deno#19533)

devoutdrawai · 2023-10-27T01:33:35Z

@xenova Yeah I got Error: OffscreenCanvas not supported by this browser. after trying to create a new RawImage class rather than a blob. Is there a workaround way to pass in a image for inferencing that I just don't know about?

xenova · 2023-11-16T18:07:01Z

Update from the sharp.js team about installation with Deno: lovell/sharp#3750 (comment) 🥳

A new sharp v0.33.0-alpha.11 pre-release is now available, which has been tested as working out of the box with Deno and has (drum roll please) a WebAssembly version - please see #3750 (comment)

Will be a good idea to revisit proper Deno support (instead of using the web/WASM version).

birkskyum · 2023-11-16T18:15:56Z

@xenova , I think that's a good idea too to avoid paying the sometimes steep wasm price, unless strictly necessary

Sharp - WASM PR: As you can see, Wasm performs at around native speed in some benchmarks, but is ~2x slower where SIMD is required - which is expected, given that libvips heavily relies on runtime SIMD code generation, which is not yet supported for Wasm.

kyeshmz · 2023-12-15T00:28:59Z

It seems v0.33 is now publically avaliable!

birkskyum · 2023-12-19T10:51:31Z

Have this been working since transformers.js 2.9 where sharp was bumped to v0.33?

DuncanLHS · 2024-01-03T15:50:00Z

transformers.js v 2.13.1

Not sure if this should be a new issue but is also Deno related. I've been using remote models without issue in a Supabase edge function (Deno environment), now experimenting with using the same model locally and hitting errors related to the local filesystem.

I'm getting Invalid URL error

[Error] Unable to load from local path "../_models/Supabase/gte-small/tokenizer.json": "TypeError: Invalid URL: '../_models/Supabase/gte-small/tokenizer.json'"

I've had a pick through the source and don't think this should be happening if the 'fs' package is available, which AFAIK it should be in deno.

So I tried adding env.FS = true and got a different error, again appears related to fs (or lack thereof)

[Error] Unable to load from local path "../_models/Supabase/gte-small/tokenizer.json": "TypeError: r.existsSync is not a function"

Folder structure is:
,,/functions/_models/Supabase/gte-small - model files are here
../functions/embed/embed.ts - pipeline is here

embed.ts

env.useBrowserCache = false
env.allowLocalModels = true
env.allowRemoteModels = false
env.localModelPath = '../_models'
env.useFS = true

const generateEmbedding = await pipeline(
  'feature-extraction',
  'Supabase/gte-small'
)

xenova · 2024-01-03T16:00:00Z

@DuncanLHS Could you try test if fs is available (outside of transformers.js)? e.g., open a local text file.

DuncanLHS · 2024-01-03T16:13:56Z

@xenova Yes, but before I do I've just spotted that any node packages available in deno need to use the 'node:' specifier.
https://docs.deno.com/deploy/api/runtime-node

xenova · 2024-01-03T16:17:18Z

Also note that filesystem access is disabled when loading the library from a CDN, so you would need to wait until the library functions correctly with Deno.

DuncanLHS · 2024-01-03T16:22:43Z

OK, I'll move the functionality into node for now.

cotyhamilton · 2024-09-01T20:18:33Z

The alpha is working in deno https://huggingface.co/posts/Xenova/681836693682285

working example

DENO_FUTURE=1 deno add npm:@huggingface/[email protected]

// main.ts
import { pipeline } from "@huggingface/transformers";

async function main() {
  const generateEmbeddings = await pipeline("feature-extraction");
  const embeddings = await generateEmbeddings("Hello, World!");
  console.log(embeddings);
}

if (import.meta.main) {
  main();
}

DENO_FUTURE=1 deno run -RWN --allow-ffi --deny-env main.ts

No model specified. Using default model: "Xenova/all-MiniLM-L6-v2".
dtype not specified for "model". Using the default dtype (fp32) for this device (cpu).
Tensor {
  ort_tensor: Tensor {
    cpuData: Float32Array(2304) [
        0.09704522788524628,   0.04466759413480759,   0.03342921659350395,
         0.2942180037498474,  -0.09638233482837677,  -0.39724844694137573,
       0.029586339369416237,  -0.09695737808942795,    -0.233986034989357,
       -0.11687380075454712,  0.031189898028969765,   0.28859788179397583,
        0.12789778411388397,  0.007472040131688118,   0.09201933443546295,
        -0.2740094065666199,   -0.0917656198143959,   0.09536977857351303,
       0.011544985696673393,  -0.07689643651247025,  -0.08625669777393341,
        -0.0317402184009552,   0.19354557991027832,  0.056520186364650726,
       -0.18216514587402344,    0.1443641483783722,  -0.06378541141748428,
        0.22648848593235016,  -0.09405055642127991,   -0.7146937251091003,
       -0.02438972145318985,   -0.1423027515411377,    0.3438223600387573,
       -0.24876512587070465,  -0.09658709168434143,    0.3035530149936676,
        0.04076158255338669,   -0.2887788712978363,   -0.1621636301279068,
        0.07270413637161255,    0.1354200392961502,  -0.18576160073280334,
       -0.11635178327560425,   0.14100773632526398,   0.03162102401256561,
       -0.01900934800505638,  -0.21902813017368317,   0.23496025800704956,
       0.041369009763002396,    0.1376398801803589,   -0.3159908354282379,
       -0.05101257562637329,   0.08553100377321243,   -0.1915268898010254,
         0.1287413239479065,   0.12561288475990295, -0.013958334922790527,
        0.09618371725082397, -0.046709202229976654,  -0.19115103781223297,
       -0.04283100366592407,  -0.09430375695228577,  -0.23719236254692078,
      -0.027194540947675705,   0.06842745095491409,   -0.3157595992088318,
        0.08512159436941147,   0.10566434264183044, -0.053132228553295135,
        -0.0351015068590641,  -0.15570200979709625,    0.0818176120519638,
         0.1027064248919487,   0.04647187143564224,   -0.1836329847574234,
       -0.11254461109638214,   0.06946499645709991,   0.02888619899749756,
         0.2119930386543274,    0.0845736488699913, -0.038595687597990036,
        0.02478068694472313,   0.09258890151977539, -0.021482625976204872,
       -0.05618399381637573,   0.15647916495800018,   0.00866264570504427,
        0.03660000488162041,  -0.12824304401874542,  -0.09475530683994293,
       -0.47616755962371826,  -0.18034975230693817,  0.041318655014038086,
           0.09013681858778,   -0.6151973009109497,  0.008081695064902306,
        0.11617103964090347,  -0.17217235267162323,   -0.3506837487220764,
           5.51707649230957,
      ... 2204 more items
    ],
    dataLocation: "cpu",
    type: "float32",
    dims: [ 1, 6, 384 ],
    size: 2304
  }
}

Unfortunately only able to use cpu device though

omar2205 added the enhancement New feature or request label Apr 8, 2023

xenova mentioned this issue Dec 19, 2023

Update sharp.js version (+ add Deno NPM support) #463

Closed

xenova linked a pull request Jan 27, 2024 that will close this issue

[WIP] 🚀🚀🚀 Transformers.js V3 🚀🚀🚀 #545

Open

18 tasks

[Feature request] Deno Support #78

[Feature request] Deno Support #78

Comments

omar2205 commented Apr 8, 2023

xenova commented Apr 8, 2023 • edited Loading

omar2205 commented Apr 8, 2023

xenova commented Apr 8, 2023 • edited Loading

omar2205 commented Apr 8, 2023

xenova commented Apr 8, 2023

xenova commented May 15, 2023

omar2205 commented May 15, 2023

takoyaro commented May 16, 2023

xenova commented May 16, 2023

omar2205 commented May 16, 2023

xenova commented May 16, 2023 • edited Loading

omar2205 commented May 16, 2023

xenova commented May 16, 2023

omar2205 commented May 17, 2023

xenova commented May 17, 2023

xenova commented May 17, 2023

omar2205 commented May 17, 2023

xenova commented May 17, 2023

omar2205 commented May 17, 2023

xenova commented May 17, 2023

omar2205 commented May 17, 2023

xenova commented May 17, 2023

josephrocca commented May 17, 2023

omar2205 commented May 17, 2023

xenova commented May 17, 2023 • edited Loading

omar2205 commented May 18, 2023

xenova commented May 18, 2023 • edited Loading

omar2205 commented May 18, 2023

xenova commented May 18, 2023

birkskyum commented Aug 31, 2023

kivlor commented Sep 22, 2023 • edited Loading

xenova commented Sep 22, 2023 • edited Loading

kivlor commented Sep 27, 2023 • edited Loading

devoutdrawai commented Oct 26, 2023

xenova commented Oct 26, 2023 • edited Loading

devoutdrawai commented Oct 27, 2023

xenova commented Nov 16, 2023

birkskyum commented Nov 16, 2023

kyeshmz commented Dec 15, 2023

birkskyum commented Dec 19, 2023 • edited Loading

DuncanLHS commented Jan 3, 2024 • edited Loading

xenova commented Jan 3, 2024

DuncanLHS commented Jan 3, 2024

xenova commented Jan 3, 2024

DuncanLHS commented Jan 3, 2024

cotyhamilton commented Sep 1, 2024 • edited Loading

xenova commented Apr 8, 2023 •

edited

Loading

xenova commented Apr 8, 2023 •

edited

Loading

xenova commented May 16, 2023 •

edited

Loading

xenova commented May 17, 2023 •

edited

Loading

xenova commented May 18, 2023 •

edited

Loading

kivlor commented Sep 22, 2023 •

edited

Loading

xenova commented Sep 22, 2023 •

edited

Loading

kivlor commented Sep 27, 2023 •

edited

Loading

xenova commented Oct 26, 2023 •

edited

Loading

birkskyum commented Dec 19, 2023 •

edited

Loading

DuncanLHS commented Jan 3, 2024 •

edited

Loading

cotyhamilton commented Sep 1, 2024 •

edited

Loading