Cannot construct SharedArrayBuffer with BackingStore of ArrayBuffer in mmap.

[creationix]

I'm trying to use mmap, but I'm getting V8 errors when it tries to create the buffer for the return value.

const fd = fs.open("data.idx", fs.O_RDWR)
const stat = new BigUint64Array(18)
const size = Number(stat[7])
const buf = sys.mmap(fd, size, sys.PROT_READ)

And it fails with:

#
# Fatal error in v8_SharedArrayBuffer_New
# Cannot construct SharedArrayBuffer with BackingStore of ArrayBuffer
#

[billywhizz]

hi tim. hope you are having fun! 😄

yes. i came across this recently myself and have fixed it. i should be able to get a new release out this weekend. if you can build yourself the fix needs to be applied here,

and this should fix it

  std::unique_ptr<BackingStore> backing = SharedArrayBuffer::NewBackingStore(
      data, len, [](void*, size_t, void*){}, nullptr);
  Local<SharedArrayBuffer> ab = SharedArrayBuffer::New(isolate, std::move(backing));

verified locally running this (you were missing your fstat call)

const { fs } = just.library('fs')
const { sys } = just.library('sys')

const { SystemError } = just
const { open, fstat } = fs
const { mmap } = sys

const fd = open('data.idx', fs.O_RDWR)
if (fd < 0) throw new SystemError('open')
const stat = new BigUint64Array(18)
const r = fstat(fd, stat)
if (r < 0) throw new SystemError('fstat')
const size = Number(stat[7])
const buf = mmap(fd, size, sys.PROT_READ)
if (!buf) throw new SystemError('mmap')
just.print(buf.byteLength)

[billywhizz]

also, just to be clear on this. this assumes you are managing the lifetime of that memory now. v8 will not free the underlying memory when the SharedArrayBuffer goes out of scope and is garbage collected.

feel free to ping me any questions if you are trying to put something together and get stuck. is there a best place for communicating? i set up a gitter but i never check it. https://gitter.im/just-js/community

[billywhizz]

if you get that working (i'll push latest up later today as 0.1.1 which should have a fix) here's a little example of POSIX shared memory using shm_open and mmap to share 1 GB memory between two processes and use Atomics to read/write the shared memory. https://gist.github.com/billywhizz/73aa1c01b1e35cbb6fc6aa13b2cc479e

[billywhizz]

i pushed a new release out with a fix for this. you can try it out with working mmap example here. example is in examples/shmem/mmap.js.

release: https://github.com/just-js/just/releases/tag/0.1.1 (the attached binary seems to be broken for me but the tarball should build fine)

[creationix]

Thanks. I've been away from my computer for a few days.

I was going to try and build it myself from source, but got stuck on figuring out the exact way to build everything. It's interesting how there is the tiny core and then a set of blessed modules. I remember noticing some other bugs too when reading the sources to the bindings (I've written enough V8 bindings that it works as docs for me :D) I'll make sure and check against the latest tree to see which have been fixed already.

[billywhizz]

thanks tim - nice to get some feedback from one of my heroes from the early days of node.js!

would love a second pair of eyes - much of it i know myself is broken but i am only one person and have been more focused on using it to build things and fixing as i go for now. i'm going to be releasing some new bits very shortly and then major focus will be on cleaning up and finalising the internal code, standard library, documentation etc.

as long as you have make/g++/curl available you should just be able to run

make clean runtime

and everything should work. there may be issues with the v8 lib i am linking against in your environment. because v8 takes so outrageously long to build i build the v8 monolithic lib separately and download and link against that from here. if you do have issues with linkage, you can try building in docker using debian:buster-slim which the v8 lib was built on.

if you do take an interest and have any suggestions around how to manage that process of linking against v8 better i am all ears. at the moment it is just something i hacked together based on my limited knowledge and got to work. i'd particularly like to know how to reduce the size of the resulting binary and memory usage by excluding some v8 features that may not be needed.

feel free to ping me on here or by email if you have any questions or thoughts. in general, u think it could be a useful thing? i am really enjoying hacking on various lower level OS internals using JS myself - is nice having all the power (and danger) of C with all the flexibility and ease of development of JS. kinda what i had hoped node.js was originally going to be back in the day.

[billywhizz]

also, re. modules - i am working on allowing external modules to be imported and built through github which would allow anyone to create and use each others libs and modules. i have all the pieces in place now to put this together and should have it working and released soon, hopefully with some nice docs and examples.

[creationix]

You're doing great. I'm sorry I've been busy the last few weeks working on other stuff. I hope to get back to this some time. I see a lot of potential in this project.