PoC of sharing Lua interpreter memory between multiple processes.

It features two processes that take turns calling functions f() and g() in the same Lua state (loaded from demo.lua).

How?

There’s nothing new in Lua being flexible: it allows to override all the memory management by passing a custom allocator to lua_newstate. The only things left are to create a shared memory mapping and implement an allocator working in it.

This is more or less safe as long as Lua knows nothing about parallelism. Some things in the standard library do know, though; see the “Caveats” section.

On systems other that Linux, shared memory mappings have to be of fixed size; by default, this PoC allocates 16 Mb on these systems.

In fact, Linux shared memory mappings also have to be of fixed “size” — of fixed virtual size: thanks to the overcommit feature and the MAP_NORESERVE mmap flag, which tells Linux to only allocate physical pages on demand (this does not depend on which overcommit policy your system is configured to use), we can only pay for what we use, and not a page more. And with madvise(..., MADV_REMOVE), we can reclaim the pages we don’t need anymore. Yay!

On Linux, this PoC goes on to allocate 16 Gb of virtual memory (1 Gb on 32-bit systems). Because this ought to be enough for anybody. If your virtual memory is, in some reason, limited, run it as ./main -p, which forces it to fall back to a portable 16 Mb mapping.

Caveats

• Some functions in Lua’s standard library really don’t expect being called with such a setup. These functions are os.execute, io.popen, os.exit and os.setlocale.

• All actions on files (including opening and closing) are local to the current process.

• My memory allocator sucks. Implementing a better one — possibly leveraging the aforementioned Linux features — is left as an exercise for the reader.

• Does not work with LuaJIT.

• Be careful with external modules.