Allow loadable modules

[mattklein123]

In general, everywhere we use static registration (filters, stats sinks, loggers, tracers, etc.) it would not be difficult to also support dynamic loading. While some set of people want everything to be statically linked, others want dynamic loading flexibility. I have no objection to adding optional dynamic loading support if someone wants to take on this project.

[glyn]

Please can we define (either here or in a separate issue) in what circumstances it would be necessary to recompile loadable modules?

For example, envoy could apply semantic versioning:

• loadable modules would need recompiling for distinct major versions of envoy,
• loadable modules could be reused without recompilation between versions of envoy with the same major version.

[ggreenway]

Without a large amount of work on ensuring binary compatibility of all the infrastructure that a module could depend on, my guess is that we would require modules to be compiled from the exact same source as the envoy that will load them.

[mattklein123]

I agree with @ggreenway until #3390 is solved, which if I'm being realistic is probably 1-2 years out.

[glyn]

Thanks. That's a clear definition. ;-)

[atrifan]

I would think that you could have a generic http filter and network filter in which you specify what .so file to load and there you would call a predefined method - proxying the call to the dynamic .so file - passing as parameters - callback functions to be invoked for logging, setting meta variables or some other process in which you would do the logic in your dynamic .so file but logging and other envoy stuff would be delegated to the native implementation of the network / http filter.

Thoughts on this?

[mattklein123]

Thoughts on this?

My preference is to actually support it more directly in the code at each extension point. Meaning, today we statically register all extension points and then search for them at load time by name. I think what we need to do is actually allow extensions to optionally be specified by an .so/.dll, and then just load that.

[davinci26]

fyi @pravb

[cpakulski]

I would like to go back to idea of dynamic loading of libraries. We have several extension mechanisms: Lua, wasm and golang. I would like to propose another one: very simple http filter which would load *.so file and look for C bindings. Those bindings would be basically equivalent to GRPC methods used in extern_proc: init, on req/resp headers/body/trailers.
The config specific to the loaded *.so would be delivered via xDS. The *.so library could be packaged along with Envoy in a container or xDS could be extended by binaryDS, which upon delivery to Envoy would load the *.so directly into memory or mount it into file system and then load. Envoy should provide to the library callbacks for local_reply, logging, metadata and stats.
My use case is requirement for high speed body validation/transformation and I would like to implement it in C++.
I can take on this work if there is an agreement.

[ggreenway]

I don't think this would be too difficult as long as you accept that there's no ABI (or even API stability, see #3390), and thus all extensions must be compiled from the exact same source tree as the envoy they'll be run with, using the same compiler, same compiler settings, etc.

[mattklein123]

yeah I'm fine with this also. My main request would be to factor out all of the SO loading config so it's shared among all users (you can take/move the config/code from the Go filter I think).

[kyessenov]

What about Wasm ABI "null" version: https://github.com/proxy-wasm/proxy-wasm-cpp-sdk/blob/master/proxy_wasm_externs.h?

That would provide some stability, and should interop well with ext_proc, Rust, etc. AFAIK ext_proc can convert to proxy-wasm host. We'd certainly need to revise it before committing, there are some mistakes made in the first version.

[cpakulski]

I don't think this would be too difficult as long as you accept that there's no ABI (or even API stability, see #3390), and thus all extensions must be compiled from the exact same source tree as the envoy they'll be run with, using the same compiler, same compiler settings, etc.

I understand that. I remember a case when a struct had #ifdef X statement and a library was compiled without X defined and main code was compiled with X defined and it took me ages to figure it out. You are right, structures has to be packed the same way and have to have the same sizes. But this becomes a user's problem to make sure that libraries are compatible.

My main request would be to factor out all of the SO loading config so it's shared among all users (you can take/move the config/code from the Go filter I think).

Agree. I looked at go L7 code and there is *so manager to share the library among multiple users.

What about Wasm ABI "null" version: https://github.com/proxy-wasm/proxy-wasm-cpp-sdk/blob/master/proxy_wasm_externs.h?

I was thinking about that, but heard that C++ may not compile nicely to wasm and also that there is some speed penalty. Need to investigate it more because I am not super familiar with null vm. Thanks for bringing it up.

[kyessenov]

I was thinking about that, but heard that C++ may not compile nicely to wasm and also that there is some speed penalty. Need to investigate it more because I am not super familiar with null vm. Thanks for bringing it up.

The penalty is a general penalty for having a narrow and low-level interface. Unless you want some tighter integration with Envoy (e.g. stats system), the overhead is minimal and has to do with data serialization. C is great in that sense, it's painful to figure out C++ compilation artifacts (like packed strings, etc)

[kyessenov]

There's an open PR to add dynamically loaded Wasm ABI modules proxy-wasm/proxy-wasm-cpp-host#379. I like that approach since it brings the benefits of the decoupled build with less of a runtime hit. It's difficult to build a cross-language ABI without even taking Wasm into account - and the existing Envoy interfaces are C++-native - even for Rust it's non-trivial ownership/conversion implications.

Separately, I'm curious about the operational implications of the dlopen in k8s. I wonder if there's some accounting hit on the memory used caused by dlopen.

[cpakulski]

Makes sense. I made a similar prototype to test idea of dynamically linking modules and it seems to work. The disadvantage is obviously increased memory requirements. For example, if the module uses json parser code, it must use it's own (linked with the module) as json parser linked with Envoy is not visible to such module.

[Protryon]

I wrote that dynamic NullVM model. What I do for the same use case in NGINX is link against the same libraries it's linking against (i.e. openssl) to reduce binary size and therefore memory. The same concept can apply here, you could import symbols from any library linked by Envoy provided you link against the same version -- dlopen will handle that case the same as importing symbols from Envoy/proxy-wasm-cpp-host.

[kyessenov]

@Protryon does that require dynamic linking for the common shared library? Envoy is mostly statically linked right now. I was thinking you could use FFI ABI call to expose crypto, for example.

[htuch]

I think part of the goal here is to be writing dynamic modules against a stable, versioned ABI. FFI ABI to crypto violates that, since we can't make compatability guarantees for modules across Envoy releases. The hope was that Wasm ABI is this stable versioned ABI.

[kyessenov]

@htuch Yeah, FFI is more of a loophole solution. It might be worth looking into the component model instead where some components are provided by Envoy natively.

[Protryon]

@kyessenov Yeah, it requires the component to be dynamically linked by both Envoy and the dynamic module.

As an alternative, the dynamic module can leave the shared library unlinked, and have unresolved symbols that Envoy must have or the module will fail to load.

In NGINX, since I rely on version-specific symbols, I'm recompiling for each version of NGINX supported. We are using Dynamic NullVM in production now, and just don't rely on any native symbols. Only Proxy-WASM defined ones, and we are calling system calls/doing custom threading work, so we haven't had to recompile for each version of Envoy.

[atrifan]

I would like to go back to idea of dynamic loading of libraries. We have several extension mechanisms: Lua, wasm and golang. I would like to propose another one: very simple http filter which would load *.so file and look for C bindings. Those bindings would be basically equivalent to GRPC methods used in extern_proc: init, on req/resp headers/body/trailers. The config specific to the loaded *.so would be delivered via xDS. The *.so library could be packaged along with Envoy in a container or xDS could be extended by binaryDS, which upon delivery to Envoy would load the *.so directly into memory or mount it into file system and then load. Envoy should provide to the library callbacks for local_reply, logging, metadata and stats. My use case is requirement for high speed body validation/transformation and I would like to implement it in C++. I can take on this work if there is an agreement.

It's nice we are revisiting this 4 years later :)