Evolution, epiphany: Webkitgtk bubblewrapper crashes when LD_LIBRARY path points into /run/current-system/** (the case on a pipewire system with jack emulation)

[Emantor]

Describe the bug
Mails are not displayed within evolution. I'm running sway, so evolution is run under the wayland backend. It's also broken when forcing evolution to use X11.

To Reproduce
Steps to reproduce the behavior:

1. start evolution
2. Try to open a mail by double clicking
3. New window for the mail displays:

Something has gone wrong when displaying the message

A WebKitWebProcess crashed when displaying the message. You can try again by moving to another message and back. If the issue persists, please file a bug report in GNOME Gitlab.

Expected behavior
Being able to read the mail instead of the error message.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
The only change touching webkitgtk was #96140.

Notify maintainers
@NixOS/gnome

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

- system: `"x86_64-linux"`
- host os: `Linux 5.10.9, NixOS, 21.03pre265130.f217c0ea7c1 (Okapi)`
- multi-user?: `yes`
- sandbox: `yes`
- version: `nix-env (Nix) 2.3.10`
- channels(root): `"nixos-21.03pre265130.f217c0ea7c1"`
- channels(phoenix): `"ptxpkgs"`
- nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[Emantor]

Epiphany is also broken.

[jtojnar]

cc @matthewbauer

[matthewbauer]

I think this is from not being able to find the libwpe-fdo plugin. You shouldn't need it on X11, I think, but on wayland it needs to find the plugin somewhere. The situation is annoying (WebPlatformForEmbedded/libwpe#59), but for now we can just hardcode it to avoid the issue (albeit breaking the ability to add more plugin). Should have a fix available soon.

[matthewbauer]

I've opened #110738 to revert. Can't quite figure out what needs to change - WPE_RENDERER is enabled on other distros without issue, so there's got to be some way to get it working.

[worldofpeace]

It appears that the plugin loader in its current state could not work on nixos, as most plugin system unfortunately.
The two PRs created https://github.com/Igalia/WPEBackend-fdo/pull/99/files https://github.com/WebPlatformForEmbedded/libwpe/pull/64/files changes how backend implementations are loaded by searching in a directory WPE_BACKENDS_DIR=\"${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}/wpe-${WPE_API_VERSION}\" which will be compiled in. Not even that will work for as us everything is installed into separate prefixes. They could however, to support us, include an environment variable to recurse for plugins.

So this requires creating a wpe-with-plugins just to use the fdo plugin for webkitgtk.
The code that changes the plugin loading isn't merged yet so we could suggest a solution to support our usecase or PR against the PR. Or just roll our own patches.

[worldofpeace]

... realizes these aren't executable

[worldofpeace]

So this would then require 😁 changing the loader so that wpe_loader_init can accept an absolute path to a plugin. And we'd have to perform code substitution on the users of the function to load the intended plugin from an absolute path.

diff --git a/Source/WebKit/UIProcess/glib/WebProcessPoolGLib.cpp b/Source/WebKit/UIProcess/glib/WebProcessPoolGLib.cpp
index a380a25fa4b..d96d23bce2e 100644
--- a/Source/WebKit/UIProcess/glib/WebProcessPoolGLib.cpp
+++ b/Source/WebKit/UIProcess/glib/WebProcessPoolGLib.cpp
@@ -114,7 +114,7 @@ void WebProcessPool::platformInitializeWebProcess(const WebProcessProxy& process
 #if PLATFORM(WAYLAND)
     if (WebCore::PlatformDisplay::sharedDisplay().type() == WebCore::PlatformDisplay::Type::Wayland) {
 #if USE(WPE_RENDERER)
-        wpe_loader_init("libWPEBackend-fdo-1.0.so");
+        wpe_loader_init("@[email protected]");
         if (wpe_fdo_initialize_for_egl_display(WebCore::PlatformDisplay::sharedDisplay().eglDisplay())) {
             parameters.hostClientFileDescriptor = wpe_renderer_host_create_client();
             parameters.implementationLibraryName = FileSystem::fileSystemRepresentation(wpe_loader_get_loaded_implementation_library_name());

diff --git a/Source/WebKit/UIProcess/glib/WebProcessPoolGLib.cpp b/Source/WebKit/UIProcess/glib/WebProcessPoolGLib.cpp
index a380a25fa4b..d96d23bce2e 100644
--- a/Source/WebKit/UIProcess/glib/WebProcessPoolGLib.cpp
+++ b/Source/WebKit/UIProcess/glib/WebProcessPoolGLib.cpp
@@ -114,7 +114,7 @@ void WebProcessPool::platformInitializeWebProcess(const WebProcessProxy& process
 #if PLATFORM(WAYLAND)
     if (WebCore::PlatformDisplay::sharedDisplay().type() == WebCore::PlatformDisplay::Type::Wayland) {
 #if USE(WPE_RENDERER)
-        wpe_loader_init("libWPEBackend-fdo-1.0.so");
+        wpe_loader_init("/nix/store/dp94mlfkz0yxj53yk4bhcdxkcbax764q-wpebackend-fdo-1.7.1/lib/libWPEBackend-fdo-1.0.so");
         if (wpe_fdo_initialize_for_egl_display(WebCore::PlatformDisplay::sharedDisplay().eglDisplay())) {
             parameters.hostClientFileDescriptor = wpe_renderer_host_create_client();
             parameters.implementationLibraryName = FileSystem::fileSystemRepresentation(wpe_loader_get_loaded_implementation_library_name());

[worldofpeace]

Reasonably, I think wpe could support loading a plugin relatively and absolutely. It shouldn't be a very intrusive change, although we have to do this madness to actually use it.

[Emantor]

I have now switched to nixos-unstable-small to test the revert however evolution and epiphany are still broken. Could this be due to my local setup? Can others reproduce this?

[Emantor]

This is due to me using pipewire instead of pulseaudio, using pipewire outputs

bwrap: Can't mkdir parents for /run/current-system/sw/lib/pipewire: No such file or directory

when starting epiphany or evolution.

[Emantor]

The nixos pipewire module sets LD_LIBRARY_PATH to /run/current-system/sw/lib/pipewire if cfg.jack.enable is enabled, this is parsed by the webkitgtk bubblewrap launcher, which tries to bind this path. However this bind fails, crashes the bubblewrapper and webkitgtk fails to start. Unsetting LD_LIBRARY_PATH lets me use evolution with pipewire only.

[gshipunov]

Can others reproduce this?

Run into the same issue, nixos-unstable 536fe36e23ab0fc8b7f35c24603422eee9fc17a2. The peculiarity is the fact that I'm not explicitly enabling pipewire and/or jack.

pactl info:

Server String: /run/user/1000/pulse/native
Library Protocol Version: 34
Server Protocol Version: 34
Is Local: yes
Client Index: 4
Tile Size: 65472
User Name: grigory
Host Name: dishwasher
Server Name: pulseaudio
Server Version: 14.2-rebootstrapped

When running evolution (or epiphany) from the terminal bwrap suggests enabling non-privileged user namespaces:
$ evolution:

bwrap: No permissions to creating new namespace, likely because the kernel does not allow non-privileged user namespaces. On e.g. debian this can be enabled with 'sysctl kernel.unprivileged_userns_clone=1'.

Doing exactly as it suggests fixes evolution.

If anyone has any idea how to debug that, feel free to provide instructions

[hmenke]

Just updated from nixos-20.09 to nixos-21.05 with services.pipewire.jack.enable = true and get the same error.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/headphone-volume-resets-itself-to-100/13866/2

[mm--]

From looking at what @Emantor posted, it seems like bubblewrap has some issues with bind mounting symlinks. WebKitGTK should probably try to canonicalize source paths before bind mounting them on a destination.

In the meantime, a workaround for people that want to use PipeWire+JACK and WebKitGTK at the same time is to set the LD_LIBRARY_PATH to the absolute path in the nix store instead of /run/current-system/sw/lib/pipewire. Here's how you can do this in your config:

{ config, pkgs, lib, ... }:

{
  environment.sessionVariables.LD_LIBRARY_PATH = lib.mkForce "${config.services.pipewire.package.jack}/lib";
  services.pipewire = {
    enable = true;
    # ...
    jack.enable = true;
    # ...
  };
}

[fufexan]

I have a problem caused by the same issue with pipewire, where I can't add some accounts to Gnome, including Google, Facebook, Microsoft.

[fufexan]

Using @Artturin's patch fixed the account login screen not appearing and epiphany instantly crashing.

However, now I have another issue. Starting either epiphany/Gnome account login works fine but the window is blank.

Oddly enough, https://html.duckduckgo.com works

This is a relevant issue I found, but it's from 2 years ago https://gitlab.gnome.org/GNOME/epiphany/-/issues/570

EDIT: prepending WEBKIT_DISABLE_COMPOSITING_MODE=1 fixed it.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/sound-only-works-while-pavucontrol-is-open/15280/2

[danwdart]

This being done does that mean that we no longer need to use the workaround?

[jtojnar]

The workaround from #110468 (comment) was applied to the NixOS module in #136475.

But we should not need it since we no longer have webkitgtk use fdo backend either.