Wayland target, like sway

[SimonBiggs]

Hi @m1k1o,

Do you have an idea how I might get this working with a wayland target like sway. Also, for clarification, I don't mean something like this:

cat <<EOF | sudo tee /etc/neko/supervisord/sway.conf >/dev/null
[program:sway]
environment=HOME="/home/%(ENV_USER)s",
    USER="%(ENV_USER)s",
    DISPLAY="%(ENV_DISPLAY)s",
    XDG_RUNTIME_DIR="/run/user/%(ENV_USER_UID)s",
    WLR_BACKENDS=x11,
    WLR_LIBINPUT_NO_DEVICES=1,
    WLR_X11_DISPLAY=%(ENV_DISPLAY)s
command=/usr/bin/sway --verbose
stopsignal=INT
autorestart=true
priority=500
user=%(ENV_USER)s
stdout_logfile=/var/log/neko/sway.log
stdout_logfile_maxbytes=100MB
stdout_logfile_backups=10
redirect_stderr=true

As that is using sway with an x11 backend. The above config works fine enough. But I explicitly don't want to be using x11 so that I can have many nested sway instances without needing to have them all in RAM

[m1k1o]

To get just the streaming done, it needs to be adjused in gstreamer. I would expect that there is similar element to ximagexrc like waylandsrc but there is not.

To get the controls ported to wayland, there would need to be implemented a new driver (possibly using libei) that satisfies this interace:

neko/server/internal/types/desktop.go

Lines 43 to 77 in 5d62e56

	type DesktopManager interface {
	Start()
	Shutdown() error
	GetScreenSizeChangeChannel() (before chan bool) // true - before, false - after
	// clipboard
	ReadClipboard() string
	WriteClipboard(data string)
	// xorg
	Move(x, y int)
	GetCursorPosition() (int, int)
	Scroll(x, y int)
	ButtonDown(code uint32) error
	KeyDown(code uint32) error
	ButtonUp(code uint32) error
	KeyUp(code uint32) error
	ButtonPress(code uint32) error
	KeyPress(codes ...uint32) error
	ResetKeys()
	ScreenConfigurations() map[int]ScreenConfiguration
	SetScreenSize(ScreenSize) error
	GetScreenSize() *ScreenSize
	SetKeyboardMap(KeyboardMap) error
	GetKeyboardMap() (*KeyboardMap, error)
	SetKeyboardModifiers(mod KeyboardModifiers)
	GetKeyboardModifiers() KeyboardModifiers
	GetCursorImage() *CursorImage
	GetScreenshotImage() *image.RGBA
	// xevent
	GetCursorChangedChannel() chan uint64
	GetClipboardUpdatedChannel() chan struct{}
	GetEventErrorChannel() chan DesktopErrorMessage
	}

[SimonBiggs]

Also... I imagine multi cursor / seat would be quite helpful?

swaywm/wlroots#352
https://youtu.be/WO2L_ihO_rI

[SimonBiggs]

SEAT CONFIGURATION¶
Configure options for multiseat mode.

A seat is a collection of input devices that act independently of each other. Seats are identified by name and the default seat is seat0 if no seats are configured. While sway is running, - (hyphen) can be used as an alias for the current seat. Each seat has an independent keyboard focus and a separate cursor that is controlled by the pointer devices of the seat. This is useful for multiple people using the desktop at the same time with their own devices (each sitting in their own "seat"). The wildcard character, *, can also be used in place of to change settings for all seats.

https://manpages.debian.org/experimental/sway/sway-input.5.en.html

[m1k1o]

Definitely, that would be great to have.

[SimonBiggs]

I think what I'll need to do is just set up the "view only" no inputs version first with webRTC, and then potentially in the next couple of weeks I'll come back to this and see if I can sort out the inputs.

Do you have any preferences for how to do PRs or anything?

[SimonBiggs]

Do you have a recommendation on how best to use / edit / refactor your tool if all I want to do with it initially is pass through Wayland screen captures through webRTC?

[SimonBiggs]

This seems like it would be something appropriate to use to pipe the video and audio through to Neko?
https://github.com/ammen99/wf-recorder

[SimonBiggs]

Actually, maybe as a first step entirely I'll just have wf-recorder go to the video loopback and then just stream it with this for now (until I get Neko inputs sorted):

https://github.com/mpromonet/webrtc-streamer

[m1k1o]

For neko you could just replace all x11 dependencies with dummy functions so that you get rid of it, and then just try to setup gstreamer pipeline that captures video & audio. That way you shuld be able to acheive the first step.

[SimonBiggs]

For now I think I won't actually pick this up 😞, but I might pick this up in a few weeks. I give it ~15% chance

[SimonBiggs]

An update here:

Setting up a loopback video device with:

sudo modprobe v4l2loopback devices=1 video_nr=0 exclusive_caps=1

Then using wf-recorder:

wf-recorder --muxer=v4l2 --codec=rawvideo --file=/dev/video0 -x yuv420p -r 30 -D

Followed by this for gstreamer:

gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! x264enc tune=zerolatency ! video/x-h264,profile=baseline ! webrtcsink

Seems promising :)

[SimonBiggs]

webrtcsink is from:

git clone https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs.git

[m1k1o]

That's great news, thanks for updating. Unfortunately loading kernel module in docker is not possible. It would have to be done on host. I wonder if this is the only way or there is someone containerization friendly alternative.

[SimonBiggs]

Do you think this would work? (Given it uses gstreamer and doesn't use a loopback?)

https://github.com/ammen99/wf-recorder/wiki#streaming-rtsp-with-gstreamer

wf-recorder -a -R 48000 -o DP-2 -y -c h264_vaapi -m rawvideo -D -r 15 -f /dev/stdout | ./test-launch '( fdsrc fd=0 ! decodebin ! videoconvert ! video/x-raw,format=I420,width=1920,height=1080 ! x264enc tune="zerolatency" byte-stream=true ! rtph264pay name=pay0 pt=96 autoaudiosrc ! audioconvert ! avenc_aac ! rtpmp4gpay name=pay1 pt=97 )'