Boombox requirements

[mat-hek]

MVP

Boombox features

👍 - let's do it
👎 - not for MVP
🪝 - do it hacky

• Input and output up to one audio and video stream 👍
• Detecting stream type by protocol (like rtsp://my.stream:2137/stream) or file type (like file.mp4) 👍
• Determine when to transcode, allow force enabling and disabling transcoding 🪝
• Determine when to use real-time, allow force enabling real-time 👍
• APIs:
  • Command line 🪝
  • Elixir 👍
  • Membrane Bin 👎
  • Python 👎
• Delivery:
  • Precompiled binary, usable via cmd (👍 ), as an Elixir node or via Port (👎 )
  • Elixir dependency - should it contain the precompiled binary? plugins as optional deps? (👎 )
• Inspect metrics with KinoMembrane 👍 - if for free
• Work on Mac OS, Debian & friends (GlibC and MUSL (👎 )), and Windows via WSL (possibly without AV capture/playback)
• Drop dependency on FFmpeg (?) 👎
• Demos & examples TBD
  • Cmdline usage
  • Livebook
  • Integrating with Nx / Axon / ...
  • Integrating with Membrane Pipeline
  • Integrating with FFmpeg / some SFUs (Janus?)

Membrane features

Name	Features	Comments
H264/5	decode ✅, encode ✅, RTP ✅, MP4 ✅, CMAF ✅, FLV ✅	Encoder and decoder are FFmpeg-based
VP8	decode ⏳, encode ⏳ , RTP ✅, IVF ✅	Community POC encoder and decoder: https://github.com/spscream/membrane_vp8_ffmpeg_plugin
Opus	decode ✅, encode ✅, RTP ✅, OGG demux ✅, OGG mux ✅ , MP4 ✅	elixir_webrtc has OGG muxing and demuxing implemented
AAC	decode ✅, encode ✅, RTP ✅, MP4 ✅, CMAF ✅, FLV ✅, ADTS ✅
FLAC	decode ✅, encode ✅, save/read from file ✅
MP3	decode ✅, encode ✅, ID3 ✅, save/read from file ✅
WAV	decode ✅, encode ✅, ID3 ✅, save/read from file ✅
Raw audio	Play ✅, record ✅, resample ✅	Play & record not working on WSL, resample FFmpeg-based
Raw video	Play ✅, capture ✅, overlay ✅, scale ✅	Capture and scaling is FFmpeg-based, not sure if play and capture work on WSL
MP4	demux ✅, mux ✅
FLV	demux ✅, mux ✅
MSR	demux ✅, mux ✅	Membrane Stream Recording, used by Jellyfish
WebRTC	receive ✅ , send ✅ , WHIP/WHEP ❌	both are currently implemented in elixir_webrtc but not in the plugin
RTMP	receive ✅
HLS	send ✅
RTSP	receive ✅
HTTP	receive ✅, Send ✅
File	read ✅, Write ✅, Stdio ✅	Stdio may not work correctly with Mix.install
Elixir	receive ❌, send ❌	Send / receive a stream from an Elixir process
Thumbnails	❌	low priority
Kino Membrane	⏳	It basically works, but only in Chrome and it's not released

Roadmap

The first thing to do should be to implement a basic version of Elixir API and support for one input and one output endpoint (see the Endpoints section below). Then, the development can be split in the following phases. Tasks in each phase can be done in parallel.

1. When we have basic Elixir API and one input and output endpoint:

• Implement lacking features in Membrane
• Add more endpoints
• Create a precompiled binary and interface to it

2. When we have at least a few endpoints and more or less stable Elixir API:

• Add detecting stream type by protocol
• Add transcode and realtime options

3. When we have most endpoints implemented:

• Make sure we transcode when needed
• Implement the rest of the APIs
  • Cmdline
  • Membrane Bin
  • Python
• Integrate with KinoMembrane
• Test integration with FFmpeg / Some SFU / ?
• Create examples

Endpoints

Boombox should have multiple endpoints - inputs and outputs:

Status	Endpoint	Type	Missing features
	WebRTC	input, output	WHIP/WHEP
	RTMP	input
	HLS	output
	File	input, output
	HTTP	input, output
	RTP	input, output
	Elixir	input, output	Elixir process plugin
	Membrane pad	input, output
	RTSP	input
	Thumbnails	output	Thumbnail plugin

Elixir endpoints should support the following payload:

• Raw audio
• Raw video

HTTP, File endpoints should support the following payload:

• MP4
• OGG
• IVF
• AAC
• H264
• WAV
• Membrane Stream Recording

Post-MVP

Boombox features

• Multiple audio and video streams
• Print the code to create a Membrane pipeline that Boombox would create for a given command
• Integration with Compositor (?)
• Integration with Jellyfish (?)
• AV capture / playback on WSL

Membrane Features

Name	Features	Comments
AV1	decode ❌, encode ❌, RTP ❌, MP4 ❌
MKV	mux ❓, demux ❓	Could allow keeping VP8/9 and Opus, currently unstable
RTSP send	❌	We could have an RTSP server and no major tool has it
RTMP send	✅	FFmpeg-based
LL HLS	❌	Implemented in Jellyfish
VP9	decode ❌, encode ❌, RTP ❌, IVF ✅
Audio mix	✅	Would require support for multiple audio streams