Skip to content

Commit

Permalink
Final updates (#20)
Browse files Browse the repository at this point in the history 
* status updates

* add track composite stream test

* kill pipeline after eos timeout

* fix deadlock

* update readme

* livekit-egress -> egress

* dockerfile updates

* update testing config

* update readme again

* delete files, better track muting

* put test back

* better muting/blank frames

* even better muting

* fix opus playback with muting

* kill filewriter

* remove SkipPipeline

* fix dockerfiles

* fix room composite stream test
  • Loading branch information
@frostbyte73
frostbyte73 authored May 19, 2022
1 parent 7c90380 commit a1b7e24
Show file tree
Hide file tree
Showing 31 changed files with 989 additions and 1,150 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/buildtest.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ jobs:
~/go/bin
~/bin/protoc
~/.cache
key: livekit-egress
key: egress

- name: Build docker images
run: docker build -t livekit-egress-test -f ./build/test/Dockerfile .
run: docker build -t egress-test -f ./build/Dockerfile-test .

- name: Run tests
run: docker run --rm livekit-egress-test
run: docker run --rm egress-test
265 changes: 90 additions & 175 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,81 +8,15 @@ Depending on your request type, the egress service will either launch a web temp
(room composite requests), or it will use the sdk directly (track and track composite requests). It uses GStreamer to
encode, and can output to a file or to one or more streams.

## Capabilities
## Supported Output

### Egress Types
| Egress Type | MP4 File | OGG File | IVF File | Rtmp(s) Stream | Websocket Stream |
|-----------------|----------|----------|----------|----------------|------------------|
| Room Composite | ✅ | ✅ | | ✅ | |
| Track Composite | ✅ | ✅ | | ✅ | |
| Track | ✅ | ✅ | ✅ | | ✅ |

* Room Composite - export an entire room's video, audio, or both using a web layout
* Track Composite - transcode up to one audio track and one video track
* Track - export tracks directly, without transcoding

| Egress Type | File Output | Stream Output | Websocket Output |
|-----------------|-------------|---------------|------------------|
| Room Composite | ✅ | ✅ | |
| Track Composite | Coming soon | Coming soon | |
| Track | Coming soon | | Coming soon |

### Supported Codecs and Output Types

* Room Composite and Track Composite egress can be output to an mp4 or ogg file, or an rtmp or rtmps stream.

| Output type | H264 Baseline | H264 Main | H264 High | AAC | OPUS |
|--------------|---------------|-------------|-----------|-------------|-------------|
| mp4 file | ✅ | ✅ (default) | ✅ | ✅ (default) | ✅ |
| ogg file | | | | | ✅ (default) |
| rtmp stream | ✅ | ✅ (default) | ✅ | ✅ (default) | |
| rtmps stream | ✅ | ✅ (default) | ✅ | ✅ (default) | |

* Track egress is exported directly, without transcoding.

| Track codec | File Output |
|-------------|-------------|
| h264 | h264 file |
| vp8 | ivf file |
| opus | ogg file |

## Running locally

These changes are **not** recommended for a production setup.

To run against a local livekit server, you'll need to do the following:
* open `/usr/local/etc/redis.conf` and comment out the line that says `bind 127.0.0.1`
* change `protected-mode yes` to `protected-mode no` in the same file
* find your IP as seen by docker
* `ws_url` needs to be set using the IP as Docker sees it
* on linux, this should be `172.17.0.1`
* on mac or windows, run `docker run -it --rm alpine nslookup host.docker.internal` and you should see something like
`Name: host.docker.internal
Address: 192.168.65.2`
* your livekit-server must be run using `--node-ip` set to the above IP

These changes allow the service to connect to your local redis instance from inside the docker container.

Create a directory to mount. In this example, we will use `~/livekit-egress`.

Create a config.yaml in the above directory.
* `redis` and `ws_url` should use the above IP instead of `localhost`
* `insecure` should be set to true

```yaml
log_level: debug
api_key: your-api-key
api_secret: your-api-secret
ws_url: ws://192.168.65.2:7880
insecure: true
redis:
address: 192.168.65.2:6379
```
Then to run the service:
```shell
docker run --rm \
-e EGRESS_CONFIG_FILE=/out/config.yaml \
-v ~/livekit-egress:/out \
livekit-egress
```

You can then use our [cli](https://github.com/livekit/livekit-cli) to submit egress requests to your server.
Files can be uploaded to any S3 compatible storage, Azure, or GCP.

## API

Expand All @@ -93,76 +27,42 @@ and [JS Egress Client](https://github.com/livekit/server-sdk-js/blob/main/src/Eg
[cli](https://github.com/livekit/livekit-cli/blob/main/cmd/livekit-cli/egress.go) can be used to submit requests manually.
The API is part of LiveKit Server, which uses redis to communicate with the Egress service.

The following examples are using the Go SDK.

```go
ctx := context.Background()
egressClient := lksdk.NewEgressClient(url, apiKey, secret)
```
See our [docs](https://docs.livekit.io/guides/egress) for code examples.

### StartRoomCompositeEgress

Starts a web composite egress (uses a web template).

File example:
```go
fileRequest := &livekit.RoomCompositeEgressRequest{
RoomName: "my-room",
Layout: "speaker-dark",
Output: &livekit.RoomCompositeEgressRequest_File{
File: &livekit.EncodedFileOutput{
FileType: livekit.EncodedFileType_MP4,
Filepath: "my-room-test.mp4",
Output: &livekit.EncodedFileOutput_S3{
S3: &livekit.S3Upload{
AccessKey: "<AWS access key>",
Secret: "<AWS access secret>",
Region: "<AWS region>",
Bucket: "my-bucket",
},
},
},
},
}

info, err := egressClient.StartRoomCompositeEgress(ctx, fileRequest)
egressID := info.EgressId
```
Export an entire room's video and/or audio using a web layout rendered by chrome. Always transcoded.

* `fileRequest.Output.File.Output` can be left empty if one of `s3`, `azure`, or `gcp` is supplied with your config (see [below](#config)).
* `fileRequest.Output.File.Filepath` can be left empty, and a unique filename will be generated based on the date and room name.

Stream example:
```go
streamRequest := &livekit.RoomCompositeEgressRequest{
RoomName: "my-room",
Layout: "speaker-dark",
Output: &livekit.RoomCompositeEgressRequest_Stream{
Stream: &livekit.StreamOutput{
Protocol: livekit.StreamProtocol_RTMP,
Urls: []string{"rtmp://live.twitch.tv/app/<stream-key>"},
},
},
}

info, err := egressClient.StartRoomCompositeEgress(ctx, streamRequest)
streamEgressID := info.EgressId
```
Example use case: recording a meeting for team members to watch later.

The `File.Output` field can be left empty if one of `s3`, `azure`, or `gcp` is supplied with your config (see [below](#config)).
The `File.Filepath` field can be left empty, and a unique filename will be generated based on the date and room name.

Built-in layouts include `speaker-dark`, `speaker-light`, `grid-dark`, and `grid-light`.
To create your own web templates, see our [web README](https://github.com/livekit/egress/blob/main/web/README.md).

Egress will end when the room is closed or a StopEgress request is sent.

### StartTrackCompositeEgress

Sync and export up to one audio and one video track. Avoids transcoding when possible.

Example use case: exporting audio+video from many cameras at once during a production, for use in additional post-production.

Egress will end when the participant disconnects or stops publishing, or a StopEgress request is sent.

### StartTrackEgress

Exports a track without transcoding or processing. You can either export the file via an HTTP POST to a URL,
or streaming it via WebSocket.
Export individual tracks directly. Video tracks are not transcoded or processed, and audio tracks are decoded.

In this mode, `Content-Type` header will be set to the MIME-Type of the track. The streams will be
exported without a container. I.e. `audio/opus` for audio tracks, and `video/h264` or `video/vp8` for video tracks.
You can either export the file to cloud storage, or stream it via WebSocket.

Egress will end when the track is unpublished or a StopEgress request is sent.

#### Websocket stream

Websocket streaming is only available for audio tracks. The tracks will be exported as raw PCM data (`Content-Type` audio/x-raw).

When a `TrackEgressRequest` is started with a websocket URL, we'll initiate a WebSocket request to the desired URL.

This gives you a way of receiving a real-time stream of media from LiveKit rooms. It's helpful for when additional
Expand All @@ -185,52 +85,26 @@ And when unmuted:

The WebSocket connection will terminate when the track is unpublished (or if the participant leaves the room).

### UpdateLayout (coming soon)
### UpdateLayout

Used to change the web layout on an active RoomCompositeEgress.

```go
info, err := egressClient.UpdateLayout(ctx, &livekit.UpdateLayoutRequest{
EgressId: egressID,
Layout: "grid-dark",
})
```

### UpdateStream

Used to add or remove stream urls from an active stream

```go
info, err := egressClient.UpdateStream(ctx, &livekit.UpdateStreamRequest{
EgressId: streamEgressID,
AddOutputUrls: []string{"rtmp://a.rtmp.youtube.com/live2/<stream-key>"}
})
```
Used to add or remove stream urls from an active RoomComposite or TrackComposite stream.

### ListEgress

Used to list active egress. Does not include completed egress.

```go
res, err := egressClient.ListEgress(ctx, &livekit.ListEgressRequest{})
for _, item := range res.Items {
fmt.Println(item.EgressId)
}
```


### StopEgress

Stops an active egress.

```go
info, err := egressClient.StopEgress(ctx, &livekit.StopEgressRequest{
EgressId: egressID,
})
```

## Deployment

See our [docs](https://docs.livekit.io/deploy/egress) for more information on deploying an egress cluster.

### Config

The Egress service takes a yaml config file:
Expand Down Expand Up @@ -271,39 +145,72 @@ gcp:
The config file can be added to a mounted volume with its location passed in the EGRESS_CONFIG_FILE env var, or its body can be passed in the EGRESS_CONFIG_BODY env var.
### Version Compatibility
### Running locally
These changes are **not** recommended for a production setup.
To run against a local livekit server, you'll need to do the following:
* open `/usr/local/etc/redis.conf` and comment out the line that says `bind 127.0.0.1`
* change `protected-mode yes` to `protected-mode no` in the same file
* find your IP as seen by docker
* `ws_url` needs to be set using the IP as Docker sees it
* on linux, this should be `172.17.0.1`
* on mac or windows, run `docker run -it --rm alpine nslookup host.docker.internal` and you should see something like
`Name: host.docker.internal
Address: 192.168.65.2`

These changes allow the service to connect to your local redis instance from inside the docker container.

Create a directory to mount. In this example, we will use `~/egress-test`.

Egress is still in beta and subject to change. The following versions are compatible:
Create a config.yaml in the above directory.
* `redis` and `ws_url` should use the above IP instead of `localhost`
* `insecure` should be set to true

| livekit-egress | protocol | livekit-server | server-sdk-go | server-sdk-js | livekit-cli |
|----------------|----------|----------------|---------------|---------------|-------------|
| v0.4+ | v0.13.0+ | v0.15.7+ | v0.9.3+ | v0.5.11+ | v0.7.2+ |
```yaml
log_level: debug
api_key: your-api-key
api_secret: your-api-secret
ws_url: ws://192.168.65.2:7880
insecure: true
redis:
address: 192.168.65.2:6379
```

### Autoscaling
Then to run the service:
```shell
docker run --rm \
-e EGRESS_CONFIG_FILE=/out/config.yaml \
-v ~/egress-test:/out \
livekit/egress
```

The `livekit_egress_available` Prometheus metric is provided to support autoscaling. `prometheus_port` must be defined in your config.
You can then use our [cli](https://github.com/livekit/livekit-cli) to submit egress requests to your server.

## FAQ

### I get a `"no response from egress service"` error when sending a request

* Your livekit server cannot an egress instance through redis. Make sure they are both able to reach the same redis db.
* Each instance currently only accepts one RoomCompositeRequest at a time - if it's already in use, you'll need to deploy more instances or set up autoscaling.
* Each instance currently only accepts one RoomCompositeRequest at a time - if it's already in use,
* you'll need to deploy more instances or set up autoscaling.

### I get a different error when sending a request

* Make sure your egress, livekit-server, server-sdk-go, server-sdk-js, and livekit-cli are compatible (see [version compatibility](#version-compatibility)).
* Make sure your egress, livekit, server-sdk-go, server-sdk-js, and livekit-cli repos and deployments are all up to date.

### I'm getting a broken mp4 file
### I'm getting a broken (0 byte) mp4 file

* GStreamer needs to be properly shut down - if the process is killed, the file will be unusable. Make sure you're stopping the recording with a `StopEgressRequest`.
* This is caused by the process being killed - GStreamer needs to be properly shut down to close the file.
* Make sure your instance has enough CPU and memory, and is being stopped correctly.

### I'm seeing GStreamer warnings/errors. Is this normal?

* `GStreamer-CRITICAL **: 20:22:13.875: gst_mini_object_unref: assertion 'GST_MINI_OBJECT_REFCOUNT_VALUE (mini_object) > 0' failed`
* Occurs during audio-only egress - this is a gst bug, and is safe to ignore.
* `WARN flvmux ... Got backwards dts! (0:01:10.379000000 < 0:01:10.457000000)`
* Occurs when streaming to rtmp - safe to ignore. These warnings occur due to live sources being used for the flvmux. The dts difference should be small (under 150ms).
* Occurs when streaming to rtmp - safe to ignore. These warnings occur due to live sources being used for the flvmux.
The dts difference should be small (under 150ms).

### Can I run this without docker?

Expand All @@ -313,17 +220,25 @@ The `livekit_egress_available` Prometheus metric is provided to support autoscal

## Testing and Development

Running `mage test` will run the test suite on your machine, and will dump the resulting files into livekit-egress/test.
Running `mage test` will run the Room Composite tests on your machine, and will dump the resulting files into egress/test/output.

To run these tests against your own LiveKit rooms, a deployed LiveKit server with a secure websocket url is required.
First, create a `livekit-egress/test/config.yaml`:
First, create `egress/test/config.yaml`:

```yaml
log_level: info
log_level: debug
api_key: your-api-key
api_secret: your-api-secret
ws_url: your-wss-url
ws_url: wss://your-livekit-url.com
room_name: your-room
room: true
track_composite: true
track: true
file: true
stream: true
muting: false
gst_debug: 1
```

Join a room using https://example.livekit.io/#/ or your own client, then run `LIVEKIT_ROOM_NAME=my-room mage integration test/config.yaml`.
Join a room using https://example.livekit.io/#/ or your own client, then run `mage integration test/config.yaml`.
This will test recording different file types, output settings, and streams against your room.
Loading

0 comments on commit a1b7e24

Please sign in to comment.