Use media_kit to play videos

[deckerst]

(originally explored as part of #555)

The good:

• improved seek accuracy (fixing bug: Skipping in Videos is Widely Divided #660 and making it possible to implement Fast forward/backward on long press video sides #262),
• improved looping transition (but not fixing Seemless loops #356, cf [Enhancement] seamless loop media-kit/media-kit#150 loop-file: add seamless-looping option to ensure that there are no delayed frames at beginning of each loop mpv-player/mpv#10748),
• improved speed selection (works fine at least from x0.25 to x4,
• HDR support (fixing HDR videos look pale #486),
• AV1 support (fixing AV1 video codec support #555),
• some corrupted videos become playable (e.g. mp4 with inconsistent duration in their metadata).

The bad:

• often fails to load the first frame of local videos (cf [Android] First frame is sometimes a solid color media-kit/media-kit#339)

Flavor	Package	media_kit v1.1.1 + ffmpeg_kit v5.1.0-LTS (min)	media_kit v1.1.1 + ijk	ijk only
play	aab	83.6	63.7	47.4
izzy	armeabi-v7a	34.2	23.1	18.1
izzy	arm64-v8a	27.5	23.1	17.9
izzy	x86_64	29.3	24.7	18.7

[deckerst]

@alexmercerind

1. subtitles

• Using the subtitle stream works fine, but the stream only contains the unstyled text. Not a big deal, but it's a step down from the subtitle stream I was receiving from ijkplayer, where things like bold, italics, etc. was inlined with the text.

• I currently style SubtitleViewConfiguration with a transparent color to hide the embedded subtitle view, but it would be cleaner if there was a way to remove the SubtitleView from the widget tree altogether (and without using libass).

2. screenshot

It works fine!

3. codec selection

I appreciate that you reduced the library size by tightening the codec selection. However, I can no longer play some of my video/audio streams (e.g. avi, ac3). Is it possible for consuming apps to customize which codecs are bundled? Should I rebuild mpv, and then fork media_kit to point to that custom build?

4. errors

• I get the following exception when destroying video views (loaded from Android content URIs). It seems final fd = int.parse(data.pathSegments.last) is assuming pathSegments are not empty. But it may be a deeper issue.

E/flutter ( 3507): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Bad state: No element
E/flutter ( 3507): #0      _Array.last (dart:core-patch/array.dart:56:5)
E/flutter ( 3507): #1      Media._finalizer.<anonymous closure> (package:media_kit/src/models/media/media_native.dart:47:48)
E/flutter ( 3507): #2      _RootZone.runUnaryGuarded (dart:async/zone.dart:1594:10)
E/flutter ( 3507): #3      _RootZone.bindUnaryCallbackGuarded.<anonymous closure> (dart:async/zone.dart:1633:26)
E/flutter ( 3507): #4      _FinalizerImpl._runFinalizers (dart:core-patch/finalizer_patch.dart:63:18)
E/flutter ( 3507): #5      _FinalizerImpl._handleFinalizerMessage (dart:core-patch/finalizer_patch.dart:76:15)

• last but not least, I can still make my app crash by quickly going through my videos (after about 30 videos? maybe related to the dispose cleanup issue above). I'm willing to provide additional details, just let me know how I can help you debug this.

[alexmercerind]

Replacing mediacodec to mediacodec-copy can improve stability to some degree. The VideoControllerConfiguration allows this. An overall efficient approach will be to reuse existing Players (if possible). I'll ask for logcat; we have debug symbols now.

I'll have a look at the rest. I possibly messed up the Media GC finalizer.

EDIT: media-kit/media-kit@38d2bc9

[alexmercerind]

I did enable AVI. Something seems missing.

EDIT: An AVI sample I grabbed does play.

[deckerst]

I did enable AVI. Something seems missing.

EDIT: An AVI sample I grabbed does play.

My bad, I was imprecise: the AVI container I tried contained an MJPEG video stream. This is niche, but useful to me. Your codec selection is reasonable for most users, that's why I was asking how to customize it for my needs.

Edit: from libmpv logs:

PlayerLog(prefix: vd, level: error, text: Failed to initialize a decoder for codec 'theora'.)
PlayerLog(prefix: vd, level: error, text: Failed to initialize a decoder for codec 'msmpeg4v3'.)
PlayerLog(prefix: ad, level: error, text: Failed to initialize a decoder for codec 'wmav2'.)
PlayerLog(prefix: vd, level: error, text: Failed to initialize a decoder for codec 'mjpeg'.)
PlayerLog(prefix: ad, level: error, text: Failed to initialize a decoder for codec 'ac3'.)
PlayerLog(prefix: ad, level: error, text: Failed to initialize a decoder for codec 'eac3'.)

For me ac3, eac3 and mjpeg are important, the rest not so.

[alexmercerind]

You can create your own builds for libmpv + FFmpeg + media_kit by:

1. Fork https://github.com/media-kit/libmpv-android-video-build & enable GitHub actions.
2. Modify the FFmpeg configure step.
   It may look overwhelming (it is in-fact). Mostly you need to only look at the --enable-decoder, --enable-demuxer & --enable-parser for your codec selection; other things are configured already.
3. The GitHub actions will compile the project(s) & resulting binaries will be automatically uploaded as release & artifact.
4. Remove media_kit_libs_android_video from pubspec.yaml.
5. Bundle your *.jar(s) with your application by modifying build.gradle, something like this.

[alexmercerind]

I picked the FFmpeg configuration from fijkplayer; kind of surprising things are different. Still, you have the freedom.

[deckerst]

I picked the FFmpeg configuration from fijkplayer; kind of surprising things are different. Still, you have the freedom.

I customized the FFmpeg config for Aves, so it's not the same as the basic fijkplayer.

[alexmercerind]

Interesting.

For the note, you won't need image encoders for screenshot; we are encoding pixel-buffer with package:image.

[deckerst]

we are encoding pixel-buffer with package:image

I noticed that the screenshot function is a tad slow. Not a big deal, but maybe it's because you use package:image, which is pure dart and usually slower compared to native encoders. That package is actually more suitable for server usage, or when performance are no concern. On Android for example, you may use instead android.graphics.Bitmap and its compress(Bitmap.CompressFormat.JPEG, ...).

[alexmercerind]

It runs on Isolate, so it may be slow (but definitely non-blocking (doesn't freeze UI) unlike native interop); portable too.

The performance isn't really crucial for screenshots (not at all). When we want hardware-level performance at the time of rendering, we don't even use pixel buffers.

[deckerst]

I was not talking about native in the sense of FFI, but rather to use Android JPEG encoder instead of the slow package:image. And of course it's non-blocking when done on the appropriate threads, but being slow means you don't know whether it worked successfully after a long time.

[alexmercerind]

The performance gain from using the Android JPEG encoder may be lesser than sending pixel-buffer across platform-channels/FFI & then receiving back encoded image buffer.

screenshot-to-file will use FFmpeg. I'm not really going to invest time into this at the moment.

[Quackdoc]

do you have a sample mkv that is unplayable that you would be willing to share? I have not been able to replicate the issue.

[deckerst]

@Quackdoc please contact me by email. By the way, that video does not play well with the vanilla MPV player app either.

[alexmercerind]

Hi!

Apologies for the day; catching up again.

• SubtitleViewConfiguration.visible can now be used to enable/disable internal subtitle rendering in Video.
• The file-descriptor close crash should now be resolved.
• I added the codecs you specified above & few more to package:media_kit_libs_android_video itself.

I think it's all green now. This update will make a lot of things better (especially for Android).

---

As for decoders/demuxers: I noticed I had some of these videos. I don't think anything else is left... I also added Microsoft's msmpeg* & wmv*. Chromium's Audio/Video page also has theora specified (even though our selection is far bigger); which I missed.

Still... I have attached full variants to the release above; in-case you want them instead.

[alexmercerind]

I monitored the screenshot performance:

I can make it use a native C/C++ JPEG/PNG encoder based on stb_image if you wish.
For the time being, I exposed raw pixel-buffer which you can use to manually encode (main branch).

pixel-buffer = 57ms
pixel-buffer = 21ms
pixel-buffer = 35ms
pixel-buffer = 36ms
pixel-buffer = 34ms
pixel-buffer = 33ms
pixel-buffer = 30ms
pixel-buffer = 39ms
pixel-buffer = 30ms
pixel-buffer = 41ms
JPEG         = 525ms
JPEG         = 489ms
JPEG         = 442ms
JPEG         = 481ms
JPEG         = 385ms
JPEG         = 369ms
JPEG         = 381ms
JPEG         = 425ms
JPEG         = 451ms
JPEG         = 403ms
PNG          = 2679ms
PNG          = 3392ms
PNG          = 3211ms
PNG          = 2814ms
PNG          = 2713ms
PNG          = 2759ms
PNG          = 2559ms
PNG          = 2156ms
PNG          = 2394ms
PNG          = 2441ms

[alexmercerind]

What kind of additional metadata are you exactly looking for?

[deckerst]

Am i looking for additional metadata? What makes you ask this?

At this stage, the last hurdle with your library is that the first frame of each video is a plain solid color frame, but that aside I'm quite happy with your library!

[alexmercerind]

I mean why do you require separate FFmpeg or IJK?

Yeah, I also saw that issue on Android. I thought it was resolved after this:

At this stage, the last hurdle with your library is that the first frame of each video is a plain solid color frame.

https://github.com/media-kit/media-kit/blob/5ea2c84b5c7b074a45b3887dc64a997f9a5e3172/media_kit_video/lib/src/video_controller/android_video_controller/real.dart#L146-L148

EDIT: To be honest, increasing the delay to be slightly higher can be best solution for now (you can provide me with a value). My device doesn't show solid color frame anymore. For the cause, initially SurfaceView is probably 1x1 size.

[deckerst]

I mean why do you require separate FFmpeg or IJK?

Aves shows stream details, etc. It's not just a viewer.

https://github.com/media-kit/media-kit/blob/5ea2c84b5c7b074a45b3887dc64a997f9a5e3172/media_kit_video/lib/src/video_controller/android_video_controller/real.dart#L146-L148

I don't mind the delay, but couldn't the surface be transparent while it's initializing?

[alexmercerind]

You mean that solid color is black (or ?)

[alexmercerind]

I mean it should be possible to show stream details within media_kit. There's a large list of properties:
https://mpv.io/manual/stable/

[deckerst]

You mean that solid color is black (or ?)

No, it depends on the video. When the player initializes, it shows as the first frame a solid color which can be pink, brown, purple, etc. It seems derived from the video, and it's always the same color for a given video.

I mean it should be possible to show stream details within media_kit.

Maybe it's possible with MPV, but I didn't see an API in your package to retrieve the same info as I get from ffprobe.

Also, I don't need it now but ffmpeg kit could allow Aves to edit videos someday.

[alexmercerind]

Just try to increase that delay & provide me with a value that works fine. That will avoid that first glitchy frame.

I personally don't see that first uni-color 1x1 texture frame after that delay.

[deckerst]

Cf media-kit/media-kit#339

[alexmercerind]

I'd be really glad if we're able to avoid package:ffmpeg_kit_flutter. Is it really a good deal for it's size?

You can download mpv on your macOS/PC & try out different properties/options, I think you'll find something relevant (if not all, certainly to some extent).

[deckerst]

I am just a consumer of media_kit. If its API allows probing and editing like ffmpeg_kit_flutter I won't have to use that package, but until then I do. Please do not get distracted with Aves needs, and follow your own roadmap.

[alexmercerind]

It won't really take much effort from my end to expose an API that allows listening to native property changes like we already provide an API to make native property changes. It'd be up to you whichever you wish to listen to & display (but you have to find them out).

---

The ultimate goal of media_kit is to meet the community's demands.

Now things mostly work... improving performance for other platforms & some features like caching etc. are still in interest.

Our new Android support has been well-tested by your project & now it is at its sweet spot. This stability can now be enjoyed by others, that's open-source. From a personal usage stand-point, I only created package:media_kit for my music app & wasn't even going to add video support.

Lastly, public contributions are also welcomed :)

---

At your project level, you may even consider forking our media-kit/libmpv-android-video-build & create a build that dynamically links to FFmpeg (instead of static, like we have to avoid conflicts). Then, you can use that "shared" FFmpeg to query metadata (while it'll be used by media_kit too). With our compilation optimizations in-place, the resulting bundle size is also decent (~5 MB for ARM64).

[alexmercerind]

Few more updates on my part:

• #350 : feat: NativePlayer.observeProperty & NativePlayer.unobserveProperty

For allowing you to listen to custom properties.

import 'package:media_kit/media_kit.dart';

Future<void> main() async {
  MediaKit.ensureInitialized();
  final player = Player();
  await (player.platform as NativePlayer).observeProperty(
    'demuxer-cache-state',
    (data) async {
      print(data);
    },
  );
  await player.open(Media('https://user-images.githubusercontent.com/28951144/229373695-22f88f13-d18f-4288-9bf1-c3e078d83722.mp4'));
}

• #355 : build(windows/android/macOS/iOS): bump dependencies

Fixed support for AV1.

• #358 : feat: VideoTrack/AudioTrack/SubtitleTrack expose more parameters

For retrieving video, audio & subtitle streams data.

VideoTrack(auto, null, null, image: null, albumart: null, codec: null, decoder: null, w: null, h: null, channelscount: null, channels: null, samplerate: null, fps: null, bitrate: null, rotate: null, par: null, audiochannels: null)
VideoTrack(no, null, null, image: null, albumart: null, codec: null, decoder: null, w: null, h: null, channelscount: null, channels: null, samplerate: null, fps: null, bitrate: null, rotate: null, par: null, audiochannels: null)
VideoTrack(1, null, null, image: false, albumart: false, codec: h264, decoder: null, w: 1920, h: 1080, channelscount: null, channels: null, samplerate: null, fps: 60.0, bitrate: null, rotate: null, par: null, audiochannels: null)
VideoTrack(2, null, null, image: false, albumart: false, codec: h264, decoder: null, w: 1280, h: 720, channelscount: null, channels: null, samplerate: null, fps: 60.0, bitrate: null, rotate: null, par: null, audiochannels: null)
VideoTrack(3, null, null, image: false, albumart: false, codec: h264, decoder: null, w: 854, h: 480, channelscount: null, channels: null, samplerate: null, fps: 30.0, bitrate: null, rotate: null, par: null, audiochannels: null)
VideoTrack(4, null, null, image: false, albumart: false, codec: h264, decoder: null, w: 640, h: 360, channelscount: null, channels: null, samplerate: null, fps: 30.0, bitrate: null, rotate: null, par: null, audiochannels: null)
VideoTrack(5, null, null, image: false, albumart: false, codec: h264, decoder: null, w: 426, h: 240, channelscount: null, channels: null, samplerate: null, fps: 30.0, bitrate: null, rotate: null, par: null, audiochannels: null)
VideoTrack(6, null, null, image: false, albumart: false, codec: h264, decoder: null, w: 256, h: 144, channelscount: null, channels: null, samplerate: null, fps: 30.0, bitrate: null, rotate: null, par: null, audiochannels: null)
VideoTrack(7, null, null, image: false, albumart: false, codec: h264, decoder: null, w: 256, h: 144, channelscount: null, channels: null, samplerate: null, fps: 15.0, bitrate: null, rotate: null, par: null, audiochannels: null)
AudioTrack(auto, null, null, image: null, albumart: null, codec: null, decoder: null, w: null, h: null, channelscount: null, channels: null, samplerate: null, fps: null, bitrate: null, rotate: null, par: null, audiochannels: null)
AudioTrack(no, null, null, image: null, albumart: null, codec: null, decoder: null, w: null, h: null, channelscount: null, channels: null, samplerate: null, fps: null, bitrate: null, rotate: null, par: null, audiochannels: null)
AudioTrack(1, null, null, image: false, albumart: false, codec: aac, decoder: null, w: null, h: null, channelscount: 2, channels: stereo, samplerate: 44100, fps: null, bitrate: 1172, rotate: null, par: null, audiochannels: 2)
AudioTrack(2, null, null, image: false, albumart: false, codec: aac, decoder: null, w: null, h: null, channelscount: 2, channels: stereo, samplerate: 44100, fps: null, bitrate: 443, rotate: null, par: null, audiochannels: 2)
AudioTrack(3, null, null, image: false, albumart: false, codec: aac, decoder: null, w: null, h: null, channelscount: 2, channels: stereo, samplerate: 44100, fps: null, bitrate: 278, rotate: null, par: null, audiochannels: 2)
SubtitleTrack(auto, null, null, image: null, albumart: null, codec: null, decoder: null, w: null, h: null, channelscount: null, channels: null, samplerate: null, fps: null, bitrate: null, rotate: null, par: null, audiochannels: null)
SubtitleTrack(no, null, null, image: null, albumart: null, codec: null, decoder: null, w: null, h: null, channelscount: null, channels: null, samplerate: null, fps: null, bitrate: null, rotate: null, par: null, audiochannels: null)