[VideoToolbox] Force SW decoder for interlaced h264 videos on iOS/tvOS devices

[sy6sy2]

On iOS and tvOS devices VideoToolbox is not able to correctly decode some interlaced h264 videos.
But on macOS everything works fine.

This patch, found here https://patchwork.ffmpeg.org/project/ffmpeg/patch/[email protected]/#16694 (thank you @tmm1), force FFMPEG to use SW decoding in the special case "iOS/tvOS + interlaced h264 video".

This path will fix the issue given here xbmc/xbmc#18287

[lrusak]

@tmm1 can you comment on the state of the patch as I looks like it was never applied upstream?

[sy6sy2]

@lrusak, You can subscribe on #tvos-freenode, tmm1 answered for your question on it ;-)

[lrusak]

@sy6sy2 are you willing to take this patch on and try to get it merged upstream? Otherwise we will start taking on extra debt here which isn't nice in the long run.

[sy6sy2]

Not sure to correctly understand your message, but on my side I am just waiting your review and/or your merge ;-)

[fuzzard]

It's the stance that patches should be upstreamed. So what lrusak is saying is can you run with it with ffmpeg and try and get the fix/change merged there. @phunkyfish might be able to give some insight/guidance into that process, and tmm1 is available in slack

[tmm1]

As I noted in my original patch, my heuristic here is not comprehensive and will return an error on some videos which actually can be decoded by VT on iOS.

Decoding interlaced.ts actually works as expected, even though its SPS matches the other two samples which fail. This means my test is not comprehensive, and it is not possible to fully detect if a file is compatible with VideoToolbox using its SPS alone. Still, though this method produces false positives, it does not produce false negatives.

So to merge this upstream, we need either:

1. A more accurate heuristic which allows VT to decode interlaced.ts

2. A new option which allows the user to opt-in to this behavior so files like interlaced.ts don't fail to decode all of a sudden.

I am happy to review patches for either approach and merge upstream.

https://s3.amazonaws.com/tmm1/videotoolbox/interlaced.ts
https://s3.amazonaws.com/tmm1/videotoolbox/interlaced2.ts
https://s3.amazonaws.com/tmm1/videotoolbox/interlaced3.ts

$ mediainfo https://s3.amazonaws.com/tmm1/videotoolbox/interlaced.ts | grep ^Scan
Scan type                                : MBAFF
Scan type, store method                  : Interleaved fields
Scan order                               : Top Field First

$ mediainfo https://s3.amazonaws.com/tmm1/videotoolbox/interlaced2.ts | grep ^Scan
Scan type                                : Interlaced
Scan type, store method                  : Separated fields
Scan order                               : Top Field First

$ mediainfo https://s3.amazonaws.com/tmm1/videotoolbox/interlaced3.ts | grep ^Scan
Scan type                                : MBAFF
Scan type, store method                  : Separated fields
Scan order                               : Top Field First

[sy6sy2]

Thank you Fuzzard for explained me (still some issue with English...).

I have no problem to open a PR in ffmpeg repo but I'm not sure to be the right person to do that ; I have absolutely no background on video stuff :-/.
I simply copy paste the patch from tmm1 and checked if it solve the issue.

Maybe if we open a PR someone from ffmpeg will help us to have a more accurate check?

[tmm1]

ffmpeg -hide_banner -i https://s3.amazonaws.com/tmm1/videotoolbox/interlaced.ts -map v -bsf:v trace_headers -vframes 1 -c copy -f null -y /dev/null 2>&1 | egrep '(mbs|mb_)'

$ ffmpeg -i interlaced.ts ...
[AVBSFContext @ 0x7f8921e21080] 257         pic_width_in_mbs_minus1                         0000001111000 = 119
[AVBSFContext @ 0x7f8921e21080] 281         frame_mbs_only_flag                                         0 = 0
[AVBSFContext @ 0x7f8921e21080] 282         mb_adaptive_frame_field_flag                                1 = 1


$ ffmpeg -i interlaced2.ts ...
[AVBSFContext @ 0x7fcc78c1c040] 59          pic_width_in_mbs_minus1                         0000001111000 = 119
[AVBSFContext @ 0x7fcc78c1c040] 83          frame_mbs_only_flag                                         0 = 0
[AVBSFContext @ 0x7fcc78c1c040] 84          mb_adaptive_frame_field_flag                                0 = 0


$ ffmpeg -i interlaced3.ts ...
[AVBSFContext @ 0x7fcce1c0c2c0] 641         pic_width_in_mbs_minus1                         0000001111000 = 119
[AVBSFContext @ 0x7fcce1c0c2c0] 665         frame_mbs_only_flag                                         0 = 0
[AVBSFContext @ 0x7fcce1c0c2c0] 666         mb_adaptive_frame_field_flag                                1 = 1

mediainfo says interlaced.ts is "Interleaved fields", whereas the other two are "Separated fields". If we can find where in the PPS/SPS that information comes from that would help improve the heuristic.

[phunkyfish]

I have no problem to open a PR in ffmpeg repo but I'm not sure to be the right person to do that ; I have absolutely no background on video stuff :-/.
I simply copy paste the patch from tmm1 and checked if it solve the issue.

This is exactly what my first patch was like 😉

[sy6sy2]

mediainfo says interlaced.ts is "Interleaved fields", whereas the other two are "Separated fields". If we can find where in the PPS/SPS that information comes from that would help improve the heuristic.

Maybe can we ask to FFMPEG guys?

[phunkyfish]

Technically @tmm1 is an ffmpeg guy 😉

Similar to kodi there are parts of ffmpeg without maintainers. Generally the best way to get something fixed is to do some deep learning.

[sy6sy2]

I will try to understand something about that, but I do not promise anything 😜

[sy6sy2]

• Separated vs Interleaved Fields:
  • https://github.com/MediaArea/MediaInfoLib/blob/de8a7b8a27fe5e0e97b9ffdbaaf4a5fbf498e3ce/Source/MediaInfo/Multiple/File_Mpeg4_Elements.cpp#L7006-L7018
  • https://github.com/MediaArea/MediaInfoLib/blob/8b4bcb2c2251dcda3ddc4955f87c74660b83a3cf/Source/MediaInfo/Multiple/File_Mk.cpp#L4103-L4115

[sy6sy2]

So, based on tmm1's samples, we have (ATM):

• interlaced.ts --> Ok
• interlaced2.ts --> Ko
• interlaced3.ts --> Ko

This is what I have:

	interlaced.ts	interlaced2.ts	interlaced3.ts
Apple TV Kodi test	Ok	Ko	Ko
MediaInfo Scan type	MBAFF	Interlaced	MBAFF
MediaInfo Scan type, store method	Interleaved fields	Separated fields	Separated fields
frame_mbs_only_flag	0	0	0
field_pic_flag	0	1	1
mb_adaptive_frame_field_flag	1	0	1

I have absolutely no idea what these flags are supposed to describe but with a combination of both frame_mbs_only_flag and field_pic_flag we should eliminate false positives.

Edit:

Also found that (source: http://forum.doom9.org/archive/index.php/t-96598.html):

• frame_mbs_only_flag = 1 means progressive frames only.
• mb_adaptive_frame_field_flag = 1 means that pictures encoded with filed_pic_flag = 0 will use mbaff
• field_pic_flag = 1 means that the picture is encoded as two interlaced fields.

[sy6sy2]

I tried this code:

    case kCMVideoCodecType_H264 : {
        const H264Context *h = avctx->priv_data;
        const H264SliceContext *sl = h->slice_ctx;
        const int field_pic_flag = get_bits1(&sl->gb);
        if (TARGET_OS_IPHONE && h->ps.sps->frame_mbs_only_flag == 0 && field_pic_flag == 1) {
            av_log(avctx, AV_LOG_ERROR, "VideoToolbox cannot decode interlaced fields on iOS\n");
            CFRelease(avc_info);
            goto fail;
        }

But for the three interlaced.ts files this if is true... :-(

[sy6sy2]

I also found that there is a field_picture field in H264Picture struct with the ///< whether or not picture was encoded in separate fields comment.
This variable seems interesting for us but in videotoolbox_decoder_config_create function if I do:

const H264Context *h = avctx->priv_data;
const H264Picture *pic = h->cur_pic_ptr;

I have a nullptr on pic...

I have absolutely no idea how to do now :-(

[sy6sy2]

FYI: This is how VLC handle the issue: https://code.videolan.org/videolan/vlc/-/commit/547ce8c47e1abadb18d6999332762e3266ebf245
I will continue to perform test with FFMPEG...

[sy6sy2]

So, some months later iOS and tvOS Kodi users that use Live TV feature with channels using H264 interlaced stream (like in France) are still with the following issue :

• If you enable hardware video decoding in Kodi settings most of your TV channels will only show black screen.
• If you disable hardware video decoding your TV channels will work BUT if you want HW decoding for your movies you will have to manually enable hardware decoding each time (and it's a pain so you keep this button off all the time).

As we saw, it's look like hard to fix the problem upstream, so this is some solutions I thought and I would like your opinion about that:

• Solution 1: Add a second "Hardware video decoding" switch in Kodi settings especially for Live TV streams (so in iOS/tvOS case you can keep this switch off and keep the other (for movies and so on) on.
• Solution 2: Integrate MediaInfo in Kodi (only for iOS/tvOS) so that we can use it to check if the stream is interlaced or not and force SW decoding if it's the case. (Absolutely no idea if it's technically possible, especially for live stream from PVR...)

Thank you for your help!

[BTopbas]

This is still open. I have reported this issue 1 year ago... And I'm really tired to manually switch every time between LiveTV and 4k movies... If it's hard to fix deeply Solution1 or Solution2 will be very good options for now.

[phunkyfish]

This would not be our oldest bug by far. Please bear in mind that the way to fix this is to get it repaired upstream. Getting things into ffmpeg can be a very involved process.