Encrypted video, sent over rtp + udp, requires specific 'order' of receiver launch

[inobelar]

I'm testing realtime streaming capabilites, and discovered, that sometimes, we cannot listen to video stream, if its not pre-launched. I uploaded annotated videos to YouTube, due to GitHub restritions for attachments size (max 10 mb).

For quicker consumption, sender's stream configured for:

• sending full frames each 15 frames, x264enc::key-int-max=15
• sending config each 1 second: rtph264pay::config-interval=1

I recorded 2 videos (used "pr4-test-fix" branch):

1. H264 + rtp + udp (localhost), without encryption: https://youtu.be/QPSbk_f_LeA
   • This video show, that 'Receiver' & 'Sender' can be launched in any order, its important.
   • This behaviour I assume as 'reference'/'expected'/'needed'.
   • Works without rtpjitterbuffer well, as expected on 'localhost', without many packets lost.
2. H264 + rtp + udp (localhost), with enctyption: https://youtu.be/qCyWFBbGNhw
   • This video show the next issues:
     • Without and with rtpjitterbuffer, 'Receiver' cannot be launched after 'Sender' (~immediately), only after pre-launched 'Sender' - its important current disadvantage.
     • Without rtpjitterbuffer video shown with glitches
   • As side-efect, due to usage of rtpjitterbuffer, strange lag or video buffering was added.

[ReFormationPro]

Thanks for testing this plugin and sharing your findings again.

• Error at the start of video 2 is GST_H264_PARSER_BROKEN_LINK, whose description in GStreamer is "The link to structure needed for the parsing couldn't be found". Nal parser has not seen the config yet and thus it is unable to parse this slice iirc. Instead of giving an error and exiting, the plugin can try to ignore the slices it cannot parse. It will eventually read the config and become able to parse them. I think this is what the rest of the pipeline elements, ie. decoder, do. Maybe putting h264parse after rtph264depay can help. I think it drops the frames that are not parsable due to this reason.
• I cannot think of any reason for rtpjitterbuffer to have this effect. I would like to test it myself a bit. Can you share the pipelines to save me from typing them?
• There seems to be quite a few things wrong with strange lag issue. First, after the receiver disconnects, the sender jumps ahead in time. By default, timeoverlay is in buffer-time mode and uses PTS of the buffer it receives for the overlay. Hence, maybe too many buffers are produced after the receiver disconnects. These too many buffers maybe queued and this may lead to the receiver keeping to play after the sender disconnects. I cannot confidently say what could cause this. On h264encrypt's part, I realized I am not explicitly setting PTS / DTS / DUR and not sure if the base class does. I need to check this.

[inobelar]

@ReFormationPro Sorry, my mistake. I should have attached the text to the video right away, I checked my email quite late. Thanks again for your quick response!

without_encryption.txt

                            WITHOUT ENCRYPTION
SENDER:

  GST_DEBUG=3 \
  gst-launch-1.0 --verbose \
      videotestsrc do-timestamp=true pattern=ball ! \
          "video/x-raw, width=640, height=480, framerate=30/1" ! \
      timeoverlay ! \
      x264enc \
          speed-preset=ultrafast \
          tune=zerolatency \
          key-int-max=15 ! \
      rtph264pay config-interval=1 ! \
      udpsink \
          host=127.0.0.1 \
          port=5000

RECEIVER:

  GST_DEBUG=3 \
  DISPLAY=:0.0 \
  gst-launch-1.0 --verbose \
      udpsrc \
          port=5000 ! \
          "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! \
      rtph264depay ! \
      avdec_h264 ! \
      videoconvert ! \
      ximagesink \
          sync=false

with_encryption.txt

                             WITH ENCRYPTION
SENDER:

  GST_PLUGIN_PATH=/home/vuser/work/Gstreamer-H264-Encryption/build \
  GST_DEBUG=3 \
  gst-launch-1.0 --verbose \
      videotestsrc do-timestamp=true pattern=ball ! \
          "video/x-raw, width=640, height=480, framerate=30/1" ! \
      timeoverlay ! \
      x264enc \
          speed-preset=ultrafast \
      	  tune=zerolatency \
      	  key-int-max=15 ! \
      h264encrypt \
          iv-seed=1869052520 \
          key=01234567012345670123456701234567 \
          encryption-mode=aes-cbc ! \
      rtph264pay config-interval=1 ! \
      udpsink \
          host=127.0.0.1 \
          port=5000

RECEIVER:

  GST_PLUGIN_PATH=/home/vuser/work/Gstreamer-H264-Encryption/build \
  GST_DEBUG=3 \
  DISPLAY=:0.0 \
  gst-launch-1.0 --verbose \
      udpsrc \
          port=5000 ! \
          "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! \
      rtph264depay ! \
      h264decrypt \
          key=01234567012345670123456701234567 \
          encryption-mode=aes-cbc ! \
      avdec_h264 ! \
      videoconvert ! \
      ximagesink \
          sync=false

[inobelar]

"The link to structure needed for the parsing couldn't be found".

Maybe putting h264parse after rtph264depay can help

Wow! I didn't notice this many times! Thanks, I added h264parse after rtph264depay, and it fixed 'major problem of the issue' (rtpjitterbuffer is not too critical for me, but lag/buffering kinda annoying), but if you say, it can be fixed too - its awesome! 🎉

In general, I find this plugin of yours to be a gem! 💎 Excellently and clearly written (if reader take the trouble to delve into the domain a little), it works, almost flawlessly, 95% of flawlessly :)

It became for me both an excellent reading, an introduction to encryption and h264 low-level format (nal-units), and a cool reference for writing plugins! That's why I would like to "make it as cool as possible". I would like to fix all the bugs myself, but I'm not that cool yet 😆 As they say - "I would do everything myself if I could", but for now I'm meticulously testing it, understanding the nuances, so that later I can improve the documentation, explain the implementation more clearly for other beginners (like me).

It seems to me that this plugin is a powerful tool 💪 for those who need it.

[ReFormationPro]

Thanks for your kind comments again and thank you also for testing and bringing issues. It is nice to hear that this project is beneficial. It would be nice if this plugin had better documentation, faq etc. I will be considering improving those aspects. Good luck with your work too.