[Camera Support]: Hikvision doorbell in Frigate. Either jumpy video, or lagging and choppy audio

[fedd]

Describe the problem you are having

I'm guessing it's the timestamps problem, however I a complete noob and my expertise lies in other fields. Currently I'm having the audio lagging for several seconds behind the video, the lag seems growing over time. Also, the sound makes annoying micropauses every couple of seconds (probably every 50 frames but sometimes not).

Previously I've tried different setups including using the substreams for object detection, restreaming the main stream in go2rtc as it was and adding aac stream just before recording, but the result was SO bad as when the 10(?) second video chunks were only recorded partially and the MSE player cursor was skipping many seconds of the timeline every 10 seconds, I was imagining how out of sync the videostream timestamps were. Other experiments gave me green flashes every couple of seconds in the live view. Those errors mostly were appearing after a several hours of uptime.

In the doorbell itself I've also tried in desperation to change the source audio codec from alaw to ulaw or vice versa, and tomake the video bitrate constant, to no avail. Constant (and large) bitrate seemed to make the player itself to lag, but I may be wrong.

My current setup adds aac audio in ro2rtc and supposedly fixes the timestamps with the /#async preset, copying the video supposedly intact. I've also decided to use the big screen to run the detection against, to capture cats at last, and also to avoid any extra sync problems. It's the best result so far, I need it to work for a couple of days to measure audio the lag, and if it is nearly constant will be hacking some delay for the videostream. I've tried to add /#video-h264 just to (imaginably) fix the timestamps in the video stream too, but that did'nt help and I would like to avoid to reencode with the same codec.

However I suspect this is a known problem with the Hikvision doorbells, though the search only gave me the /#async advice. Maybe someone happens to know how to fix the audio lag problem?

Version

0.13.2-6476f8a

Frigate config file

mqtt:
  enabled: true
  host: 192.168.9.10

#logger:
#  logs:
#    frigate.record.maintainer: debug  

ffmpeg:
  hwaccel_args: preset-nvidia-h264
  global_args: -hide_banner -loglevel warning -threads 6

detect:
  enabled: true
  fps: 5
#  height: 720
#  width: 1280

record:
  enabled: true
  retain: 
    days: 10
    mode: motion

snapshots:
  enabled: true

detectors:
  coral:
    type: edgetpu
    device: pci

objects:
  track:
  - person
  - car
  - cat
  - bicycle
  - motorcycle
  - dog
  - horse
  - sheep
  - cow
  - bear
  - bird

birdseye:
  enabled: True
  mode: continuous

go2rtc:
  streams:
    frontdoorphone:
      - rtsp://admin:[email protected]/Streaming/Channels/101/
      - isapi://admin:[email protected]:80/
    frontdoor:
      - "ffmpeg:frontdoorphone#async#video=copy#audio=aac#hardware=cuda"
    backleftout:
      - rtsp://user:[email protected]/cam/realmonitor?channel=1&subtype=0

cameras:
  frontdoor:
    enabled: true
    audio: 
      enabled: true
      min_volume: 100
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
      - path: rtsp://127.0.0.1:8554/frontdoor?video&audio
        input_args: preset-rtsp-restream
        roles:
        - record
        - audio
        - detect
    zones:  
      cul_de_sac:
        coordinates: 69,860,330,1026,655,1014,909,976,1255,839,1408,742,1341,630,1252,586,1094,590,916,591,650,596,524,621,458,652,218,699,94,772
    motion:
      mask:    
      - 0,0,1713,0,1696,379,0,382
      - 1355,871,1748,880,1743,839,1362,837

  backleftout:
    enabled: true
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/backleftout
        input_args: preset-rtsp-restream
        roles:
        - record
        - detect
    motion:
      mask:
      - 1120,1440,0,1440,0,0,980,0,1098,563,1133,1034

Relevant log output

2024-07-18 08:19:45.797184240  [INFO] Preparing Frigate...
2024-07-18 08:19:45.806738101  [INFO] Starting Frigate...
2024-07-18 08:19:46.915164720  [2024-07-18 08:19:46] frigate.app                    INFO    : Starting Frigate (0.13.2-6476f8a)
2024-07-18 08:19:52.879960378  [2024-07-18 08:19:52] peewee_migrate.logs            INFO    : Starting migrations
2024-07-18 08:19:52.946749390  [2024-07-18 08:19:52] peewee_migrate.logs            INFO    : There is nothing to migrate
2024-07-18 08:19:52.952793089  [2024-07-18 08:19:52] frigate.app                    INFO    : Recording process started: 1425
2024-07-18 08:19:52.955255676  [2024-07-18 08:19:52] frigate.app                    INFO    : go2rtc process pid: 89
2024-07-18 08:19:52.978535589  [2024-07-18 08:19:52] detector.coral                 INFO    : Starting detection process: 1435
2024-07-18 08:19:52.980927583  [2024-07-18 08:19:52] frigate.app                    INFO    : Output process started: 1437
2024-07-18 08:19:52.994634455  [2024-07-18 08:19:52] frigate.app                    INFO    : Camera processor started for frontdoor: 1444
2024-07-18 08:19:52.998110652  [2024-07-18 08:19:52] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci
2024-07-18 08:19:52.999213829  [2024-07-18 08:19:52] frigate.app                    INFO    : Camera processor started for backleftout: 1452
2024-07-18 08:19:53.003062223  [2024-07-18 08:19:52] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2024-07-18 08:19:53.003250694  [2024-07-18 08:19:53] frigate.app                    INFO    : Capture process started for frontdoor: 1457
2024-07-18 08:19:53.007562484  [2024-07-18 08:19:53] frigate.app                    INFO    : Capture process started for backleftout: 1462
2024-07-18 08:19:53.011864231  [2024-07-18 08:19:53] frigate.app                    INFO    : Audio process started: 1466

FFprobe output from your camera

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"0/0","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":1080,"width":1920},{"avg_frame_rate":"0/0","codec_long_name":"AAC(AdvancedAudioCoding)"}]}}]

Frigate stats

No response

Operating system

Debian

Install method

Docker Compose

Object Detector

Coral

Network connection

Wired

Camera make and model

Hikvision DS-KV6113-WPE1(B)

Any other information that may be helpful

/opt/frigate# ffprobe -v quiet  -print_format json -show_streams "rtsp://admin:[email protected]/Streaming/Channels/101/"
{
    "streams": [
        {
            "index": 0,
            "codec_name": "h264",
            "codec_long_name": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10",
            "profile": "Baseline",
            "codec_type": "video",
            "codec_tag_string": "[0][0][0][0]",
            "codec_tag": "0x0000",
            "width": 1920,
            "height": 1080,
            "coded_width": 1920,
            "coded_height": 1080,
            "closed_captions": 0,
            "film_grain": 0,
            "has_b_frames": 0,
            "pix_fmt": "yuvj420p",
            "level": 42,
            "color_range": "pc",
            "color_space": "bt709",
            "color_transfer": "bt709",
            "color_primaries": "bt709",
            "chroma_location": "left",
            "field_order": "progressive",
            "refs": 1,
            "is_avc": "false",
            "nal_length_size": "0",
            "r_frame_rate": "25/1",
            "avg_frame_rate": "25/1",
            "time_base": "1/90000",
            "start_pts": 7200,
            "start_time": "0.080000",
            "bits_per_raw_sample": "8",
            "extradata_size": 28,
            "disposition": {
                "default": 0,
                "dub": 0,
                "original": 0,
                "comment": 0,
                "lyrics": 0,
                "karaoke": 0,
                "forced": 0,
                "hearing_impaired": 0,
                "visual_impaired": 0,
                "clean_effects": 0,
                "attached_pic": 0,
                "timed_thumbnails": 0,
                "captions": 0,
                "descriptions": 0,
                "metadata": 0,
                "dependent": 0,
                "still_image": 0
            }
        },
        {
            "index": 1,
            "codec_name": "pcm_alaw",
            "codec_long_name": "PCM A-law / G.711 A-law",
            "codec_type": "audio",
            "codec_tag_string": "[0][0][0][0]",
            "codec_tag": "0x0000",
            "sample_fmt": "s16",
            "sample_rate": "8000",
            "channels": 1,
            "channel_layout": "mono",
            "bits_per_sample": 8,
            "r_frame_rate": "0/0",
            "avg_frame_rate": "0/0",
            "time_base": "1/8000",
            "start_pts": 0,
            "start_time": "0.000000",
            "bit_rate": "64000",
            "disposition": {
                "default": 0,
                "dub": 0,
                "original": 0,
                "comment": 0,
                "lyrics": 0,
                "karaoke": 0,
                "forced": 0,
                "hearing_impaired": 0,
                "visual_impaired": 0,
                "clean_effects": 0,
                "attached_pic": 0,
                "timed_thumbnails": 0,
                "captions": 0,
                "descriptions": 0,
                "metadata": 0,
                "dependent": 0,
                "still_image": 0
            }
        }
    ]
}

[NickM-27]

sounds like you have already tried most things that I would suggest

[fedd]

What does it mean "fps unknown" though? Camera settings tell it's 25 but it doesn't seem to be stated in the stream metadata, does it?

I've also tried to add -fpsmax to the go2rtc ffmpeg #video=h264... command, afair it wasn't any good

Would update the doorbell firmware though the release notes don't mention anything codec related

[xavan83]

"ffmpeg:rtsp://admin:[email protected]/Streaming/Channels/101/#input=rtsp/udp#video=copy#audio=aac#hardware

Still issue after about 1hour

2024-07-19 08:16:01.178771989 [INFO] Preparing new go2rtc config...
2024-07-19 08:16:02.027544462 [INFO] Starting go2rtc...
2024-07-19 08:16:02.363567795 08:16:02.363 INF go2rtc platform=linux/amd64 revision=b2399f3 version=1.9.2
2024-07-19 08:16:02.363574767 08:16:02.363 INF config path=/dev/shm/go2rtc.yaml
2024-07-19 08:16:02.365183719 08:16:02.365 INF [rtsp] listen addr=:8554
2024-07-19 08:16:02.365576128 08:16:02.365 INF [api] listen addr=:1984
2024-07-19 08:16:02.365725927 08:16:02.365 INF [webrtc] listen addr=:8555/tcp
2024-07-19 08:16:11.174679484 [INFO] Starting go2rtc healthcheck service...
2024-07-19 09:39:17.670953093 09:39:17.666 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error=EOF url=ffmpeg:rtsp://admin:[email protected]/Streaming/Channels/101/#input=rtsp/udp#video=copy#audio=aac#hardware

[fedd]

"ffmpeg:rtsp://admin:[email protected]/Streaming/Channels/101/#input=rtsp/udp#video=copy#audio=aac#hardware

Still issue after about 1hour

I had major issues after it worked for a night: the out of sync went so badly the video is barely watchable, see screencast. The clock jumps, the car jumps, even the playhead jumps.

Screencast.from.19.07.2024.09.31.23.webm

Then I added just "#raw=-use_wallclock_as_timestamps 1": the video is ok as the audio but the audio lagged for a second in an hour.

Currently testing this:

frontdoor: - "ffmpeg:rtsp://admin:[email protected]/Streaming/Channels/101/#async#input=rtsp/udp#video=copy#audio=aac#audio=opus#hardware=cuda"

Where "async" means both "use_wallclock_as_timestamps" and ffmpeg's own "async" (which supposedly stands for "audio sync", not " asynchronous"). "opus" supposedly needed for the "webrtc" player to work, "aac" is needed for the frigate recorder to work/

It already has my "micropauses" at the live view, I'll see what it gives in the recordings in a couple of hours

upd: with "#async" the snapshots somehow became distorted, but the audio lag didn't go. Will revert to "#raw=-use_wallclock_as_timestamps 1" for a while.

[fedd]

"ffmpeg:rtsp://admin:[email protected]/Streaming/Channels/101/#input=rtsp/udp#video=copy#audio=aac#hardware

Still issue after about 1hour

2024-07-19 09:39:17.670953093 09:39:17.666 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error=EOF url=ffmpeg:rtsp://admin:[email protected]/Streaming/Channels/101/#input=rtsp/udp#video=copy#audio=aac#hardware

By the way I can't really comment the EOF error. I just realized I didn't watch my logs! Went straight to the actual recordings in the morning, saw the mess and started modifying the config

edit: could be a GPU issue. I have and Nvidia card to do the encoding, thus I have "#hardware=cuda". Can't really say if I had EOFs too

edit2: or a network issue!

[fedd]

Well, the udp advice seemed to help make the audio better. The #async switch somewhat does it's job keeping the lag tolerable for security purposes, though not fully eliminating it. It is ahead of the video or behind it at times, and sometimes even in sync. Sticking with this for a while:

go2rtc:
  streams:
#    frontdoorphone:
#      - rtsp://admin:[email protected]/Streaming/Channels/101/
    frontdoor:
      - "ffmpeg:rtsp://admin:[email protected]/Streaming/Channels/101/#async#input=rtsp/udp#video=copy#audio=aac#audio=opus#hardware=cuda"
      - isapi://admin:[email protected]:80/
cameras:
  frontdoor:
    enabled: true
    audio: 
      enabled: true
      min_volume: 100
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
      - path: rtsp://127.0.0.1:8554/frontdoor?video&audio
        input_args: preset-rtsp-restream
        roles:
        - record
        - audio
        - detect

[vk496]

Thank you @fedd . I was facing the same issue and your config worked like a charm