[Detector Support]: Tensorrt is very slow

[LuisPeregrina]

Describe the problem you are having

Every once in a couple of days tensorrt is unresponsive, no video feeds are receiving frames.
I have a GTX 1060 6GB.

Version

0.14.1-f4f3cfa

Frigate config file

version: 0.14

mqtt:
  enabled: true
  host: mosquitto
  user: admin
  password: abc123


ffmpeg:
  hwaccel_args: preset-nvidia-h264

detectors:
  tensorrt:
    type: tensorrt
    device: 0

model:
  path: /config/model_cache/tensorrt/yolov7-320.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 320
  height: 320

record:
  enabled: true
  retain:
    days: 12
    mode: all
  events:
    retain:
      default: 30
      mode: motion

motion:
  threshold: 50

snapshots:
  enabled: true
  retain:
    default: 30

      #audio:
      #  enabled: false
      #  listen:
      #    - bark
      #    - alarm
      #    - fire_alarm
      #    - scream
      #    - yell
      #    - gunshot
      #    - machine_gun
      #    - fusillade
      #    - artillery_fire
      #    - eruption
      #    - boom

go2rtc:
  streams:
    principal:
      - rtsp://[email protected]/ch0
      - ffmpeg:principal
    principal_detect:
      - rtsp://[email protected]/ch1
      - ffmpeg:principal_detect

cameras:
  principal:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://127.0.0.1:8554/principal?video&audio
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/principal_detect
          input_args: preset-rtsp-restream
          roles:
            - detect
    motion:
      mask:
        - 0.413,0.547,0.369,1,0,1,0,0.336,0.186,0.381,0.187,0.251,0.268,0.267
        - 0.538,0.615,0.628,1,1,1,1,0,0.63,0,0.545,0.245
    zones:
      escaleras:
        coordinates: 
          0.276,0.631,0.41,0.639,0.373,1,0.625,1,0.553,0.642,0.439,0.631,0.44,0.53,0.285,0.512
        inertia: 3
        loitering_time: 0
      garage:
        coordinates: 0,0.232,0.275,0.165,0.264,0.547,0.241,0.547,0.23,0.379,0,0.33
        loitering_time: 0
        inertia: 3
      porton:
        coordinates: 0.296,0.189,0.535,0.187,0.542,0.63,0.414,0.633,0.415,0.438,0.294,0.417
        loitering_time: 0
        inertia: 3
    objects:
      filters:
        dog:
          mask: 0.25,0.423,0.53,0.445,0.541,0.643,0.625,1,0.373,1,0.407,0.559,0.236,0.554

  afuera:
    ffmpeg:
      inputs:
        - path: rtsp://[email protected]:8554/afuera
          roles:
            - record
        - path: rtsp://[email protected]:8554/afuera-sub
          roles:
            - detect
    objects:
      track:
        - person
    detect:
      enabled: false
      fps: 5

    motion:
      mask: 0.715,0.931,0.984,0.935,0.984,0.989,0.714,0.987

  doorbell:
    ffmpeg:
      inputs:
        - path: rtsp://[email protected]:8554/doorbell
          roles:
            - record
            - detect
            # - audio
    objects:
      track:
        - person
    detect:
      width: 480
      height: 640
    motion:
      mask:
        - 0.598,0.963,1,0.958,1,1,0.598,1
        - 0.514,0.416,0.24,0.427,0.188,0.555,0,0.56,0,0.779,0.521,0.72

docker-compose file or Docker CLI command

frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable-tensorrt
    env_file: stack.env
    shm_size: "128mb" # update for your cameras based on calculation above
    #devices:
      #- /dev/bus/usb:/dev/bus/usb  # Passes the USB Coral, needs to be modified for other versions
      #- /dev/apex_0:/dev/apex_0    # Passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      #- /dev/video11:/dev/video11  # For Raspberry Pi 4B
      #- /dev/dri/renderD128:/dev/dri/renderD128 # For intel hwaccel, needs to be updated for your hardware
    volumes:
      - ${CONFIG_ROOT}/frigate:/config
      - ${VIDEO_ROOT}/storage:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8971:8971"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
      - "1935:1935" # RTMP, not sure why people want this
    environment:
      - YOLO_MODELS=yolov7-320
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    networks:
      - default
      - internal

Relevant Frigate log output

Logs lost, over a week ago.

Install method

Docker Compose

Object Detector

TensorRT

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

Restarting frigate is the only option to fix this, not sure if tensorrt crashes and its not being restarted.

[NickM-27]

we would need logs otherwise there is no telling what happened