[Config Support]: Frigate only detect person, no cat

[Honusnap]

Describe the problem you are having

I've installed Frigate 2 days ago and tried many parameters in settings.yaml to try to improve things, i also tried Yolonas model .. with the same result. I increased frame_height to see if it improve the result .. it did but only on person.
I got some videos with cat moving and being detected as a movement and rectangled is tracked correctly all the way.. but it is never tagged as a cat while the human near it is tagged correctly.
What strange is that a near barrel and wood parts just nearby is sometimes detected as an animal (i tried grouping all animals in an "animal" label. if i remember between 14 and 19)
I'm starting to think that my environment is just not working well with Frigate... and Nick said that i should try config support to see if i was not making a dumb mistake.

Version

0.14.1

Frigate config file

mqtt:
  enabled: true
  host: 192.168.1.62 #Insert the IP address of your Home Assistant
  port: 1883 #Leave as default 1883 or change to match the port set in yout MQTT Broker configuration
  topic_prefix: Frigate
  client_id: Frigate
  user: mosquito #Change to match the username set in your MQTT Broker
  password: JackTheRipper #Change to match the password set in your MQTT Broker
  stats_interval: 60

birdseye:
  enabled: true
  mode: motion # continuous / motion / objects
  inactivity_threshold: 300

detectors:
  ov:
    type: openvino
    device: GPU.0
model:
  # width: 300
  # height: 300
  # input_tensor: nhwc
  # input_pixel_format: bgr
  # path: /openvino-model/FP16/ssdlite_mobilenet_v2.xml
  # labelmap_path: /openvino-model/coco_91cl_bkgr.txt
  model_type: yolonas
  width: 320
  height: 320
  input_tensor: nchw
  input_pixel_format: bgr
  path: /openvino-model/yolo_nas_s.onnx
  labelmap_path: /openvino-model/coco-80.txt
  # labelmap:
  #   14: animal
  #   15: animal
  #   16: animal
  #   17: animal
  #   18: animal
  #   19: animal

logger:
  # Optional: Default log verbosity (default: shown below)
  default: info

detect:
  # Optional: enables detection for the camera (default: True)
  # This value can be set via MQTT and will be updated in startup based on retained value
  #enabled: true
  # Optional: width of the frame for the input with the detect role (default: shown below)
  width: 640
  # Optional: height of the frame for the input with the detect role (default: shown below)
  height: 480
  # Optional: desired fps for your camera for the input with the detect role (default: shown below)
  # NOTE: Recommended value of 5. Ideally, try and reduce your FPS on the camera.
  fps: 5
  # Optional: Number of frames without a detection before frigate considers an object to be gone. (default: 5x the frame rate)
  #max_disappeared: 40
  # Optional: Configuration for stationary object tracking # since v0.10
  stationary:
    # Optional: Frequency for running detection on stationary objects (default: shown below)
    # When set to 0, object detection will never be run on stationary objects. If set to 10,
    # it will be run on every 10th frame.
    #interval: 10
    # Optional: Number of frames without a position change for an object to be considered
    #                 stationary (default: 10x the frame rate or 10s)
    #threshold: 40
    # Optional: Define a maximum number of frames for tracking a stationary object (default: not set, track forever)
    # This can help with false positives for objects that should only be stationary for a limited amount of time.
    # It can also be used to disable stationary object tracking. For example, you may want to set a value for person,
    # but leave car at the default.
    max_frames:
      # Optional: Default for all object types (default: not set, track forever)
      default: 1500
      # Optional: Object specific values
      objects:
        person: 500

objects:
  track:
    - person
    - bicycle
    - car
    - motorcycle
    - airplane
    - bus
    - train
    - truck
    - boat
    - traffic light
    - fire hydrant
    - stop sign
    - parking meter
    - bench
    - bird
    - cat
    - dog
    - horse
    - sheep
    - cow
    - elephant
    - bear
    - zebra
    - giraffe
    - backpack
    - umbrella
    - handbag
    - tie
    - suitcase
    - frisbee
    - skis
    - snowboard
    - sports ball
    - kite
    - baseball bat
    - baseball glove
    - skateboard
    - surfboard
    - tennis racket
    - bottle
    - wine glass
    - cup
    - fork
    - knife
    - spoon
    - bowl
    - banana
    - apple
    - sandwich
    - orange
    - broccoli
    - carrot
    - hot dog
    - pizza
    - donut
    - cake
    - chair
    - couch
    - potted plant
    - bed
    - dining table
    - toilet
    - tv
    - laptop
    - mouse
    - remote
    - keyboard
    - cell phone
    - microwave
    - oven
    - toaster
    - sink
    - refrigerator
    - book
    - clock
    - vase
    - scissors
    - teddy bear
    - hair drier
  filters:
    person:
      # min_area: 1000
      # max_area: 40000
      threshold: 0.60
    animal:
      # min_area: 200
      # max_area: 70000
      threshold: 0.30
motion:
  enabled: true
  # Optional: The threshold passed to cv2.threshold to determine if a pixel is different enough to be counted as motion. (default: shown below)
  # Increasing this value will make motion detection less sensitive and decreasing it will make motion detection more sensitive.
  # The value should be between 1 and 255.
  threshold: 22
  # Optional: Minimum size in pixels in the resized motion image that counts as motion (default: shown below)
  # Increasing this value will prevent smaller areas of motion from being detected. Decreasing will
  # make motion detection more sensitive to smaller moving objects.
  # As a rule of thumb:
  #  - 10 - high sensitivity
  #  - 30 - medium sensitivity
  #  - 50 - low sensitivity
  contour_area: 12
  # Optional: The percentage of the image used to detect lightning or other substantial changes where motion detection
  #           needs to recalibrate. (default: shown below)
  # Increasing this value will make motion detection more likely to consider lightning or ir mode changes as valid motion.
  # Decreasing this value will make motion detection more likely to ignore large amounts of motion such as a person approaching
  # a doorbell camera.
  lightning_threshold: 0.8 
  # Optional: Alpha value passed to cv2.accumulateWeighted when averaging frames to determine the background (default: shown below)
  # Higher values mean the current frame impacts the average a lot, and a new object will be averaged into the background faster.
  # Low values will cause things like moving shadows to be detected as motion for longer.
  # https://www.geeksforgeeks.org/background-subtraction-in-an-image-using-concept-of-running-average/
  frame_alpha: 0.01
  # Optional: Height of the resized motion frame  (default: 100)
  # Higher values will result in more granular motion detection at the expense of higher CPU usage.
  # Lower values result in less CPU, but small changes may not register as motion.
  frame_height: 150
  # Optional: motion mask
  # NOTE: see docs for more detailed info on creating masks
  #Cameras Timestamp
  mask: 0.025,0.036,0.320,0.036,0.320,0,0.025,0
  # Optional: improve contrast (default: shown below)
  # Enables dynamic contrast improvement. This should help improve night detections at the cost of making motion detection more sensitive
  # for daytime.
  improve_contrast: true
  # Optional: Delay when updating camera motion through MQTT from ON -> OFF (default: shown below).
  mqtt_off_delay: 30

record:
  enabled: true
  retain:
    #The number of days a recording will be kept for after a motion is detected.
    days: 2
    mode: motion
  events:
    # Optional: Retention settings for recordings of events
    retain:
      # Required: Default retention days (default: shown below)
      default: 60
      # continuous / motion / active_objects / all
      mode: active_objects
      # Optional: Per object retention days
      #objects:
      # person: 60
      # animal: 60
    # Optional: Number of seconds before the event to include (default: shown below)
    pre_capture: 2
    # Optional: Number of seconds after the event to include (default: shown below)
    post_capture: 5
    # Optional: Objects to save recordings for. (default: all tracked objects)
    #objects:
    #  - person
    # Optional: Restrict recordings to objects that entered any of the listed zones (default: no required zones)
    # required_zones: []

# Optional: Configuration for the jpg snapshots written to the clips directory for each event
# NOTE: Can be overridden at the camera level
snapshots:
  # Optional: Enable writing jpg snapshot to /media/frigate/clips (default: shown below)
  enabled: true
  # Optional: save a clean PNG copy of the snapshot image (default: shown below)
  clean_copy: false
  # Optional: print a timestamp on the snapshots (default: shown below)
  timestamp: false
  # Optional: draw bounding box on the snapshots (default: shown below)
  bounding_box: true
  # Optional: crop the snapshot (default: shown below)
  crop: false
  # Optional: height to resize the snapshot to (default: original size)
  #height: 175
  # Optional: Restrict snapshots to objects that entered any of the listed zones (default: no required zones)
  #required_zones: []
  # Optional: Camera override for retention settings (default: global values)
  retain:
    # Required: Default retention days (default: shown below)
    default: 60
    # Optional: Per object retention days
    #objects:
      #person: 15
  # Optional: quality of the encoded jpeg, 0-100 (default: shown below)
  quality: 80

live:
  # Optional: Set the height of the live stream. (default: 720)
  # This must be less than or equal to the height of the detect stream. Lower resolutions
  # reduce bandwidth required for viewing the live stream. Width is computed to match known aspect ratio.
  height: 480
  # Optional: Set the encode quality of the live stream (default: shown below)
  # 1 is the highest quality, and 31 is the lowest. Lower quality feeds utilize less CPU resources.
  quality: 5
# Optional: in-feed timestamp style configuration
# NOTE: Can be overridden at the camera level
# timestamp_style:
#   # Optional: Position of the timestamp (default: shown below)
#   #           "tl" (top left), "tr" (top right), "bl" (bottom left), "br" (bottom right)
#   position: "tr"
#   # Optional: Format specifier conform to the Python package "datetime" (default: shown below)
#   #           Additional Examples:
#   #             german: "%d.%m.%Y %H:%M:%S"
#   format: "%d/%m/%Y %H:%M:%S"
#   # Optional: Color of font
#   color:
#     # All Required when color is specified (default: shown below)
#     red: 255
#     green: 255
#     blue: 255
#   # Optional: Line thickness of font (default: shown below)
#   thickness: 1
#   # Optional: Effect of lettering (default: shown below)
#   #           None (No effect),
#   #           "solid" (solid background in inverse color of font)
#   #           "shadow" (shadow for font)
#   effect: shadow

cameras:
  # AtelierTest:
  #   ffmpeg:
  #     inputs:
  #       - path: /media/frigate/exports/Test.mp4
  #         input_args: -re -stream_loop -1 -fflags +genpts
  #         roles:
  #           - detect
  #   motion:
  #     threshold: 20
  #     contour_area: 8
  #     improve_contrast: 'true'

  AtelierFacade:
    ffmpeg:
      hwaccel_args: preset-vaapi
      inputs:
        - path: rtsp://127.0.0.1:8554/AtelierFacade
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/AtelierFacadeSub
          roles:
            - detect
    objects:
      filters:
        person:
          mask: 0.267,0.546,0.37,0.546,0.371,0.623,0.271,0.618
        animal:
          mask: 0.269,0.614,0.265,0.549,0.368,0.547,0.37,0.624
    live:
      stream_name: AtelierFacade
    motion:
      threshold: 30
      contour_area: 8
      improve_contrast: 'false'
    zones:
      FacadeChemin:
        coordinates: 
          0.067,0.855,0.244,0.841,0.414,0.796,0.563,0.759,0.782,0.79,0.99,0.837,0.998,0.76,0.79,0.675,0.716,0.604,0.691,0.53,0.677,0.444,0.678,0.384,0.59,0.308,0.563,0.281,0.602,0.262,0.678,0.23,0.773,0.194,0.752,0.18,0.658,0.213,0.59,0.23,0.535,0.261,0.53,0.305,0.545,0.376,0.545,0.503,0.525,0.606,0.43,0.716,0.265,0.753,0,0.763,0.002,0.861
        inertia: 10
        loitering_time: 2
    review:
      alerts:
        required_zones: FacadeChemin

  AtelierHaut:
    ffmpeg:
      hwaccel_args: preset-vaapi
      inputs:
        - path: rtsp://127.0.0.1:8554/AtelierHaut
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/AtelierHautSub
          roles:
            - detect
    objects:
      filters:
        person:
          mask: 0.267,0.546,0.37,0.546,0.371,0.623,0.271,0.618
        animal:
          mask: 0.269,0.614,0.265,0.549,0.368,0.547,0.37,0.624
    live:
      stream_name: AtelierHaut
    motion:
      threshold: 25
      contour_area: 8
      improve_contrast: 'false'
      mask:
        - 0.025,0.036,0.32,0.036,0.32,0,0.025,0
        - 0.83,0.457,0.83,0.251,0.808,0.148,0.786,0.085,0.806,0.067,0.837,0.077,0.867,0,0.998,0,1,0.546,0.931,0.671,0.897,0.592,0.865,0.525
        - 0.563,0.324,0.617,0.283,0.661,0.169,0.6,0.155,0.507,0.171,0.482,0.082,0.405,0,0.327,0,0.234,0.146,0.213,0.197,0.215,0.279,0.26,0.318
    zones:
      HautChemin:
        coordinates: 
          0.305,1,0.467,0.832,0.518,0.745,0.56,0.671,0.598,0.583,0.618,0.496,0.634,0.375,0.673,0.269,0.705,0.218,0.746,0.153,0.807,0.168,0.784,0.209,0.753,0.283,0.726,0.36,0.72,0.461,0.709,0.597,0.659,0.735,0.601,0.853,0.526,1
        inertia: 10
        loitering_time: 2
      HautChamp:
        coordinates: 
          0.292,1,0.002,0.997,0,0.356,0.21,0.199,0.211,0.286,0.253,0.322,0.392,0.33,0.567,0.329,0.669,0.267,0.625,0.371,0.6,0.565,0.565,0.648,0.468,0.819
        inertia: 10
        loitering_time: 2
    review:
      alerts:
        required_zones:
          - HautChamp
          - HautChemin

  AtelierBas:
    ffmpeg:
      hwaccel_args: preset-vaapi
      inputs:
        - path: rtsp://127.0.0.1:8554/AtelierBas
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/AtelierBasSub
          roles:
            - detect
    live:
      stream_name: AtelierBas
    motion:
      threshold: 20
      contour_area: 8
      improve_contrast: 'false'
      mask:
        - 0.025,0.036,0.32,0.036,0.32,0,0.025,0
        - 0.236,1,0.184,0.916,0.16,0.809,0.154,0.723,0.166,0.654,0.216,0.61,0.309,0.584,0.355,0.546,0.4,0.563,0.427,0.593,0.441,0.627,0.46,0.678,0.473,0.729,0.493,0.779,0.519,0.843,0.539,0.916,0.551,1
    zones:
      BasChemin:
        coordinates: 
          0.67,0.726,0.696,0.606,0.648,0.431,0.589,0.315,0.539,0.266,0.516,0.219,0.506,0.174,0.536,0.134,0.562,0.164,0.562,0.219,0.59,0.254,0.634,0.311,0.673,0.382,0.729,0.48,0.768,0.56,0.877,0.609,0.995,0.663,1,0.701,0.848,0.633,0.811,0.673,0.793,0.764,0.876,0.896,0.993,0.932,0.995,0.997,0.703,1,0.605,0.897,0.539,0.893,0.506,0.801,0.459,0.667,0.433,0.586,0.345,0.527,0.325,0.573,0.28,0.586,0.201,0.612,0.161,0.647,0.142,0.758,0.178,0.92,0.22,0.997,0.096,0.996,0,0.866,0.008,0.806,0.092,0.739,0.082,0.596,0.062,0.508,0.045,0.458,0,0.473,0,0.358,0.041,0.317,0.056,0.262,0.09,0.252,0.119,0.27,0.089,0.309,0.084,0.366,0.105,0.4,0.1,0.477,0.126,0.542,0.158,0.604,0.261,0.562,0.308,0.474,0.383,0.457,0.445,0.541,0.503,0.652,0.559,0.733,0.629,0.741
        inertia: 10
        loitering_time: 2
      BasPotager:
        coordinates: 
          0.16,0.599,0.104,0.484,0.111,0.399,0.088,0.359,0.092,0.311,0.123,0.27,0.169,0.255,0.374,0.247,0.422,0.272,0.42,0.335,0.423,0.442,0.382,0.454,0.307,0.468,0.261,0.561
        inertia: 10
        loitering_time: 2
    review:
      alerts:
        required_zones:
          - BasChemin
          - BasPotager
version: 0.14
camera_groups:
  BirdsEye:
    order: 1
    icon: LuBird
    cameras:
      - AtelierBasSub
      - AtelierFacadeSub
      - AtelierHautSub
      - birdseye
  Camera_Bas:
    order: 2
    icon: LuArrowDown
    cameras: AtelierBas
  Camera_Haut:
    order: 3
    icon: LuArrowUp
    cameras: AtelierHaut
  Camera_Facade:
    order: 4
    icon: LuArrowRight
    cameras: AtelierFacade

Relevant Frigate log output

2024-10-12 12:40:27.966788  [INFO] Preparing Frigate...
2024-10-12 12:40:27.966907  [INFO] Starting Frigate...
2024-10-12 12:40:30.750871  [2024-10-12 12:40:30] frigate.app                    INFO    : Starting Frigate (0.14.1-)
2024-10-12 12:40:30.751051  [2024-10-12 12:40:30] frigate.util.config            INFO    : Checking if frigate config needs migration...
2024-10-12 12:40:30.843685  [2024-10-12 12:40:30] frigate.util.config            INFO    : frigate config does not need migration...
2024-10-12 12:40:31.147771  [2024-10-12 12:40:31] frigate.util.services          INFO    : Automatically detected vaapi hwaccel for video decoding
2024-10-12 12:40:31.387247  [2024-10-12 12:40:31] peewee_migrate.logs            INFO    : Starting migrations
2024-10-12 12:40:31.387571  [2024-10-12 12:40:31] peewee_migrate.logs            INFO    : There is nothing to migrate
2024-10-12 12:40:31.404196  [2024-10-12 12:40:31] frigate.app                    INFO    : Recording process started: 246107
2024-10-12 12:40:31.411406  [2024-10-12 12:40:31] frigate.app                    INFO    : Recording process started: 246109
2024-10-12 12:40:31.416314  [2024-10-12 12:40:31] frigate.app                    INFO    : go2rtc process pid: 128
2024-10-12 12:40:31.487483  [2024-10-12 12:40:31] detector.ov                    INFO    : Starting detection process: 246144
2024-10-12 12:40:31.487766  [2024-10-12 12:40:31] frigate.app                    INFO    : Output process started: 246145
2024-10-12 12:40:31.570918  [2024-10-12 12:40:31] frigate.app                    INFO    : Camera processor started for AtelierFacade: 246161
2024-10-12 12:40:31.584588  [2024-10-12 12:40:31] frigate.app                    INFO    : Camera processor started for AtelierHaut: 246166
2024-10-12 12:40:31.613475  [2024-10-12 12:40:31] frigate.app                    INFO    : Camera processor started for AtelierBas: 246171
2024-10-12 12:40:31.630320  [2024-10-12 12:40:31] frigate.app                    INFO    : Capture process started for AtelierFacade: 246176
2024-10-12 12:40:31.633726  [2024-10-12 12:40:31] frigate.app                    INFO    : Capture process started for AtelierHaut: 246187
2024-10-12 12:40:31.646331  [2024-10-12 12:40:31] frigate.app                    INFO    : Capture process started for AtelierBas: 246192

Relevant go2rtc log output

go2rtc config file is not updated by frigate in my case, i just configured streams in go2rtc webUI and used them in Frigate Config.yaml, it work normally.

Frigate stats

No response

Operating system

Proxmox

Install method

Proxmox via TTeck Script

docker-compose file or Docker CLI command

Do not know what to write here.

Object Detector

OpenVino

Screenshots of the Frigate UI's System metrics pages

With Yolonas 33ms, with default model : 13ms
Here with yolonas

Any other information that may be helpful

I'm in the country side ... lot of things in the way of the camera


This cat is not detected .. it gone from top to bottom .. then 15min after from bottom to top, there is a box around it .. no detection

AtelierHaut_8ht59v.640.mp4

I'm tried using the main stream and resizing it to full hd to see if detection is better ..... it do not seems it was while it was taxing my weak CPU a lot.

[NickM-27]

I see a number of issues in this config:

1. you copied a lot of the reference config which is explicitly recommended against, this includes a few specific issues
2. you have a non-default frame_height which is going to increase your CPU usage as well as make motion detection work harder, you should remove the entire global motion config.

also, the best way to debug object issues is to use the debug view in settings. It would be great to see a screen recording of that with regions and bounding boxes enabled while it is trying to detect the cat. That will allow us to see what Frigate is seeing.

It would also be really helpful to see a screenshot of the camera metrics page as well

[Honusnap]

Another thing that might help you is increasing the detect fps a bit. This gives Frigate more frames to be able to identify the cat. Perhaps try 8 or at the most 10.

I don't have much experience with the yolonas notebook model so I'm not sure what to expect in terms of performance there

For every test we did, i tried Yolonas AND Frigate stock openvino : /openvino-model/FP16/ssdlite_mobilenet_v2.xml model.
Tried 10 fps with 1280x720 on the same clip = no cat detected.

So ... it seems i was right ... the model is the problem, the stock model can't detect anything else then person, i got some boat .. umbrella.. banc... false positive but no cat .. :D. Maybe it can but in some special positive situations.

What model did you use for the test to work ?

[NickM-27]

So ... it seems i was right ... the model is the problem, the stock model can't detect anything else then person, i got some boat .. umbrella.. banc... false positive but no cat .. :D. Maybe it can but in some special positive situations.

it should work fine with car as well. If you look at the COCO dataset you can see it is primarily comprised of person. The video you are using is also a rather difficult scenario for detecting the cat as it is walking away from the camera instead of side to side. Many users have no issues with the default model.

[NickM-27]

I was testing with the default included coral model

[Honusnap]

So ... it seems i was right ... the model is the problem, the stock model can't detect anything else then person, i got some boat .. umbrella.. banc... false positive but no cat .. :D. Maybe it can but in some special positive situations.

it should work fine with car as well. If you look at the COCO dataset you can see it is primarily comprised of person. The video you are using is also a rather difficult scenario for detecting the cat as it is walking away from the camera instead of side to side. Many users have no issues with the default model.

I do not have cars passing by here i 'm in a forest, there is a small road 100m from the cameras.
I will move the cameras on the side of the workshop farther.... moving object will not come "AT" the camera but more on the side, it's better anyway.

What does it prove infine ? That an "unknown" moving object is important, not saying the AI should say unknown cause i suppose 90% of the pictures it receive are unknown objects (seems even 99% in the case of the openvino default one =) ), no .. but the analysis of the moving group of pixels should be able to categorise motion... in the case of this cat, movement speed and it variation is an information...size of the detected object is an other. Cause this video of the cat moving .. is buried among many others... ! I found it by watching closely. Of course i could reduce motion detection ... it's easy when the camera is looking at a concrete parking, not in my case, i got trees and grass everywhere... if i mask the most annoying of them .. i remove like 50% of the camera angular surface of vision.
The cat motion was "special" vs the small tree one.... thius information is important even if object clasification fail.

About something else :
Is there a posibility to apply motion detection/AI interpretation on an already recorded clip, kind like on the debug view ? Cause having to modify config to create an articifial camera, restart frigate ... and test a clip ... is time consuming.

[NickM-27]

I want to reiterate that your particular circumstances are more challenging for object detection. Many users have great success with the default openvino model detecting objects. But that is the challenge with the default models, they are not trained on security camera images so the results vary from user to user based on their specific circumstances. This is one reason why object detection models trained on security camera images and tuned on an individuals camera images are more accurate and precise. Frigate+ is one such solution, along with CodeProject.AI training, training your own model "by hand", etc.

Frigate motion detection is very simple because it is just a first step to know where to look for objects and then Frigate uses object detection to track objects of interest, if you prefer an NVR that supports line crossing, zones, etc. using motion blocks without object detection then Frigate is likely not the NVR for you.

Cause having to modify config to create an articifial camera, restart frigate ... and test a clip ... is time consuming.

currently this is the only way, there is a pinned feature request for building this into the UI.