[Config Support]: High memory usage and ffmpeg keeps crashing #11326
Unanswered
plsfixrito
asked this question in
Config Support
Replies: 7 comments 8 replies
-
|
we will need information from the container about what is using memory. This can be retrieved using the system page in the UI as well as |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Here is an image from the system page Here is an image from btop Do note that the pid climbs up from 300 all the way to over 100k in around 2 days |
Beta Was this translation helpful? Give feedback.
2 replies
-
sounds like there are a lot of camera issues then, seems like a good place to start. Logs are full of timeouts |
Beta Was this translation helpful? Give feedback.
-
|
The camera with timeout is a HIKVISION DS-2CD2663G2-IZS. I have it setup to run with a POE network switch. |
Beta Was this translation helpful? Give feedback.
-
|
im also facing similar issue: mqtt:
host: home-assistant-mq
user: hass
password: 'xxx'
detectors:
ov:
type: openvino
device: GPU
model:
path: /models/yolo_nas_s7.onnx
labelmap_path: /labelmap/coco-80.txt
width: 640
height: 640
input_tensor: nchw
input_pixel_format: bgr
model_type: yolonas
detect:
width: 1088
height: 640
go2rtc:
ffmpeg:
cam: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -use_wallclock_as_timestamps
1 -rtsp_transport tcp -i {input}
streams:
cam1:
- ffmpeg:rtsp://[email protected]/stream1#input=cam#video=h264#hardware#audio=aac
cam2:
- ffmpeg:rtsp://[email protected]/stream1#input=cam#video=h264#hardware#audio=aac
ffmpeg:
hwaccel_args: preset-intel-qsv-h264
output_args:
record: preset-record-generic-audio-copy
input_args: preset-rtsp-restream
objects:
filters:
person:
min_score: 0.7
threshold: 0.7
cat:
min_score: 0.5
threshold: 0.7
track:
- person
- cat
record:
enabled: true
expire_interval: 5
retain:
days: 7
mode: all
events:
retain:
default: 7
mode: motion
snapshots:
enabled: true
timestamp: false
bounding_box: true
retain:
default: 7
motion:
threshold: 30
contour_area: 10
improve_contrast: false
cameras:
cam1:
onvif:
host: 192.168.20.21
port: 2020
user: camera
password: xxx
ffmpeg:
inputs:
- path: rtsp://127.0.0.1:8554/cam1
roles:
- record
- detect
detect:
fps: 5
min_initialized: 2
motion:
mask: 0,0.071,0.328,0.068,0.328,0,0,0
threshold: 30
contour_area: 10
improve_contrast: 'true'
zones:
inside:
coordinates: 0,0.523,0.336,0.341,0.348,0.476,0.656,0.269,0.661,0,1,0,1,1,0,1
inertia: 1
outside:
coordinates: 0.002,0,0,0.52,0.337,0.337,0.349,0.472,0.654,0.266,0.658,0
inertia: 1
review:
detections:
required_zones:
- inside
- outside
alerts:
required_zones: inside
cam2:
onvif:
host: 192.168.20.22
port: 2020
user: camera
password: xxx
ffmpeg:
inputs:
- path: rtsp://127.0.0.1:8554/cam2
roles:
- record
- detect
detect:
fps: 5
min_initialized: 2
motion:
mask: 0,0,0.336,0,0.336,0.07,0,0.072
objects:
mask: 0.519,0.374,0.615,0.355,0.63,0.673,0.534,0.663
zones:
inside:
coordinates: 0,0,1,0,1,1,0,1
inertia: 1
review:
alerts:
required_zones: inside
detections:
required_zones: inside
version: 0.14logs: i limit the docker memory usage to 2gb and currently it is at 1.8gb of usage. processor is i5 6500T. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Check go2rtc logs |
Beta Was this translation helpful? Give feedback.
-
|
Ok, i've also encountered this. Here's the go2rtc logs: Here's the output of the main frigate log: This is my config: Capture of htop: This is on a RPi4, cameras are running Thingino firmware, RTSP h264 1920x1080, 24fps I understand that there might have been a timeout on reading the camera's output (probably bad WiFi signal), but why would it hog the ram? Also, here's the ffmpeg command for one camera (the other one is identical): |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Also having this issue, and it's bringing down my entire system regularly. |
Beta Was this translation helpful? Give feedback.
4 replies
-
|
what is your free memory % usually? |
Beta Was this translation helpful? Give feedback.
-
|
40-50%. Plenty. I have the container's memory limited to 2g because anything higher causes much more frequent crashes. |
Beta Was this translation helpful? Give feedback.
-
|
huh interesting. i usually have 4gb as the container mem limit (combined my free memory is around 40 and 50 percent) and around once a week my system freezes for no clear reason. i dropped it to 2gb and will see what happen. |
Beta Was this translation helpful? Give feedback.
-
|
i can definitely see that my swap is increasing at a constant rate until i restarted the frigate container making me believe that this is indeed a memory leak |
Beta Was this translation helpful? Give feedback.
-
|
I still have this issue with frigate. I have upgraded to the latest commit 0.14.1-e01b6ee. It would basically continue using more and more ram then freeze or crash my RPI. A temporary solution that I wouldn't recommend is creating a cronjob to restart frigate docker container every 12 or 24 hours. But this obviously creates safety issues for you if you are using it for surveillance. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
|
Beta Was this translation helpful? Give feedback.
1 reply
-
|
even after bumping the memory limit to 4gb and allocating more to the swap file, the processes are still crashing that causes no recording for the last few hours which thankfully i noticed and restarted the container manually. the memory usage was 5gb in total (3.8gb on ram, and the rest on swap). i use openvino with intel igpu and no hardware acceleration for encoding. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Describe the problem you are having
I ran frigate version 0.13.2-6476f8a on RPI 4B 8gb ram installed on docker without HASS. The frigate process starts with 1.9gb on ram and uses an extra 1gb each day until it runs out of memory and my RPI stops being responsive. Not using coral currently using CPU.
Version
0.13.2-6476f8a
Frigate config file
Relevant log output
Frigate stats
{"cameras":{"cam_1":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":1.1,"capture_pid":333,"detection_enabled":1,"detection_fps":1.4,"ffmpeg_pid":106393,"pid":327,"process_fps":1.0,"skipped_fps":0.0},"cam_2":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":1.0,"capture_pid":336,"detection_enabled":1,"detection_fps":1.5,"ffmpeg_pid":106396,"pid":329,"process_fps":1.1,"skipped_fps":0.0},"cam_3":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":1.0,"capture_pid":340,"detection_enabled":1,"detection_fps":1.1,"ffmpeg_pid":106390,"pid":331,"process_fps":1.0,"skipped_fps":0.0}},"cpu_usages":{"1":{"cmdline":"/package/admin/s6/command/s6-svscan -d4 -- /run/service","cpu":"0.0","cpu_average":"0","mem":"0.0"},"16":{"cmdline":"s6-supervise s6-linux-init-shutdownd","cpu":"0.0","cpu_average":"0","mem":"0.0"},"18":{"cmdline":"/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B","cpu":"0.0","cpu_average":"0","mem":"0.0"},"25":{"cmdline":"s6-supervise frigate","cpu":"0.0","cpu_average":"0","mem":"0.0"},"26":{"cmdline":"s6-supervise go2rtc-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"27":{"cmdline":"s6-supervise s6rc-oneshot-runner","cpu":"0.0","cpu_average":"0","mem":"0.0"},"28":{"cmdline":"s6-supervise nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"29":{"cmdline":"s6-supervise s6rc-fdholder","cpu":"0.0","cpu_average":"0","mem":"0.0"},"30":{"cmdline":"s6-supervise go2rtc","cpu":"0.0","cpu_average":"0","mem":"0.0"},"31":{"cmdline":"s6-supervise go2rtc-healthcheck","cpu":"0.0","cpu_average":"0","mem":"0.0"},"32":{"cmdline":"s6-supervise nginx-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"33":{"cmdline":"s6-supervise frigate-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"42":{"cmdline":"/package/admin/s6-2.11.3.2/command/s6-fdholderd -1 -i data/rules","cpu":"0.0","cpu_average":"0","mem":"0.0"},"43":{"cmdline":"/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --","cpu":"0.0","cpu_average":"0","mem":"0.0"},"81":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"82":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate","cpu":"0.0","cpu_average":"0","mem":"0.0"},"83":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc","cpu":"0.0","cpu_average":"0","mem":"0.0"},"98":{"cmdline":"/usr/local/go2rtc/bin/go2rtc -config=/dev/shm/go2rtc.yaml","cpu":"0.0","cpu_average":"0","mem":"0.2"},"110":{"cmdline":"bash ./run.user go2rtc-healthcheck","cpu":"0.0","cpu_average":"0","mem":"0.0"},"113":{"cmdline":"python3 -u -m frigate","cpu":"4.0","cpu_average":"12","mem":"39.4"},"127":{"cmdline":"nginx: master process nginx","cpu":"0.0","cpu_average":"0","mem":"0.1"},"145":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.1"},"146":{"cmdline":"nginx: worker process","cpu":"0.1","cpu_average":"0","mem":"0.1"},"150":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.1"},"151":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.1"},"152":{"cmdline":"nginx: cache manager process","cpu":"0.0","cpu_average":"0","mem":"0.1"},"296":{"cmdline":"frigate.logger ","cpu":"0.0","cpu_average":"0","mem":"1.3"},"298":{"cmdline":"frigate.recording_manager","cpu":"32.6","cpu_average":"31","mem":"8.5"},"307":{"cmdline":"/usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(49)","cpu":"0.1","cpu_average":"0","mem":"0.1"},"308":{"cmdline":"frigate.detector.cpu1","cpu":"115.3","cpu_average":"102","mem":"1.5"},"310":{"cmdline":"frigate.output ","cpu":"0.5","cpu_average":"0","mem":"1.5"},"313":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 640x480 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.5"},"314":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 640x480 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.5"},"315":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 640x480 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.5"},"316":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.8"},"327":{"cmdline":"frigate.process:cam_1","cpu":"2.6","cpu_average":"1","mem":"5.7"},"329":{"cmdline":"frigate.process:cam_2","cpu":"1.7","cpu_average":"2","mem":"6.0"},"331":{"cmdline":"frigate.process:cam_3","cpu":"4.5","cpu_average":"3","mem":"5.7"},"333":{"cmdline":"frigate.capture:cam_1","cpu":"0.4","cpu_average":"0","mem":"5.5"},"336":{"cmdline":"frigate.capture:cam_2","cpu":"0.2","cpu_average":"0","mem":"5.5"},"340":{"cmdline":"frigate.capture:cam_3","cpu":"0.3","cpu_average":"0","mem":"5.5"},"106389":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -c:v:1 h264_v4l2m2m -user_agent FFmpeg Frigate/0.13.2-6476f8a -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://*:*@192.168.8.161:554/Streaming/Channels/101/ -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /tmp/cache/cam_2@%Y%m%d%H%M%S%z.mp4","cpu":"2.8","cpu_average":"2","mem":"0.7"},"106390":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -c:v:1 h264_v4l2m2m -user_agent FFmpeg Frigate/0.13.2-6476f8a -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://*:*@192.168.8.162:554/Streaming/Channels/102/ -r 1 -vf fps=1,scale=640:480 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"1.0","cpu_average":"0","mem":"0.6"},"106392":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -c:v:1 h264_v4l2m2m -user_agent FFmpeg Frigate/0.13.2-6476f8a -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://*:*@192.168.8.162:554/Streaming/Channels/101/ -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /tmp/cache/cam_3@%Y%m%d%H%M%S%z.mp4","cpu":"2.7","cpu_average":"2","mem":"0.8"},"106393":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -c:v:1 h264_v4l2m2m -user_agent FFmpeg Frigate/0.13.2-6476f8a -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://*:*@192.168.8.160:554/Streaming/Channels/102/ -r 1 -vf fps=1,scale=640:480 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"1.0","cpu_average":"0","mem":"0.6"},"106395":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -c:v:1 h264_v4l2m2m -user_agent FFmpeg Frigate/0.13.2-6476f8a -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://*:*@192.168.8.160:554/Streaming/Channels/101/ -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /tmp/cache/cam_1@%Y%m%d%H%M%S%z.mp4","cpu":"0.7","cpu_average":"0","mem":"0.6"},"106396":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -c:v:1 h264_v4l2m2m -user_agent FFmpeg Frigate/0.13.2-6476f8a -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://*:*@192.168.8.161:554/Streaming/Channels/102/ -r 1 -vf fps=1,scale=640:480 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"1.0","cpu_average":"1","mem":"0.6"},"110806":{"cmdline":"sleep 30s","cpu":"0.0","cpu_average":"0","mem":"0.0"}},"detection_fps":4.0,"detectors":{"cpu1":{"detection_start":1715332574.803795,"inference_speed":112.42,"pid":308}},"gpu_usages":{"rpi-v4l2m2m":{"gpu":-1,"mem":-1}},"processes":{"go2rtc":{"pid":98},"logger":{"pid":296},"recording":{"pid":298}},"service":{"last_updated":1715332574,"latest_version":"0.13.2","storage":{"/dev/shm":{"free":1019.5,"mount_type":"tmpfs","total":1024.0,"used":4.5},"/media/frigate/clips":{"free":153716.5,"mount_type":"ext4","total":1877270.3,"used":1628167.2},"/media/frigate/recordings":{"free":153716.5,"mount_type":"ext4","total":1877270.3,"used":1628167.2},"/tmp/cache":{"free":940.2,"mount_type":"tmpfs","total":953.7,"used":13.4}},"temperatures":{},"uptime":56293,"version":"0.13.2-6476f8a"}}Operating system
Debian
Install method
Docker Compose
Coral version
CPU (no coral)
Any other information that may be helpful
No response
Beta Was this translation helpful? Give feedback.
All reactions