frigate works with 2 cams, but not with more

[blaess]

frigate 7.3 and 8.0beta1 runs fine with 2 cams. But if I'm adding one more, I run into the following problem (on both 7.3 and 8).
Two stream are working, the 3rd not. It's random, always one goes into a timeout (20s?).
Image: 0.8.0-beta1-aarch64 on Raspberry 4 (8Gb)

frigate | * Starting nginx nginx
frigate | ...done.
frigate | frigate.app INFO : Creating directory: /media/frigate/recordings
frigate | frigate.app INFO : Camera processor started for east: 36
frigate | detector.coral INFO : Starting detection process: 34
frigate | frigate.edgetpu INFO : Attempting to load TPU as usb
frigate | frigate.app INFO : Camera processor started for sud-east: 38
frigate | frigate.app INFO : Camera processor started for sud-west: 40
frigate | frigate.app INFO : Capture process started for east: 42
frigate | frigate.app INFO : Capture process started for sud-east: 44
frigate | frigate.app INFO : Capture process started for sud-west: 47
frigate | frigate.edgetpu INFO : TPU found
frigate | watchdog.sud-west INFO : No frames received from sud-west in 20 seconds. Exiting ffmpeg...
frigate | watchdog.sud-west INFO : Waiting for ffmpeg to exit gracefully...
frigate | watchdog.sud-west INFO : FFmpeg didnt exit. Force killing...
frigate | frigate.video INFO : sud-west: ffmpeg sent a broken frame. something is wrong.
frigate | frigate.video INFO : sud-west: ffmpeg sent a broken frame. something is wrong.
frigate | frigate.video INFO : sud-west: ffmpeg sent a broken frame. something is wrong.
frigate | frigate.video INFO : sud-west: ffmpeg process is not running. exiting capture thread...
frigate | [mov,mp4,m4a,3gp,3g2,mj2 @ 0x5579885c10] moov atom not found
frigate | /tmp/cache/sud-west-20201217173754.mp4: Invalid data found when processing input
frigate | frigate.events INFO : bad file: sud-west-20201217173754.mp4
frigate | watchdog.sud-west INFO : No frames received from sud-west in 20 seconds. Exiting ffmpeg...
frigate | watchdog.sud-west INFO : Waiting for ffmpeg to exit gracefully...
frigate | watchdog.sud-west INFO : FFmpeg didnt exit. Force killing...
frigate | frigate.video INFO : sud-west: ffmpeg sent a broken frame. something is wrong.
frigate | frigate.video INFO : sud-west: ffmpeg sent a broken frame. something is wrong.
frigate | frigate.video INFO : sud-west: ffmpeg sent a broken frame. something is wrong.
frigate | frigate.video INFO : sud-west: ffmpeg sent a broken frame. something is wrong.
frigate | frigate.video INFO : sud-west: ffmpeg sent a broken frame. something is wrong.
frigate | frigate.video INFO : sud-west: ffmpeg process is not running. exiting capture thread...

[walthowd]

Can you check if your /tmp/cache directory inside the container is filling up?

[blaess]

yes, looks good, for all 3 streams

root@db30f959632c:/tmp/cache# ls
east-20201218112440.mp4 east-20201218112510.mp4 east-20201218112540.mp4 east-20201218112610.mp4 sud-east-20201218112459.mp4 sud-east-20201218112529.mp4 sud-east-20201218112559.mp4
east-20201218112450.mp4 east-20201218112520.mp4 east-20201218112550.mp4 sud-east-20201218112439.mp4 sud-east-20201218112509.mp4 sud-east-20201218112539.mp4 sud-east-20201218112609.mp4
east-20201218112500.mp4 east-20201218112530.mp4 east-20201218112600.mp4 sud-east-20201218112449.mp4 sud-east-20201218112519.mp4 sud-east-20201218112549.mp4 sud-west-20201218112530.mp4

[blakeblackshear]

If the resolution is high, you may also need to increase your shm-size

[blaess]

Looks good too!?

root@db30f959632c:/dev/shm# df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
shm 1.0G 13M 1012M 2% /dev/shm

But I see a lot of files from the cam which is not working on the /dev/shm directory.

[rhatguy]

I'm having similar issues on a RPi 4 8GB running the aarch64 image. I have 5 Reolink cameras. I've increased gpu mem in the /boot/config.txt to 512M (RPi4 won't boot with this set to 1024M). I've bumped the shm-size in the docker-compose.yml up to 2g. I'm actually only running the small stream on my reolink cameras right now (640x480) as I can't seem to get it stable. I keep getting weird "smeared" images when I watch the streams at frigate:5000/cam_name like the one attached here. I'm beginning to wonder how much the RPi 4 should be able to decode. It seems to do ok with 1 camera, but when I start enabling more cameras or high bandwidth streams things get unstable. Does anyone know how many streams a RPi4 8GB should reasonably be able to handle? I know the page says a "small" number, but how small is small?

ffmpeg:
hwaccel_args:
- -c:v
- h264_v4l2m2m
detectors:
coral:
type: edgetpu
device: usb
cameras:
front_door_cam:
ffmpeg:
inputs:
- path: rtsp://user:password@front_door_cam.travisandliz.com:554/h264Preview_01_sub
roles:
- detect
- rtmp
height: 480
width: 640
fps: 7

[flaektrem]

FYI, I loaded the new firmware on two of my cameras. One is a POE RLC-410-5MP and the other is a wifi based RLC-511-W. Now RTMP doens't work on either of these cameras. It just gets an Input/Ourput error when I try to load the stream via ffprobe. VLC or rtmpdump similarly don't work anymore. Neither does the HTML web gui. It seems like the RTMP service is just completely non-functional after the upgrade. I'm working through the issue with support, but as per usual you only get one email form them per day, so it takes a while.

On the plus side, I have been able to get all 5 RTSP streams working with my RPi4 8GB. I've bumped up a couple of different options and turned the cameras down a bit from what I had them at previously. I'm not to concerned about the memory usage on this PI since its dedicated and has 8GB, so I may have gone overboard a bit on some of the settings.

* /boot/config.txt - set gpu_mem=512 #(setting this to 1024 caused the PI to not boot.  512 seemed to be as high as it could go)

* docker-compose.yaml - tmpfs size 1000000000 #(set to 1GB.  only seeing ~500MB of utilization)

* docker-compose.yaml - shm_size '2g' #(set this to 2GB.  Only seeing about 1% utilization inside the container on df -h /dev/sdm)

* config.yaml - set "sub" stream for detect role, set "main" stream for clips and rtmp

* config.yaml - set ffmpeg input_args to match https://github.com/blakeblackshear/frigate/blob/a1cc9ad1f0a3ea3e3a1c8163c5ff3c27065a9be4/docs/cameras.md

* cameras - set "clear" stream to 2560_1920 at 15 FPS, with max bitrate of 8192.  Set "fluent" stream to 640_480 at 4 FPS with max bitrate of 512.

It seemed like when I was testing the biggest difference was when I dropped the FPS to 4 on the "fluent" sub stream. I suspect this may be the case since thats the stream that frigate is always running detection against, while the full bandwidth streams are only used after a detection or while streaming to Home Assistant. I may play with reducing some of the memory usage a bit on some of these things, but I want to get to a stable working config first, maybe switch to RTMP if I can figure out whats going on with the cameras, then start reducing stuff.

Since this thread was about frigate working with 2 cams, but not more, I wanted to confirm that the RPI4 8GB is capable of working with at least 5 streams with the settings above.

Solved this with support yet?

[rhatguy]

No. I've been emailing back and forth with them but unless you want to email them at 3AM EST it seems like it takes 2 days for a round trip on email. They said they were going to get back with me yesterday after escalating the issue to a "senior" engineer but I haven't heard back yet. Not only does RTMP not work now but my live view on the camera webpage doesn't work and the cameras seem generally less stable even with rtsp as more than once now I've noticed one of the new upgraded ones being down in BlueIris (using RTSP). I am avoiding upgrading the rest of my cameras to the newer HTML5 based Reolink firmware until we can sort out this issue.

[flaektrem]

No. I've been emailing back and forth with them but unless you want to email them at 3AM EST it seems like it takes 2 days for a round trip on email. They said they were going to get back with me yesterday after escalating the issue to a "senior" engineer but I haven't heard back yet. Not only does RTMP not work now but my live view on the camera webpage doesn't work and the cameras seem generally less stable even with rtsp as more than once now I've noticed one of the new upgraded ones being down in BlueIris (using RTSP). I am avoiding upgrading the rest of my cameras to the newer HTML5 based Reolink firmware until we can sort out this issue.

Still the same issues?

[rhatguy]

Actually I got them working. Its a very strange situation though. I have a local DNS server on my router and have my cameras setup such that under normal circumstances I can connect to them through the DNS name. garage_cam.mylocaldns.com for instance. With this new firmware, the RTMP streams WILL NOT work when using the DNS name. If I connect to them using the IP directly however, the RTMP stream works fine. The same is true whether connecting to the web gui through the browser or using frigate. I can't explain it as I thought DNS was strictly a client side action, but apparently the camera is aware in some way of the fact that I connected through a DNS name and doesn't work. I gave up on investigating with Reolink and changed the cameras I use inside frigate (that I've upgraded to the new firmware) over to using IP instead of DNS. I haven't upgraded any more of my cameras since then, so I now have 2 reolinks running the newer firmware and rtmp streaming to frigate via IP, and 3 reolinks running older firmware still using RTSP and DNS names. Knock on wood, but my RPI4 setup with these has been rock solid for a while now, so I haven't messed with anything lately.

[flaektrem]

Ah I see. I just use regular IP to connect to my cameras so maybe i will not have any issues. I´ll try :)
Thanks for your reply.

[tubalainen]

Some factors comes into play when it comes to CPU usage.
1.) Resolution of the rtsp stream - try to lower it on the camera.
2.) Frame rate of the rtsp stream - try to lower it on the camera.
3.) Make a mask and mask out all things that moves around in the picture such as trees, clouds, bushes etc. Mark out all areas where you do not want to detect movement. For me, running six cameras (on an old laptop) that made a lot of difference to the CPU load.
https://github.com/blakeblackshear/frigate#masks
4.) Test with only the "detect" setting
5.) Also, make sure that you have setup the tmpfs correctly - spooling to an SD-card can cause bottle necks if there are simultaneous writes from different cameras (not likely, but please check)
6.) And, of course, shm-size if your logs reports any issues with that.

[blaess]

Adding the line "gpu_mem=256" at /boot/config.txt fixed the issue.
Four cams with 1080p/5fps just running fine now!

[dgillier]

@blaess
I'm very curious of what we can get on a RPI4 + coral:

• I'm assuming your are using hardware acceleration on ffmpeg H264 decoding ?
• What are the CPU use in your case ?
  Thanks for your help :)
  Denis

[rhatguy]

I have a RPi4 with a coral and 5 reolink cams with detection on a small 640x480 stream from each, then doing rtmp and clips on the larger 5MP streams. I'm using the RPIs built in H264 decoding acceleration. CPU usage goes up and down pending whether the system is actively detecting motion or if you're watching streams through the HASS interface but I generally see it between 20-40%. I get the sense this setup might handle another couple of cameras, but not to many more without running out of horsepower. Given the cost of the hardware and the power usage considering its running 24x7 I feel like this would be a hard setup to beat given what it can handle.

[dgillier]

@rhatguy Thanks a lot for your input !