Motion vs MotionPlus benchmark

[algoryunov]

Hello Motion-Project Team,

On behalf of Cloudastructure Company, we want to express our heartfelt appreciation for the outstanding Motion Project, which has been an invaluable asset to our operations for years.

We recently did a study comparing Motion and MotionPlus products in terms of performance and quality. We would like to share with you the results of our study.
All studies were conducted on the same hardware:

• Intel(R) Core(TM) i7-10710U, 6 Cores, 12 Threads
• 32 GB RAM

The applications were running in a docker container, that used ubuntu:22.04 as a base image.

In the graphs below, the abscissa axis represents the number of RTSP streams simultaneously consumed by a single process.
All processes in all configurations uses the same RTSP stream as an input.

The RTSP stream contained looped video in which each frame contained its current sequence number in encoded form:

We analyzed videos recorded by the Motion app. We checked that the frames are sequential, i.e. a frame with the number N must be followed by a frame with the number N+1.
If the sequence was broken (for example, a frame was duplicated or skipped), we increased the skipped frames counter.
If the sequence number could not be decoded (artifacts/glitches), we incremented the Corrupted frames counter.

Also, we analyzed the quality of the recorded video. Since we have the original video, and we know the frame number, we can measure the quality of each recorded frame by comparing it with the original frame. For this purpose we measured the SSIM value (https://arxiv.org/abs/1503.06680):

(SSIM Min/Max/STD are the relatively the same for all these tests):

Video parameters:
Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p(progressive), 1920x1080, 3573 kb/s, 24.60 fps, 24.67 tbr, 12800 tbn (default)

Main config that we used:

    daemon off
    setup_mode off
    log_level 5
    log_type ALL
    netcam_keepalive on
    noise_level 64
    noise_tune off
    minimum_motion_frames 1
    movie_max_time 10
    picture_output off
    picture_quality 100
    movie_output on
    movie_duplicate_frames false
    locate_motion_mode on
    text_right <...>
    text_changes on
    movie_filename %Y/%m/%d/%H-%M-%S
    stream_limit 1
    min_motion_frames 1

Camera Config Template:

	auto_brightness off
	movie_quality 29
	movie_codec mp4
	despeckle_filter EedDl
	post_capture 125
	threshold 75
	width 1920
	height 1080
	framerate 25
	pre_capture 70
	smart_mask_speed  0
	netcam_url <...> 
	target_dir <...> 
	text_left "<...>
	text_scale 3
	record_sound false
	status_file 
	on_camera_lost echo "lost" > <...> 
	on_camera_found echo "found" > <...> 

Our company would like to thank you for your product. Please let us know if you have any recommendations on what we can improve in the MotionPlus configuration.

Thanks,
Cloudastructure Team

[Mr-Dave]

Thanks. The differences associated with the user specified parameter for capture_rate included and made available in the recent versions are my thoughts on the cause for variation.

[algoryunov]

(Sorry for delayed response). Thanks for your response. capture_rate didn't help me.
I don't want to mix things, I did much more analysis in Motion app, not in Motionapp, but my analysis shows that the reason for the issue with skipped frames is that operations are processed in serial manner, so if motion detection or anything else for frame N takes more time to proceed than expected - we may skip the N+1 frame.

[Mr-Dave]

Actually, that brings up a additional point on the "skipped frames" Reading your description I note that you label it as "skipped frames" but in actual fact, the description indicates that it is skipped as well as duplicated or out of sequence. Given that the default for capture_rate is intentionally faster than the actual FPS of the camera, duplication is intentional. Duplication is also a lot different than "skipped"

[algoryunov]

Maybe I'm using the wrong terms, but there are no duplicated frames in the video (no frames N after N, but after frame N I see N+2). I will prepare and provide you a source video and scripts for analysis to capture statistics.

[Mr-Dave]

I would not suggest spending additional time preparing anything for my use. I do not anticipate that I will spend any additional time on such analysis.

[algoryunov]

ok, got it

[Mr-Dave]

An additional cause for the difference in performance is that Motion silently ignores the framerate parameter for motion detection and uses 3fps versus Motionplus honors the user provided framerate specified and (in this case) performs the detection at 25fps.

[StuartIanNaylor]

So does that mean when we are looking at the above motionplus is really 8x faster (lower load) than motion?
That the motion bench are 3fps in reality whilst motion+ is 25fps?

[Mr-Dave]

No. It means that for CPU usage, they are going to be similar once the parameters are aligned a bit better. i.e. Remove the limit in Motion or lower the framerate in each to 3.

[StuartIanNaylor]

Oh I thought the above was with motion hard limited to 3fps but config set to 25 and that was the results.
Apols my bad

[Mr-Dave]

I was curious so I checked the performance with the following results.

Basically, the following confirms a substantial CPU usage difference resulting from the additional detection. This is seen as the difference between Motion 4.6 and Motion 4.6-Adj

CPU Utilization					

Tests	motion-4.3	Motion-4.6	motion-4.6Adj		motionplus	motionplus-adj
Test 1	79.25		63.78		94.58			97.21		96.86
Test 2	78.70		64.76		95.20			96.67		96.94
					
4.6 Adj:  A special version that implements motion detection on each frame					

Motionplus-adj:  A special version that removes the standard deviation calculations.					

Each Test:	Application start up time (not in measurements)			15 seconds	
		Durations between samples					1 second	
		Total duration of each test					10 minutes	
		Camera			Pi camera via libcamerify		
		Computer		Pi zero		
		Frame rate		24		
		Resolution		1280x720