At the end of the sliced video via split-video there is a frame of the next video

[whatforiam]

Description:

At the end of the sliced video via split-video there is a frame of the next video. I cut the anime into moments and on some videos it worked poorly, as a result, either 1-2 frames from the last moment are shown at the beginning, or the next one appears at the end for 1 frame

Command:

scenedetect -i input/input.mp4 split-video -o output -hq -p fast

Output:

BUG_REPORT.txt

Environment:

[PySceneDetect] PySceneDetect 0.6.4

System Info

OS Windows-10-10.0.19045-SP0
Python 3.9.13

Packages

av 10.0.0
click 8.1.7
cv2 4.10.0
moviepy Not Installed
numpy 1.26.4
platformdirs 4.2.2
scenedetect 0.6.4
tqdm 4.66.4

Tools

ffmpeg 6.0-full_build-www.gyan.dev
mkvmerge Not Installed

Media/Files:

https://github.com/user-attachments/assets/cf4dac57-632f-44a6-8893-d0ad1bf64cde
https://github.com/user-attachments/assets/379f5981-7e44-41a7-a277-583907542869
https://github.com/user-attachments/assets/c79a5b5a-1ade-4cee-8a43-66624aa839b9
https://github.com/user-attachments/assets/6ac030fc-a95d-4cde-a68e-03fd14209f8b
https://github.com/user-attachments/assets/ca551609-7a9e-46a4-a54d-6de1921a8ff8
https://github.com/user-attachments/assets/2bc46619-b1db-4917-a9f5-8e0c1c3ad124

[Breakthrough]

Can you share the original clip? Does the issue happen with all videos or only a few?

[whatforiam]

Can you share the original clip? Does the issue happen with all videos or only a few?

I haven't checked, but I suspect on all anime videos, because they are drawn the same way

its original clip
https://drive.google.com/file/d/1a2el3LT2A_EOhzCWws3jvk2hLmOeEMFG/view?usp=sharing

[Breakthrough]

Thanks for sharing, I was able to repro the issue. The issue isn't with detection but with PySceneDetect converting timecodes to frame numbers and back. This causes issues with certain containers/streams especially when there are corrupt frames. It should be fixed by #168 but I don't have a timeline for that fix yet unfortunately, since it is a very big change.

[Breakthrough]

@whatforiam sorry my initial theory might be incorrect - can you try setting the backend to PyAV with --backend pyav e.g.:

scenedetect -i input/input.mp4 -b pyav split-video -o output -hq -p fast

And let me know if that fixes the issue? You can also try moviepy if you have it.

---

So far from my testing this video has different output with each supported backend (moviepy, opencv, and pyav) which is very confusing. My theory about the timecodes was close - after doing a test on the sample you provided I found the frame rate is different depending on the backend used:

import scenedetect
>>> opencv = scenedetect.open_video("issue-431.mp4", backend="opencv") 
>>> pyav = scenedetect.open_video("issue-431.mp4", backend="pyav")     
>>> moviepy = scenedetect.open_video("issue-431.mp4", backend="moviepy") 
>>> opencv.frame_rate
23.97601106386279
>>> pyav.frame_rate  
23.976023976023978
>>> moviepy.frame_rate
23.976023976023978
>>> 

This is probably leading the timecodes calculated for split-video to be different since they are based off of frames, making this bug more likely to occur near the end of longer videos. Unfortunately this means each backend may require specialized timecode handling for this case.

With PyAV we use av_guess_frame_rate. This is what OpenCV did until recently, when it was changed about a year ago:

https://github.com/opencv/opencv/blob/3901426d8515cd8d1c07654ec7e9e8bc3b51a06f/modules/videoio/src/cap_ffmpeg_impl.hpp#L1897-L1918

[whatforiam]

scenedetect -i input/input.mp4 -b pyav split-video -o output -hq -p fast

Thank you so much. This command solved the problem!

But I have another question. Is there any way to improve the detect of places where need to cut? As I understand it, the program is designed primarily for real videos, but are there any settings for a better detect for such 2d animations that I gave as an example? It's just that it either cuts too often, or vice versa in one moment 2 different angles. I just wanted it to separate into a separate file every time I changed the angle

Of course, I wrote a script for bat to combine these files when dragging into one, but considering how many such videos I have, and how much I have to do this (>5000 series), I would like to make the work easier

here is my code for combining videos when dragging, if it is useful, I wrote it myself (but the sound disappears a little at the joints of the video, I did not come up with how to solve this, so I wanted to at least reduce the number when it is necessary to combine)

chcp 65001
for %%i in (%*) do (
    if /i not "%%~xi"==".exe" (
        echo file '%%~i' >> txt.txt
    )
)
ffmpeg -f concat -safe 0 -i "txt.txt" -c copy output.mp4
for %%i in (%*) do (del /s /q "%%~i")
ren output.mp4 "%~n1.mp4"
del txt.txt

[Breakthrough]

You can change the weight placed on the luma channel for detect-content/detect-adaptive and tweak other parameters. For example try adding this to your command:

detect-adaptive -w 0.5 1.0 0.0 0.0 -c 27 -t 5

You can try saving a statsfile and using that to help guide tuning each detector.

in one moment 2 different angles

Keep in mind that the default setting for minimum scene length is 0.6 seconds, which can lead to some issues like this where it seems to "miss" these shot changes. You might want to lower it further to say 0.2-0.3 seconds. You can also customize how sensitive each detector is by lowering the threshold. See the docs for whatever detector you use for a full list of params.

Please do let me know what parameters end up working best for your use case :)

[wjs018]

the frame rate is different depending on the backend used

Could this be due to precision differences between the different libraries? It looks like moviepy and pyav are the same while OpenCV is different. I wonder if a solution for this specific framerate (23.976 comes from 3:2 pulldown), would be to check to see if the detected framerate is very, very close to 23.976, and if so, then simply define it as such, overriding what the library automatically detects? Not sure what that would break though.

[whatforiam]

You can change the weight placed on the luma channel for detect-content/detect-adaptive and tweak other parameters. For example try adding this to your command:

detect-adaptive -w 0.5 1.0 0.0 0.0 -c 27 -t 5

Yes, thank you very much for your help. I came to the conclusion that it is either better without any additional settings at all

scenedetect -i input/input.mp4 -b pyav split-video -o output -hq -p fast

or with these (This is a script .bat that processes each video in the input folder and saves it to the output folders_(File name))

for %%i in ("input/*") do (
    scenedetect -i "%input%" -d 2 -b pyav detect-adaptive -w 0.1 0.1 1.0 0.0 -c 34 -t 2.7 split-video -o "%folder%" -a "-c:v libx265 -crf %crf%" -p fast
)

That 's another one who needs mine .bat file for auto-cutting anime series

@echo off
setlocal enabledelayedexpansion
cd /D "%~dp0"
set /p folder=FOLDER NAME=
set /p num=STARTING WITH=
set /a num=num-1
set /p crf=CRF=
set num2=-1
if not defined crf (set crf=19)
if %crf%==0 (goto copy)
:crf
cls
for %%i in ("input/*.mp4") do (
    set /a num2=num2+1
    if !num2! GEQ !num! (cls & set /a "num=num+1" & echo +++ Processing [%%~ni] to [%folder%\EP!num!] +++ & scenedetect -i "input/%%~i" -d 2 -b pyav detect-adaptive -w 0.1 0.1 1.0 0.0 -c 34 -t 2.7 split-video -o "anime\%folder%\EP!num!" -a "-c:v libx265 -crf %crf%")
)
exit
:copy
cls
for %%i in ("input/*.mp4") do (
    set /a num2=num2+1
    if !num2! GEQ !num! (if not defined skipping_stop (echo skip !num2! & set skipping_stop=1) & cls & set /a "num=num+1" & set /a "num2=num2+1" & echo +++ Processing [%%~ni] to [%folder%\EP!num!] +++ & scenedetect -i "input/%%~i" -d 2 -b pyav detect-adaptive -w 0.1 0.1 1.0 0.0 -c 34 -t 2.7 split-video -o "anime\%folder%\EP!num!" -c) else (echo skip !num2!)
)
exit

of course, this did not completely solve the problem, but it reduced the number of unnecessary triggers

Should I close the repo or do you still need any information?

[Breakthrough]

the frame rate is different depending on the backend used

Could this be due to precision differences between the different libraries? It looks like moviepy and pyav are the same while OpenCV is different. I wonder if a solution for this specific framerate (23.976 comes from 3:2 pulldown), would be to check to see if the detected framerate is very, very close to 23.976, and if so, then simply define it as such, overriding what the library automatically detects? Not sure what that would break though.

@wjs018 I looked into this just now, but it looks like both ffmpeg and OpenCV are using double for these calculations. Initially I thought it was the same thing, 32-bit float issues. Your theory though is still plausible, could be some intermediate calculations or differences in what options the libs were compiled with.

Hard-coding a value for cases like this is a reasonable one, and would probably not break too many things. The most important thing is that we get the timecode calculations aligned with what ffmpeg expects for split-video functionality, as all the other output commands work on the VideoStream directly so there will never be visual off-by-one issues like this. I'll do some digging into how ffmpeg translates these timestamps back into timecodes or how it uses start seek and -t.

It's a bit worrying that OpenCV doesn't seem to work even when I use CAP_PROP_POS_MSEC since I thought that would fix this, but it didn't. Makes me wonder if we should consider switching the default backend to PyAV until we can sort that out. Thoughts?

---

@whatforiam sure, please do. You might also want to try the new detect-hist command with a very high threshold, e.g. detect-hist -t 0.5. Lastly, note that downscaling might be rendering some details out, so you might want to lower downscale factor so the processing happens at higher resolution. That will come with performance cost however.

Thanks for the report and sharing the params you are using, I'll close the issue once I've done some further investigation into OpenCV/FFmpeg differences here.

---

Before closing this out I think we should also add an FAQ for this to the website and the docs since this does come up frequently enough.

Tasks:

• Add FAQ entry to docs/website so folks know what to do when this happens (e.g. have workarounds available and can understand the problem)
• Investigate changing the default backend to PyAv
• Investigate alternative implementations for the split-video command that might avoid the issue:
  • We could encode frames on the fly using PyAV, but then we need to parse encoder options which there are way too many
  • Since ffmpeg already handles we could use MoviePy to stream frames out to the ffmpeg process, making it work with user-specified args like today

[IncredibleLaser]

Something that could be looked into is also how these timecodes are exported with list-scenes. I once noticed that the results for split-video and using the values from the split file with ffmpeg separately are different for videos that have framerates like 24000/1001. This seems to be due to the rounding error introduced in scene_manager.py by formatting values with %.3f. This can round up, causing ffmpeg to include an additional frame from the next scene IIRC (or the other way around, I'm not quite sure anymore and would need to find some sample).

[whatforiam]

Something that could be looked into is also how these timecodes are exported with list-scenes. I once noticed that the results for split-video and using the values from the split file with ffmpeg separately are different for videos that have framerates like 24000/1001. This seems to be due to the rounding error introduced in scene_manager.py by formatting values with %.3f. This can round up, causing ffmpeg to include an additional frame from the next scene IIRC (or the other way around, I'm not quite sure anymore and would need to find some sample).

I noticed that after I was advised to add pyav, most of my frames became normal, but sometimes not one frame is added to the end, but a whole second, I do not know how to fix it

[Breakthrough]

Something that could be looked into is also how these timecodes are exported with list-scenes. I once noticed that the results for split-video and using the values from the split file with ffmpeg separately are different for videos that have framerates like 24000/1001. This seems to be due to the rounding error introduced in scene_manager.py by formatting values with %.3f. This can round up, causing ffmpeg to include an additional frame from the next scene IIRC (or the other way around, I'm not quite sure anymore and would need to find some sample).

@IncredibleLaser thanks for bringing that up, I wasn't aware that was an issue. Filed #464 to track this issue.

I noticed that after I was advised to add pyav, most of my frames became normal, but sometimes not one frame is added to the end, but a whole second, I do not know how to fix it

@whatforiam could you provide a logfile to help debug the issue and also provide the version of PyAV you are using (run scenedetect version)? You can generate a debug log by adding -v debug -l BUG_REPORT.txt to the beginning of your command.

[whatforiam]

Something that could be looked into is also how these timecodes are exported with list-scenes. I once noticed that the results for split-video and using the values from the split file with ffmpeg separately are different for videos that have framerates like 24000/1001. This seems to be due to the rounding error introduced in scene_manager.py by formatting values with %.3f. This can round up, causing ffmpeg to include an additional frame from the next scene IIRC (or the other way around, I'm not quite sure anymore and would need to find some sample).

@IncredibleLaser thanks for bringing that up, I wasn't aware that was an issue. Filed #464 to track this issue.

I noticed that after I was advised to add pyav, most of my frames became normal, but sometimes not one frame is added to the end, but a whole second, I do not know how to fix it

@whatforiam could you provide a logfile to help debug the issue and also provide the version of PyAV you are using (run scenedetect version)? You can generate a debug log by adding -v debug -l BUG_REPORT.txt to the beginning of your command.

[PySceneDetect] PySceneDetect 0.6.4

System Info

OS Windows-10-10.0.19045-SP0
Python 3.9.13

Packages

av 10.0.0
click 8.1.7
cv2 4.10.0
moviepy Not Installed
numpy 1.26.4
platformdirs 4.2.2
scenedetect 0.6.4
tqdm 4.66.4

Tools

ffmpeg 6.0-full_build-www.gyan.dev
mkvmerge v45.0.0 ('Heaven in Pennies') 64-bit

BUG_REPORT.txt