Question regarding effects and GPU

[space-lux]

[TL;DR] : what about GPU computation for effects ?

The end of Hardware Acceleration mentions future improvements based on Vulkan Compute Shaders.

With my (limited) knowledge of OpenGL, I think that it would already be possible to do some GPU computations with framebuffers and fragment shaders, and thus improve performance a bit.

If I have the time, I may propose implementation for this in a not-so-far future, but first I would like to know what are the thought of developers about all this. It may require to rewrite big part of the code, and it would probably need to manage frames duplicates.

Actually, I have tons of thing to ask and say on the subject.
What are your (main developers, contributors, users) opinions? Would it be useful? How do you (fore)see this? I haven't seen much about it in the issues, neither in the forums (though there are lots of questions about encoding/decoding).

[ferdnyc]

@space-lux Funny you should mention that, as some of these topics recently came up in OpenShot/openshot-qt#3448 as well. I'll quote part of my comment there...

RadeonOpenCompute, which is basically an OpenCL implementation AIUI, is mentioned in passing as a possible way to accelerate effects. Which is about right -- GPU computing might be good for some intensive computations, including the image processing operations performed by some effects and filters, but video encoding doesn't necessarily benefit from it that much. At least, not automatically.

Blender benefits from CUDA/OpenCL/etc. Hugely. Possibly some of our effect algorithms could. (They'd have to be rewritten for the GPU, though, just like Blender's rendering algorithms have been.)

Then I digressed into some notes about NVENC, before coming back around to CUDA/OpenCL...

MOST consumer video cards don't have the kind of power to accelerate CUDA algorithms significantly. So the user base for this kind of general-purpose GPU computing is relatively small even compared to the OpenShot user base as a whole.

My video card is mostly meant for playback, and it accelerates decoding fine, plus gives me a reasonable boost for video encoding. I think last I checked it was something like 2-4x as fast as my CPU when encoding x264.

But for GPU computing? Like I've said, it's SLOWER than my CPU is, when rendering in Blender. There is no real benefit to CUDA & friends for anyone with a system like mine, one that wasn't built for power and uses a non-"gamer"-level, sub-$200 graphics processor. So, while there might be speedups to be had for some users, I wouldn't want to leave the remaining majority out in the cold by focusing on features exclusively targeted towards users with high-end hardware. (Or, even worse, making OpenShot only support those systems.)

Now, all that being said, OpenGL is a different thing because, like hardware-accelerated decoding/encoding, it is well-supported by nearly every GPU currently out there, and even the least capable of them will tend to show significant performance gains.

So I think you're probably right, there's at least a possibility of accelerated OpenGL effects improving performance in a way that'd be generally useful to (and usable by) most users. It's much less of an exclusionary/power-user-only feature, which is the thing that concerns me with other types of hardware acceleration. I definitely think there's room for at least a selection of effects implemented in OpenGL, to achieve better performance compared to the strictly software-based effects we have now.

My personal preference, though I can't speak for anyone else (including and especially other OpenShot developers), would still be that the creation of any OpenGL-based effects take the form of adding more effect choices alongside the existing unaccelerated versions, rather than for the current effects to be replaced/rewritten. That way OpenGL doesn't necessarily become a hard requirement to use OpenShot and/or its effects. (Though that may be a moot point, really — it's possible just using Qt means that there's an implicit OpenGL requirement already.)

Dynamic effect modules

The work that @kirillmakhonin started in #345 to modularize our effects library so that additional effects could be made available at runtime in the form of dynamically loadable modules, rather than all having to be compiled into libopenshot, might also be of interest. It'd be great if deciding to make libopenshot available with the OpenGL effects vs. without them was a runtime choice of which pieces to install, rather than a compile-time decision where providing both options would require two separate builds of the entire library.

Current pipeline

Currently in OpenShot, effects are attached to timeline clips.

(Although in libopenshot's internal implementation that's not necessarily true, the backend support for effects is more flexible and allows them to be placed anywhere on the timeline, with their own start and stop points that don't necessarily correspond to any one clip. Limiting the placement of each effect to be aligned only with a single clip is a restriction imposed by OpenShot.)

Nevertheless, at least in OpenShot an effect modifies a particular clip, meaning conceptually it can be thought of as an input processing step. So it seems likely that the benefits from OpenGL-accelerated effects would be greatest when combined with hardware decoding, especially if the effect could process a just-decoded frame while it's still on the hardware, without it having to leave the GPU until after the effect is applied.

I suspect that libopenshot's decoding implementation currently isn't designed to accommodate that very well, meaning even if they're presented in a similar fashion, OpenGL effects may require a completely different implementation from the existing effects, one that can tie the effect into the hardware decoding pipeline before the decoded frames are even read from the GPU by FFmpegReader.

So that part could get somewhat involved, if it's necessary. Maybe it's not — I don't have a great deal of experience with OpenGL, really, so I'm mostly making the assumption there's a significant performance penalty involved if we'd be decoding each input frame, copying it to application memory, sending it back to the GPU in order to process it using an effect, then retrieving the modified result. (And perhaps then lather, rinse, repeat for each additional effect.) But maybe there are some techniques for doing that so it doesn't have too great an impact on performance?

[space-lux]

Thank you for your answer!
Since then I've read a bit libopenshot's code, especially the Frame class and what you said about effects on Timelines, and I checked some details in Qt's doc.

About requirements

From what I've read, OpenGL is not a hard requirement for Qt (see Qt for X11 requirement) but it seems to be strongly advised, since there are many Qt classes that depend from it. Regarding Windows, OpenGL is (almost?) always available, and OpenGL ES seems to be required, but I don't know how it works for MacOS.
In any case, OpenGL-related classes (such as QopenGLContext) are included in the gui package, that is already included for QImages... So, it wouldn't need to change anything in libopenshot's build system, buttt... Finally, nothing is assured concerning OpenGL availability. I guess we would need to know what people do with the lib.

OpenGL in the current pipeline

Assuming that we have access to OpenGL, it is possible to write some effects in a very similar way they are for now: taking a Frame in input, returning another. Input Frame video data (::bits()) could be considered as a texture buffer, and the returned one as a render buffer for the GPU. This is a terrible technique with data going back and forth between CPU and GPU, fortunately I believe that would still be faster than doing the computation on CPU. The reason for that is that GPUs have pretty fast access to RAM, and usually perform filter-like (convolutions...) operations way faster than CPU. Furthermore, the parallelism that is currently in place (with OpenMP) is a good thing, but not as perfect as what happens on GPUs.

OpenGL with a slightly modified pipeline

I thought of some additions to the Frame class : it could internally handle transfers between main RAM and GPU's memory. In that way, transfers would occur only when needed, and a Frame that lies between two GPU effects would keep things in the right place. At least that's what I suppose. It would take the form of an additional method, Frame::getGLBuffer() or something like that.

OpenGL with a very modified pipeline

This is the first idea I had, but I hadn't read the code about effects in timeline yet.
In the case of a stack of effects, I think there is a way to compile a giant shader that applies all effects in just one GPU call. It raises several questions (Are fragment shaders adapted to this case? I don't believe so. Are Compute Shaders a good way to go? Probably, but they need a really later version of OGL, 4.2+ if I'm correct.) but is probably the best way to the fastest render.

I have not yet installed libopenshot's build stack, but I will. Then I will write GPU versions of some effects, and carry performance tests – with different hardware configurations.
Also and not related, I suspect OpenMP usage is not optimal (at least in some cases), so I will also test some changes around this and open another issue if that's relevant.

EDIT :
I also looked at what was done about the dynamic effect modules, and it seems to me that, for now, the topic is not very active. It would be great to see it work but I will not rely on it for my experiments.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.