Render graph slots revival: a render graph runner overhaul proposal

[nickguletskii]

I've been experimenting with Bevy for a while and I have some concerns about the direction of the renderer's development. It can be a bit challenging to work with and there are some areas where things are tightly coupled. This means that making even a simple change or addition to the render graph requires copying and modifying a lot of components that I feel should be more composable.

This proposal is designed to address some of the issues I have with the render graph execution engine. It is supposed to make writing and composing render nodes easier by reintroducing slots as the primary method of passing buffers and textures between nodes in a render graph.

The problem

The current render graph runner delegates texture/buffer management to the nodes. This pretty much defeats the purpose of a render graph because all built-in nodes make use of hard-coded source and destination textures. In general, I don't like the idea of nodes relying on components such as ViewTarget for communication.

The solution

Slots should be brought back from the dead and the render graph runner should be modified to allocate, assign and recycle buffers and textures used by nodes. All existing rendering nodes should be refactored to request the required resources before being executed, instead of accessing resources and components like the TextureCache and ViewTarget directly.

A rough specification

• All textures and buffers that can be read by a node should be declared as input slots of the node.
• All textures and buffers produced by the current node that can be read by other nodes should be declared as output slots of the node.
• All internal textures and buffers that can be read or written by the current node and current node only should be declared as workspace slots.
• All internal textures and buffers that need to be persisted between frames should be managed by the nodes themselves.
• When composing a graph, the developer has to link output slots to compatible input slots.
• RenderTargets should be represented as special sink nodes in the graph.

When executing a subgraph, the graph runner should:

1. Perform a topological sorting of the nodes in accordance to slot and node edges.
2. Query each node (in topological order) for its output slot and workspace slot descriptors given the descriptors of the output slots connected to its input slots.
3. Determine the lifespans of each texture/buffer for each output slot. While traversing the subgraph in topological order, for each output and workspace slot, we either take an existing texture or buffer whose reference count has reached zero (thus recycling this texture/buffer), or request a new one if no matching texture/buffer exists. When leaving a node, we decrement the reference count to any textures/buffers used as input buffers.
4. Execute the nodes in topological order, passing textures/buffers according to the plan created in the previous step.

For example, let's say nodes A, B, and C all have a single input and a single output texture, and that all the texture descriptors are the same. If we execute them sequentially, only two textures will be allocated, because once node A has been executed, we can reuse its input texture as the output of B, and after B has been executed, we can use its input texture as the output of C.

A similar algorithm is described in the MXNet whitepaper, but in our case, we don't have to deal with inplace operations.

Implementation status

I've started writing a draft implementation of the graph runner, but it might take me a while to make a working prototype because I don't have much free time.

Impact

These changes will break pretty much every 3rd party implementation of Node. However, I don't believe the current render graph API is sufficient for any serious use anyway, so I think breaking compatibility will be worth it in this case.

Additionally, texture labels will be meaningless since textures will be reused between nodes.

Questions

• Why are Entity and Sampler valid variants of Slot?
• Would it make sense to "flip" slot descriptor calculation and make the nodes calculate the required inputs given the desired outputs rather than outputs given some inputs?

I apologize for any mistakes or unclear descriptions here, I'll read over and correct anything I notice when I am less tired, I just wanted to post it here and get some feedback before I decide to put it into my backlog again...

[IceSentry]

I'm not sure I'd want to go in that direction. Slots add a lot of boilerplate and in most situations simply aren't needed. We almost considered removing them entirely because they simply aren't used in practice. We decided to keep them optional instead of entirely removing them for people that prefer using them.

In practice, you can simply use the ECS to store everything and just use the graph to order everything correctly. Using the graph to also pass the data introduces a lot of boilerplate and new concepts. Using the ECS for everything means you don't need to learn as many apis just to pass data around.

Your idea of recycling textures is interesting, but I'm not entirely sure using slots is the only way to achieve this.

I disagree with the idea that using slots would lead to simpler code. We've had slots for a while and they were much harder to follow than the current approach used in bevy main. Of course, if you can make a render graph using slots with a simple api then it would be interesting, but as it currently is slots are very boilerplate heavy and forces the users to learn a bigger api surface just to do some simple things.

I'm not sure why you think the current render graph is not sufficient for any serious use. Could you expand on this a bit.

Why are Entity and Sampler valid variants of Slot?

Because the render graph can be used for more than just textures. Until very recently, the view target of the current graph being executed was passed as an entity slot. We've now made this view entity a first party of the render graph, but I'm sure there would be other similar use cases. Not sure about samplers though.

Would it make sense to "flip" slot descriptor calculation and make the nodes calculate the required inputs given the desired outputs rather than outputs given some inputs?

I wouldn't be against that change, I've been confused way too many times by this, but I feel like switching it would also end up confusing people.

[nickguletskii]

Sorry for the (very) late reply. Bevy's birthday pushed me to write a reply after all these months :)

Slots add a lot of boilerplate and in most situations simply aren't needed.

The same can be said about most abstractions borrowed from Destiny's renderer architecture as well. I would hope that Bevy aims to compete with the likes of Unreal Engine some day, and I don't think a rendering engine can be considered production-ready without a fully-fledged render graph nowadays (see the links below for reasons why).

We almost considered removing them entirely because they simply aren't used in practice.

They were never implemented properly, so it's no wonder they weren't being used in practice. It seems that they were only used for specifying the execution order of the nodes, not everything else that they are usually responsible for in other game engines.

We decided to keep them optional instead of entirely removing them for people that prefer using them.

At the moment, they don't seem to do much other than determine the order of execution of the nodes, so it's easy to see why you don't consider them important.

In practice, you can simply use the ECS to store everything and just use the graph to order everything correctly.

Storing entities in the ECS does not solve the problems a modern render graph is designed to solve. For instance, how would you implement safe VRAM aliasing if you store everything in the ECS?

I disagree with the idea that using slots would lead to simpler code. We've had slots for a while and they were much harder to follow than the current approach used in bevy main. Of course, if you can make a render graph using slots with a simple api then it would be interesting, but as it currently is slots are very boilerplate heavy and forces the users to learn a bigger api surface just to do some simple things.

As it stands right now, the API surface of Bevy's renderer is already very hard to understand, considering that things don't really work the way you would expect when you have experience with the more established game engines. I would argue that forcing users to do their own resource management and to manually track texture usage descriptions makes everything much harder than having to specify the resource dependencies of each render node.

Anyway, I highly recommend you take a look at what the modern game engines are doing with their render graphs any why they are essential for modern games: Frostbite, Anvil, Unreal. These presentations & pages do a much better job at explaining why slots and slot edges are essential than I ever could in a short forum post.

By the way, I did some thinking about how the rendering API could be simplified and it seems to me that the scheduling API for render nodes is currently very similar to the scheduling API for systems. Perhaps Bevy's system scheduling system could be extended to support transient resource management and dynamic execution schedules and the render graph API could be replaced with plain old Bevy systems?

[JMS55]

Rather than revive render graph slots, I'm playing around with a RenderTask trait that wraps the existing pattern of:

1. Initialize a bunch of specialized pipelines
2. Query cameras with marker component, queue pipeline creation
3. Query cameras with marker component, create resources
4. Query cameras with marker component, and query for other textures in the ECS, create bind groups
5. Add a render node, and in it get all the bind groups and pipelines and dispatch all the passes

It's basically just an automated wrapper around all the duplicate boilerplate I've been writing as I've developed TAA/SSAO/Bloom/EnvMaps/Solari/etc.

WIP is here: https://github.com/JMS55/bevy/blob/render_task/crates/bevy_render/src/render_task