Is there an ability to automatically assign vf with GPU affinity to pods?

[cyclinder]

If the gpu and nic are on the same PCIe bridge or their topology distance is at least PHB, then communication between them can be accelerated by enabling GPU Direct RDMA.

[SchSeba]

that is a kubernetes feature. you can configure device manager and check the topology type

https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/#policy-single-numa-node

[cyclinder]

Thanks for your reply, I think even if GPU and Nic are in the same NUMA nodes, they may still cross the PCIe bridge, as shown in the figure above, GPU0 and mlx5_3, so in this case, we cannot enable GPU Direct RDMA. The same NUMA nodes may be a large distance, we may need a smaller distance.

[adrianchiris]

currently there is no solution that im aware of which takes into account PCIe topology.

DRA (Dynamic Resource Allocation) aims to solve that, but there is still a way to go....

[aojea]

This is on DRA roadmap as @adrianchiris mentions, it will be beta in 1.32

[SchSeba]

Hi @cyclinder, do you think we can close this issue?
I don't think there is something the sriov-operator can do about this.

I think for now the only solution is to manually have a resourcePool for every pcie and do a static request for GPU pool and VF pool from the same pcie :(

[cyclinder]

@SchSeba Yes we can close the issue, we are planning to implement this in these days.