How to allocate different static IP addresses for pod replicas of statefulset using multus

[rookrunner]

I have 2 NetworkAttachmentDefinition with same definition like below, only IP are changed:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  annotations:
    meta.helm.sh/release-name: multus
    meta.helm.sh/release-namespace: test
  creationTimestamp: "2024-07-13T01:46:19Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: vlan1
  namespace: test
  resourceVersion: "306722382"
  uid: 43bac009-6d45-4730-b8ee-db1e54528bc1
spec:
  config: |-
    { "cniVersion": "0.4.0", "name": "vlan1", "plugins": [{ "type": "ipvlan", "master": "vlan1",
    "ipam": { "type": "static", "addresses":[{ "address": "10.13.19.10/24", "gateway": "10.13.19.1" }] } },{ "type": "sbr" }] }
    

Now, I want to assign these 2 NetworkAttachmentDefinition to 2 different pod replicas of a statefulset such that both the pods are having different IP's. Is it possible. Can you please help here

[rookrunner]

I tried using whereabout for this. And, now I am able to assign different IP's to the replicaset.

But, facing another issue. When I am doing deployment IP's were properly allocated to the replicas of the statefulset.
But, when I tried upgrade looks like the IP's are not getting released by multus:

Warning FailedCreatePodSandBox <invalid> kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0889f8fc31b1fa54ee3a89d8864e7e943c8a93a952c6451b3c3ad5feb56f586f": [testnamespace/pod-1:multustestnamespace2]: error adding container to network "multustestnamespace2": Error at storage engine: Could not allocate IP in range: ip: 10.93.97.12 / - 10.93.97.13 / range: net.IPNet{IP:net.IP{0xa, 0x5d, 0x60, 0x0}, Mask:net.IPMask{0xff, 0xff, 0xf0, 0x0}}

Because of this my pod is not getting the IP and it remains in ContainerCreating state.

I have found this ticket which suggests to use cniVersion: 0.4.0. I am already using cniVersion: 0.4.0. Still facing the issue:
#738

This is my net-attach-def:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
annotations:
meta.helm.sh/release-name: multus
meta.helm.sh/release-namespace: testnamespace
creationTimestamp: "2024-07-26T12:44:11Z"
generation: 2
labels:
app.kubernetes.io/managed-by: Helm
name: multustest
namespace: testnamespace
resourceVersion: "493917757"
uid: 9779f5ab-e969-4e04-9368-5d022746093c
spec:
config: |-
{ "plugins": [ { "cniVersion": "0.4.0", "type": "ipvlan", "master": "eth1",
"mode": "l2", "ipam": { "type": "whereabouts", "range": "10.93.96.0/20", "range_start": "10.93.97.12", "range_end": "10.93.97.13", "log_file" : "/tmp/whereabouts.log", "log_level" : "error" } }, { "type": "sbr" } ] }

To be noted I have only 2 IP's in the range(10.93.97.12 - 10.93.97.13). Also, when I dont use sbr, it seems to be working.
So, IP not releasing problem only exists with "sbr"

Could anyone please help

[rookrunner]

@dougbtv Could you please help here. Is this a known issue with chained CNI plugins.

[rookrunner]

An update:
This config has worked. The only difference being "cniVersion" outside of plugins.
Seems "cniVersion" is part of both "plugins" as well as "config" structs. I had put it in wrong place.

  config: '{ "cniVersion": "0.4.0", "plugins": [ { "type": "ipvlan", "master": "eth1",
    "mode": "l2", "ipam": { "type": "whereabouts", "range": "10.93.96.0/20", "range_start":
    "10.93.97.12", "range_end": "10.93.97.13" } },{"type": "sbr"}] }'

Thanks!