VrfContext object not found

[vitality411]

Describe the bug

I am trying to use AKO with NSX-T cloud in a dedicated non-admin tenant. I have deployed all the required infrastructure in NSX-T and the AVI tenant. A manually created VS works fine. But AKO cannot set up static routes on VRF because it tries to look for them in the admin tenant (note the words tenant admin):

ako-0 ako 2024-03-19T15:23:37.717Z      INFO    api/api.go:52   Setting route for GET /api/status
ako-0 ako 2024-03-19T15:23:37.718Z      INFO    ako-main/main.go:77     AKO is running with version: v1.11.3
ako-0 ako 2024-03-19T15:23:37.718Z      INFO    api/api.go:110  Starting API server at :8080
ako-0 ako 2024-03-19T15:23:37.718Z      INFO    ako-main/main.go:87     We are running inside kubernetes cluster. Won't use kubeconfig files.
ako-0 ako 2024-03-19T15:23:37.718Z      INFO    lib/control_config.go:314       Setting the client version to AVI Max supported version 22.1.3
ako-0 ako 2024-03-19T15:23:37.730Z      INFO    ako-main/main.go:174    Kubernetes cluster apiserver version 1.27
ako-0 ako 2024-03-19T15:23:37.736Z      INFO    utils/utils.go:171      Initializing configmap informer in avi-system
ako-0 ako 2024-03-19T15:23:39.674Z      INFO    cache/controller_obj_cache.go:2359      Avi cluster state is CLUSTER_UP_HA_ACTIVE
ako-0 ako 2024-03-19T15:23:39.978Z      INFO    cache/controller_obj_cache.go:3131      Setting cloud vType: CLOUD_NSXT
ako-0 ako 2024-03-19T15:23:39.978Z      INFO    cache/controller_obj_cache.go:3134      Setting cloud uuid: cloud-77594459-fc93-467c-9588-5821fb8d9360
ako-0 ako 2024-03-19T15:23:39.978Z      INFO    lib/lib.go:301  Setting AKOUser: ako-albVS-ORG0001-k8s for Avi Objects
ako-0 ako 2024-03-19T15:23:40.085Z      INFO    cache/controller_obj_cache.go:3413      Skipping the check for Node Network
ako-0 ako 2024-03-19T15:23:40.365Z      INFO    cache/controller_obj_cache.go:3583      Setting VRF T1_ORG0001-001-DATA found that matches the T1Lr /infra/tier-1s/2990ecf4-f802-4157-8ec9-8b28abfac69f
ako-0 ako 2024-03-19T15:23:40.366Z      INFO    record/event.go:285     Event(v1.ObjectReference{Kind:"Pod", Namespace:"avi-system", Name:"ako-0", UID:"7dad8c70-4398-4a51-9066-8675a2753ee5", APIVersion:"v1", ResourceVersion:"53128292", FieldPath:""}): type: 'Normal' reason: 'ValidatedUserInput' User input validation completed.
ako-0 ako 2024-03-19T15:23:40.370Z      INFO    lib/lib.go:240  Setting Disable Sync to: false
ako-0 ako 2024-03-19T15:23:40.373Z      INFO    k8s/ako_init.go:276     avi k8s configmap created
ako-0 ako 2024-03-19T15:23:41.725Z      WARN    lib/avi_api.go:65       msg: Unable to fetch data from uri /api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360 Encountered an error on GET request to URL https://nsx-alb.tld.de//api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360: HTTP code: 400; error from Avi: map[error:Ipam Type invalid or configuration not found in Avi]
ako-0 ako 2024-03-19T15:23:41.816Z      WARN    lib/avi_api.go:65       msg: Unable to fetch data from uri /api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360 Encountered an error on GET request to URL https://nsx-alb.tld.de//api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360: HTTP code: 400; error from Avi: map[error:Ipam Type invalid or configuration not found in Avi]
ako-0 ako 2024-03-19T15:23:41.907Z      WARN    lib/avi_api.go:65       msg: Unable to fetch data from uri /api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360 Encountered an error on GET request to URL https://nsx-alb.tld.de//api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360: HTTP code: 400; error from Avi: map[error:Ipam Type invalid or configuration not found in Avi]
ako-0 ako 2024-03-19T15:23:41.907Z      WARN    cache/controller_obj_cache.go:2521      DNSProperty Get uri /api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360 returned err msg: AviGet retried 3 times, aborting
ako-0 ako 2024-03-19T15:23:41.907Z      WARN    cache/controller_obj_cache.go:2466      Cloud: albC-ORG0001 does not have a dns provider configured
ako-0 ako 2024-03-19T15:23:43.059Z      WARN    nodes/avi_model_routeingr_hostname_shard.go:359 key: Ingress/nginx/nginx, msg: nothing to delete for route: nginx
ako-0 ako 2024-03-19T15:23:43.059Z      WARN    nodes/avi_model_routeingr_hostname_shard.go:359 key: Ingress/oauth/dex, msg: nothing to delete for route: dex
ako-0 ako 2024-03-19T15:23:43.059Z      WARN    nodes/avi_model_l7_translator.go:253    key: Ingress/oauth/dex, msg: secret: dex-tls has been deleted, err: secret "dex-tls" not found
ako-0 ako 2024-03-19T15:23:43.071Z      WARN    status/svc_status.go:39 key: syncstatus, msg: Service hostname not found for service [nginx-ingress-controller/nginx-ingress-controller] status update
ako-0 ako 2024-03-19T15:23:43.100Z      WARN    rest/dequeue_nodes.go:65        key: ORG0001/DummyVSForStaleData, msg: no model found for the key
ako-0 ako 2024-03-19T15:23:43.406Z      WARN    rest/rest_operation.go:304      key: ORG0001/T1_ORG0001-001-DATA, msg: RestOp method PUT path /api/vrfcontext/vrfcontext-d59343ab-8a36-4208-a3c9-42cb30b36b71 tenant admin Obj {"_last_modified":"1710833802833745","attrs":[{"key":"tier1path","value":"/infra/tier-1s/2990ecf4-f802-4157-8ec9-8b28abfac69f"}],"cloud_ref":"https://nsx-alb.tld.de/api/cloud/cloud-77594459-fc93-467c-9588-5821fb8d9360","lldp_enable":true,"name":"T1_ORG0001-001-DATA","static_routes":[{"next_hop":{"addr":"10.70.27.33","type":"V4"},"prefix":{"ip_addr":{"addr":"0.0.0.0","type":"V4"},"mask":0},"route_id":"1"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.35","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.1.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-1"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.34","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.2.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-2"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.36","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.0.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-3"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.40","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.4.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-4"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.41","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.5.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-5"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.42","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.6.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-6"}],"system_default":false,"tenant_ref":"https://nsx-alb.tld.de/api/tenant/tenant-1e7619a4-a24f-4e9c-baa7-edd2c2dab69f","url":"https://nsx-alb.tld.de/api/vrfcontext/vrfcontext-d59343ab-8a36-4208-a3c9-42cb30b36b71","uuid":"vrfcontext-d59343ab-8a36-4208-a3c9-42cb30b36b71"} returned err {"code":0,"message":"map[error:VrfContext object not found!]","Verb":"PUT","Url":"https://nsx-alb.tld.de//api/vrfcontext/vrfcontext-d59343ab-8a36-4208-a3c9-42cb30b36b71","HttpStatusCode":404} with response null
ako-0 ako 2024-03-19T15:23:43.406Z      WARN    rest/dequeue_nodes.go:624       key: ORG0001/T1_ORG0001-001-DATA, msg: there was an error sending the macro Error during PUT: Encountered an error on PUT request to URL https://nsx-alb.tld.de//api/vrfcontext/vrfcontext-d59343ab-8a36-4208-a3c9-42cb30b36b71: HTTP code: 404; error from Avi: map[error:VrfContext object not found!]
ako-0 ako 2024-03-19T15:23:43.406Z      WARN    rest/dequeue_nodes.go:659       key: ORG0001/T1_ORG0001-001-DATA, msg: Avi model not set, possibly a DELETE call
ako-0 ako 2024-03-19T15:23:44.098Z      WARN    nodes/avi_model_l7_translator.go:253    key: Endpoints/oauth/dex, msg: secret: dex-tls has been deleted, err: secret "dex-tls" not found

Reproduction steps

1. Create all required networks and firewall rules in NSX-T manager
2. Create tenant, user, cloud, IPAM profile, SEG, network profiles in NSXALB manager (VRFs are auto-created during cloud creation)
3. Deploy AKO with nodeNetworkList and ClusterIP mode

AKOSettings:
  clusterName: 'k8stest'   # A unique identifier for the kubernetes cluster, that helps distinguish the objects for this cluster in the avi controller. // MUST-EDIT
  cniPlugin: 'cilium' # Set the string if your CNI is calico or openshift or ovn-kubernetes. For Cilium CNI, set the string as cilium only when using Cluster Scope mode for IPAM and leave it empty if using Kubernetes Host Scope mode for IPAM. enum: calico|canal|flannel|openshift|antrea|ncp|ovn-kubernetes|cilium
  disableStaticRouteSync: 'false' # If the POD networks are reachable from the Avi SE, set this knob to true.

### This section outlines the network settings for virtualservices.
NetworkSettings:
  nsxtT1LR: '/infra/tier-1s/2990ecf4-f802-4157-8ec9-8b28abfac69f' # Unique ID (note: not display name) of the T1 Logical Router for Service Engine connectivity. Only applies to NSX-T cloud.

  # Network information of the VIP network. Multiple networks allowed only for AWS Cloud.
  # Either networkName or networkUUID should be specified.
  # If duplicate networks are present for the network name, networkUUID should be used for appropriate network.
  vipNetworkList:
   - networkName: 'segO-T1_ORG0001-001-VIP-0001'

  ## This list of network and cidrs are used in pool placement network for vcenter cloud.
  ## Node Network details are not needed when in nodeport mode / static routes are disabled / non vcenter clouds.
  ## Either networkName or networkUUID should be specified.
  ## If duplicate networks are present for the network name, networkUUID should be used for appropriate network.
  nodeNetworkList:
    - networkName: "segO-T1_ORG0001-001-0001"
      cidrs:
        - 10.244.0.0/16

L4Settings:
  defaultDomain: '' # If multiple sub-domains are configured in the cloud, use this knob to set the default sub-domain to use for L4 VSes.
  autoFQDN: disabled  # ENUM: default(<svc>.<ns>.<subdomain>), flat (<svc>-<ns>.<subdomain>), "disabled" If the value is disabled then the FQDN generation is disabled.

#L7Settings:
#  serviceType: NodePort        # enum NodePort|ClusterIP|NodePortLocal

### This section outlines settings on the Avi controller that affects AKO's functionality.
ControllerSettings:
  serviceEngineGroupName: 'albSEG-ORG0001'   # Name of the ServiceEngine Group.
  cloudName: 'albC-ORG0001'   # The configured cloud name on the Avi controller.
  controllerHost: 'nsx-alb.tld.de' # IP address or Hostname of Avi Controller
  controllerVersion: '22.1.5' # The controller API version
  tenantsPerCluster: true    # If set to true, AKO will map each k8s cluster uniquely to a tenant in AVI.
  tenantName: 'ORG0001'   # Name of the tenant where all the AKO objects will be created in AVI.

avicredentials:
  username: 'ako-ORG0001'
  password: ''
  authtoken: 'redacted'
  certificateAuthorityData: |-
    -----BEGIN CERTIFICATE-----
    redacted
    -----END CERTIFICATE-----

4. See that AKO is unable to set up static routes on VRF in non-admin tenant

Expected behavior

AKO is able to set up static routes on VRF in non-admin tenant

Additional context

AVI Controller Version: 22.1.5
NSX-T Version: 4.1.2.3