Towards Beta

[simplysoft]

Please find a draft PR of our current variation/extension of this provider. With those changes we are closer to be able to consider using it outside of just experimental setups:

• Run CAPPX with non-priviledged PVE user / permissions
  • Workaround proxmox shortcomings (VNC shell login, storage folder permissions)
  • Add VMs to proxmox resource pool (CAPPX only having permission to manage VMs on that resource pool)
• Reworked storage approach to use import-from to allow use different storage for VMs (related to Support different types of storages #93)
• Control over VM ids & node placement
  • Configure nodes on ProxmoxCluster and ProxmoxMachineTemplate level
  • Initial support for CAPI Failure Domains (Node as Failure Domain Strategy)
  • Configure VM id ranges
• Network Config:
  • Configurable bridge & vlan tag
  • Initial support for CAPI IPAM (Provide IP auto population feature #4)
• Various smaller fixes / changes

We are currently testing with Cluster API Provider RKE2, so have some example for #68

Sample using new features & RKE2

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
  name: ${CLUSTER_NAME}
  namespace: ${NAMESPACE}
spec:
  controlPlaneEndpoint:
    host: ${CONTROLPLANE_ADDRESS}
    port: 6443
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: RKE2ControlPlane
    name: ${CLUSTER_NAME}
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: ProxmoxCluster
    name: ${CLUSTER_NAME}
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: ProxmoxCluster
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
  name: ${CLUSTER_NAME}
  namespace: ${NAMESPACE}
spec:
  controlPlaneEndpoint:
    host: ${CONTROLPLANE_ADDRESS}
    port: 6443
  serverRef:
    endpoint: https://${PROXMOX_ADDRESS}:8006/api2/json
    secretRef:
      name: ${CLUSTER_NAME}
      namespace: ${NAMESPACE}
  resourcePool: capi-proxmox
  failureDomain:
    nodeAsFailureDomain: true
  nodes:
    - node1
    - node2
    - node3

---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: RKE2ControlPlane
metadata:
  name: ${CLUSTER_NAME}
  namespace: ${NAMESPACE}
spec:
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: ProxmoxMachineTemplate
    name: ${CLUSTER_NAME}-controlplane
  replicas: 3
  agentConfig:
    version: ${K8S_VERSION}+rke2r1
    kubelet:
      extraArgs:
        - 'cloud-provider=external'
  nodeDrainTimeout: 2m
  registrationMethod: "address"
  registrationAddress: "${CONTROLPLANE_ADDRESS}"
  postRKE2Commands:
    - curl https://kube-vip.io/manifests/rbac.yaml > /var/lib/rancher/rke2/server/manifests/kube-vip-rbac.yaml
    - /var/lib/rancher/rke2/bin/crictl -r "unix:///run/k3s/containerd/containerd.sock" pull ghcr.io/kube-vip/kube-vip:latest
    - CONTAINERD_ADDRESS=/run/k3s/containerd/containerd.sock /var/lib/rancher/rke2/bin/ctr -n k8s.io run --rm --net-host ghcr.io/kube-vip/kube-vip:latest vip /kube-vip manifest daemonset --arp --address ${CONTROLPLANE_ADDRESS} --controlplane --leaderElection --taint --services --inCluster | tee /var/lib/rancher/rke2/server/manifests/kube-vip.yaml
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: ProxmoxMachineTemplate
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
  name: ${CLUSTER_NAME}-controlplane
  namespace: ${NAMESPACE}
spec:
  vmIDs:
    start: 1000
    end: 1010
  template:
    spec:
      hardware:
        cpu: 4
        memory: 8192
        disk: 16G
        storage: ${PROXMOX_VM_STORAGE}
      image:
        checksum: ${CLOUD_IMAGE_HASH_SHA256}
        checksumType: sha256
        url: ${CLOUD_IMAGE_URL}
      network:
        bridge: ${VM_BRIDGE}
        vlanTag: ${VM_VLAN}
        nameServer: ${IPV4_NAMESEVER}
        ipConfig:
          IPv4FromPoolRef:
            name: ${IPV4_POOL_NAME}
            apiGroup: ipam.cluster.x-k8s.io
            kind: InClusterIPPool
      options:
        onBoot: true
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
  name: ${CLUSTER_NAME}-md-0
  namespace: ${NAMESPACE}
spec:
  clusterName: ${CLUSTER_NAME}
  replicas: 3
  selector:
    matchLabels: {}
  template:
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1
          kind: RKE2ConfigTemplate
          name: ${CLUSTER_NAME}-md-0
      clusterName: ${CLUSTER_NAME}
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: ProxmoxMachineTemplate
        name: ${CLUSTER_NAME}-md-0
      version: ${K8S_VERSION}
---
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1
kind: RKE2ConfigTemplate
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
  name: ${CLUSTER_NAME}-md-0
  namespace: ${NAMESPACE}
spec:
  template:
    spec:
#      preRKE2Commands:
#        - sleep 30 # fix to give OS time to become ready
      agentConfig:
        version: ${K8S_VERSION}+rke2r1
        kubelet:
          extraArgs:
            - "cloud-provider=external"
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: ProxmoxMachineTemplate
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
  name: ${CLUSTER_NAME}-md-0
  namespace: ${NAMESPACE}
spec:
  nodes:
    - node1
    - node2
  vmIDs:
    start: 1010
    end: 1019
  template:
    spec:
      hardware:
        cpu: 4
        memory: 8192
        disk: 16G
        storage: ${PROXMOX_VM_STORAGE}
      image:
        checksum: ${CLOUD_IMAGE_HASH_SHA256}
        checksumType: sha256
        url: ${CLOUD_IMAGE_URL}
      network:
        bridge: ${VM_BRIDGE}
        vlanTag: ${VM_VLAN}
        nameServer: ${IPV4_NAMESEVER}
        ipConfig:
          IPv4FromPoolRef:
            name: ${IPV4_POOL_NAME}
            apiGroup: ipam.cluster.x-k8s.io
            kind: InClusterIPPool
      options:
        onBoot: true
---
apiVersion: v1
kind: Secret
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
  name: ${CLUSTER_NAME}
  namespace: ${NAMESPACE}
stringData:
  PROXMOX_PASSWORD: "${PROXMOX_PASSWORD}"
  PROXMOX_SECRET: ""
  PROXMOX_TOKENID: ""
  PROXMOX_USER: "${PROXMOX_USER}"
type: Opaque
---
apiVersion: addons.cluster.x-k8s.io/v1beta1
kind: ClusterResourceSet
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
  name: ${CLUSTER_NAME}-crs-0
  namespace: ${NAMESPACE}
spec:
  clusterSelector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
  resources:
    - kind: ConfigMap
      name: ${CLUSTER_NAME}-cloud-controller-manager
  strategy: Reconcile
---
apiVersion: v1
data:
  cloud-controller-manager.yaml: |
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: proxmox-cloud-controller-manager
      namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: system:proxmox-cloud-controller-manager
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: cluster-admin
    subjects:
      - kind: ServiceAccount
        name: proxmox-cloud-controller-manager
        namespace: kube-system
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      labels:
        k8s-app: cloud-controller-manager
      name: cloud-controller-manager
      namespace: kube-system
    spec:
      selector:
        matchLabels:
          k8s-app: cloud-controller-manager
      template:
        metadata:
          labels:
            k8s-app: cloud-controller-manager
        spec:
          serviceAccountName: proxmox-cloud-controller-manager
          containers:
            - name: cloud-controller-manager
              image: ghcr.io/sp-yduck/cloud-provider-proxmox:latest
              command:
                - /usr/local/bin/cloud-controller-manager
                - --cloud-provider=proxmox
                - --cloud-config=/etc/proxmox/config.yaml
                - --leader-elect=true
                - --use-service-account-credentials
                - --controllers=cloud-node,cloud-node-lifecycle
              volumeMounts:
                - name: cloud-config
                  mountPath: /etc/proxmox
                  readOnly: true
              livenessProbe:
                httpGet:
                  path: /healthz
                  port: 10258
                  scheme: HTTPS
                initialDelaySeconds: 20
                periodSeconds: 30
                timeoutSeconds: 5
          volumes:
            - name: cloud-config
              secret:
                secretName: cloud-config
          tolerations:
            - key: node.cloudprovider.kubernetes.io/uninitialized
              value: "true"
              effect: NoSchedule
            - key: node-role.kubernetes.io/control-plane
              operator: Exists
              effect: NoSchedule
            - key: node-role.kubernetes.io/master
              operator: Exists
              effect: NoSchedule
          nodeSelector:
            node-role.kubernetes.io/control-plane: "true"
    ---
    apiVersion: v1
    kind: Secret
    metadata:
      name: cloud-config
      namespace: kube-system
    stringData:
      config.yaml: |
        proxmox:
          url: https://${PROXMOX_ADDRESS}:8006/api2/json
          user: ""
          password: ""
          tokenID: "${PROXMOX_CCM_TOKENID}"
          secret: "${PROXMOX_CCM_SECRET}"
kind: ConfigMap
metadata:
  name: ${CLUSTER_NAME}-cloud-controller-manager
  namespace: ${NAMESPACE}
 

Beyond those current changes, based on our assessment, the existing CRD would further require some bigger refactoring / changes to enable other (for us required) use cases, such as support for multiple network devices & disks. Those changes will definitely be breaking changes.

Any suggestions & feedback welcome

[3deep5me]

Nice to see this!
I especially appreciate the feature with the failure-domains and the auth / resource pool!
Multiple network-interfaces and especially disks are definitively welcome from my site!

[sp-yduck]

Thank you for the PR. It's difficult to check all of them in one single PR. let's breakdown into separated PRs/Issues. and let's get merged one by one.
As you mentioned my issues in the document, some of the changes are already in my mind and very welcome. However, I haven't have clear understanding for some of them yet. so it would be appreciated if you can split off them and I can find more detailed description. Again, thank you for the preparation !

[sp-yduck]

Control over VM ids & node placement
Configure nodes on ProxmoxCluster and ProxmoxMachineTemplate level
Initial support for CAPI Failure Domains (Node as Failure Domain Strategy)
Configure VM id ranges

Since different user have different strategy of how to assign vmid/node to their vm, I am not thinking providing a single solution. It should be a configurable by user. We can refer how kubernetes assign nodes to containers. like kube-scheduler we can provide multiple scheduling solution as plugin. Then user can choose which plugin they want to use. After achieving this idea (I have already started some poc), we can assign vmid from specified range, assign node from specified node list, assign node by calculating allocatable cpu/mem of nodes or whatever we want.

[sp-yduck]

the feature I mentioned in this comment is now available on #102