nodeselector, tolerations and e2e (#92)

* init k8s test

* set spec for uc correctly

* ...

* add nodeselector and tolderation

* add test with tainted nodes

* start reconcilers

* start goroutine at the right place

* test basic with 5 workers and typo

* use unused function

* install fluxcd with toleration

* use minikube instead of k3d for local

* typo in cli name

* use github action to setup flux cli

* rename

* fix flux command name

* add correct flag for toleration keys

* compare only required condition parameters

* fix flip

* add single toleration key

* ok if tainting fails

* remove type

* change tolerations datatype

* test without prints

* attempt reading codecov

* add more conditions for testing

* don't print diff

* reduce one parameter read

* overengineered functions

* typo

* fix reg

* skip basic test is cluster is tainted'

* simplify

* fix nodeselector type

* make minikube reusable locally

* use nodeselector when necessary

* fix nodeselector config

* set nodeselector if necessary

* node selector fix

* fix test

* spec not being read

* more precise argument parsing and placement in one place

* make changes to k3s not k8s syncer args

* update

* enable k8s test

* fix manager clusterrole

* fix manager clusterrole

* increase basic ephemeral storage for k8s to work

* fix etcd deployment, make standard storage class as the default one for etcd

* refactor

* fix crd yaml gen

* create new toleration type with it's own nice little sting function

* fix toleration notation

* fix notation again

* remove tolerations from redundant places

* add interval and timeout to helmrepos created for source-controller

* try out other flags for flux

* update vcluster version

* increase timeout

* add persistence option

* set persistence on the syncer

* remove unused enableHA

* set defaults for persistence

* nil ref check for persistence

* use k3s instead of minikube

* correct args for k3d for setting up the cluster

* update k3d command to disable traefik

* add missing setup k3d

* use minikube and add nodeselector and tolerations on all workload abstractions

* update tolerations and nodeselector using a patch

* taint after the deployments are updated

* iterate over namespaces as well

* fix

* apply taint first

* minikube start:

* derefeerence namespace properly

* use hack script:

* fix hack script location

* set nodeselector and tolerations as common config:

* correct operator for toleration test

* use default storage class

* *bool to bool

* default storage class is standard

* i'm a little tired now

* add tolerations directly to certain pods

.github/actions/k3d/action.yaml

@@ -0,0 +1,8 @@
+name: setup-k3d
+description: "Install k3d and create a cluster"
+
+runs:
+  using: composite
+  steps:
+    - run: "curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash"
+      shell: bash

.github/kubectl-patch/nodeselector-pod-toleration.yaml

@@ -0,0 +1,8 @@
+spec:
+  tolerations:
+    - key: "testkey"
+      operator: "Equal"
+      value: "testvalue"
+      effect: "NoSchedule"
+  nodeSelector:
+    testkey: "testvalue"

.github/kubectl-patch/nodeselector-toleration.yaml

@@ -0,0 +1,10 @@
+spec:
+  template:
+    spec:
+      nodeSelector:
+        testkey: "testvalue"
+      tolerations:
+        - key: "testkey"
+          operator: "Equal"
+          value: "testvalue"
+          effect: "NoSchedule"

.github/workflows/e2e-with-cluster.yaml

@@ -11,22 +11,44 @@ permissions:
   id-token: write
 
 jobs:
-  minikube:
-    name: Run E2E tests with Minikube
+  minikube-default:
+    name: Run e2e tests against Minikube
     if: ${{ github.event_name == 'pull_request' && github.event.action != 'closed' }}
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
         uses: actions/checkout@v3
 
-      - name: Setup Minikube
-        run: minikube start --driver=docker
+      - name: Setup Flux CLI
+        uses: fluxcd/flux2/action@main
 
-      - name: Install FluxCD
+      - name: Setup k3d
+        uses: ./.github/actions/k3d
+
+      - name: Run e2e tests against current cluster
         run: |
-          curl -s https://fluxcd.io/install.sh | sudo bash
-          flux install --namespace=flux-system --components="source-controller,helm-controller"
-          
-      - name: Run e2e tests
+          make test-e2e-with-cluster-local
+
+      - name: Upload coverage reports to Codecov
+        uses: codecov/codecov-action@v3
+
+  minikube-tainted:
+    name: Run e2e tests against Minikube (TAINTED)
+    if: ${{ github.event_name == 'pull_request' && github.event.action != 'closed' }}
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+
+      - name: Setup Flux CLI
+        uses: fluxcd/flux2/action@main
+
+      - name: Setup k3d
+        uses: ./.github/actions/k3d
+
+      - name: Run e2e tests against current tainted cluster
         run: |
-          make e2e-test-with-cluster
+          make test-e2e-with-tainted-cluster-local
+
+      - name: Upload coverage reports to Codecov
+        uses: codecov/codecov-action@v3

.github/workflows/pull_request.yaml

@@ -17,8 +17,10 @@ jobs:
         run: |
           (cd ./chart && helm dep update .)
           helm lint ./chart --with-subcharts
+
       - name: Test
         run: | 
           go test -coverprofile=coverage.txt -covermode=atomic -v ./src/controllers/...
+
       - name: Upload coverage reports to Codecov
         uses: codecov/codecov-action@v3

Makefile

@@ -93,7 +93,7 @@ help: ## Display this help.
 
 .PHONY: manifests
 manifests: controller-gen ## Generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects.
-	$(CONTROLLER_GEN) rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
+	$(CONTROLLER_GEN) rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=src/config/crd/bases
 
 .PHONY: generate
 generate: controller-gen ## Generate code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations.
@@ -107,15 +107,75 @@ fmt: ## Run go fmt against code.
 vet: ## Run go vet against code.
 	go vet ./...
 
+.PHONY: install-flux-prereq
+install-flux-prereq: ## Install the fluxcd if not preset
+    which flux || curl -s https://fluxcd.io/install.sh | sudo bash
+
+.PHONY: install-fluxcd-controllers
+install-fluxcd-controllers: install-flux-prereq ## Install the fluxcd controllers.
+	flux install --namespace=flux-system --components="source-controller,helm-controller" --network-policy=false --insecure-skip-tls-verify
+
+.PHONE: install-fluxcd-controllers-with-toleration
+install-fluxcd-controllers-with-toleration: install-flux-prereq ## Install the fluxcd controllers with toleration.
+	flux install --namespace=flux-system --components="source-controller,helm-controller" --toleration-keys="testkey" --network-policy=false --insecure-skip-tls-verify
+
+.PHONY: start-test-k3d
+start-test-k3d: ## Start a k3d cluster for testing.
+	k3d cluster create basic agents=1
+	$(MAKE) install-fluxcd-controllers
+
+.PHONY: start-test-minikube
+start-test-minikube: ## Start a minikube cluster for testing.
+	minikube start --addons default-storageclass,storage-provisioner --driver=docker
+	kubectl taint nodes minikube testkey- || true
+	kubectl label nodes minikube testkey- || true
+	$(MAKE) install-fluxcd-controllers
+
+.PHONY: stop-test-minikube
+stop-test-minikube: ## Stop the minikube cluster for testing.
+	minikube stop
+
+.PHONY: start-test-minikube-tainted
+start-test-minikube-tainted: ## Start a minikube cluster with a tainted node for testing.
+	minikube start --addons default-storageclass,storage-provisioner,hostpat --driver=docker
+	sh ./hack/minikube-patch-pod-tolerations.sh
+	kubectl taint nodes minikube testkey=testvalue:NoSchedule || true
+	kubectl label nodes minikube testkey=testvalue || true
+	$(MAKE) install-fluxcd-controllers-with-toleration
+	sh ./hack/minikube-patch-workload-tolerations.sh
+
+.PHONY : stop-test-k3d
+stop-test-k3d: ## Stop the k3d cluster for testing.
+	k3d cluster delete basic
+
+.PHONY: start-test-k3d-tainted
+start-test-k3d-tainted: ## Start a k3d cluster with a tainted node for testing.
+	k3d cluster create tainted --agents=1 --k3s-arg="--kubelet-arg=node-labels=testkey=testvalue@agent:0" --k3s-arg="--kubelet-arg=taints=testkey=testvalue:NoSchedule@agent:0"
+	$(MAKE) install-fluxcd-controllers
+
+.PHONY : stop-test-k3d-tainted
+stop-test-k3d-tainted: ## Stop the k3d cluster with a tainted node for testing.
+	k3d cluster delete tainted
+
 ##@ Test
 
-.PHONY: e2e-test-without-cluster
-e2e-test-without-cluster: manifests generate fmt vet envtest ## Run test.
-	KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" go test ./... -coverprofile cover.out
+.PHONY: test-e2e-without-cluster
+test-e2e-without-cluster: manifests generate fmt vet envtest ## Run test.
+	KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" go test ./... -coverprofile=coverage.txt
+
+.PHONY: test-e2e-with-cluster
+test-e2e-with-cluster: manifests generate fmt vet envtest ## Run test.
+	KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" ENVTEST_REMOTE=true go test ./... -coverprofile=coverage.txt -v
+
+.PHONY: test-e2e-with-cluster-local
+test-e2e-with-cluster-local: start-test-minikube test-e2e-with-cluster ## Run test.
+
+.PHONY: test-e2e-with-tainted-cluster
+test-e2e-with-tainted-cluster: manifests generate fmt vet envtest ## Run test.
+	KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" ENVTEST_REMOTE=true E2E_ARG_IS_TAINTED=true go test ./... -coverprofile=coverage.txt -v
 
-.PHONY: e2e-test-with-cluster
-e2e-test-with-cluster: manifests generate fmt vet envtest ## Run test.
-	KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" ENVTEST_REMOTE=true go test ./... -coverprofile cover.out -v
+.PHONY: test-e2e-with-tainted-cluster-local
+test-e2e-with-tainted-cluster-local: start-test-minikube-tainted test-e2e-with-tainted-cluster ## Run test.
 
 ##@ Build
 

chart/templates/uffizziclusters.uffizzi.com_customresourcedefinition.yaml

@@ -148,6 +148,10 @@ spec:
                 type: object
               manifests:
                 type: string
+              nodeSelector:
+                additionalProperties:
+                  type: string
+                type: object
               resourceQuota:
                 description: UffizziClusterResourceQuota defines the resource quota which defines the quota of resources a namespace has access to
                 properties:
@@ -199,7 +203,7 @@ spec:
                         default: 1Gi
                         type: string
                       storage:
-                        default: 5Gi
+                        default: 10Gi
                         type: string
                     type: object
                   services:
@@ -216,8 +220,37 @@ spec:
                 type: object
               sleep:
                 type: boolean
-              ttl:
-                type: string
+              storage:
+                properties:
+                  persistence:
+                    default: true
+                    type: boolean
+                  size:
+                    default: 5Gi
+                    type: string
+                type: object
+              tolerations:
+                items:
+                  description: The pod this Toleration is attached to tolerates any taint that matches the triple <key,value,effect> using the matching operator <operator>.
+                  properties:
+                    effect:
+                      description: Effect indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute.
+                      type: string
+                    key:
+                      description: Key is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be Exists; this combination means to match all values and all keys.
+                      type: string
+                    operator:
+                      description: Operator represents a key's relationship to the value. Valid operators are Exists and Equal. Defaults to Equal. Exists is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category.
+                      type: string
+                    tolerationSeconds:
+                      description: TolerationSeconds represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system.
+                      format: int64
+                      type: integer
+                    value:
+                      description: Value is the taint value the toleration matches to. If the operator is Exists, the value should be empty, otherwise just a regular string.
+                      type: string
+                  type: object
+                type: array
             type: object
           status:
             description: UffizziClusterStatus defines the observed state of UffizziCluster