Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

✨ Add fake K8s API server #1610

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+1,029 −0
Open

✨ Add fake K8s API server #1610

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ export GO111MODULE=on
# Full directory of where the Makefile resides
ROOT_DIR := $(shell dirname $(realpath $(firstword $(MAKEFILE_LIST))))
TOOLS_DIR := hack/tools
FAKE_APISERVER_DIR := hack/fake-apiserver
APIS_DIR := api
TEST_DIR := test
BIN_DIR := bin
Expand Down Expand Up @@ -319,6 +320,7 @@ lint: $(GOLANGCI_LINT) ## Lint codebase
$(GOLANGCI_LINT) run -v $(GOLANGCI_LINT_EXTRA_ARGS) --timeout=10m
cd $(APIS_DIR) && $(GOLANGCI_LINT) run -v $(GOLANGCI_LINT_EXTRA_ARGS) --timeout=10m
cd $(TEST_DIR) && $(GOLANGCI_LINT) run -v $(GOLANGCI_LINT_EXTRA_ARGS) --timeout=10m
cd $(FAKE_APISERVER_DIR) && $(GOLANGCI_LINT) run -v $(GOLANGCI_LINT_EXTRA_ARGS) --timeout=10m

.PHONY: lint-fix
lint-fix: $(GOLANGCI_LINT) ## Lint the codebase and run auto-fixers if supported by the linter
Expand All @@ -328,6 +330,7 @@ lint-full: $(GOLANGCI_LINT) ## Run slower linters to detect possible issues
$(GOLANGCI_LINT) run -v --fast=false --timeout=30m
cd $(APIS_DIR) && $(GOLANGCI_LINT) run -v --fast=false --timeout=30m
cd $(TEST_DIR) && $(GOLANGCI_LINT) run -v --fast=false --timeout=30m
cd $(FAKE_APISERVER_DIR) && $(GOLANGCI_LINT) run -v --fast=false --timeout=30m

# Run manifest validation
.PHONY: manifest-lint
Expand All @@ -349,6 +352,8 @@ modules: ## Runs go mod to ensure proper vendoring.
cd $(APIS_DIR) && $(GO) mod verify
cd $(TEST_DIR) && $(GO) mod tidy
cd $(TEST_DIR) && $(GO) mod verify
cd $(FAKE_APISERVER_DIR) && $(GO) mod tidy
cd $(FAKE_APISERVER_DIR) && $(GO) mod verify
mquhuy marked this conversation as resolved.
Show resolved Hide resolved

.PHONY: generate
generate: ## Generate code
Expand Down Expand Up @@ -449,6 +454,10 @@ docker-build: ## Build the docker image for controller-manager
docker-push: ## Push the docker image
docker push $(CONTROLLER_IMG)-$(ARCH):$(TAG)

.PHONY: build-fake-api-server
build-fake-api-server: ## Build the fake api server
cd $(FAKE_APISERVER_DIR) && $(CONTAINER_RUNTIME) build --build-arg ARCH=$(ARCH) -t "quay.io/metal3-io/api-server:$(ARCH)" .

## --------------------------------------
## Docker — All ARCH
## --------------------------------------
Expand Down
1 change: 1 addition & 0 deletions hack/fake-apiserver/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
fake-apiserver
49 changes: 49 additions & 0 deletions hack/fake-apiserver/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Copyright 2021 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Support FROM override
ARG BUILD_IMAGE=docker.io/golang:1.22.6@sha256:d5e49f92b9566b0ddfc59a0d9d85cd8a848e88c8dc40d97e29f306f07c3f8338
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're at 1.22.7 now

ARG BASE_IMAGE=gcr.io/distroless/static:nonroot@sha256:9ecc53c269509f63c69a266168e4a687c7eb8c0cfd753bd8bfcaa4f58a90876f

# Build the manager binary on golang image
FROM $BUILD_IMAGE as builder
WORKDIR /workspace

# Run this with docker build --build_arg $(go env GOPROXY) to override the goproxy
ARG goproxy=https://proxy.golang.org
ENV GOPROXY=$goproxy

# Copy the Go Modules manifests
COPY go.mod go.sum ./

# Cache deps before building and copying source so that we don't need to re-download as much
# and so that source changes don't invalidate our downloaded layer
RUN go mod download

# Copy the sources
COPY main.go .

# Build
ARG ARCH=amd64
RUN CGO_ENABLED=0 GOOS=linux GOARCH=${ARCH} \
go build -a -ldflags '-extldflags "-static"' \
-o manager .

# Copy the controller-manager into a thin image
FROM $BASE_IMAGE
WORKDIR /
COPY --from=builder /workspace/manager .
# Use uid of nonroot user (65532) because kubernetes expects numeric user when applying pod security policies
USER 65532
ENTRYPOINT ["/manager"]
197 changes: 197 additions & 0 deletions hack/fake-apiserver/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
# Fake API server

Fake API server is a tool running inside a kubernetes cluster,
and generates "fake" k8s api server endpoints on demand.

## Purpose

When CAPI+CAPM3 provisions a baremetal node, after BMO and Ironic finished
provisioning the node, a node image with kubernetes components (kubelet,
containerd, etcd, etc.) is supposed to be installed inside the new node,
and as it boots up afterwards, it will start itself up as a kubernetes node.
At the end of the process, CAPI will send queries towards the newly launched
cluster's API server to verify that the cluster is fully up and running.
The address of this API server is defined in `CLUSTER_APIENDPOINT_HOST`
and `CLUSTER_APIENDPOINT_PORT` variables, which users need to provide to the
cluster template.

In a test setup with fake nodes, i.e. nodes that only "exist" in either ironic database,
(like in case of FakeIPA system), or nodes faked by BMO in its simulation mode,
the "nodes" are not able to boot up any kubernetes api server,
hence the needs of having mock API servers on-demands.

## How it works

After a BMH associating to a fake node is provisioned to `available` state,
the user can send a request towards the Fake API server endpoint `/register`,
which will spawn a new API server, with an unique IP address.

The generated API server endpoint responds to any request normally responded
by the apiserver of target clusters created in a normal CAPI workflow.

User can, then, use this IP address to feed the cluster template, and start
"provisioning" the node to a kubernetes cluster.

## How to use

You can build the `fake-api-server` image that is suitable for
your local environment with

```shell
make build-fake-api-server
```

The result is an image with label `quay.io/metal3-io/fake-apiserver:<your-arch-name>`

Alternatively, you can also build a custom image with

```shell
cd hack/fake-apiserver
docker build -t <custom tag> .
```

For local tests, it's normally needed to load the image into the cluster.
For e.g. with `minikube`

```shell
docker image save -o /tmp/api-server.tar <image-name>
minikube image load /tmp/api-server.tar
```

Now you can deploy this container to the cluster, for e.g. with a deployment

```yaml
# api-server-deployment.yaml
---
mquhuy marked this conversation as resolved.
Show resolved Hide resolved
apiVersion: apps/v1
kind: Deployment
metadata:
name: metal3-fake-api-server
namespace: default
spec:
replicas: 2
selector:
matchLabels:
app: capim
strategy:
type: Recreate
template:
metadata:
labels:
app: capim
spec:
containers:
- image: quay.io/metal3-io/api-server:amd64
imagePullPolicy: IfNotPresent
name: capim
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
name: apiserver
```

```shell
kubectl apply -f api-server-deployment.yaml
```

After building the container image and deploy it to a kubernetes cluster,
mquhuy marked this conversation as resolved.
Show resolved Hide resolved
you need to create a tunnel to send request to it and get response, by using
a LoadBalancer, or a simple port-forward

```shell
api_server_name=$(kubectl get pods -l app=capim -o jsonpath="{.items[0].metadata.name}")
kubectl port-forward pod/${api_server_name} ${api_server_port}:3333 2>/dev/null&
```

Now, you can generate a fake API server endpoint by sending
a GET request to the fake API server. But first, let's generate some needed certificates

```shell
openssl req -x509 -subj "/CN=Kubernetes API" -new -newkey rsa:2048 \
-nodes -keyout "/tmp/ca.key" -sha256 -days 3650 -out "/tmp/ca.crt"
openssl req -x509 -subj "/CN=ETCD CA" -new -newkey rsa:2048 \
-nodes -keyout "/tmp/etcd.key" -sha256 -days 3650 -out "/tmp/etcd.crt"
```

```shell
caKeyEncoded=$(cat /tmp/ca.key | base64 -w 0)
caCertEncoded=$(cat /tmp/ca.crt | base64 -w 0)
etcdKeyEncoded=$(cat /tmp/etcd.key | base64 -w 0)
etcdCertEncoded=$(cat /tmp/etcd.crt | base64 -w 0)
namespace="metal3"
cluster_name="test_cluster"

cluster_endpoint=$(curl localhost:${api_server_port}/register?resource=$namespace/$cluster_name&caKey=$caKeyEncoded&caCert=$caCertEncoded&etcdKey=$etcdKeyEncoded&etcdCert=$etcdCertEncoded")
```

The fake API server will return a response with the ip and port of the newly
generated api server. These information can be fed to a CAPI infrastructure provider
(for e.g. CAPM3) to create a cluster. Notice that you need to manually create
the `ca-secret` and `etcd-secret` of the new cluster, so that it matches the certs
used by the api server.

```shell
host=$(echo ${cluster_endpoints} | jq -r ".Host")
port=$(echo ${cluster_endpoints} | jq -r ".Port")

cat <<EOF > "/tmp/${cluster}-ca-secrets.yaml"
apiVersion: v1
kind: Secret
metadata:
labels:
cluster.x-k8s.io/cluster-name: ${cluster}
name: ${cluster}-ca
namespace: ${namespace}
type: kubernetes.io/tls
data:
tls.crt: ${caCertEncoded}
tls.key: ${caKeyEncoded}
EOF

kubectl -n ${namespace} apply -f /tmp/${cluster}-ca-secrets.yaml

cat <<EOF > "/tmp/${cluster}-etcd-secrets.yaml"
apiVersion: v1
kind: Secret
metadata:
labels:
cluster.x-k8s.io/cluster-name: ${cluster}
name: ${cluster}-etcd
namespace: ${namespace}
type: kubernetes.io/tls
data:
tls.crt: ${etcdCertEncoded}
tls.key: ${etcdKeyEncoded}
EOF

kubectl -n ${namespace} apply -f /tmp/${cluster}-etcd-secrets.yaml

# Generate metal3 cluster
export CLUSTER_APIENDPOINT_HOST="${host}"
export CLUSTER_APIENDPOINT_PORT="${port}"
echo "Generating cluster ${cluster} with clusterctl"
clusterctl generate cluster "${cluster}" \
--from "${CLUSTER_TEMPLATE}" \
--target-namespace "${namespace}" > /tmp/${cluster}-cluster.yaml
kubectl apply -f /tmp/${cluster}-cluster.yaml
```

After the cluster is created, CAPI will expect that information like node name
and provider ID is registered in the API server. Since our API server doesn't
live inside the node, we will need to feed the info to it, by sending a
GET request to `/updateNode` endpoint:

```shell
curl "localhost:${api_server_port}/updateNode?resource=${namespace}/${cluster}&nodeName=${machine}&providerID=${providerID}"
```

## Acknowledgements

This was developed thanks to the implementation of
[Cluster API Provider In Memory (CAPIM)](https://github.com/kubernetes-sigs/cluster-api/tree/main/test/infrastructure/inmemory).

**NOTE:**:
This is intended for development environments only.
Do **NOT** use it in production.
Loading
Loading