Merge pull request kubernetes#4019 from brightbox/brightbox-cluster-a…

…utoscaler-provider

Brightbox cluster autoscaler provider

cluster-autoscaler/README.md

@@ -17,6 +17,7 @@ You should also take a look at the notes and "gotchas" for your specific cloud p
 * [Azure](./cloudprovider/azure/README.md)
 * [AWS](./cloudprovider/aws/README.md)
 * [BaiduCloud](./cloudprovider/baiducloud/README.md)
+* [Brightbox](./cloudprovider/brightbox/README.md)
 * [CloudStack](./cloudprovider/cloudstack/README.md)
 * [HuaweiCloud](./cloudprovider/huaweicloud/README.md)
 * [Hetzner](./cloudprovider/hetzner/README.md)
@@ -150,6 +151,7 @@ Supported cloud providers:
 * AWS https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md
 * Azure https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/azure/README.md
 * Alibaba Cloud https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/alicloud/README.md
+* Brightbox https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/brightbox/README.md
 * OpenStack Magnum https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/magnum/README.md
 * DigitalOcean https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/digitalocean/README.md
 * CloudStack https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/cloudstack/README.md

cluster-autoscaler/cloudprovider/brightbox/Makefile

@@ -0,0 +1,44 @@
+export BUILD_TAGS=brightbox
+export REGISTRY=brightbox
+export GOARCH?=$(shell go env GOARCH)
+ifndef TAG
+	override TAG=dev
+endif
+export TAG
+ifeq ($(TAG), dev)
+	deploydeps=build
+endif
+
+.PHONY: deploy
+deploy: examples/config.rb $(deploydeps)
+	helm repo update
+	ruby $< | \
+	helm template release autoscaler/cluster-autoscaler \
+	--namespace kube-system -f - | \
+	kubectl -n kube-system apply -f -
+
+.PHONY: remove
+remove: examples/config.rb
+	helm repo update
+	ruby $< | \
+	helm template release autoscaler/cluster-autoscaler \
+	--namespace kube-system -f - | \
+	kubectl -n kube-system delete -f -
+
+.PHONY: secret
+secret: ${HOME}/.docker/config.json
+	-kubectl create secret generic regcred \
+	    --from-file=.dockerconfigjson=$? \
+	    --type=kubernetes.io/dockerconfigjson
+
+../../cluster-autoscaler: brightbox_cloud_provider.go brightbox_node_group.go
+	$(MAKE) -C $(@D) container
+	docker tag ${REGISTRY}/cluster-autoscaler-${BUILD_TAGS}-${GOARCH}:${TAG} ${REGISTRY}/cluster-autoscaler-${BUILD_TAGS}:${TAG}
+	docker push ${REGISTRY}/cluster-autoscaler-${BUILD_TAGS}:${TAG}
+
+.PHONY: build
+build: ../../cluster-autoscaler
+
+.PHONY: clean
+clean: 
+	$(MAKE) -C ../.. $@

cluster-autoscaler/cloudprovider/brightbox/README.md

@@ -0,0 +1,162 @@
+# Cluster Autoscaler for Brightbox Cloud
+
+This cloud provider implements the autoscaling function for
+[Brightbox Cloud](https://www.brightbox.com). The autoscaler should
+work on any Kubernetes clusters running on Brightbox Cloud, however
+the approach is tailored to clusters built with the [Kubernetes Cluster
+Builder](https://github.com/brightbox/kubernetes-cluster)
+
+# How Autoscaler works on Brightbox Cloud
+
+The autoscaler looks for [Server
+Groups](https://www.brightbox.com/docs/guides/cli/server-groups/) named
+after the cluster-name option passed to the autoscaler (--cluster-name).
+
+A group named with a suffix of the cluster-name
+(e.g. k8s-worker.k8s-test.cluster.local) is a candidate to be a scaling
+group. The autoscaler will then check the description to see if it is
+a pair of integers separated by a colon (e.g. 1:4). If it finds those
+numbers then they will become the minimum and maximum server size for
+that group, and autoscaler will attempt to scale the group between those sizes.
+
+The type of server, the image used  and the target zone will be
+dynamically determined from the existing members. If these differ, or
+there are no existing servers, autoscaler will log an error and will not
+scale that group.
+
+A group named precisely the same as the cluster-name
+(e.g. k8s-test.cluster.local) is considered to be the default cluster
+group and all autoscaled servers created are placed within it as well
+as the scaling group.
+
+The Brightbox Cloud provider only supports auto-discovery mode using
+this pattern. `node-group-auto-discovery` and `nodes` options are
+effectively ignored.
+
+## Cluster configuration
+
+If you are using the [Kubernetes Cluster
+Builder](https://github.com/brightbox/kubernetes-cluster) set the
+`worker_min` and `worker_max` values to scale the worker group, and the
+`storage_min` and `storage_max` values to scale the storage group.
+
+The Cluster Builder will ensure the group name and description are
+updated with the correct values in the format that autoscaler can recognise.
+
+Generally it is best to keep the `min` and the `count` values to be the
+same within the Cluster Buider and let autoscaler create and destroy
+servers dynamically up the the `max` value.
+
+While using autoscaler you may find that the Cluster Builder recreates
+servers that have been scaled down, if you use the manifests to maintain
+the cluster for other reasons (changing the management address for
+example). This is a limitation of the Terraform state database, and
+autoscaler will scale the cluster back down during the next few minutes.
+
+# Autoscaler Brightbox cloudprovider configuration
+
+The Brightbox Cloud cloudprovider is configured via Environment Variables
+suppied to the autoscaler pod. The easiest way to do this is to [create
+a secret](https://kubernetes.io/docs/concepts/configuration/secret/#creating-a-secret-manually) containing the variables within the `kube-system` namespace.
+
+```
+apiVersion: v1
+kind: Secret
+metadata:
+  name: brightbox-credentials
+  namespace: kube-system
+type: Opaque
+data:
+  BRIGHTBOX_API_URL: <base 64 of api URL>
+  BRIGHTBOX_CLIENT: <bas64 of Brighbox Cloud client id>
+  BRIGHTBOX_CLIENT_SECRET: <base64 of Brightbox Cloud client id secret>
+  BRIGHTBOX_KUBE_JOIN_COMMAND: <base64 of cluster join command>
+  BRIGHTBOX_KUBE_VERSION: <base 64 of installed k8s version>
+```
+
+The join command can be obtained from the kubeadm token command
+
+```
+$ kubeadm token create --ttl 0 --description 'Cluster autoscaling token' --print-join-command
+```
+
+[Brightbox API
+Clients](https://www.brightbox.com/docs/guides/manager/api-clients/)
+can be created in the [Brightbox
+Manager](https://www.brightbox.com/docs/guides/manager/)
+
+## Cluster Configuration
+
+The [Kubernetes Cluster
+Builder](https://github.com/brightbox/kubernetes-cluster) creates a
+`brightbox-credentials` secret in the `kube-system` namespace ready
+to use.
+
+## Checking the environment
+
+You can check the brightbox-credentials secret by running the `check-env` job from the examples directory.
+
+```
+$ kubectl apply -f examples/check-env.yaml
+job.batch/check-env created
+$ kubectl -n kube-system logs job/check-env
+PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+HOSTNAME=check-env-hbh6m
+_BASH_GPG_KEY=7C0135FB088AAF6C66C650B9BB5869F064EA74AB
+_BASH_VERSION=5.0
+_BASH_PATCH_LEVEL=0
+_BASH_LATEST_PATCH=11
+BRIGHTBOX_KUBE_VERSION=1.17.0
+...
+$ kubectl delete -f examples/check-env.yaml
+job.batch "check-env" deleted
+```
+
+# Running the Autoscaler
+
+1. Clone this repository and change into this directory.
+1. Edit the `examples/config.rb` file and adjust the config hash.
+2. Alter the cluster name if
+required. (If you are using the [Kubernetes Cluster
+Builder](https://github.com/brightbox/kubernetes-cluster), this will be
+`cluster_name` and `cluster_domainname` joined with a '.')
+
+Then generate and apply the manifests
+```
+$ make deploy TAG=<version>
+```
+
+where TAG is the version you wish to use (1.17, 1.18, etc.)
+
+As the Brightbox cloud-provider auto-detects and potentially scales all
+the worker groups, the example deployment file runs the autoscaler on
+the master nodes. This avoids it accidentally killing itself.
+
+## Viewing the cluster-autoscaler options
+
+Cluster autoscaler has many options that can be adjusted to better fit the needs of your application. To view them run
+
+```
+$ kubectl create job ca-options --image=brightbox/cluster-autoscaler-brightbox:dev -- ./cluster-autoscaler -h
+$ kubectl log job/ca-options
+```
+
+Remove the job in the normal way with `kubectl delete job/ca-options`
+
+You can read more details about some of the options in the [main FAQ](../../FAQ.md)
+
+
+# Building the Brightbox Cloud autoscaler
+
+Extract the repository to a machine running docker and then run the make command
+
+```
+$ make build
+```
+
+This builds an autoscaler containing only the Brightbox Cloud provider, tagged as `brightbox/cluster-autoscaler-brightbox:dev`. To build any other version add a TAG variable
+
+```
+make build TAG=1.1x
+```
+