k8s-techworld.md

Nana's Kubernetes course

Reference.

This is a continuation of Kubernetes course from Microsoft Learn. So, it's helpful to finish this section before reading this.

Practice more kubectl commands

1. Get Logs from Pod

   kubectl logs [pd name]
   

2. Describe Pod

   kubectl describe pod [pod name]
   

3. Debugging by getting into the pod and get the terminal (bash for eg.) from in there

   kubectl exec -it [pod name] -- bin/bash
   

   

4. Delete Pods

   kubectl delete deployment [name]
   

   

5. Using config files to do deployments

   kubectl apply -f [some-config-file.yaml]
   

   The first spec is for deployment and the second spec is for pods. template is the blueprint for the pods.

   

Summary:

Kubernetes YAML config file

YAML is a human friendly data serialization standard for all programming languages.
Its syntax uses strict indentation.
Best practice is to store config file with your code (or have a separate repo for config files).

Each configuration file has 3 parts:

1. Metadata

2. Specification

3. Status (This part is automatically generated and added by K8s)
   

   K8s gets status data from etcd database. etcd holds the current status of any K8s component.

Blueprint for Pods (Template)

Connecting Components (Labels & Selectors & Ports)

metadata part contains labels and spec part contains selectors.

1. Connecting Deployment to Pods
   Way for deployment to know which Pods belong to it.

   

   Pods get the label through the template blueprint.
   Then we tell the deployment to connect or to match all the labels app: nginx.

2. Connecting Services to Deployments
   Way for service to know which Pods belong to it.

   

   Example:
   Service has spec:ports configuration and deployment has spec:template:spec:containers:ports configuration:
   

Ports in Service and Pod

Example

View the status of the deployment generated by K8s from etcd and compare it to original

kubectl get deployment nginx-deployment -o yaml > nginx-deployment-result.yaml

Delete the deployment and service

kubectl delete -f nginx-deployment.yaml  
kubectl delete -f nginx-service.yaml

Elaborate example

Create MongoDb deployment

Create secret first before you run deployment. Secret is created in the section that immediately follows this one.

mongodb-deployment.yaml

# This file will get checked into app's Git Repo, so don't put secret value of settings like MONGO_INITDB_ROOT_USERNAME
kind: Deployment
metadata:
    name: mongodb-deployment
    # A service could use 'app:mongodb' selector to route network traffic to the pods created by this deployment. A label is used to identify the deployment itself.
    labels:
        app: mongodb
spec:
    replicas: 1
    selector:
        matchLabels:
            app: mongodb
    template:
        metadata:
            labels:
                app: mongodb
        spec:
            containers:
            # Config this image from info here: https://hub.docker.com/_/mongo
            - name: mongodb
              image: mongo
              
              ports:
              - containerPort: 27017
              
              env:
              - name: MONGO_INITDB_ROOT_USERNAME
                valueFrom:
                    secretKeyRef:
                        name: mongodb-secret
                        key: mongodb-root-username
              - name: MONGO_INITDB_ROOT_PASSWORD
                valueFrom:
                    secretKeyRef:
                        name: mongodb-secret
                        key: mongodb-root-password

Create Deployment:

[~]$ kubectl apply -f mongodb-deployment.yaml

Create secret (this will live in K8s, not in repo)

Create secret file:

mongodb-secret.yaml

apiVersion: v1
kind: secret
metadata:
    name: mongodb-secret
# Basic key:value secret type. Other types include TLS
type: Opaque
data:
    # The value you put here should be base64 encoded
    # Get the base64 value by going to terminal and typing:
    # echo -n 'username' | base64
    # Similarly using 'password' for password
    mongodb-root-username: dXNlcm5hbWU=
    mongodb-root-password: cGFzc3dvcmQ=

Create Secret:

[~]$ cd k8s-config/
[~]$ ls
mongodb-deployment.yaml      mongo.yaml
[~]$ kubectl apply -f mongodb-secret.yaml
[~]$ secret/mongodb-secret created

k8s-config is just a folder on your local computer.

You can view secrets with command:

kubectl get secret

Create MongoDb Internal Service

Add this section to the mongodb-deployment.yaml file.

---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-service
spec:
  selector:
    # To connect to Pod through label
    app: mongodb
  ports:
    - protocol: TCP
      # Service port
      port: 27017
      # This is the containerPort of deployment
      targetPort: 27017

Create Service:

[~]$ kubectl apply -f mongodb-deployment.yaml
deployment.apps/mongodb-deployment unchanged
service/mongodb-service created

To check that the service is attached to the correct pod:

kubectl describe service mongodb-service

That's the POD IP address and 27017 is the port where the application inside the Pod is listening.

Check that IP on the Pod by running:

kubectl get pod -o wide

View all components created so far

kubectl get all | grep mongodb

Create External configuration to put Db url for MongoDb

mongodb-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: mongodb-configmap
data:
  # Servername is just the name of the service
  database_url: mongodb-service

Create config map:

kubectl apply -f mongodb-configmap.yaml

Create MongoExpress deployment

Use config from here: https://hub.docker.com/_/mongo-express

You need these info:

1. Which Db it should connect to?
   Environment var: ME_CONFIG_MONGODB_SERVER
2. Credentials to authenticate using.
   Environment var: ME_CONFIG_ADMINUSERNAME and ME_CONFIG_ADMINPASSWORD

mongoexpress-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongo-express
  labels:
    app: mongo-express
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongo-express
  template:
    metadata:
      labels:
        app: mongo-express
    spec:
      containers:
      - name: mongo-express
        image: mongo-express
        ports:
        - containerPort: 8081
        env:
        - name: ME_CONFIG_MONGODB_ADMINUSERNAME
          valueFrom:
            secretKeyRef:
              name: mongodb-secret
              key: mongo-root-username
        - name: ME_CONFIG_MONGODB_ADMINPASSWORD
          valueFrom: 
            secretKeyRef:
              name: mongodb-secret
              key: mongo-root-password
        - name: ME_CONFIG_MONGODB_SERVER
          valueFrom: 
            configMapKeyRef:
              name: mongodb-configmap
              key: database_url

Create deployment:

kubectl apply -f mongoexpress-deployment.yaml

Create MongoExpress External Service

To access mongoexpress from the browser.

Add this section to the mongoexpress-deployment.yaml file.

---
apiVersion: v1
kind: Service
metadata:
  name: mongo-express-service
spec:
  selector:
    app: mongo-express
  # Add this guy that's different from internal service
  # Bad naming choice because internal service also works as a load balancer
  # This makes the service accept external requests by assigning the service an external IP address
  type: LoadBalancer  
  ports:
    - protocol: TCP
      port: 8081
      targetPort: 8081
      # Add this guy that's different from internal service
      # Port where the external IP address will be open. Must be between 30000 to 32767
      nodePort: 30000

Create external service:

kubectl apply -f mongoexpress-deployment.yaml

Check it out:

Internal Service or Cluster IP is DEFAULT.
LoadBalancer is also assigned an External IP.

Give a URL to external service in minikube

minikube service mongo-express-service

Namespace

Way to organize resources.

You can put your object/ resource into some namespace in the .yaml configuration file.

Why?

1. To group resources into different namespaces.
   

2. Use same cluster by different teams.
   

3. Resource sharing: Staging and Development.
   

4. Set access and resource limits.
   

You can access service in another Namespace:

You can't access most resources from another namespace.
For eg: Each namespace must define its own ConfigMap, Secret etc.

There are some components that can't live inside a namespace.

Ingress

To make an app accessible through the browser, you can use external service or ingress.

External Service looks like below. It uses http protocol and IP address:port of the node which is ok for testing but not good for final product.

Ingress makes it better:

One thing to note is that the http attribute under rules doesn't correspond to http protocol in the browser.
This just means that the incoming request gets forwarded to internal service.

Ingress and internal service configuration

In the host: myapp.com, myapp.com must be a valid domain address.
You should map domain name to Node's IP address, which is the entrypoint.

How to configure ingress in your cluster

You need an implementation for Ingress which is called Ingress Controller.

Install an ingress controller which is basically a pod or a set of pods which run on your node in your K8s cluster.

Ingress Controller

1. Evaluates all the rules defined in the cluster.
   
2. Manages redirections.
3. Entrypoint to cluster.
4. Many third-party implementations.

If you're running your K8s cluster on some cloud environment, you'll already have a cloud load balancer. External requests will first hit the load balancer and that will redirect the request to Ingress Controller.

But if you're deploying your K8s cluster on a bare-metal, you'll have to configure some kind of entrypoint yourself. That entrypoint can be either inside of your cluster or outside as a separate server.
For eg: An external Proxy Server shown below.

Install Ingress Controller in Minikube

minikube addons enable ingress

With 1 simple command, I can have Nginx ingress controller pod setup.

Create Ingress rule for K8s dashboard

kubectl get ns just lists all the namespaces in your current K8s context.

kubectl get all -n kubernetes-dashboard shows all the components in kubernetes-dashboard.
It already has internal service and pod for K8s dashboard (you can see that if you run the command above).

Now we can configure an ingress rule for dashboard so it's accessible from our browser using some domain name.

dashboard-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard-ingress
  # Same as service and pod!
  namespace: kubernetes-dashboard
spec:
  rules:
  - host: dashboard.com
    http:
      paths:
      - backend:
          service:
            name: kubernetes-dashboard
            port: 
              number: 80

Above is the ingress config for forwarding every request that is directed to dashboard.com to internal kubernetes-dashboard service. kubernetes-dashboard service is internal service as seen by it being of ClusterIP type.

Create ingress rule:

Resolve IP address to host name dashboard.com

Go to your hosts file and put that mapping.

[~]$ sudo vim /etc/hosts

Type Esc and :wq to save and exit.

Now go to your browser and type dashboard.com and you'll reach the K8s dashboard.

Ingress default backend

A good usage for this is to provide user with some custom error messages when a page isn't found.

For this, all you have to do is create an internal service with the name default-http-backend and create a pod or application that sends the custom error response.

Multiple paths for same host

Multiple sub-domains or domains

You can also have multiple hosts with 1 path. Each host represents a subdomain.

Configuring TLS certificate - https

Notes:

1. Data keys need to be "tls.crt" and "tls.key".
2. Values are file contents NOT file paths/ locations.
3. Secret component must be in the same namespace as the Ingress component. default in the example.

Helm

Package manager for Kubernetes.
To package YAML files and distribute them in public and private repositories.

For eg: You have deployed your app in a K8s cluster. Now you want to deploy/ add elastic stack for logging.
If I wanted to do this myself, I'd have to take care of the following components:

1. Stateful set: For stateful apps like Dbs
2. ConfigMap: For external config
3. K8s user with permissions
4. Secret: For secret data
5. Services

If I had to do this manually, it'd be tedious.
But since something like Elasticsearch is a standard deployment, someone already created the YAML files required for those deployments, packaged them up and made them available in repositories so other people could use them.

That bundle of YAML files is called HELM chart.

Helm Charts

1. Bundle of YAML files
2. Create your own Helm Charts with Helm
3. Push them to Helm Repository
4. Download and use existing ones

Standard apps with complex setup like: Database apps: MongoDb, MySQL, Elasticsearch, Monitoring apps: Prometheus etc. all have charts already available.

Helm is a templating engine

If you have an app that has a bunch of microservices and deployment and service configuration of each of those microservices look almost the same. For eg: The only difference is something simple like version and image name.
In this scenario, you can just define a base/ common blueprint and use placeholders to populate dynamic values.

The values come from values.yaml file.

Values injection into template files

helm install --values=my-values.yaml <chartname>

Helm helps deploying same app across different environments

Helm chart structure

1. mychart folder: Name of the chart.
2. Chart.yaml: Meta info about the chart. For eg: name, version, dependencies etc.
3. values.yaml: Values for the template files.
4. charts folder: Chart dependencies. For eg: If this chart depends on other charts.
5. templates folder: Actual template files.

Install chart using the command:

helm install <chartname>

Helm version 3 got rid of Tiller for good reasons

Kubernetes Volumes

Persist data in K8s using following ways:

1. Persistent Volume
2. Persistent Volume Claim
3. Storage Class

Let's say you have an app with a Db.

If you make changes to the db and restart the mysql pod, the data gets lost because K8s doesn't provide data persistence out of the box.

Storage Requirements:

1. We need a storage that doesn't depend on the pod lifecycle.
   
2. Storage must be available on all nodes.
   
3. Storage needs to survive even if cluster crashes.

Persistent Volume

1. Cluster resource.
2. Created via YAML file.
3. Needs actual physical storage.
   
4. Example that uses Google cloud:
   
5. They are NOT namespaced.
   

Persistent Volume Claim

Application has to claim the Persistent volume.

In other words, PVC specifies claims about a volume with storage size, access mode etc. and whichever persistent volume matches those claims will be used for the application.

Note: PVClaims must exist in the same namespace as the pod.

You use that claim in the Pods configuration like so:

Mounting volume:

Different volume types in a Pod

Storage Class

Storage class provisions persistent volumes dynamically when PersistentVolumeClaim claims it.

Using Storage class:

Flow:

Volume Types: ConfigMap and Secret

1. They both are local volumes.
2. They are not created via PV or PVC.
3. They are managed by K8s.

StatefulSet

Stateful vs Stateless apps

Deployment vs StatefulSet

Deployment:

When you delete one, one of those pods can be deleted randomly.

StatefulSet:

Pod Identity

Let's say ID-2 pod goes down and gets created again. Even when it's created newly this time, it'll have the same Id of ID-2.

Scaling Db apps

Only let one instance to write data to prevent data inconsistencies.

Let's say you add mysql-3 instance.

The workers continuously synchronize data.
PV means Persistent volume.

Pod state

Pod state is stored in its storage. When a pod dies and it gets replaced, the persistent pod identifiers make sure that the storage volume gets reattached to the replacement pod.

Pod Identity

For eg: StatefulSet with 3 replica:

1. mysql-0 : Master
2. mysql-1 : Worker
3. mysql-2 : Worker

Next pod is only created if previous one is up and running.
Deletion in reverse order, starting from the last one.

Pod Endpoints

Each pod in a stateful set gets it own DNS endpoint from a service.

2 characteristics:

It's complex so just use cloud db services.

Stateful apps are not perfect for containerized apps, stateless are.

Kubernetes Services

Pods in K8s are destroyed frequently, so it's not practical to reach them using their ip addresses. Services address this problem by providing a stable IP address even when the pod dies.

Different Services:

1. ClusterIP Services
2. Headless Services
3. NodePort Services
4. LoadBalancer Services

ClusterIP Services

This is a default type, so when you create a service and don't specify the type, this is what you'll get.

For eg:
Let's say we have deployed our microservice app in our cluster. Let's say this pod contains the app container and a sidecar container that collects logs from the app and sends it to some log store. The deployment yaml file looks like this:

Pod gets an IP address from a range that is assigned to a Node.

Let's say you access the app through the browser:

The ingress rule and app service yaml contents look like this:

As you can see, the ingress forwards to port 3200 where the service is, and the service forwards to port 3000 in the container where the app is running.

The servicePort is arbitrary while targetPort is not. targetPort has to match the port where container inside the pod is listening at.

Which pods to forward request to?
By using selectors which are labels of pods.

Which port to forward request to?

When we create the service, it will find all the pods that match the selector (1), so these pods will become endpoints of the service. When service gets a request, it will pick one of those pod replicas randomly beause it's a load balancer and it will send the request it received to that specific pod on a port defined by targetPort attribute (2).

Service Endpoints:
When we create a service, K8s creates an endpoints object that has the same name as the service and it will use the endpoints object to keep track of which pods are members/ endpoints of the service.

Multi-Port Services:

When you have multiple ports defined in a service, you have to name them.

Headless Services

Useful in scenario like:

1. Client wants to communicate with 1 specific Pod directly.
2. Pods wants to talk directly with specific Pod.
3. Pod can't be randomly selected.

Use case: Stateful applications, like databases where Pod replicas aren't identical.

To create a headless service, set clusterIP to none.

We have these 2 services alongside each other. ClusterIP one will handle load balanced requests, Headless one will handle data synchronization requests.

NodePort Services

Recall that ClusterIP is only accessible within the cluster so no external traffic can directly address the cluster IP service. The node port service however makes the external traffic accessible on static or fixed port on each worker node.

So in this case, instead of a browser request coming through the Ingress, it can directly come to the worker node at the port that the service specification defines. This way external traffic has access to fixed port on each Worker Node.

This is how NodePort service is created. Whenever we create a node port service, a cluster ip service to which the node port service will route is automatically created.

The nodePort value has a predefined range of 30000 - 32767.
Here you can reach the app at 172.90.1.2:30008.

View the service:

The node port has the cluster IP address and for that IP address, it also has the port open where the service is accessible at.

Service spans all the worker nodes:

Service is able to handle a request coming on any of the worker nodes and then forward it to one of the pod replicas. Like 172.90.1.1:30008 or 172.90.1.2:30008 or 172.90.1.3:30008. This type of service exposure is not secure.

Don't use it for external connection. Use it only for testing.

Configure Ingress or LoadBalancer for production environments.

LoadBalancer Services

A service becomes accessible externally through a cloud provider's load balancer service.

LoadBalancer service is an extension of NodePort service.
NodePort service is an extension of ClusterIP service.
So whenever we create LoadBalancer service, NodePort and ClusterIP services are created automatically.

The YAML file looks like this:

It works like this:

NodePort is the port that is open on the worker node but it's not directly accessible externally but only through the load balancer. So the entry point becomes the load balancer first and it can then direct the traffic to node port on the worker node and the cluster IP, the internal service.

The load balancer service looks like this:

Configure Ingress or LoadBalancer for production environments.