Merge pull request #99 from deefreak/deepakfork/ossCharts

Charts , Installation Guide and Remove some integrations
flipkart-incubator · Jan 3, 2024 · d7e7d1d · d7e7d1d
2 parents 3aa2233 + d06d834
commit d7e7d1d
Show file tree

Hide file tree

Showing 23 changed files with 465 additions and 752 deletions.
diff --git a/INSTALLATION.md b/INSTALLATION.md
@@ -12,11 +12,84 @@ Ottoscalr can be installed in any kubernetes cluster which meets the prerequisit
 ## Building the image
 
 The following section outlines the docker image build process for the ottoscalr. 
-
-TBA
+The following commands builds the image for the platform linux/amd64. Change the TARGETOS and TARGETARCH if it is required to be built for other platform.
+```console
+$ TARGETOS=linux;TARGETARCH=amd64;make docker-build docker-push IMG={repository}:{tag}
+```
 
 ## Deploying the ottoscalr
 
-TBA
+Ottoscalr can be deployed via this [helm chart](https://github.com/flipkart-incubator/ottoscalr/tree/main/charts/ottoscalr). 
+
+This chart bootstraps ottoscalr on a Kubernetes cluster using the Helm package manager. As part of that, it will install all the required Custom Resource Definitions (CRD).
+
+### Installing the Chart
+
+To install the chart with the release name `ottoscalr`:
+
+```console
+$ kubectl create namespace ottoscalr
+$ helm install ottoscalr <path-of-the-helm-chart> --namespace ottoscalr -f <path-of-the-helm-chart>/values.yaml
+```
+
+### Upgrading the Chart
+
+To update the already installed chart `ottoscalr`:
+
+```console
+$ helm upgrade ottoscalr <path-of-the-helm-chart> --namespace ottoscalr -f <path-of-the-helm-chart>/values.yaml
+```
+
+### Uninstalling the Chart
+
+To uninstall/delete the `ottoscalr` Helm chart:
+
+```console
+helm uninstall ottoscalr
+```
+The command removes all the Kubernetes components associated with the chart and deletes the release.
+
+
+
+### Configuration
+
+The following table lists the configurable parameters of the KEDA chart and
+their default values.
+
+#### Ottoscalr Deployment
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `image.repository` | string | `""` | Image name of ottoscalr deployment |
+| `image.tag` | string | `""` | Image tag of ottoscalr deployment |
+| `replicaCount` | int | `1` | Capability to configure the number of replicas for ottoscalr operator. While you can run more replicas of our operator, only one operator instance will be the leader and serving traffic. You can run multiple replicas, but they will not improve the performance of ottoscalr, it could only reduce downtime during a failover. |
+| `resources` | object | `{"limits":{"cpu":1,"memory":"2Gi"},"requests":{"cpu":"1","memory":"2Gi"}}` | Manage [resource request & limits] of ottoscalr operator pod |
+
+#### Operations
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `ottoscalr.config.metricsScraper.prometheusUrl` | string | `""` | URL where prometheus for the kubernetes cluster is running.   |
+| `ottoscalr.config.metricsScraper.queryTimeoutSec` | int | `300` | Time in seconds within which the response for any query should be served by the prometheus  |
+| `ottoscalr.config.metricsScraper.querySplitIntervalHr` | int | `24` | The shortest period in hour for which data will be fetched from prometheus |
+| `ottoscalr.config.policyRecommendationController.maxConcurrentReconciles` | int | `1` | Maximum number of concurrent Reconciles which can be run. |
+| `ottoscalr.config.policyRecommendationController.minRequiredReplicas` | int | `3` | The hpa.spec.minReplicas  recommended by the controller will not have replicas minimum than this.  |
+| `ottoscalr.config.policyRecommendationController.policyExpiryAge` | string | `3h` | Target Recommendation will be reached in multiple iterations and through different policies. This is the time after which a policy expires and next policy in the list can be applied. |
+| `ottoscalr.config.breachMonitor.pollingIntervalSec` | int | `300` | Time in seconds to check for any breaches happened during this time |
+| `ottoscalr.config.breachMonitor.cpuRedLine` | float | `0.75` | Conisder it a breach if CPU utilization goes above this number. |
+| `ottoscalr.config.breachMonitor.concurrentExecutions` | int | `50` | Concurrent execution of breach queries. Configure it according to the number of concurrent connections that can be handled safely by the prometheus  |
+| `ottoscalr.config.periodicTrigger.pollingIntervalMin` | int |  `180` | Duration in minutes to periodically trigger the recommendation generation |
+| `ottoscalr.config.policyRecommendationRegistrar.excludedNamespaces` | string | `""` | Comma separated namespaces where ottoscalr will not generate recommendations. Example: "namespace1,namespace2,namespace3" |
+| `ottoscalr.config.cpuUtilizationBasedRecommender.metricWindowInDays` | int | `28` | Number of days for which cpu utilization metrics should be fetched for generating the recommendation. |
+| `ottoscalr.config.cpuUtilizationBasedRecommender.minTarget` | int | `5` | The cpu utilization threshold recommended will not be less than this. |
+| `ottoscalr.config.cpuUtilizationBasedRecommender.maxTarget` | int | `60` | The cpu utilization threshold recommended will not be more than this. |
+| `ottoscalr.config.cpuUtilizationBasedRecommender.metricsPercentageThreshold` | int | `25` | Minimum percentage of metrics to be present to generate any recommendation. Example: if `ottoscalr.config.cpuUtilizationBasedRecommender.metricWindowInDays` is 28 days and `ottoscalr.config.cpuUtilizationBasedRecommender.metricsPercentageThreshold` is 25%, then atleast 7 days worth of data should be present. |
+| `ottoscalr.config.hpaEnforcer.maxConcurrentReconciles` | int | `1` | Maximum number of concurrent Reconciles which can be run by hpaenforcement controller |
+| `ottoscalr.config.hpaEnforcer.excludedNamespaces` | string | `""` | Comma separated namespaces where ottoscalr will not create HPAs. Example: "namespace1,namespace2,namespace3" |
+| `ottoscalr.config.hpaEnforcer.includedNamespaces` | string | `""` | If provided, ottoscalr will only create HPAs for these namespaces. If it is empty, it will include all except for the excluded ones. Example: "namespace1,namespace2,namespace3" |
+| `ottoscalr.config.hpaEnforcer.whitelistMode` | bool | `true` | hpa controller will act only on deployments having `ottoscalr.io/enable-hpa-enforcement: true` annotation and create hpa for them. If false, hpaEnforcer runs on blacklistMode where it will create hpas for every workload except for ones having `ottoscalr.io/disable-hpa-enforcement: true`. It is recommended to run with whitelistMode as true. |
+| `ottoscalr.config.hpaEnforcer.isDryRun` | bool | `false` | If true, hpa controller will not create any HPAs.   |
+| `ottoscalr.config.hpaEnforcer.enableMetricsTransformer` | bool | `true` | This metrics transformer can be used to interpolate any known period of data that should not be used for generating recommendation.  |
+| `ottoscalr.config.autoscalerClient.scaledObjectConfigs.enableScaledObject` | bool | `false` | Flag whether to use KEDA ScaledObjects or HPA for autoscaling. KEDA needs to be deployed on your cluster for enabling it. |
+| `ottoscalr.config.autoscalerClient.hpaConfigs.hpaAPIVersion` | string | `"v2"` | Set this if using HPA for autoscaling. By default, `autoscaling/v2` api is supported. If you wish to use `autoscaling/v1` api for HPA, change this to `"v1"`. |
 
-### Configuration
diff --git a/charts/ottoscalr/envconfigs/ottoscalr_config.yaml b/charts/ottoscalr/envconfigs/ottoscalr_config.yaml
@@ -1,26 +1,51 @@
 port: 9443
 metricBindAddress: ":8080"
 healthProbeBindAddress: ":8081"
+pprofBindAddress: ":8082"
 enableLeaderElection: false
 leaderElectionID: "85d48caf.fcp.ottoscalr.io"
 metricsScraper:
-  prometheusUrl: {{ .Values.prometheusUrl }}
-  queryTimeoutSec: 30
-  querySplitIntervalHr: 24
+  prometheusUrl: {{ .Values.ottoscalr.config.metricsScraper.prometheusUrl }}
+  queryTimeoutSec: {{ .Values.ottoscalr.config.metricsScraper.queryTimeoutSec | default "30" }}
+  querySplitIntervalHr: {{ .Values.ottoscalr.config.metricsScraper.querySplitIntervalHr | default "24" }}
 breachMonitor:
-  pollingIntervalSec: 300
-  cpuRedLine: 0.85
+  pollingIntervalSec: {{ .Values.ottoscalr.config.breachMonitor.pollingIntervalSec | default "300" }}
+  cpuRedLine: {{ .Values.ottoscalr.config.breachMonitor.cpuRedLine | default "0.75" }}
   stepSec: 30
+  concurrentExecutions: {{ .Values.ottoscalr.config.breachMonitor.concurrentExecutions | default "50" }}
 periodicTrigger:
-  pollingIntervalMin: 360
+  pollingIntervalMin: {{ .Values.ottoscalr.config.periodicTrigger.pollingIntervalMin | default "1440" }}
 policyRecommendationController:
-  maxConcurrentReconciles: 1
+  maxConcurrentReconciles: {{ .Values.ottoscalr.config.policyRecommendationController.maxConcurrentReconciles | default "1" }}
+  minRequiredReplicas: {{ .Values.ottoscalr.config.policyRecommendationController.minRequiredReplicas | default "3" }}
+  policyExpiryAge: {{ .Values.ottoscalr.config.policyRecommendationController.policyExpiryAge | default "48h" }}
 policyRecommendationRegistrar:
   requeueDelayMs: 500
+  excludedNamespaces: {{ .Values.ottoscalr.config.policyRecommendationRegistrar.excludedNamespaces | default "kube-system,monitoring,gatekeeper-system,webhook" }}
+  includedNamespaces: {{ .Values.ottoscalr.config.policyRecommendationRegistrar.includedNamespaces }}
 cpuUtilizationBasedRecommender:
-  metricWindowInDays: 28
+  metricWindowInDays: {{ .Values.ottoscalr.config.cpuUtilizationBasedRecommender.metricWindowInDays | default "28" }}
   stepSec: 30
-  minTarget: 10
-  maxTarget: 60
+  minTarget: {{ .Values.ottoscalr.config.cpuUtilizationBasedRecommender.minTarget | default "5" }}
+  maxTarget: {{ .Values.ottoscalr.config.cpuUtilizationBasedRecommender.maxTarget | default "60" }}
+  metricsPercentageThreshold: {{ .Values.ottoscalr.config.cpuUtilizationBasedRecommender.metricsPercentageThreshold | default "0" }}
 metricIngestionTime: 15.0
 metricProbeTime: 15.0
+enableMetricsTransformer: {{ .Values.ottoscalr.config.enableMetricsTransformer | default false }}
+eventCallIntegration:
+  customEventDataConfigMapName: {{ .Values.ottoscalr.config.eventCallIntegration.customEventDataConfigMapName | default "custom-event-data-config" }}
+hpaEnforcer:
+  maxConcurrentReconciles: {{ .Values.ottoscalr.config.hpaEnforcer.maxConcurrentReconciles | default 1 }}
+  excludedNamespaces: {{ .Values.ottoscalr.config.hpaEnforcer.excludedNamespaces | default "" }}
+  includedNamespaces: {{ .Values.ottoscalr.config.hpaEnforcer.includedNamespaces | default "" }}
+  isDryRun: {{ kindIs "invalid" .Values.ottoscalr.config.hpaEnforcer.isDryRun |  ternary true .Values.ottoscalr.config.hpaEnforcer.isDryRun }}
+  whitelistMode: {{ kindIs "invalid" .Values.ottoscalr.config.hpaEnforcer.whitelistMode |  ternary true .Values.ottoscalr.config.hpaEnforcer.whitelistMode }}
+  minRequiredReplicas: {{ .Values.ottoscalr.config.hpaEnforcer.minRequiredReplicas | default 2 }}
+autoscalerClient:
+  scaledObjectConfigs:
+    enableScaledObject: {{ kindIs "invalid" .Values.ottoscalr.config.autoscalerClient.scaledObjectConfigs.enableScaledObject |  ternary true .Values.ottoscalr.config.autoscalerClient.scaledObjectConfigs.enableScaledObject }}
+    enableEventAutoscaler: {{ kindIs "invalid" .Values.ottoscalr.config.autoscalerClient.scaledObjectConfigs.enableEventAutoscaler |  ternary true .Values.ottoscalr.config.autoscalerClient.scaledObjectConfigs.enableEventAutoscaler }}
+  hpaConfigs:
+    hpaAPIVersion: {{ .Values.ottoscalr.config.autoscalerClient.hpaConfigs.hpaAPIVersion | default "v2" }}
+enableArgoRolloutsSupport: {{ kindIs "invalid" .Values.ottoscalr.config.enableArgoRolloutsSupport |  ternary true .Values.ottoscalr.config.enableArgoRolloutsSupport }}
+
diff --git a/charts/ottoscalr/templates/configmap.yaml b/charts/ottoscalr/templates/configmap.yaml
@@ -10,4 +10,4 @@ metadata:
     {{- include "ottoscalr.labels" . | nindent 4 }}
   namespace: {{ .Release.Namespace }}
 data:
-{{ ( tpl (.Files.Glob "envconfigs/*").AsConfig .) | indent 2 }}
+{{ ( tpl (.Files.Glob "envconfigs/ottoscalr_config.yaml").AsConfig .) | indent 2 }}
diff --git a/...templates/crd/ottoscaler.io_policies.yaml → ...emplates/crds/ottoscaler.io_policies.yaml b/...templates/crd/ottoscaler.io_policies.yaml → ...emplates/crds/ottoscaler.io_policies.yaml
@@ -35,19 +35,16 @@ spec:
           spec:
             description: PolicySpec defines the desired state of Policy
             properties:
-              id:
-                type: string
               isDefault:
                 type: boolean
-              min:
+              minReplicaPercentageCut:
                 type: integer
               riskIndex:
-                type: string
+                type: integer
               targetUtilization:
                 type: integer
             required:
-            - id
-            - min
+            - minReplicaPercentageCut
             - riskIndex
             - targetUtilization
             type: object

diff --git a/.../ottoscaler.io_policyrecommendations.yaml → .../ottoscaler.io_policyrecommendations.yaml b/.../ottoscaler.io_policyrecommendations.yaml → .../ottoscaler.io_policyrecommendations.yaml
@@ -36,49 +36,24 @@ spec:
           spec:
             description: PolicyRecommendationSpec defines the desired state of PolicyRecommendation
             properties:
+              currentHPAConfig:
+                properties:
+                  max:
+                    type: integer
+                  min:
+                    type: integer
+                  targetMetricValue:
+                    type: integer
+                required:
+                - max
+                - min
+                - targetMetricValue
+                type: object
               generatedAt:
                 format: date-time
                 type: string
               policy:
-                description: Policy is the Schema for the policies API
-                properties:
-                  apiVersion:
-                    description: 'APIVersion defines the versioned schema of this
-                      representation of an object. Servers should convert recognized
-                      schemas to the latest internal value, and may reject unrecognized
-                      values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
-                    type: string
-                  kind:
-                    description: 'Kind is a string value representing the REST resource
-                      this object represents. Servers may infer this from the endpoint
-                      the client submits requests to. Cannot be updated. In CamelCase.
-                      More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-                    type: string
-                  metadata:
-                    type: object
-                  spec:
-                    description: PolicySpec defines the desired state of Policy
-                    properties:
-                      id:
-                        type: string
-                      isDefault:
-                        type: boolean
-                      min:
-                        type: integer
-                      riskIndex:
-                        type: string
-                      targetUtilization:
-                        type: integer
-                    required:
-                    - id
-                    - min
-                    - riskIndex
-                    - targetUtilization
-                    type: object
-                  status:
-                    description: PolicyStatus defines the observed state of Policy
-                    type: object
-                type: object
+                type: string
               queuedForExecution:
                 type: boolean
               queuedForExecutionAt:
@@ -97,6 +72,9 @@ spec:
                 - min
                 - targetMetricValue
                 type: object
+              transitionedAt:
+                format: date-time
+                type: string
               workload:
                 properties:
                   apiVersion:
@@ -113,26 +91,13 @@ spec:
                     type: string
                   name:
                     type: string
-                  namespace:
-                    type: string
-                required:
-                - name
-                - namespace
                 type: object
-            required:
-            - policy
-            - queuedForExecution
-            - targetHPAConfig
-            - workload
             type: object
           status:
             description: PolicyRecommendationStatus defines the observed state of
               PolicyRecommendation
             properties:
               conditions:
-                description: 'INSERT ADDITIONAL STATUS FIELD - define observed state
-                  of cluster Important: Run "make" to regenerate code after modifying
-                  this file'
                 items:
                   description: "Condition contains details for one aspect of the current
                     state of this API Resource. --- This struct is intended for direct
@@ -200,6 +165,9 @@ spec:
                   - type
                   type: object
                 type: array
+                x-kubernetes-list-map-keys:
+                - type
+                x-kubernetes-list-type: map
             type: object
         type: object
     served: true

diff --git a/charts/ottoscalr/templates/customeventdataconfig.yaml b/charts/ottoscalr/templates/customeventdataconfig.yaml
@@ -0,0 +1,16 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: custom-event-data-config
+  labels:
+    app.kubernetes.io/name: configmap
+    app.kubernetes.io/instance: ottoscalr-manager
+    app.kubernetes.io/created-by: ottoscalr
+    app.kubernetes.io/part-of: ottoscalr
+    {{- include "ottoscalr.labels" . | nindent 4 }}
+  namespace: {{ .Release.Namespace }}
+data:
+{{- if .Values.addCustomEventData }}
+  7f8b9c83: '{"eventId":"7f8b9c83","eventName":"Outlier","startTime":"2023-07-27 04:00","endTime":"2023-07-27 05:00"}'
+  7f8b9c84: '{"eventId":"7f8b9c84","eventName":"Outlier","startTime":"2023-08-02 21:00","endTime":"2023-08-02 23:59"}'
+{{- end }}
diff --git a/charts/ottoscalr/templates/deployment.yaml b/charts/ottoscalr/templates/deployment.yaml
@@ -4,6 +4,7 @@ metadata:
   name: ottoscalr-manager
   namespace: {{ .Release.Namespace }}
   labels:
+    app: ottoscalr
     app.kubernetes.io/name: deployment
     app.kubernetes.io/instance: ottoscalr-manager
     app.kubernetes.io/component: manager
@@ -17,12 +18,14 @@ spec:
       {{- include "ottoscalr.selectorLabels" . | nindent 6 }}
   template:
     metadata:
-      {{- with .Values.podAnnotations }}
       annotations:
         kubectl.kubernetes.io/default-container: manager
+        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
+      {{- with .Values.podAnnotations }}
         {{- toYaml . | nindent 8 }}
       {{- end }}
       labels:
+        app: ottoscalr
         {{- include "ottoscalr.selectorLabels" . | nindent 8 }}
     spec:
       {{- with .Values.imagePullSecrets }}
@@ -47,6 +50,10 @@ spec:
           env:
             - name: OTTOSCALR_CONFIG
               value: {{ .Values.ottoscalrConfigPath }}
+            - name: DEPLOYMENT_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
           ports:
             - name: http
               containerPort: 8080
@@ -70,6 +77,7 @@ spec:
             {{- toYaml .Values.resources | nindent 12 }}
           volumeMounts:
             {{- toYaml .Values.volumeMounts | nindent 12 }}
+
       {{- with .Values.nodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}

diff --git a/charts/ottoscalr/templates/p8s-role.yaml b/charts/ottoscalr/templates/p8s-role.yaml
@@ -0,0 +1,22 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: p8s-ottoscalr
+  labels:
+    app.kubernetes.io/name: clusterrole
+    app.kubernetes.io/component: rbac
+    app.kubernetes.io/created-by: ottoscalr
+    app.kubernetes.io/part-of: ottoscalr
+    {{- include "ottoscalr.labels" . | nindent 4 }}
+rules:
+  - apiGroups:
+      - ""
+      - apps
+      - discovery.k8s.io
+    resources:
+      - pods
+      - services
+      - endpointslices
+      - endpoints
+
+    verbs: ["get", "list", "watch"]
diff --git a/charts/ottoscalr/templates/p8s-rolebinding.yaml b/charts/ottoscalr/templates/p8s-rolebinding.yaml
@@ -0,0 +1,19 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: p8s-ottoscalr
+  namespace: {{ .Release.Namespace }}
+  labels:
+    app.kubernetes.io/name: clusterrole
+    app.kubernetes.io/component: rbac
+    app.kubernetes.io/created-by: ottoscalr
+    app.kubernetes.io/part-of: ottoscalr
+      {{- include "ottoscalr.labels" . | nindent 4 }}
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: p8s-ottoscalr
+subjects:
+  - namespace: monitoring
+    kind: ServiceAccount
+    name: prometheus-k8s