From 580da206038b0fe6a6ad1593189ad4985c487a48 Mon Sep 17 00:00:00 2001 From: Nguyen Nhu Viet Date: Wed, 24 May 2023 17:45:56 +0200 Subject: [PATCH] feat: K8S IaC - GitOps with ArgoCD (#93) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: enable helm chart inflation * feat: add loki and promtail * fix: some changes on the names - tags - azs * feat: remove redis * feat: move obsy from argo to kust * feat: rename folder * feat: try different configuration * Connectivity working * enable cni - argocd manage addonsg * undo - to _ * cleanup commented code * fix: change name to root * fix: wrong name * feat: pending code to push * fix: app generation * refactor: split backend.tf file * feat: update apps * feat: add comments - remove branch * feat: add tooling app * feat: split in folders * feat: add branch * fix: change target revision value in root app * fix: update tools path * fix: values file * feat: add influxdb for tg-sidecar * feat: fix influx * try to use containerd sock * undo containerd sock * feat: upgrade versions - some tests * some updates and fixes - cluster can be created now * specify ns * feat: add argocd values file - more nodes/masters * feat: change ami zone to eu-west-1 * feat: add comments * feat: add resources required when creating a cluster with kops * feat: add efs helm chart installation * feat: add context name * feat: change tag * feat: comment not create sa * resources for efs * feat: add rbac * feat: remove efs roles * fix: issues with EFS - update AMI * feat: add nfs client installation to the nodes * feat: test create sa from helm need permission cannot get resource leases in API group coordination.k8s.io * remove comments * temp commented * feat: uncomment networking * update registry * fix: cleanup ds_store * feat: add delete tf resources + cluster * feat: create tf resoruces + add description * feat: move ns to kustomization * feat: comment influx observability out * feat: add grafana resources * feat: remove influxdb * feat: add grafana crdsb * feat: add olm + grafana opeartor * feat: add ns * cleanup comments * test another version * move resources * add monitoring * rename monitoring * add ns observability * add crds * feat: redis change name * feat: redis add ns * feat: redis not specify ns * feat: redis add url * add permissions * redis - bitnami * add influxdb charts - bitnami * add redis charts - bitnami * fix prometheus * add ns * change service to redis * change img * fix: lb permissions issues * fix: crds names * fix: olm * fix: prometheus admission controller * add ns * add system-tools * add grafana-agent-operator * add crd * add grafana-agent crd - 2 * add crd grafana-dashboard * add loki config * add storage class * feat: add multi master in 3 pols, rename cluster - rename tf resources * fix: CNI to AWS-CNI * fix: update telegraf-operator * fix: move to default nsl * fix: comment telegraf - manual installation * fix: move telegraf to tools * fix: install telegraf again * refactor: cleanup not needed files * add serversiderender to all * fix: commete * fix: values inline * feat: use argocd app instead * feat: rename app * network resources updated * deactivate telegraf * activate telegraf * activate networking * refactor: cleanup * use files for telegraf * feat: add clusterautoscaler - tags * feat: add clusterautoscaler - increase number of nodes * feat: test versions * feat: update comments * feat: cleanup not needed files * feat: update variables * feat: cleanup not needed files * remove comments * docs: update readme + content * fix: blank spaces * feat: resources increased * feat: multus resources increased * feat: increase timeouts * fix: solve comments in the PR * feat: add more replicas to influxdb * feat: update cidr - nodes - types * feat: reduce number of instances on the infra pool * feat: update to the latest version 900c23e * feat: influxdb changes url * feat: add autoscaler * feat: update sync-service - increase timeouts * feat: use latest commit for sync-service * feat: update sync-service - increase timeouts * feat: update resources - running 6k nodes * feat: update resources * feat: dont autosync * feat: undo autosync * feat: move observability to default ns * feat: update redis svc * feat: change references to observability ns * feat: use testground org and specify the tag v0.7.0 * feat: remove comments - move to v1.23.4 * feat: reduce number of instances in tginfra nodepool --------- Signed-off-by: Smuu <18609909+Smuu@users.noreply.github.com> Co-authored-by: Smuu <18609909+Smuu@users.noreply.github.com> Co-authored-by: Jose Ramon Mañes --- .editorconfig | 12 + argocd-root/kustomization.yaml | 5 + argocd-root/root.yaml | 22 + argocd/aws-applicationset.yaml | 34 + argocd/kustomization.yaml | 11 + argocd/networking-applicationset.yaml | 35 + argocd/observability-applicationset.yaml | 35 + argocd/telegraf-operator.yaml | 41 + argocd/testground-applicationset.yaml | 35 + argocd/testground-tools-applicationset.yaml | 35 + docs/argocd_dashboard.png | Bin 0 -> 332375 bytes docs/argocd_root_app.png | Bin 0 -> 67586 bytes k8s/eks/bash/functions.sh | 34 +- k8s/eks/yaml/otel/configmap-collector.yml | 37 + k8s/eks/yaml/otel/deployment.yml | 47 + k8s/eks/yaml/otel/service.yml | 26 + k8s/eks/yaml/telegraf-operator/values.yaml | 47 + k8s/kops/01_install_k8s.sh | 109 - k8s/kops/02_efs.sh | 86 - k8s/kops/03_ebs.sh | 52 - k8s/kops/04_testground_daemon.sh | 32 - k8s/kops/README.md | 155 +- k8s/kops/argocd/values.yaml | 27 + k8s/kops/cluster.yaml | 339 +- k8s/kops/delete_ebs.sh | 25 - k8s/kops/delete_efs.sh | 49 - k8s/kops/delete_kops.sh | 9 + k8s/kops/ebs-terraform/backend.tf | 3 - k8s/kops/ebs-terraform/ebs.tf | 13 - k8s/kops/ebs-terraform/outputs.tf | 8 - k8s/kops/ebs-terraform/variables.tf | 13 - k8s/kops/ebs/pv.yml.spec | 15 - k8s/kops/ebs/storageclass.yml | 11 - k8s/kops/efs-terraform/backend.tf | 3 - k8s/kops/efs-terraform/efs.tf | 23 - k8s/kops/efs-terraform/outputs.tf | 23 - k8s/kops/efs-terraform/variables.tf | 11 - k8s/kops/efs/manifest.yaml.spec | 74 - k8s/kops/efs/rbac.yaml | 54 - k8s/kops/influxdb/values.yaml | 7 - k8s/kops/install-playbook/validation.sh | 9 +- k8s/kops/install_k8s.sh | 144 + k8s/kops/kops-weave/dummy.yml | 29 - k8s/kops/kops-weave/flannel.yml | 227 - k8s/kops/kops-weave/genie-plugin.yaml | 231 - k8s/kops/kops-weave/weave-metrics-service.yml | 15 - k8s/kops/kops-weave/weave-service-monitor.yml | 19 - k8s/kops/kops-weave/weave.yml | 260 - k8s/kops/limit-range/limit-range.yaml | 12 - k8s/kops/packer/Makefile | 3 +- k8s/kops/packer/docker-pull-images.sh | 6 + k8s/kops/prometheus-operator/values.yaml | 155 - k8s/kops/sidecar.yaml | 73 - k8s/kops/testground-daemon/service.yml | 12 - k8s/kops/testground-infra/Chart.yaml | 8 - .../charts/testground-dashboards/Chart.yaml | 8 - .../dashboards/README.md | 59 - .../dashboards/dashboards/coredns.json | 1400 --- .../dashboards/dashboards/k8s-api-server.json | 2193 ---- .../dashboards/dashboards/k8s-kubelet.json | 2322 ----- .../dashboards/k8s-networking-cluster.json | 1703 ---- .../k8s-networking-namespace-pods.json | 1369 --- .../dashboards/k8s-networking-pod.json | 1155 --- .../dashboards/k8s-stateful-sets.json | 873 -- .../dashboards/dashboards/nodes.json | 933 -- .../dashboards/dashboards/prometheus.json | 1265 --- .../dashboards/dashboards/redis.json | 1168 --- .../dashboards/use-method-cluster.json | 1026 -- .../dashboards/use-method-node.json | 1052 -- .../dashboards/weavenet-cluster.json | 3347 ------- .../dashboards/dashboards/weavenet.json | 2605 ----- .../dashboards/datasources/prometheus.json | 1 - .../testground-dashboards/dashboards/main.go | 194 - .../templates/dashboard.yaml | 8 - .../templates/datasource.yaml | 8 - k8s/kops/testground-infra/requirements.lock | 9 - k8s/kops/testground-infra/requirements.yaml | 9 - k8s/kops/testground-infra/values.yaml | 44 - k8s/kops/tools/README.md | 9 - k8s/kops/tools/authorize-19999.sh | 12 - k8s/kops/tools/confirm_dns_route.sh | 6 - k8s/kops/tools/redeploy-daemon.sh | 19 - manifests/aws/ebs/kustomization.yaml | 5 + .../ebs/pvc.yml => manifests/aws/ebs/pvc.yaml | 0 .../efs/charts/aws-efs-csi-driver/.helmignore | 22 + .../charts/aws-efs-csi-driver/CHANGELOG.md | 181 + .../efs/charts/aws-efs-csi-driver/Chart.yaml | 18 + .../aws-efs-csi-driver/templates/NOTES.txt | 3 + .../aws-efs-csi-driver/templates/_helpers.tpl | 56 + .../templates/controller-deployment.yaml | 132 + .../templates/controller-serviceaccount.yaml | 62 + .../templates/csidriver.yaml | 10 + .../templates/node-daemonset.yaml | 171 + .../templates/node-serviceaccount.yaml | 12 + .../templates/storageclass.yaml | 26 + .../efs/charts/aws-efs-csi-driver/values.yaml | 147 + manifests/aws/efs/kustomization.yaml | 17 + manifests/aws/efs/pvc.yaml | 11 + manifests/aws/efs/rbac.yaml | 100 + manifests/aws/efs/values.yaml | 32 + manifests/aws/kustomization.yaml | 7 + manifests/kustomization.yaml | 11 + manifests/networking/kustomization.yaml | 7 + .../networking/multus/kustomization.yaml | 7 + .../networking/multus/multus-daemonset.yaml | 275 + manifests/networking/multus/softlink-cm.yaml | 27 + manifests/networking/multus/softlink-ds.yaml | 48 + .../networking/weave/clusterrolebinding.yaml | 69 + manifests/networking/weave/drop-weave-cm.yaml | 29 + manifests/networking/weave/drop-weave-ds.yaml | 48 + manifests/networking/weave/kustomization.yaml | 11 + .../weave/networkattachmentdefinition.yaml | 4 + manifests/networking/weave/weave.yaml | 234 + .../grafana-agent-operator/kustomization.yaml | 14 + .../observability/grafana/base/grafana.yaml | 18 + .../grafana/base/kustomization.yaml | 10 + .../grafana/base/loki_datasource.yaml | 17 + .../grafana/base/monitoring/README.md | 1 + .../grafana/base/monitoring/dashboards.sh | 31 + .../grafana/base/monitoring/dashboards.yaml | 301 + .../base/monitoring/dashboards.yaml.tmpl | 12 + .../base/monitoring/kustomization.yaml | 9 + .../base/monitoring/kustomizeconfig.yaml | 5 + .../grafana/base/prometheus_datasource.yaml | 17 + .../observability/grafana/dashboards.yaml | 24 + .../grafana/dashboards/base-logs.json | 652 ++ .../grafana/dashboards/base-monitoring.json | 812 ++ .../observability/grafana/kustomization.yaml | 17 + manifests/observability/kustomization.yaml | 12 + .../observability/loki/kustomization.yaml | 11 + manifests/observability/loki/values.yaml | 36 + .../otel/configmap-collector.yaml | 37 + manifests/observability/otel/deployment.yaml | 47 + .../observability/otel/kustomization.yaml | 9 + manifests/observability/otel/service.yaml | 26 + .../prometheus-crds/kustomization.yaml | 21 + .../charts/kube-prometheus-stack/.helmignore | 28 + .../kube-prometheus-stack/CONTRIBUTING.md | 12 + .../charts/kube-prometheus-stack/Chart.lock | 12 + .../charts/kube-prometheus-stack/Chart.yaml | 55 + .../charts/kube-prometheus-stack/README.md | 768 ++ .../charts/grafana/.helmignore | 23 + .../charts/grafana/Chart.yaml | 22 + .../charts/grafana/README.md | 603 ++ .../charts/grafana/ci/default-values.yaml | 1 + .../grafana/ci/with-affinity-values.yaml | 16 + .../ci/with-dashboard-json-values.yaml | 53 + .../grafana/ci/with-dashboard-values.yaml | 19 + .../ci/with-extraconfigmapmounts-values.yaml | 7 + .../ci/with-image-renderer-values.yaml | 19 + .../charts/grafana/ci/with-persistence.yaml | 3 + .../grafana/dashboards/custom-dashboard.json | 1 + .../charts/grafana/templates/NOTES.txt | 54 + .../charts/grafana/templates/_helpers.tpl | 199 + .../charts/grafana/templates/_pod.tpl | 1143 +++ .../charts/grafana/templates/clusterrole.yaml | 25 + .../grafana/templates/clusterrolebinding.yaml | 24 + .../configmap-dashboard-provider.yaml | 29 + .../charts/grafana/templates/configmap.yaml | 125 + .../templates/dashboards-json-configmap.yaml | 35 + .../charts/grafana/templates/deployment.yaml | 50 + .../grafana/templates/extra-manifests.yaml | 4 + .../grafana/templates/headless-service.yaml | 23 + .../charts/grafana/templates/hpa.yaml | 49 + .../templates/image-renderer-deployment.yaml | 119 + .../image-renderer-network-policy.yaml | 73 + .../templates/image-renderer-service.yaml | 31 + .../charts/grafana/templates/ingress.yaml | 78 + .../grafana/templates/networkpolicy.yaml | 52 + .../templates/poddisruptionbudget.yaml | 22 + .../grafana/templates/podsecuritypolicy.yaml | 49 + .../charts/grafana/templates/pvc.yaml | 36 + .../charts/grafana/templates/role.yaml | 32 + .../charts/grafana/templates/rolebinding.yaml | 25 + .../charts/grafana/templates/secret-env.yaml | 14 + .../charts/grafana/templates/secret.yaml | 26 + .../charts/grafana/templates/service.yaml | 55 + .../grafana/templates/serviceaccount.yaml | 17 + .../grafana/templates/servicemonitor.yaml | 44 + .../charts/grafana/templates/statefulset.yaml | 55 + .../templates/tests/test-configmap.yaml | 20 + .../tests/test-podsecuritypolicy.yaml | 32 + .../grafana/templates/tests/test-role.yaml | 17 + .../templates/tests/test-rolebinding.yaml | 20 + .../templates/tests/test-serviceaccount.yaml | 12 + .../charts/grafana/templates/tests/test.yaml | 49 + .../charts/grafana/values.yaml | 1157 +++ .../charts/kube-state-metrics/.helmignore | 21 + .../charts/kube-state-metrics/Chart.yaml | 21 + .../charts/kube-state-metrics/README.md | 68 + .../kube-state-metrics/templates/NOTES.txt | 10 + .../kube-state-metrics/templates/_helpers.tpl | 101 + .../templates/clusterrolebinding.yaml | 20 + .../templates/deployment.yaml | 179 + .../templates/kubeconfig-secret.yaml | 12 + .../kube-state-metrics/templates/pdb.yaml | 18 + .../templates/podsecuritypolicy.yaml | 39 + .../templates/psp-clusterrole.yaml | 19 + .../templates/psp-clusterrolebinding.yaml | 16 + .../kube-state-metrics/templates/role.yaml | 196 + .../templates/rolebinding.yaml | 24 + .../kube-state-metrics/templates/service.yaml | 49 + .../templates/serviceaccount.yaml | 15 + .../templates/servicemonitor.yaml | 81 + .../templates/stsdiscovery-role.yaml | 26 + .../templates/stsdiscovery-rolebinding.yaml | 17 + .../templates/verticalpodautoscaler.yaml | 34 + .../charts/kube-state-metrics/values.yaml | 307 + .../prometheus-node-exporter/.helmignore | 21 + .../prometheus-node-exporter/Chart.yaml | 18 + .../charts/prometheus-node-exporter/README.md | 77 + .../ci/port-values.yaml | 3 + .../templates/NOTES.txt | 15 + .../templates/_helpers.tpl | 128 + .../templates/daemonset.yaml | 225 + .../templates/endpoints.yaml | 18 + .../templates/psp-clusterrole.yaml | 14 + .../templates/psp-clusterrolebinding.yaml | 16 + .../templates/psp.yaml | 49 + .../templates/service.yaml | 23 + .../templates/serviceaccount.yaml | 17 + .../templates/servicemonitor.yaml | 53 + .../templates/verticalpodautoscaler.yaml | 34 + .../prometheus-node-exporter/values.yaml | 291 + .../crds/crd-alertmanagerconfigs.yaml | 4475 +++++++++ .../crds/crd-alertmanagers.yaml | 6851 +++++++++++++ .../crds/crd-podmonitors.yaml | 666 ++ .../crds/crd-probes.yaml | 705 ++ .../crds/crd-prometheuses.yaml | 8865 +++++++++++++++++ .../crds/crd-prometheusrules.yaml | 121 + .../crds/crd-servicemonitors.yaml | 696 ++ .../crds/crd-thanosrulers.yaml | 6437 ++++++++++++ .../kube-prometheus-stack/templates/NOTES.txt | 4 + .../templates/_helpers.tpl | 254 + .../templates/alertmanager/alertmanager.yaml | 169 + .../templates/alertmanager/extrasecret.yaml | 20 + .../templates/alertmanager/ingress.yaml | 77 + .../alertmanager/ingressperreplica.yaml | 67 + .../alertmanager/podDisruptionBudget.yaml | 21 + .../templates/alertmanager/psp-role.yaml | 23 + .../alertmanager/psp-rolebinding.yaml | 20 + .../templates/alertmanager/psp.yaml | 47 + .../templates/alertmanager/secret.yaml | 27 + .../templates/alertmanager/service.yaml | 53 + .../alertmanager/serviceaccount.yaml | 20 + .../alertmanager/servicemonitor.yaml | 46 + .../alertmanager/serviceperreplica.yaml | 49 + .../templates/exporters/core-dns/service.yaml | 24 + .../exporters/core-dns/servicemonitor.yaml | 39 + .../kube-api-server/servicemonitor.yaml | 42 + .../kube-controller-manager/endpoints.yaml | 22 + .../kube-controller-manager/service.yaml | 29 + .../servicemonitor.yaml | 50 + .../templates/exporters/kube-dns/service.yaml | 28 + .../exporters/kube-dns/servicemonitor.yaml | 52 + .../exporters/kube-etcd/endpoints.yaml | 20 + .../exporters/kube-etcd/service.yaml | 27 + .../exporters/kube-etcd/servicemonitor.yaml | 56 + .../exporters/kube-proxy/endpoints.yaml | 20 + .../exporters/kube-proxy/service.yaml | 27 + .../exporters/kube-proxy/servicemonitor.yaml | 44 + .../exporters/kube-scheduler/endpoints.yaml | 22 + .../exporters/kube-scheduler/service.yaml | 29 + .../kube-scheduler/servicemonitor.yaml | 50 + .../exporters/kubelet/servicemonitor.yaml | 216 + .../grafana/configmap-dashboards.yaml | 24 + .../grafana/configmaps-datasources.yaml | 63 + .../alertmanager-overview.yaml | 614 ++ .../grafana/dashboards-1.14/apiserver.yaml | 1772 ++++ .../dashboards-1.14/cluster-total.yaml | 1882 ++++ .../dashboards-1.14/controller-manager.yaml | 1190 +++ .../grafana/dashboards-1.14/etcd.yaml | 1227 +++ .../dashboards-1.14/grafana-overview.yaml | 636 ++ .../grafana/dashboards-1.14/k8s-coredns.yaml | 1531 +++ .../k8s-resources-cluster.yaml | 3088 ++++++ .../k8s-resources-namespace.yaml | 2797 ++++++ .../dashboards-1.14/k8s-resources-node.yaml | 1026 ++ .../dashboards-1.14/k8s-resources-pod.yaml | 2469 +++++ .../k8s-resources-workload.yaml | 2024 ++++ .../k8s-resources-workloads-namespace.yaml | 2189 ++++ .../grafana/dashboards-1.14/kubelet.yaml | 2254 +++++ .../dashboards-1.14/namespace-by-pod.yaml | 1464 +++ .../namespace-by-workload.yaml | 1736 ++++ .../node-cluster-rsrc-use.yaml | 1063 ++ .../dashboards-1.14/node-rsrc-use.yaml | 1089 ++ .../grafana/dashboards-1.14/nodes-darwin.yaml | 1073 ++ .../grafana/dashboards-1.14/nodes.yaml | 1066 ++ .../persistentvolumesusage.yaml | 587 ++ .../grafana/dashboards-1.14/pod-total.yaml | 1228 +++ .../prometheus-remote-write.yaml | 1670 ++++ .../grafana/dashboards-1.14/prometheus.yaml | 1235 +++ .../grafana/dashboards-1.14/proxy.yaml | 1271 +++ .../grafana/dashboards-1.14/scheduler.yaml | 1112 +++ .../dashboards-1.14/workload-total.yaml | 1438 +++ .../job-patch/clusterrole.yaml | 33 + .../job-patch/clusterrolebinding.yaml | 20 + .../job-patch/job-createSecret.yaml | 73 + .../job-patch/job-patchWebhook.yaml | 74 + .../job-patch/networkpolicy-createSecret.yaml | 29 + .../job-patch/networkpolicy-patchWebhook.yaml | 29 + .../admission-webhooks/job-patch/psp.yaml | 47 + .../admission-webhooks/job-patch/role.yaml | 21 + .../job-patch/rolebinding.yaml | 21 + .../job-patch/serviceaccount.yaml | 17 + .../mutatingWebhookConfiguration.yaml | 42 + .../validatingWebhookConfiguration.yaml | 42 + .../aggregate-clusterroles.yaml | 31 + .../prometheus-operator/certmanager.yaml | 57 + .../prometheus-operator/clusterrole.yaml | 81 + .../clusterrolebinding.yaml | 17 + .../prometheus-operator/deployment.yaml | 168 + .../prometheus-operator/networkpolicy.yaml | 26 + .../prometheus-operator/psp-clusterrole.yaml | 22 + .../psp-clusterrolebinding.yaml | 19 + .../templates/prometheus-operator/psp.yaml | 47 + .../prometheus-operator/service.yaml | 58 + .../prometheus-operator/serviceaccount.yaml | 16 + .../prometheus-operator/servicemonitor.yaml | 47 + .../verticalpodautoscaler.yaml | 35 + .../templates/prometheus/_rules.tpl | 36 + .../additionalAlertRelabelConfigs.yaml | 16 + .../additionalAlertmanagerConfigs.yaml | 16 + .../prometheus/additionalPrometheusRules.yaml | 43 + .../prometheus/additionalScrapeConfigs.yaml | 20 + .../templates/prometheus/clusterrole.yaml | 30 + .../prometheus/clusterrolebinding.yaml | 18 + .../templates/prometheus/csi-secret.yaml | 12 + .../templates/prometheus/extrasecret.yaml | 20 + .../templates/prometheus/ingress.yaml | 77 + .../prometheus/ingressThanosSidecar.yaml | 76 + .../prometheus/ingressperreplica.yaml | 67 + .../prometheus/podDisruptionBudget.yaml | 21 + .../templates/prometheus/podmonitors.yaml | 37 + .../templates/prometheus/prometheus.yaml | 394 + .../templates/prometheus/psp-clusterrole.yaml | 22 + .../prometheus/psp-clusterrolebinding.yaml | 19 + .../templates/prometheus/psp.yaml | 58 + .../rules-1.14/alertmanager.rules.yaml | 215 + .../rules-1.14/config-reloaders.yaml | 46 + .../templates/prometheus/rules-1.14/etcd.yaml | 294 + .../prometheus/rules-1.14/general.rules.yaml | 98 + .../prometheus/rules-1.14/k8s.rules.yaml | 173 + .../kube-apiserver-availability.rules.yaml | 136 + .../kube-apiserver-burnrate.rules.yaml | 328 + .../kube-apiserver-histogram.rules.yaml | 37 + .../rules-1.14/kube-apiserver-slos.yaml | 115 + .../kube-prometheus-general.rules.yaml | 31 + .../kube-prometheus-node-recording.rules.yaml | 39 + .../rules-1.14/kube-scheduler.rules.yaml | 63 + .../rules-1.14/kube-state-metrics.yaml | 107 + .../prometheus/rules-1.14/kubelet.rules.yaml | 39 + .../rules-1.14/kubernetes-apps.yaml | 375 + .../rules-1.14/kubernetes-resources.yaml | 193 + .../rules-1.14/kubernetes-storage.yaml | 161 + .../kubernetes-system-apiserver.yaml | 130 + .../kubernetes-system-controller-manager.yaml | 46 + .../kubernetes-system-kube-proxy.yaml | 44 + .../rules-1.14/kubernetes-system-kubelet.yaml | 253 + .../kubernetes-system-scheduler.yaml | 46 + .../rules-1.14/kubernetes-system.yaml | 65 + .../rules-1.14/node-exporter.rules.yaml | 89 + .../prometheus/rules-1.14/node-exporter.yaml | 398 + .../prometheus/rules-1.14/node-network.yaml | 44 + .../prometheus/rules-1.14/node.rules.yaml | 63 + .../rules-1.14/prometheus-operator.yaml | 148 + .../prometheus/rules-1.14/prometheus.yaml | 448 + .../templates/prometheus/service.yaml | 64 + .../prometheus/serviceThanosSidecar.yaml | 39 + .../serviceThanosSidecarExternal.yaml | 46 + .../templates/prometheus/serviceaccount.yaml | 20 + .../templates/prometheus/servicemonitor.yaml | 42 + .../servicemonitorThanosSidecar.yaml | 41 + .../templates/prometheus/servicemonitors.yaml | 38 + .../prometheus/serviceperreplica.yaml | 49 + .../templates/thanos-ruler/extrasecret.yaml | 20 + .../templates/thanos-ruler/ingress.yaml | 77 + .../thanos-ruler/podDisruptionBudget.yaml | 21 + .../templates/thanos-ruler/ruler.yaml | 173 + .../templates/thanos-ruler/service.yaml | 53 + .../thanos-ruler/serviceaccount.yaml | 20 + .../thanos-ruler/servicemonitor.yaml | 45 + .../charts/kube-prometheus-stack/values.yaml | 3462 +++++++ .../prometheus/kustomization.yaml | 46 + .../observability/prometheus/values.yaml | 10 + .../observability/promtail/kustomization.yaml | 10 + manifests/observability/promtail/values.yaml | 4 + .../system-tools/kustomization.yaml | 13 + .../daemon/config-map-env-toml.yaml | 23 +- .../testground/daemon/kustomization.yaml | 9 + .../daemon/tg-daemon-cluster-role.yaml | 2 - .../daemon/tg-daemon-deployment.yaml | 36 +- .../daemon/tg-daemon-service-account.yaml | 2 +- .../testground/daemon/tg-daemon-svc.yaml | 17 + manifests/testground/kustomization.yaml | 9 + .../observability/kustomization.yaml | 5 + manifests/testground/observability/rbac.yaml | 47 + .../testground/sidecar/kustomization.yaml | 5 + .../testground/sidecar/tg-ds-sidecar.yaml | 75 + manifests/testground/sync/kustomization.yaml | 6 + .../sync/tg-sync-service-deployment.yaml | 43 + .../testground/sync/tg-sync-service.yaml | 12 + .../influxdb/charts/influxdb/.helmignore | 21 + .../influxdb/charts/influxdb/Chart.lock | 6 + .../influxdb/charts/influxdb/Chart.yaml | 26 + .../influxdb/charts/influxdb/README.md | 615 ++ .../charts/influxdb/charts/common/.helmignore | 22 + .../charts/influxdb/charts/common/Chart.yaml | 22 + .../charts/influxdb/charts/common/README.md | 335 + .../charts/common/templates/_affinities.tpl | 102 + .../charts/common/templates/_capabilities.tpl | 128 + .../charts/common/templates/_errors.tpl | 23 + .../charts/common/templates/_images.tpl | 75 + .../charts/common/templates/_ingress.tpl | 55 + .../charts/common/templates/_labels.tpl | 18 + .../charts/common/templates/_names.tpl | 52 + .../charts/common/templates/_secrets.tpl | 129 + .../charts/common/templates/_storage.tpl | 23 + .../charts/common/templates/_tplvalues.tpl | 13 + .../charts/common/templates/_utils.tpl | 62 + .../charts/common/templates/_warnings.tpl | 14 + .../templates/validations/_cassandra.tpl | 72 + .../common/templates/validations/_mariadb.tpl | 103 + .../common/templates/validations/_mongodb.tpl | 108 + .../templates/validations/_postgresql.tpl | 129 + .../common/templates/validations/_redis.tpl | 76 + .../templates/validations/_validations.tpl | 46 + .../charts/influxdb/charts/common/values.yaml | 5 + .../charts/influxdb/files/conf/README.md | 6 + .../docker-entrypoint-initdb.d/README.md | 3 + .../charts/influxdb/templates/NOTES.txt | 201 + .../charts/influxdb/templates/_helpers.tpl | 148 + .../influxdb/templates/extradeploy.yaml | 4 + .../templates/influxdb/configmap-backup.yaml | 84 + .../influxdb/configmap-initdb-scripts.yaml | 17 + .../templates/influxdb/configmap.yaml | 19 + .../templates/influxdb/cronjob-backup.yaml | 192 + .../influxdb/deployment-standalone.yaml | 313 + .../templates/influxdb/pvc-backup.yaml | 23 + .../influxdb/templates/influxdb/pvc.yaml | 23 + .../templates/influxdb/secrets-backup.yaml | 45 + .../influxdb/templates/influxdb/secrets.yaml | 27 + .../templates/influxdb/service-headless.yaml | 30 + .../templates/influxdb/service-metrics.yaml | 48 + .../influxdb/templates/influxdb/service.yaml | 61 + .../templates/influxdb/servicemonitor.yaml | 37 + .../statefulset-high-availability.yaml | 325 + .../charts/influxdb/templates/ingress.yaml | 78 + .../influxdb/templates/networkpolicy.yaml | 37 + .../influxdb/templates/podsecuritypolicy.yaml | 41 + .../influxdb/templates/relay/configmap.yaml | 18 + .../influxdb/templates/relay/deployment.yaml | 105 + .../influxdb/templates/relay/service.yaml | 48 + .../charts/influxdb/templates/role.yaml | 21 + .../influxdb/templates/rolebinding.yaml | 20 + .../influxdb/templates/service-collectd.yaml | 48 + .../influxdb/templates/serviceaccount.yaml | 11 + .../influxdb/charts/influxdb/values.yaml | 1073 ++ manifests/tooling/influxdb/kustomization.yaml | 12 + manifests/tooling/influxdb/values.yaml | 19 + manifests/tooling/kustomization.yaml | 7 + manifests/tooling/olm/crds.yaml | 8343 ++++++++++++++++ manifests/tooling/olm/kustomization.yaml | 6 + manifests/tooling/olm/olm.yaml | 346 + .../operators/grafana-agent-operator.yaml | 9 + .../tooling/operators/kustomization.yaml | 6 + .../tooling/operators/operatorgroup.yaml | 7 + manifests/tooling/operators/subscription.yaml | 9 + .../tooling/redis/charts/redis/.helmignore | 21 + .../tooling/redis/charts/redis/Chart.lock | 6 + .../tooling/redis/charts/redis/Chart.yaml | 27 + .../tooling/redis/charts/redis/README.md | 951 ++ .../charts/redis/charts/common/.helmignore | 22 + .../charts/redis/charts/common/Chart.yaml | 24 + .../charts/redis/charts/common/README.md | 233 + .../charts/common/templates/_affinities.tpl | 106 + .../charts/common/templates/_capabilities.tpl | 154 + .../redis/charts/common/templates/_errors.tpl | 23 + .../redis/charts/common/templates/_images.tpl | 80 + .../charts/common/templates/_ingress.tpl | 68 + .../redis/charts/common/templates/_labels.tpl | 18 + .../redis/charts/common/templates/_names.tpl | 66 + .../charts/common/templates/_secrets.tpl | 165 + .../charts/common/templates/_storage.tpl | 23 + .../charts/common/templates/_tplvalues.tpl | 13 + .../redis/charts/common/templates/_utils.tpl | 62 + .../charts/common/templates/_warnings.tpl | 14 + .../templates/validations/_cassandra.tpl | 72 + .../common/templates/validations/_mariadb.tpl | 103 + .../common/templates/validations/_mongodb.tpl | 108 + .../common/templates/validations/_mysql.tpl | 103 + .../templates/validations/_postgresql.tpl | 129 + .../common/templates/validations/_redis.tpl | 76 + .../templates/validations/_validations.tpl | 46 + .../charts/redis/charts/common/values.yaml | 5 + .../redis/img/redis-cluster-topology.png | Bin 0 -> 11448 bytes .../redis/charts/redis/img/redis-topology.png | Bin 0 -> 9709 bytes .../redis/charts/redis/templates/NOTES.txt | 191 + .../redis/charts/redis/templates/_helpers.tpl | 323 + .../charts/redis/templates/configmap.yaml | 59 + .../charts/redis/templates/extra-list.yaml | 4 + .../charts/redis/templates/headless-svc.yaml | 33 + .../redis/templates/health-configmap.yaml | 192 + .../redis/templates/master/application.yaml | 524 + .../charts/redis/templates/master/psp.yaml | 46 + .../charts/redis/templates/master/pvc.yaml | 30 + .../redis/templates/master/service.yaml | 61 + .../templates/master/serviceaccount.yaml | 21 + .../charts/redis/templates/metrics-svc.yaml | 41 + .../charts/redis/templates/networkpolicy.yaml | 82 + .../redis/charts/redis/templates/pdb.yaml | 23 + .../redis/templates/prometheusrule.yaml | 21 + .../charts/redis/templates/replicas/hpa.yaml | 47 + .../redis/templates/replicas/service.yaml | 58 + .../templates/replicas/serviceaccount.yaml | 21 + .../redis/templates/replicas/statefulset.yaml | 521 + .../redis/charts/redis/templates/role.yaml | 28 + .../charts/redis/templates/rolebinding.yaml | 21 + .../redis/templates/scripts-configmap.yaml | 758 ++ .../redis/charts/redis/templates/secret.yaml | 56 + .../charts/redis/templates/sentinel/hpa.yaml | 47 + .../templates/sentinel/node-services.yaml | 70 + .../templates/sentinel/ports-configmap.yaml | 100 + .../redis/templates/sentinel/service.yaml | 103 + .../redis/templates/sentinel/statefulset.yaml | 786 ++ .../redis/templates/serviceaccount.yaml | 21 + .../redis/templates/servicemonitor.yaml | 44 + .../charts/redis/templates/tls-secret.yaml | 29 + .../redis/charts/redis/values.schema.json | 156 + .../tooling/redis/charts/redis/values.yaml | 1782 ++++ manifests/tooling/redis/kustomization.yaml | 12 + manifests/tooling/redis/values.yaml | 6 + .../tooling/telegraf-operator/argocd_app.yaml | 41 + .../charts/telegraf-operator/.helmignore | 23 + .../charts/telegraf-operator/Chart.yaml | 21 + .../charts/telegraf-operator/README.md | 85 + .../telegraf-operator/templates/_helpers.tpl | 116 + .../templates/certificate.yml | 18 + .../templates/clusterrole.yml | 23 + .../templates/clusterrolebinding.yml | 14 + .../templates/deployment.yaml | 86 + .../telegraf-operator/templates/issuer.yml | 11 + .../mutatingwebhookconfiguration.yml | 31 + .../telegraf-operator/templates/pdb.yml | 16 + .../templates/secret-classes.yml | 8 + .../telegraf-operator/templates/service.yaml | 16 + .../templates/serviceaccount.yaml | 13 + .../telegraf-operator/templates/tls.yml | 3 + .../telegraf-operator/tests/influxdb.yml | 42 + .../charts/telegraf-operator/tests/redis.yml | 25 + .../charts/telegraf-operator/values.yaml | 58 + .../telegraf-operator/kustomization.yaml | 13 + .../tooling/telegraf-operator/values.yaml | 19 + terraform/.gitignore | 2 + terraform/README.md | 3 + terraform/argocd-values.yaml | 9 + terraform/backend.tf | 7 + terraform/ebs.tf | 22 + terraform/efs.tf | 54 + terraform/kops-resources/backend.tf | 7 + terraform/kops-resources/ebs.tf | 20 + terraform/kops-resources/efs.tf | 50 + terraform/kops-resources/main.tf | 34 + terraform/kops-resources/outputs.tf | 34 + terraform/kops-resources/providers.tf | 13 + terraform/main.tf | 332 + terraform/outputs.tf | 434 + terraform/providers.tf | 25 + 567 files changed, 119766 insertions(+), 24567 deletions(-) create mode 100644 .editorconfig create mode 100644 argocd-root/kustomization.yaml create mode 100644 argocd-root/root.yaml create mode 100644 argocd/aws-applicationset.yaml create mode 100644 argocd/kustomization.yaml create mode 100644 argocd/networking-applicationset.yaml create mode 100644 argocd/observability-applicationset.yaml create mode 100644 argocd/telegraf-operator.yaml create mode 100644 argocd/testground-applicationset.yaml create mode 100644 argocd/testground-tools-applicationset.yaml create mode 100644 docs/argocd_dashboard.png create mode 100644 docs/argocd_root_app.png create mode 100644 k8s/eks/yaml/otel/configmap-collector.yml create mode 100644 k8s/eks/yaml/otel/deployment.yml create mode 100644 k8s/eks/yaml/otel/service.yml create mode 100644 k8s/eks/yaml/telegraf-operator/values.yaml delete mode 100755 k8s/kops/01_install_k8s.sh delete mode 100755 k8s/kops/02_efs.sh delete mode 100755 k8s/kops/03_ebs.sh delete mode 100755 k8s/kops/04_testground_daemon.sh create mode 100644 k8s/kops/argocd/values.yaml delete mode 100755 k8s/kops/delete_ebs.sh delete mode 100755 k8s/kops/delete_efs.sh delete mode 100644 k8s/kops/ebs-terraform/backend.tf delete mode 100644 k8s/kops/ebs-terraform/ebs.tf delete mode 100644 k8s/kops/ebs-terraform/outputs.tf delete mode 100644 k8s/kops/ebs-terraform/variables.tf delete mode 100644 k8s/kops/ebs/pv.yml.spec delete mode 100644 k8s/kops/ebs/storageclass.yml delete mode 100644 k8s/kops/efs-terraform/backend.tf delete mode 100644 k8s/kops/efs-terraform/efs.tf delete mode 100644 k8s/kops/efs-terraform/outputs.tf delete mode 100644 k8s/kops/efs-terraform/variables.tf delete mode 100644 k8s/kops/efs/manifest.yaml.spec delete mode 100644 k8s/kops/efs/rbac.yaml delete mode 100644 k8s/kops/influxdb/values.yaml create mode 100755 k8s/kops/install_k8s.sh delete mode 100644 k8s/kops/kops-weave/dummy.yml delete mode 100644 k8s/kops/kops-weave/flannel.yml delete mode 100644 k8s/kops/kops-weave/genie-plugin.yaml delete mode 100644 k8s/kops/kops-weave/weave-metrics-service.yml delete mode 100644 k8s/kops/kops-weave/weave-service-monitor.yml delete mode 100644 k8s/kops/kops-weave/weave.yml delete mode 100644 k8s/kops/limit-range/limit-range.yaml delete mode 100644 k8s/kops/prometheus-operator/values.yaml delete mode 100644 k8s/kops/sidecar.yaml delete mode 100644 k8s/kops/testground-daemon/service.yml delete mode 100644 k8s/kops/testground-infra/Chart.yaml delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/Chart.yaml delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/README.md delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/coredns.json delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-api-server.json delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-kubelet.json delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-networking-cluster.json delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-networking-namespace-pods.json delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-networking-pod.json delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-stateful-sets.json delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/nodes.json delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/prometheus.json delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/redis.json delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/use-method-cluster.json delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/use-method-node.json delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/weavenet-cluster.json delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/weavenet.json delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/datasources/prometheus.json delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/dashboards/main.go delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/templates/dashboard.yaml delete mode 100644 k8s/kops/testground-infra/charts/testground-dashboards/templates/datasource.yaml delete mode 100644 k8s/kops/testground-infra/requirements.lock delete mode 100644 k8s/kops/testground-infra/requirements.yaml delete mode 100644 k8s/kops/testground-infra/values.yaml delete mode 100644 k8s/kops/tools/README.md delete mode 100755 k8s/kops/tools/authorize-19999.sh delete mode 100755 k8s/kops/tools/confirm_dns_route.sh delete mode 100755 k8s/kops/tools/redeploy-daemon.sh create mode 100644 manifests/aws/ebs/kustomization.yaml rename k8s/kops/ebs/pvc.yml => manifests/aws/ebs/pvc.yaml (100%) create mode 100644 manifests/aws/efs/charts/aws-efs-csi-driver/.helmignore create mode 100644 manifests/aws/efs/charts/aws-efs-csi-driver/CHANGELOG.md create mode 100644 manifests/aws/efs/charts/aws-efs-csi-driver/Chart.yaml create mode 100644 manifests/aws/efs/charts/aws-efs-csi-driver/templates/NOTES.txt create mode 100644 manifests/aws/efs/charts/aws-efs-csi-driver/templates/_helpers.tpl create mode 100644 manifests/aws/efs/charts/aws-efs-csi-driver/templates/controller-deployment.yaml create mode 100644 manifests/aws/efs/charts/aws-efs-csi-driver/templates/controller-serviceaccount.yaml create mode 100644 manifests/aws/efs/charts/aws-efs-csi-driver/templates/csidriver.yaml create mode 100644 manifests/aws/efs/charts/aws-efs-csi-driver/templates/node-daemonset.yaml create mode 100644 manifests/aws/efs/charts/aws-efs-csi-driver/templates/node-serviceaccount.yaml create mode 100644 manifests/aws/efs/charts/aws-efs-csi-driver/templates/storageclass.yaml create mode 100644 manifests/aws/efs/charts/aws-efs-csi-driver/values.yaml create mode 100644 manifests/aws/efs/kustomization.yaml create mode 100644 manifests/aws/efs/pvc.yaml create mode 100644 manifests/aws/efs/rbac.yaml create mode 100644 manifests/aws/efs/values.yaml create mode 100644 manifests/aws/kustomization.yaml create mode 100644 manifests/kustomization.yaml create mode 100644 manifests/networking/kustomization.yaml create mode 100644 manifests/networking/multus/kustomization.yaml create mode 100644 manifests/networking/multus/multus-daemonset.yaml create mode 100644 manifests/networking/multus/softlink-cm.yaml create mode 100644 manifests/networking/multus/softlink-ds.yaml create mode 100644 manifests/networking/weave/clusterrolebinding.yaml create mode 100644 manifests/networking/weave/drop-weave-cm.yaml create mode 100644 manifests/networking/weave/drop-weave-ds.yaml create mode 100644 manifests/networking/weave/kustomization.yaml create mode 100644 manifests/networking/weave/networkattachmentdefinition.yaml create mode 100644 manifests/networking/weave/weave.yaml create mode 100644 manifests/observability/grafana-agent-operator/kustomization.yaml create mode 100644 manifests/observability/grafana/base/grafana.yaml create mode 100644 manifests/observability/grafana/base/kustomization.yaml create mode 100644 manifests/observability/grafana/base/loki_datasource.yaml create mode 100644 manifests/observability/grafana/base/monitoring/README.md create mode 100755 manifests/observability/grafana/base/monitoring/dashboards.sh create mode 100644 manifests/observability/grafana/base/monitoring/dashboards.yaml create mode 100644 manifests/observability/grafana/base/monitoring/dashboards.yaml.tmpl create mode 100644 manifests/observability/grafana/base/monitoring/kustomization.yaml create mode 100644 manifests/observability/grafana/base/monitoring/kustomizeconfig.yaml create mode 100644 manifests/observability/grafana/base/prometheus_datasource.yaml create mode 100644 manifests/observability/grafana/dashboards.yaml create mode 100644 manifests/observability/grafana/dashboards/base-logs.json create mode 100644 manifests/observability/grafana/dashboards/base-monitoring.json create mode 100644 manifests/observability/grafana/kustomization.yaml create mode 100644 manifests/observability/kustomization.yaml create mode 100644 manifests/observability/loki/kustomization.yaml create mode 100644 manifests/observability/loki/values.yaml create mode 100644 manifests/observability/otel/configmap-collector.yaml create mode 100644 manifests/observability/otel/deployment.yaml create mode 100644 manifests/observability/otel/kustomization.yaml create mode 100644 manifests/observability/otel/service.yaml create mode 100644 manifests/observability/prometheus-crds/kustomization.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/.helmignore create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/CONTRIBUTING.md create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/Chart.lock create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/Chart.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/README.md create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/.helmignore create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/Chart.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/README.md create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/default-values.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-affinity-values.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-dashboard-json-values.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-dashboard-values.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-extraconfigmapmounts-values.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-image-renderer-values.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-persistence.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/dashboards/custom-dashboard.json create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/NOTES.txt create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/_helpers.tpl create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/_pod.tpl create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/clusterrole.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/clusterrolebinding.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/configmap-dashboard-provider.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/configmap.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/dashboards-json-configmap.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/deployment.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/extra-manifests.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/headless-service.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/hpa.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/image-renderer-deployment.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/image-renderer-network-policy.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/image-renderer-service.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/ingress.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/networkpolicy.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/poddisruptionbudget.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/podsecuritypolicy.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/pvc.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/role.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/rolebinding.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/secret-env.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/secret.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/service.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/serviceaccount.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/servicemonitor.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/statefulset.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-configmap.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-podsecuritypolicy.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-role.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-rolebinding.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-serviceaccount.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/values.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/.helmignore create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/Chart.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/README.md create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/NOTES.txt create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/_helpers.tpl create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/clusterrolebinding.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/deployment.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/kubeconfig-secret.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/pdb.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/podsecuritypolicy.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/psp-clusterrole.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/psp-clusterrolebinding.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/role.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/rolebinding.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/service.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/serviceaccount.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/servicemonitor.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/stsdiscovery-role.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/stsdiscovery-rolebinding.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/verticalpodautoscaler.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/values.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/.helmignore create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/Chart.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/README.md create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/ci/port-values.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/NOTES.txt create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/_helpers.tpl create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/daemonset.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/endpoints.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/psp-clusterrole.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/psp-clusterrolebinding.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/psp.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/service.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/serviceaccount.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/servicemonitor.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/verticalpodautoscaler.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/values.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-alertmanagerconfigs.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-alertmanagers.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-podmonitors.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-probes.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-prometheuses.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-prometheusrules.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-servicemonitors.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-thanosrulers.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/NOTES.txt create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/_helpers.tpl create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/alertmanager/alertmanager.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/alertmanager/extrasecret.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/alertmanager/ingress.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/alertmanager/ingressperreplica.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/alertmanager/podDisruptionBudget.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/alertmanager/psp-role.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/alertmanager/psp-rolebinding.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/alertmanager/psp.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/alertmanager/secret.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/alertmanager/service.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/alertmanager/serviceaccount.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/alertmanager/servicemonitor.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/alertmanager/serviceperreplica.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/core-dns/service.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/core-dns/servicemonitor.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/kube-api-server/servicemonitor.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/kube-controller-manager/endpoints.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/kube-controller-manager/service.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/kube-controller-manager/servicemonitor.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/kube-dns/service.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/kube-dns/servicemonitor.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/kube-etcd/endpoints.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/kube-etcd/service.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/kube-etcd/servicemonitor.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/kube-proxy/endpoints.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/kube-proxy/service.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/kube-proxy/servicemonitor.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/kube-scheduler/endpoints.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/kube-scheduler/service.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/kube-scheduler/servicemonitor.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/exporters/kubelet/servicemonitor.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/configmap-dashboards.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/configmaps-datasources.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/alertmanager-overview.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/apiserver.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/cluster-total.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/controller-manager.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/etcd.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/grafana-overview.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/k8s-coredns.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/k8s-resources-cluster.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/k8s-resources-namespace.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/k8s-resources-node.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/k8s-resources-pod.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/k8s-resources-workload.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/k8s-resources-workloads-namespace.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/kubelet.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/namespace-by-pod.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/namespace-by-workload.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/node-cluster-rsrc-use.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/node-rsrc-use.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/nodes-darwin.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/nodes.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/persistentvolumesusage.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/pod-total.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/prometheus-remote-write.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/prometheus.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/proxy.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/scheduler.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/workload-total.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/clusterrole.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/clusterrolebinding.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/job-createSecret.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/job-patchWebhook.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/networkpolicy-createSecret.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/networkpolicy-patchWebhook.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/psp.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/role.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/rolebinding.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/serviceaccount.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/mutatingWebhookConfiguration.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/validatingWebhookConfiguration.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/aggregate-clusterroles.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/certmanager.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/clusterrole.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/clusterrolebinding.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/deployment.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/networkpolicy.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/psp-clusterrole.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/psp-clusterrolebinding.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/psp.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/service.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/serviceaccount.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/servicemonitor.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/verticalpodautoscaler.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/_rules.tpl create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/additionalAlertRelabelConfigs.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/additionalAlertmanagerConfigs.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/additionalPrometheusRules.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/additionalScrapeConfigs.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/clusterrole.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/clusterrolebinding.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/csi-secret.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/extrasecret.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/ingress.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/ingressThanosSidecar.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/ingressperreplica.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/podDisruptionBudget.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/podmonitors.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/prometheus.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/psp-clusterrole.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/psp-clusterrolebinding.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/psp.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/alertmanager.rules.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/config-reloaders.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/etcd.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/general.rules.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/k8s.rules.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kube-apiserver-availability.rules.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kube-apiserver-burnrate.rules.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kube-apiserver-histogram.rules.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kube-apiserver-slos.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kube-prometheus-general.rules.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kube-prometheus-node-recording.rules.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kube-scheduler.rules.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kube-state-metrics.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubelet.rules.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-apps.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-resources.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-storage.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-system-apiserver.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-system-controller-manager.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-system-kube-proxy.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-system-kubelet.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-system-scheduler.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-system.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/node-exporter.rules.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/node-exporter.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/node-network.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/node.rules.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/prometheus-operator.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/prometheus.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/service.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/serviceThanosSidecar.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/serviceThanosSidecarExternal.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/serviceaccount.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/servicemonitor.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/servicemonitorThanosSidecar.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/servicemonitors.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/prometheus/serviceperreplica.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/thanos-ruler/extrasecret.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/thanos-ruler/ingress.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/thanos-ruler/podDisruptionBudget.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/thanos-ruler/ruler.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/thanos-ruler/service.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/thanos-ruler/serviceaccount.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/templates/thanos-ruler/servicemonitor.yaml create mode 100644 manifests/observability/prometheus/charts/kube-prometheus-stack/values.yaml create mode 100644 manifests/observability/prometheus/kustomization.yaml create mode 100644 manifests/observability/prometheus/values.yaml create mode 100644 manifests/observability/promtail/kustomization.yaml create mode 100644 manifests/observability/promtail/values.yaml create mode 100644 manifests/observability/system-tools/kustomization.yaml rename k8s/kops/testground-daemon/config-map-env-toml.yml => manifests/testground/daemon/config-map-env-toml.yaml (71%) create mode 100644 manifests/testground/daemon/kustomization.yaml rename k8s/kops/testground-daemon/role-binding.yml => manifests/testground/daemon/tg-daemon-cluster-role.yaml (91%) rename k8s/kops/testground-daemon/deployment.yml => manifests/testground/daemon/tg-daemon-deployment.yaml (73%) rename k8s/kops/testground-daemon/service-account.yml => manifests/testground/daemon/tg-daemon-service-account.yaml (63%) create mode 100644 manifests/testground/daemon/tg-daemon-svc.yaml create mode 100644 manifests/testground/kustomization.yaml create mode 100644 manifests/testground/observability/kustomization.yaml create mode 100644 manifests/testground/observability/rbac.yaml create mode 100644 manifests/testground/sidecar/kustomization.yaml create mode 100644 manifests/testground/sidecar/tg-ds-sidecar.yaml create mode 100644 manifests/testground/sync/kustomization.yaml create mode 100644 manifests/testground/sync/tg-sync-service-deployment.yaml create mode 100644 manifests/testground/sync/tg-sync-service.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/.helmignore create mode 100644 manifests/tooling/influxdb/charts/influxdb/Chart.lock create mode 100644 manifests/tooling/influxdb/charts/influxdb/Chart.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/README.md create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/.helmignore create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/Chart.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/README.md create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/_affinities.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/_capabilities.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/_errors.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/_images.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/_ingress.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/_labels.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/_names.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/_secrets.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/_storage.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/_tplvalues.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/_utils.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/_warnings.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/validations/_cassandra.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/validations/_mariadb.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/validations/_mongodb.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/validations/_postgresql.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/validations/_redis.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/templates/validations/_validations.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/charts/common/values.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/files/conf/README.md create mode 100644 manifests/tooling/influxdb/charts/influxdb/files/docker-entrypoint-initdb.d/README.md create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/NOTES.txt create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/_helpers.tpl create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/extradeploy.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/influxdb/configmap-backup.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/influxdb/configmap-initdb-scripts.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/influxdb/configmap.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/influxdb/cronjob-backup.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/influxdb/deployment-standalone.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/influxdb/pvc-backup.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/influxdb/pvc.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/influxdb/secrets-backup.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/influxdb/secrets.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/influxdb/service-headless.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/influxdb/service-metrics.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/influxdb/service.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/influxdb/servicemonitor.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/influxdb/statefulset-high-availability.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/ingress.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/networkpolicy.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/podsecuritypolicy.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/relay/configmap.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/relay/deployment.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/relay/service.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/role.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/rolebinding.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/service-collectd.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/templates/serviceaccount.yaml create mode 100644 manifests/tooling/influxdb/charts/influxdb/values.yaml create mode 100644 manifests/tooling/influxdb/kustomization.yaml create mode 100644 manifests/tooling/influxdb/values.yaml create mode 100644 manifests/tooling/kustomization.yaml create mode 100644 manifests/tooling/olm/crds.yaml create mode 100644 manifests/tooling/olm/kustomization.yaml create mode 100644 manifests/tooling/olm/olm.yaml create mode 100644 manifests/tooling/operators/grafana-agent-operator.yaml create mode 100644 manifests/tooling/operators/kustomization.yaml create mode 100644 manifests/tooling/operators/operatorgroup.yaml create mode 100644 manifests/tooling/operators/subscription.yaml create mode 100644 manifests/tooling/redis/charts/redis/.helmignore create mode 100644 manifests/tooling/redis/charts/redis/Chart.lock create mode 100644 manifests/tooling/redis/charts/redis/Chart.yaml create mode 100644 manifests/tooling/redis/charts/redis/README.md create mode 100644 manifests/tooling/redis/charts/redis/charts/common/.helmignore create mode 100644 manifests/tooling/redis/charts/redis/charts/common/Chart.yaml create mode 100644 manifests/tooling/redis/charts/redis/charts/common/README.md create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/_affinities.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/_capabilities.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/_errors.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/_images.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/_ingress.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/_labels.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/_names.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/_secrets.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/_storage.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/_tplvalues.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/_utils.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/_warnings.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/validations/_cassandra.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/validations/_mariadb.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/validations/_mongodb.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/validations/_mysql.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/validations/_postgresql.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/validations/_redis.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/templates/validations/_validations.tpl create mode 100644 manifests/tooling/redis/charts/redis/charts/common/values.yaml create mode 100644 manifests/tooling/redis/charts/redis/img/redis-cluster-topology.png create mode 100644 manifests/tooling/redis/charts/redis/img/redis-topology.png create mode 100644 manifests/tooling/redis/charts/redis/templates/NOTES.txt create mode 100644 manifests/tooling/redis/charts/redis/templates/_helpers.tpl create mode 100644 manifests/tooling/redis/charts/redis/templates/configmap.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/extra-list.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/headless-svc.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/health-configmap.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/master/application.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/master/psp.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/master/pvc.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/master/service.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/master/serviceaccount.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/metrics-svc.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/networkpolicy.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/pdb.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/prometheusrule.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/replicas/hpa.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/replicas/service.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/replicas/serviceaccount.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/replicas/statefulset.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/role.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/rolebinding.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/scripts-configmap.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/secret.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/sentinel/hpa.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/sentinel/node-services.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/sentinel/ports-configmap.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/sentinel/service.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/sentinel/statefulset.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/serviceaccount.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/servicemonitor.yaml create mode 100644 manifests/tooling/redis/charts/redis/templates/tls-secret.yaml create mode 100644 manifests/tooling/redis/charts/redis/values.schema.json create mode 100644 manifests/tooling/redis/charts/redis/values.yaml create mode 100644 manifests/tooling/redis/kustomization.yaml create mode 100644 manifests/tooling/redis/values.yaml create mode 100644 manifests/tooling/telegraf-operator/argocd_app.yaml create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/.helmignore create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/Chart.yaml create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/README.md create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/templates/_helpers.tpl create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/templates/certificate.yml create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/templates/clusterrole.yml create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/templates/clusterrolebinding.yml create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/templates/deployment.yaml create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/templates/issuer.yml create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/templates/mutatingwebhookconfiguration.yml create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/templates/pdb.yml create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/templates/secret-classes.yml create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/templates/service.yaml create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/templates/serviceaccount.yaml create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/templates/tls.yml create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/tests/influxdb.yml create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/tests/redis.yml create mode 100644 manifests/tooling/telegraf-operator/charts/telegraf-operator/values.yaml create mode 100644 manifests/tooling/telegraf-operator/kustomization.yaml create mode 100644 manifests/tooling/telegraf-operator/values.yaml create mode 100644 terraform/.gitignore create mode 100644 terraform/README.md create mode 100644 terraform/argocd-values.yaml create mode 100644 terraform/backend.tf create mode 100644 terraform/ebs.tf create mode 100644 terraform/efs.tf create mode 100644 terraform/kops-resources/backend.tf create mode 100644 terraform/kops-resources/ebs.tf create mode 100644 terraform/kops-resources/efs.tf create mode 100644 terraform/kops-resources/main.tf create mode 100644 terraform/kops-resources/outputs.tf create mode 100644 terraform/kops-resources/providers.tf create mode 100644 terraform/main.tf create mode 100644 terraform/outputs.tf create mode 100644 terraform/providers.tf diff --git a/.editorconfig b/.editorconfig new file mode 100644 index 00000000..a02fff6a --- /dev/null +++ b/.editorconfig @@ -0,0 +1,12 @@ +root = true + +[*] +charset = utf-8 +end_of_line = lf +indent_size = 4 +indent_style = space +insert_final_newline = true +trim_trailing_whitespace = true + +[*.{yaml,yml,tf}] +indent_size = 2 diff --git a/argocd-root/kustomization.yaml b/argocd-root/kustomization.yaml new file mode 100644 index 00000000..9b004d8e --- /dev/null +++ b/argocd-root/kustomization.yaml @@ -0,0 +1,5 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: +- root.yaml diff --git a/argocd-root/root.yaml b/argocd-root/root.yaml new file mode 100644 index 00000000..4f50c4ee --- /dev/null +++ b/argocd-root/root.yaml @@ -0,0 +1,22 @@ +# We generate this ArgoCD application with Terraform, but we keep it here as a workaround +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: root +spec: + project: default + source: + repoURL: https://github.com/testground/infra.git + path: 'argocd' + targetRevision: v0.7.0 + destination: + name: in-cluster + namespace: argocd + syncPolicy: + automated: + prune: true + allowEmpty: true + selfHeal: true + syncOptions: + - ApplyOutOfSyncOnly=true + - CreateNamespace=true diff --git a/argocd/aws-applicationset.yaml b/argocd/aws-applicationset.yaml new file mode 100644 index 00000000..434f450a --- /dev/null +++ b/argocd/aws-applicationset.yaml @@ -0,0 +1,34 @@ +apiVersion: argoproj.io/v1alpha1 +kind: ApplicationSet +metadata: + name: aws + namespace: argocd +spec: + generators: + - git: + repoURL: https://github.com/testground/infra.git + revision: v0.7.0 + directories: + - path: 'manifests/aws/*' + - path: manifests/aws/deactivated + exclude: true + template: + metadata: + name: 'aws-{{path[2]}}' + spec: + project: default + source: + repoURL: https://github.com/testground/infra.git + path: 'manifests/aws/{{path[2]}}' + targetRevision: v0.7.0 + destination: + name: in-cluster + namespace: default + syncPolicy: + automated: + prune: true + allowEmpty: true + selfHeal: true + syncOptions: + - ApplyOutOfSyncOnly=true + - CreateNamespace=true diff --git a/argocd/kustomization.yaml b/argocd/kustomization.yaml new file mode 100644 index 00000000..2734fac4 --- /dev/null +++ b/argocd/kustomization.yaml @@ -0,0 +1,11 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: argocd + +resources: +- aws-applicationset.yaml +- observability-applicationset.yaml +- testground-applicationset.yaml +- networking-applicationset.yaml +- testground-tools-applicationset.yaml diff --git a/argocd/networking-applicationset.yaml b/argocd/networking-applicationset.yaml new file mode 100644 index 00000000..c9ca4e85 --- /dev/null +++ b/argocd/networking-applicationset.yaml @@ -0,0 +1,35 @@ +apiVersion: argoproj.io/v1alpha1 +kind: ApplicationSet +metadata: + name: networking + namespace: argocd +spec: + generators: + - git: + repoURL: https://github.com/testground/infra.git + revision: v0.7.0 + directories: + - path: 'manifests/networking/*' + - path: manifests/networking/deactivated + exclude: true + template: + metadata: + name: 'networking-{{path[2]}}' + spec: + project: default + source: + repoURL: https://github.com/testground/infra.git + path: 'manifests/networking/{{path[2]}}' + targetRevision: v0.7.0 + destination: + name: in-cluster + namespace: default + syncPolicy: + automated: + prune: true + allowEmpty: true + selfHeal: true + syncOptions: + - ApplyOutOfSyncOnly=true + - CreateNamespace=true + - ServerSideApply=true diff --git a/argocd/observability-applicationset.yaml b/argocd/observability-applicationset.yaml new file mode 100644 index 00000000..3bb55f46 --- /dev/null +++ b/argocd/observability-applicationset.yaml @@ -0,0 +1,35 @@ +apiVersion: argoproj.io/v1alpha1 +kind: ApplicationSet +metadata: + name: observability + namespace: argocd +spec: + generators: + - git: + repoURL: https://github.com/testground/infra.git + revision: v0.7.0 + directories: + - path: 'manifests/observability/*' + - path: manifests/observability/deactivated + exclude: true + template: + metadata: + name: 'observability-{{path[2]}}' + spec: + project: default + source: + repoURL: https://github.com/testground/infra.git + path: 'manifests/observability/{{path[2]}}' + targetRevision: v0.7.0 + destination: + name: in-cluster + namespace: default + syncPolicy: + automated: + prune: true + allowEmpty: true + selfHeal: true + syncOptions: + - ApplyOutOfSyncOnly=true + - CreateNamespace=true + - ServerSideApply=true diff --git a/argocd/telegraf-operator.yaml b/argocd/telegraf-operator.yaml new file mode 100644 index 00000000..900165b0 --- /dev/null +++ b/argocd/telegraf-operator.yaml @@ -0,0 +1,41 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: testground-tools-telegraf-operator + namespace: argocd +spec: + project: default + source: + chart: telegraf-operator + repoURL: https://helm.influxdata.com/ + targetRevision: 1.3.11 + helm: + releaseName: telegraf-operator + values: | + replicaCount: 2 + classes: + data: + default: | + [[outputs.influxdb]] + urls = ["http://influxdb.default.svc:8086"] + database = "testground" + resources: + limits: + cpu: 400m + memory: 256Mi + requests: + cpu: 50m + memory: 64Mi + hotReload: true + destination: + namespace: default + name: in-cluster + syncPolicy: + automated: + prune: true + selfHeal: true + allowEmpty: true + syncOptions: + - ApplyOutOfSyncOnly=true + - CreateNamespace=true + - ServerSideApply=true diff --git a/argocd/testground-applicationset.yaml b/argocd/testground-applicationset.yaml new file mode 100644 index 00000000..41048622 --- /dev/null +++ b/argocd/testground-applicationset.yaml @@ -0,0 +1,35 @@ +apiVersion: argoproj.io/v1alpha1 +kind: ApplicationSet +metadata: + name: testground + namespace: argocd +spec: + generators: + - git: + repoURL: https://github.com/testground/infra.git + revision: v0.7.0 + directories: + - path: 'manifests/testground/*' + - path: manifests/testground/deactivated + exclude: true + template: + metadata: + name: 'testground-{{path[2]}}' + spec: + project: default + source: + repoURL: https://github.com/testground/infra.git + path: 'manifests/testground/{{path[2]}}' + targetRevision: v0.7.0 + destination: + name: in-cluster + namespace: default + syncPolicy: + automated: + prune: true + allowEmpty: true + selfHeal: true + syncOptions: + - ApplyOutOfSyncOnly=true + - CreateNamespace=true + - ServerSideApply=true diff --git a/argocd/testground-tools-applicationset.yaml b/argocd/testground-tools-applicationset.yaml new file mode 100644 index 00000000..06c57537 --- /dev/null +++ b/argocd/testground-tools-applicationset.yaml @@ -0,0 +1,35 @@ +apiVersion: argoproj.io/v1alpha1 +kind: ApplicationSet +metadata: + name: testground-tools + namespace: argocd +spec: + generators: + - git: + repoURL: https://github.com/testground/infra.git + revision: v0.7.0 + directories: + - path: 'manifests/tooling/*' + - path: manifests/tooling/deactivated + exclude: true + template: + metadata: + name: 'testground-tools-{{path[2]}}' + spec: + project: default + source: + repoURL: https://github.com/testground/infra.git + path: 'manifests/tooling/{{path[2]}}' + targetRevision: v0.7.0 + destination: + name: in-cluster + namespace: default + syncPolicy: + automated: + prune: true + allowEmpty: true + selfHeal: true + syncOptions: + - ApplyOutOfSyncOnly=true + - CreateNamespace=true + - ServerSideApply=true diff --git a/docs/argocd_dashboard.png b/docs/argocd_dashboard.png new file mode 100644 index 0000000000000000000000000000000000000000..6df872316f15a1f3f8c2e8d81c3856823c021ec5 GIT binary patch literal 332375 zcma%j2UJtb)-b&&ML|${QBXjNROuipy$I5K3nhf!TPPwTy>}21P${8zLQ{|uia-Jc z2uLSDKnN|ApZmV=mGyt`y?e9PS!Xgcd+#%|XJ*%PVsx~WDQ_^|z{A6%R8@Jdi-$)V zhKEO3O-6EEgUWRB$HT+7a8gpzQB_i6*YR`%IJwy4;XO*WwX*u9^7zpYYild3A22?i z8=ii;@7~4fT7gHp2D|Xzv{J0a416a}z{i(U;Gz&>qzi8A>mvIoh>jTMO@OHib=0*S zCg~n`$7E`EbiYm{Xx7z@O$|R}V;5Huc*J{+bP~Uoa8Uh*!Zstj=_j&{N9;ia@A9dv z1@UWA*|%Afd36pk3!P1z;yKOz1BCA337;D{-^U*qMamRjVB*HZ9MOi=Wm9o4NN zAtE7vp%bA;Lie6zi5Lkr6L_#wVY#iaBNV3;#1!ciFDSApHYpe=xD+}P6p{(vGQWMz zj!fl9`BuM1x2~ny?aEmw``mt_wIdb z6cS>-!>96C?M*cAvP8x&QQP}uxk1u|gy{OKEBR;gvk8ud6FK zHhif|0*^}Ao^~+TPUL#R4eX6n0h*e4yw_ziJp5=UJfiCo{`JLpeO>b<mtbB+nU`U;C^B`8%HH_FlH0PVU}LZm#Tq;I+1K^YNDD*=|sAtxy+ zD)YDg|EJ}j6#oU%=%0{JB!z|k74%=4{(n$?FMCfVH_$arZ@GW6=6`|zwef!eWd#2i z`oB!Y|IG8>YOg0+?uLxuKbR(W<2f^_6&{`fp6YYO*Z%n1^Q2b#t23l~;G>1q2e+P0 zGd(|A5V-TgthwCB6o__cXlT?20!`7MX^#!&-cgYLSO}suAR$lcI6jtfdzzW+o;vrY z`pT_yr66b|XyKp$)e#dMuzS!hKrMgnh~1rU>wbxULnn;j&Idf=M+*O?=uoBb>>0tF zA!sIsGPg0W#jiWN@Q8pxl}Q^X=^62-;CG%*5PvN%U)u64MG>!S{})34g38^)Bb<-r zzdqJKdM^&&6=LXexCHG{{WX9vhOS`hzcTVWPt}RC@PjkjliDf&(o|949mxK-sr=_N zKfdkm;}ZqM8heWTMcnJHVdXam5K%`re%*53Fl-2(X*xRscD9CKWp><9 zhXGE;Ef)gF#tRwm$6ox|rX|6~h^Y~u{>cz7pFVHV!q#=V%nV89q1Hx=CMZyv5j6~R zts1e=9`$N$kwzw8JaVrkQd3ud_39P5ygZ8A{CAtN*OS0W_dE3Toa|j}78VwoQk|ba z-}}zRMRt?Y>1V*U#-A+05dc1AAJqvK<-Yul!+-HB`I-UPxRd4h;+T*}Vw-@P+Rq!v z6xr|ALb241e=oS7);;+mumV)>J||(%{PJ-34f^toSAcnvcA}5E(E~J>AA!C)Vl>da z5}BKy+v|yA6SD2*rN!Hwc*1(s?!`+G+@X1eZ97`-dswy%?(ASy3iR%}S$GxJT%lKXS+?(dVEroE9J7?WAZc&Mg|AulajChkpCWG5F3+`~Igr4C- z$jJA4GdjLB9|=NiPJxLW(vVx!9lv>e%UFeNwOB&rb~|h;S_J1kmh~oTcz8B0-) zE@Dk*^R<-Kqg~j8{hl0&s|aF2nKB6!=DeafWuj> zqiAD4a;NF1c?zeUau-iz$X?dZGIn<&a?qZtAXaq;gdm!oF#gaWip3RNo98;_U z0Roqp)lrRGft!5bZ({{49vCpH$)0nao_;wy5!z(lJL~T@LPK=oZ@YAL8OESJM;QVQ z(EjRk4<@q*4l@B;NgO$bgPj#2E+e5Lr;a7M_>}joG2NG@7(9>1;D*%LwTiw zxVszc6VpjXb+&ArN(4f?=dPs~*j=jchi-RO0InGU3mxXn^i`Oo zRWlKErpG^IA6Sj6C-(GedIC*9T%QulI$E%hmZ7YikU3q1NH&6I-#OgLlk7%)Tvwtz zze49h;Bu|HU{4cjL_D&}ybT|GK9TB(g0fs*9@b6s;{QsNzaW5UgYPEO)OV~eMC$+y z>n@_%?y6WlC9eZqv0p%+X9}?F=c44^M?^e3$IV<7DJNDLL4)b!E?3kKcP56+iZyfL z)3x?Yp~lTbftO5f(1+_jJ+|NgMr0%An=?5c5Kud^WO;SUrmOpN?_26sL?Hx{R$kSG z#Dq=?<1t75i8#)CNnj!#P(q~Ge~d|pAq-S`>#9j#rBzeid`G)Onl`axzl5hu#@QYE zz}UVvF!=bn`JSR;IQN9###s1twMQK}fi`s8Mk9IH6bzQ%*4C0-I~&TV{PrpKVa|cv zrF-kZJ?Posz)8R*r^|dJtrO`_4-1q9x5At_v|;D`MQny*KdY?sy_X61IJlqWEX05q-E}=OsfrP4%Sia1N!k+bY0u z#;&{}G4{DU`A_KOm}c{SL=oK8`(5laLu1x5%bfyPUwZ0_yexxFUJ6rdB)($pOXy60gZA|y72~th*p%dVcbUam;$(6 zAiLW@IxRoZiI&86x306-JXLu)5fYNXGQa>|S`M7VJKP4#IzqiKEJ6G}vXlaa)nZ}u z9T&H5&C}mzIb&}%1wZ%MOsxJ^u$TBQDz|OjqiAoXPEUcbHapw+#{f)3(V}CBOWi#s&nQDH4&AW{0`IC9)nG98BUb$yTznoTq1`iBO8B%AG%0yLfEtkQ=u^%GS)( z`yi=8b7HZ)+!;*9`B-L>C>rZ;Vj|Vk<2W-dGlRvYTlC11Q` zOFPRf@i?XON$QfR9Q)LbV@s_xur#HGQR&$t&zMtpHtWY_V#o#bEe{#y1X@Pke4>{K4nH` zCV&yAFIX@-!SeWY-$I+Pi&D91%c7=W{W95u>^%n0U$qZXPTOtC(yyTYhQ}ZW-4^8Up3EZ}b`HuZ2_xY}>kUc|lG*fZSqMOos z!4TvwCjU=_4s&8yhPf|jX%R9~_NCiWw!a_ks9Pi_KfdD+L2xp3wU)Z6*hst%$ms4z zkhbPIJH<5Y)!Cg6dXZxXuh>^Vi1AM=I=h#&zq8-{FdiyO&aOg@Y9pW8THGxx{*&+G_~w{|zdxXsG=(C zz7pr`?J?B*;7W$*Bh9M4{CJW?VB}MLVq=Af#*+l%C*9z)uB*F*d^e)`)+%VTEIg0nIYVelAQdd zr!yn3z2@YtOkp#+q?lJlDL7MIs@M7-2Rj&JO1SR*dKKoU&RN~cYa|?D4kQxgF;eqy zBlW*83gxRk0%fqfPq<6@seIdb)%(Oo1#CP@H(i)wLvI%vaMm{6Bwqrzy4C@oX?Q;n zxF=6iQp;Llarwo;LO~%+{+DUL&7l2V)weF9U+weK2ub%Y<_eQ}j~hV_x|JRzSWL8B zN08NFb+>11x{9iHbQQVu@=&%>O`v%tw5*qw?8_K)AZqxsePNYuFDq|PX$=K2{AkWf^{NDY2;->!hs7FK7q=piAU$?Y!erlyT zJ}q5tp)steqNwuOcW&Z6QT|@EXeB)O{=M~_ExmW&lVJJQC^QZr?eYYXuq^Qs3*&1QTQj@ z;SYP)(Hh^6l<&8c4|!qrqt|VQU5L@URVhlQo_6!7P}Z4yi+b+|f7&z$_<#J`J}F)h zYo__DlY99{H&MyR{PqLwZ2)_2`fhE8yULph{6zs4W(G;Z7!{cdQeiR)aVs`iHHRy0> zekd&jjs|8Nl-#v6!o2=T^~rYTeL9wP{ZBNcznyN2hu24#?tm`AtNirO*RLO zEce7*k|cdDKXUnRV*RS+CFximc==Dh<4`^l{GN*mcH}!|W?&`plaQd*VC7-xiOsB1 zKsVnzFw^!wxJcaVlV&S!H&ZiLXL{m&YxjJL(h(ufdU19r#ZS-$u<=EiTtPJUH5-={zEb z&Z^pcYlP2K-gDCTA?Wfv{1q$s?9=g`a};3{@(kYe%EW-ye~JbMl=v~*t!z>=yqSwx zv+BvZ&3%vQQTSH%*sXeyo?h)K+RY(%5#pQZMhlGi)+&44f7-YRk(2j-N;`o%{XrDm z@XgD`g)dHq=J};Z_p{)$)28e8Bg<+>p4LIeUPe4tv0yTbz#Y(hh#E4-2fc(w18c($ z4h~>vosaQD)GTQ%^8S1~JbbR;>6yDQSMsSg<0agH!|7Y8s$J8=8@KO_lS9pwd!;=d z@tQ~I*z=3=g`?DBrUIY)JXe6!?Cr=95q6l})-gBmg3kNh%}&qFeeh6vd{#9Z06L9| z>OUMA9(>&-r zq@?%xr2Acv;Di0&!rJBhOt{E0$Dr|)l4+>dT^m~)4F(s90cSw*U+|%gu!cLw%!rp0 zdGkly=jqr_KRwwA^Hn*Y>iwKjVMQrKzdIWzJ9HYgy5Rzq<>jz_VIcU`9J}+laW1qX0t3V?N6%|@rRh7}1@SaS$uc){2(nLT? zpmZ&V$BEPhryWg6NdQDo7RmM{u*OIpgo=)y#{W6z?mRsp7R7t~B4@{0SYo+AW7>d( z5%VgnS=~*v8};T5k<3-k%{|N*3w}p*k%>+1l)%EfmS5s_t9Lnsg=xT{=N=~wPKn7Z z$Zkk&J%eGDr2W#*bb#%wzTx8Rm(-<)B?%2tM5IgML9P-dS+E2WXh}(lEe2{g@yB`h z!7KVXxz)}1`fqRi&4RG20OT3K{=zwXopiT@h0zhdCYarg(CV0LlyiR(HQ6Hv%OMGG zazQ32T;b8ak%yJ`Z~hg}BkYb45Kltl0Nz>}Yd7zFphPF5bbpb1waBgW!dIhhSE>#r zBEwiBy~?W#n3?LOUFxo4rrpPdLlvQT)07 z@yPw0pp?iP>jA)PQ%!n3Bm}8E?EA)baO=sM@cE`TSh%kIvB0z4ucu>nvtNJ8+}WOc zmvo@~tMKZiKY2MyMROs#*Hx}zx36LL)N@SQmwSnA^paW+lJ%zMezf&Ul43)jNu4X< z+0kyyW8(wE`wO$Yyos=3^V(p#g+PpL9HZRxh~~;CE_({dr$I+MN~%tht==wfN2@C< zzgjZOa=Rc$Od>W}l@Qm*CXECKp=ZQW+)RIAR0`p#T#LKGpRpTh^n&N);T|!%h!#fs zYY#5jjlPI&=e(;a3bR}E#F)=UuMN}gFSi|srzpAadnQ-R@qU6HX!DSoQ+A#ledZPH zU4Wqa)d#+R_9mB+^0ycB-Ik;aM)c|odHMQ%5F)|{;M6w^ivM2}`>C-j7^4se=;3_t%jb6P>@x}CJ z>_~gyG{qQT-#wv?AvnGL6_L@Qz}7nV)s@)A)zv7Fv~Q8x$f@g*3=dDf?$w*on)@>W z#Ou>62fjRi8Z&VJO)T@tq;9BLRj(Z3Hv4m>ob>yH9{D0MnPA!Eu~IQ#?wz;V`V0kB zYA@eWndx&dG^M-y$V=5%5t|t=aiZ~qx=UJ_uZF0<)xH-B2ohPOpT2VFp{gQt4UYoQ^CRQ9{b-sYA@Y@&2249D8csk^@L*h z8$aBQ>rP66npgR4zRT_agGw)K4E*OvM(vnO0`W6~3l_FTi@zZAKfEB$VZjp+n5WzCOi{r% z8uGmf#NWSv|KwU|pJ4eBxeZ`GUcOiQ`cbmDlqbI5bc|`6<6E4a16ybO2uiy+{(O~O zFC*s6FzzaNLYKHl^ZDV$-VL&FzvoZC&hz2(MCBQ-zBf?rQ=M&Nle;?~Szyij`EalM zVIerC#uxFZMnF)tyNYi2hx$z9U~SH}Rh#QW`-20BLpQRYRKh}pxcDulDol9>GB{A^SR7c&AqiFhUXftJ2GRg#-WZk?F?$?6*^r*FVWlz zkBA69Uu#$SP#3H^@oa2W-<3u*)M%=Kg9D*0~B zmC~Mnv`yl(&#PSHX6P0)|4C`-qNQwRbFa^jtcj|qS&g%Es>+@JSYc#=<a)^}md7Rbc1t=20%~e#%h3*~S4$y* zLu^=6=0nbBk+*2+W-Ckw1&aVYzDrqs zt+NS*eBzBvu2#j8p;FA0ObR}~b-Z@aOrf~l*RKZ|w>cC;$MEhk5s&EOU*J9Wb>)~_ z>7>q|{VpD}w9G#+oo*gBB>nv((BCR`1oZhJ4#cyD@?E8&fVKkBZ(D!g`%f~&{RXjF zpBkRO(oNzpv1PxrcVrKU+KRrm12nB@iNy7bxPJ2)cQT$&osA0YRyDgWSbw5RZ`#-i z^7sE>iIUsfKk7Tmmrl;{?GnGtO1{7N@yp|%ynKB72N6+?w%1;@jq%Fdw66kP@27xy zH=e#6ob8tl@|Lh&SzWdL{&i~O=a>5?{ok!e7ec19)7~t8U_acQqNQ71oo)`e>3Q&b ztkj2D%Gc3NO$~yozP(CEM|Q3LMf>hloAv1wXBnh0%$C$`)&X{;24-*9OM&LHUQM^# z^Zl~1O|B237Otz^I(#ZyApQ_9n0@)y!r7kswz{gctn04_FitrE5(!IbhRTneTOGez zob?3X&Eq?q*Gp6GjF~u27~|dwlx25Ho<=#+G5pk#U@h*F+?i>4~J(|D1%0|lG zURxrOpf&BH`n=V1;uAD2)JD=}j`=QtCvVTIdInA-q-%joLy9-%_%^25xW-W!Ubg*mE0gZRz4#@4uz`{M zAKLn*JxAeYly`tQK(^+c=+_YJ$G-%!gnv!Gjk3I2nmFOH*RewDI69JYaDLQ z-z+F#y(jF>o*`iA)Tio_8P=S2&l@vKY%b*|oGn&Gqk=fCNbg4pLOWPfNo#onD4m;YHV>1Y#D;dyLp*|&?UH5f}9sW=MRY|$ZYLSP*aGmlwCL( z8CKdCNZX>O)9bhE0GG9|!U&X!4;`L?#TkLuORsA?hU#)lrqny3SMPpXE_%~v=0AA~ z!zYv#mb3XA-{qFQNhy9%jT-xr~Z=~E>Gi5;ze{Z;dAnt07VW{>Fw%@dqsTu4>+^hJO>rXH#>0ka%WZZnC4 zYFU0(SWEC2=clH|#>GXxi*X{hWMg|19nDb$lJUjN$38QTK8o14O-|2`G#G!C)#-lw z{K3=m?B38?Uo`i??vyfYkDr%Uy!+SK*sWlBKh{AmTvKXir%c9xA+a3pH{a_YbE49% z8VMzzO0pp*;-GptaTys|F4a~Ocq=S3=utFDhTpcD@WB+&LdJvOXcr}z-B&YRZy!r8 zj<%f*_eRR*4>0AN=BDQQaie-XVCEQeO4fkIBilYn1SkRDqq>+wen(4*)RDiw{0Xj4 zpqR_e_^9Ox7pwd{vG2B4-)|q&e2ZQa)B$0jpJKmnf;unS!vWq4AGV|%EEpD-;~2DK zF5E^J3psKqgxupH`iRxsz zy!?nG2ng-t2i3?2cZEv|IyyRpTz*&$pNCSNsbyfMU)|@z+i{C%zg}>US6x$Li_L;7NY5oHUQZ({(v0I78oz;P0_*#b6`NX9E9K-49RtK;0BF|k&M6mIf)RJUG5(Kq3a7oFWUl*G z0)hGNY}`6DUi%SiE;iRso9UPEedFZESu1nhl@&lhS48-g!P>bnB!xC+Q2|^ZADiND zdeEC{pW>LEogZD17yMlLE$8E>T^S0mbWJSk5w{KbVkIi}Vm(|}` zrO|+;GgZGg(Jl9%#1>}a~fgUFVviQN8}-iB78-D&36J{?nTpV@4uT4Ni*& zgl)HLN%od{v!CW=6AKCo1{7y5#X>s8x-HFd$BRY1WE2!ho+C=b_Y6opktgCC?DqK% zqqy<7^$n!<&Cu|s3fl_%iU}%a^Zw?KA9X{8oY$5%+C>5^!|jYmPXlTunuM$mbzJ6Z z37s~TBTtVW;#90GA`fOH#b&GK$MQsFFbtQ|B57NEEOz>_M#Yhqk4^V;PYNBd#&ts647~c_eED><-%L_vq3|pw ziy{ExtQuCGWW}6zTt9+R3dN5mA|4!7DpFF0y+@R?eI`r64L|*~&Vuu*EQm^>bHf_a zHjG2FXIlDnMD}{QY%Ow4&|;OP5Y4pn3Mbz$3Ck{Z#}i(8Az&Zp!Z39~X3%bj$^ugz z_~iR}=)2gMzBsqFX#qF+b%HkB1Wv#0#%W?wW3Tayu9n<%8o24m9^eq`1G|_lEEn%q z#BV$XScKvqUN~JzZAR-4<8&ALO+X5!jw(qNmI4^d(+)e-d2q-uPT7(Pgzb39(gAJ& ztIwc~T<$zIBoZ}6%q&8394#G$gV6^=g5_b+=TsdZ^;|;CwKhU23BWb7bW(|Uqia{a z7jLvuBK7{=%J)*RGSHPWVEy6xJD*iRpx2b?s0duHb0WQeoPXmKc?g}PFtsVEyF0n3 z=<6E(g5l`q zV#ZEl&r5cv!(jUwGEEv0G&FPksb5DJ@!+M2inXlQ;oq=jLfIq>qwt>h87;0Q?c<_J zW%$My&y_5V37={ZsLr?fh*_#<2udhUVgRG*AL2+&Uc9hPt{|+?L?Bq?yoCU&fZdlo zMZx>Tv>L{1Y@r2i3G1qo4zS--29FiLgbd(32BK(768Y*AcbP;yiFkSWG`fs&KA2g3 z__k)(d;M}v1^=Guac(P_;2*bSA1k8RggmEpR@AMOMS?}@YCiyJ5uou`ukOiIcMjar z(v%MS_NI|6V|$g7W_ov_@}@wZ&8jZV z_VTpI2g=+`vL#xrZ;97zcs!C7I794z)(KJ75xKdnO!)*uNBP5Cdez1S*@Db+LA8)p z+M04WVL~69wK9T(_o#voF0@aqyLIP+ckIsV-CM{-O~W|4!~OSmbN4SDP&VJUC;`VE zO`0N#wjuIwiAb%_f<){<>yOL=yB!~tmE79iGr|}M^v*Jm> zV82m*qc-cC7J&!%X@^;!tOQfAO=7aTK3D{&3|E@2A8}h%dWG&|soin$dYTy(B=CoE z*FmN+rD{;WYjlFh`iKPnR`&%?rR6sPEgAgDc4Si^3wDzIFc$-^(uVmU)xt~&SFcsX zy`AmC8sKQ?`BJ#0t@2M+VI6MAmXK+kUq>T}x8 z?I=9*S-Y4jQ@-1Rh%u_Rql3S*U8mY{@19=bR)GctX)^etTf1qPo7R{)j4z{CBH1C( z3-VAfh97W=NI8JrkS?#fr-axXUr*C20bnR*!#G`)~3KNJK} zvO$_aCJt&})*nf;vjyu=gPKr6be5sVgpHV4+X7QmRa(#{vW{=}syCJUGen*?VIehP zq4TUeE2Fk)&8t1J?U;Kxi2Mhdl>Sf&|7^A%h;DYNx@b7``&O%u9wwD(^>t1xn3tMi zldIaX%&2(oxrJ}`kge8nvlP>v{#$wtAz>M>4b6*MiX4@Y7a1{AzH0l-Mm9zb{ft}# zNqY=@8z0FVlzXVN81E<7qd8F}Jfu}WX-{9q?0d45d)B9&to^b|eLx{|!~N4bUDpJR zvM5w;Iwy2`dngqwx;|9)B4m{fcQIT4)h){eCz!cdH)3nkwqb%hee-{%;dKmVafSSW zJ>r3Zta$y+LnYj7)unUG3_N5O;G&)kd99HghvLa!9)@-b%)aN3KB$)0{(8{Wps)BD;fsg1j8;`W)?fDqW0q0wvUr7Gy9&v99)1Lup&x2rA_?A6?o z-(N;a2B}qszW6Os>qcKGTY|rs7;5y^d!RA&It#L44L(h49sxIxq~Ra;rd*uHlQ_I_ zoTA_sWyj{c*z}V^7U;%N>YGNPc%A2z_|{KLb?FNqXJ7v+ z1}w5X+9`JfMz}IQxLi39bcw!ktxo|8ZoZi}r+wu!eX|gOa+rsb*wr zRiNfY^BYm%jwSk4-N%M})o#mtOokM`_`F}cru?e=)XL5M9yA8Q=sa{}PGfcctcnW!Vw5|QU>mYE3psN+!C*h?T=61fpaZj?o zvqH0DG_4Mfc6_;^2M)&b;wqQW8w`;`PCh5>8ksf|btPI8G!ekBjRN=^8cGasQ;I51 z{(5|IM`Ncn+|)-kT?0Uw0L<9(^SWylF>Fa#t3-129c&TE6hmX>wemTf3)T~Q&L)^M zJXg+b8MbRKWwklz(|D7MyI#E8ltoFsE>kU|y|E-EyP|u@ci*FVKWD)Ivf+4`l*b~l zc+FvP*U}sq;e8h9xTi(9^(w_!`h%q)a(cwYD0H^ziBctjd$yf6SlR(PkO%h?#jEfd zQ>Z67MvGtUMUNPRNoDpoE@%!DGb-k2+Bv#i`KHU))6^I!a3R(y`|BHKJGh3l%SM8> zGqaFw%QAOzap{SAU4>}h{WUD8@D}w2UW7RRxY@Dn7aST27W+J-dn9$aFP=X{S{M@P z@!MjGz3P$zeP9adqA5Cza4=nB`)glo^HUxK-3>&e zn#xXfqU-LOw|T?{jG{c}>E0#WkT~AnZrS#;A)@+0YV3VVUdcL^0_yEHlMx*|r50;H zk3E-my2786YQCGXtNJ48wcW-nFSFZYW4EK7f#L3gUdid)^k_l1HfA{*GS`aDD_bEi z1U^X^Q{*p?e{21Q1IQi6nt8e{(VUy4b>nJ-ia_GY43`6hYwuio?kx`TNJJb<*v1&U z$oc!qw*A$Jsdr@L&y7B?y~S?5@%5bWOs+i=-3Up`uN8W3ar96la zpQm~CU_e$d^pcRo4-*Cs78;7neY9HfkaB0@lc~FeleMPQ>huWXGpe+P5c86}!-NV? z4Z-2wi;!nRewN8ZJmLBbCHI(@*2ldHL()~Vb4n4Ciqjv z!`~Tr7GUz(SqVyZ^N`ahKqla9cOf@|gYYcJYW?FXaEzy>ax#HhBD>N=Y8vXq{xUMa zZ!S39E|)Yntm;xxUYK*T0>{jD*EUx1C5V8>Z_Ig2=%+BWu*POW#JDn!)k#jLlOx@Sd@^l2>~>gF zhMg!Eq&z1n9AJSBgTmn7tK-x*MhDF7ik_5YzzhI}HNWOurLVr#NufzEn&`=h#Jrf* z7XvvtIc{4iL*a>^_Waw)N@uayDdB!GZqsQ-WVUjU1P-LU?#$!D-_+r}!;M_h2TG`| zfnlV)RQB++xN4R(%KSQ#8l~0HLwR-+SF@o6%ljC9%KB0^8J9e)-pgy1LL5iMG9u1U zoG>y~Bgib@llBOK5-SW9LoT63UB&{ko&vHq%htcuONryns<)>;IPi+LYeq-YSKw6_ zAAc%7#=M0gO@}Sk{o6G~{BbJc2VJ2ZFr71hBMaJj@agwuG>e5g2HlFz0aazMha*mM z5tDMvZf@@C+vlC|R>$|3+Yh7=h+0PbzMU^?~TcG~tn<(gkONv;HFK0G}o z8v@x(Q>BIZP=8|UIe+iT3Ik7nTs)kk^nK`<-t028eqV+WA6-`)Uho`sri$>R6mTKPBX#(|t~by!7EO%qY}Z zcpIy9lYGCDEBBP<0H40{%?Dre65X!vv@fS}s0&5Stpr^uh!SnJyt3C5kbfw*rm~F6kxJ}Swxj;{+k1`7@g*c= zV%q{n)aaH&Tw!_buPXLCZQW_)*ph2vaSV@_RlW?Jnk=d}^RZB@sgT`9OzNR`Ivsnp z+9iP;mw{-z?digVpvwbRx!`ju=E0GE+lkI^6Mkkq`viCA_P(JvnA!CB@G8RR+k0qa zgnS6(Z~P3Hut3nQUWmUWe^4s7JruI(}3lib#{H7ILpRjclb#}Xq&ufKa zfFqLdSN(qJo*M?KY(b|pO$D1KGpsZfXnO)n#w=Z?a!M9|R)J2tcJgNE>E(Dua08=A zFy7jEJ(;QBgY2*^+N)6FAug3l<3YV|%{;ILSSWJ*UMrxsild6kU;_K@n5!%sau#vS z=+dSb5zY8`0_fP~f<1$+b=VV|`?TlyB*khbjWBlZwzPtW8qL?{8fb1!u5I}IyfE1f znC}CvsSk}KK$+^|b*(@Ks|&n004t!^VS)GI<&>4L`V1o9)T;nJ`h~<}Cs(cs{<{(7 z)>$dflq$#??I(Y!TiYP7kWnFB3EBVZll!4l>2eP#tC-L33yj6@u25gls0G|;T1wcDPK&1xsYKf$#!nW3jFvpU=)Z_W^11y z0yfmwK{291=IUL*79YOcj^eZdw)^OJhCfD~zvNjSguhu8MmXtioj*IBe>y>w=L0)9 z`Ym!5W)?J&u>#r|j zhHR>?aY;4zSBETVrgMHCVn%ck+PzKu6N3+`+vFhx7On!Hlj0V3W>QMq<)~lK>HE8x z%F&n5-Je9lh_Zi{Xm|r+4$vKq!xWN(*#aMovkbUP0TWIi3?fA-#V7+GTQ(@F^;7k= z7iMNvky1P449nvVj?@Q`*0mP9el&PfDV>ZuQ;Rhr6T|AD`8HospbR=EFDIa!q49_- zw9;!UBd77G>CpDtD?wB!mE-fp2V_z1)MbK*y;wh(s5D=Dh0#WoTp(iIaw=vUD{aRc zec}dArYbBc8J%^0K&N~(ZsT-#fFBU>4>=?)je~*RH#4h*%di<(Ktn^a2R1XF*_ zRi$gTEi7UZ2tDkJUTel`4jp9}rsX2|wc{YE4Y}H{C-5MEbLWiH=t7w2>Q^xlgUG-f zhA+oPO`CbB(+$45zOB=cz$};b8{`GsDzk>=MD@E9kAdZn>wP+XMyK|V)dDiDq~QoPHx=r9}+N)DGf6i0sJ`i zQu$ROa4l3xvyK6W-A#aYw5qy?lnum9X4uXumNic2OYj3g zn+(w!W5$+jFnW0HUUJH9h-SYNl3NR&jsg@QE=u*OOw_-4?35WV&U(jNnhTorPbgIS z~VK>?!h`2K$%^wkmxU#r5On?yS9GInWpBveVx8O6p| zo#LG5Lj_WZn0UeK&I1{yMvw801(8Yk(1o#(Q{B?pQP3q8NueKOkcsk{pO6~9m`pK!W7==A08HCJ5sxN_-Z)<**ErKTv_)kJXF() zzi>XS@wM8oEyhyDu8E(627rb-+iN=q#n21kiapM7GF)^^TzFU#>NkL9ajR`aC5gRb za#lB}5LpeuhO7$SL|yF`Mel+hNk%-a%`5g)C65{1q(d^OplP z9{USu+6zRZ;Q}&p1fz$>I&B2Y!c&go&4;p=Bkf?$u)xr3zf$&b6G0|IsxPf_k_ND^M{!kMjiMuq^YUNuHPc8Q}#d}DO8|bTXuhi zkRoTN9Dlid5MvezvnmVxr-sYZ1JSS7+%s^*Nw#-k~g$kD_bkJE)0BefJx9XcX9 z9VRY$Br3rY8;0slr6a{9kkS2rA3zz=!gmn%Uw*)JY}X*ig0u=xgYF_EyNN<1cD1_M zO3f)O-L(IBg>g@^4X>DeBRpML)oJ1W(7+4W!DM);!;zX@0)Bk#YDFGAV%*mJT0c1+dK|t z^5&S60!#DQK9}H?yU&jWSTx+#2~9!BZz_9bj3zIQ)YsvN!@CJ2)%B(4`=XRI`f57c z&&4QZoVcrl9ZOVUzb8@|Me5m%>b{5xm6oI0}1 zK;J`kcSFRl3#TRFRQzBV-Cbnq?}dk!xx;R5f`WB`BTKdxRK-&H%p&V=C>(DWBw@j2 z-zj06kYP zNwXV6Tj6dW>~gP_w`lG?2r$WX*-_P_VXTzlTh6PCETVYK*_01p{y_Qih)wUU38MN^ z6;2JOhB~f67YawTBT=@>RDF87ITu%Q|Rw!#>&6KX=MiFea0;}|DO=jA5Jsx zyJHPtpAOWTIBt09ZQoc!Xw$A#jtoM_Qn|lk*jl-tc}F@gKl)P$DKkro9UZl|6Ka-u zLeigsq-QYek3jzTef$GV$@oi7Hl+TX(5m&sXPF77U`Q~DANh914>KQ@rEk+JEAKWt zMxlb>Vqb~ym3UR2tgD{hX&xXO3XAc30_w4vd+PBH#2QcCbOHGLw*Wwnp0^0Q;(7K@ zJZ?T^!n;&e%L@HHb=)7)QRfs^{(#owuo|Dn=%c2Wh2fC)Xo`BYoLd-2#D_ z+7`@w97K6!vNNC98IfTI$EzbLY2Y_$oF&RB8wmUNWE&?27F;yS$sZHuP|FLM)a)+P_%Tab+0PNq#c(3#>T15)=%iC0DN z_v{cDC9`REyTQw9>>S0`V(!#vcM*x->0oPXXzhq=QAG{M_H6LP;QVB%8=VCf3%9MM7{&6d{Mf&>9is(Py{&}@$cr2Y$Rva?hRAUh28;@P{ zxRIxDoz!HhaZij0rJnst{N7Em@t3)!o%atr>`uq4H$(fqUYg^U8PF~=euJOL`{yz; zYT$y6B9>MP7=i0RF>#)8?sJSmWass$O~Wei=j6Vj(vBtRjJwXRL|^|%A(WyP|F?Gf zwfavMF8<{9w19nbLqu^gT<2D^rpBjYJ(J+V#PfX2jbFq+nCs~uF~{8*G=*k z=~Cy-A$OS@PanF^MA=r`2hn(640yGCACiJPTcaFpA>(_E_vK|H-%_?Ud>cR8Hnnj{ zPz)DQ{9xM#ug&m*wcdXSgc|yR;+5Cjdg^yUT7leusYb|%BfXJH0M4XAC;oWPegTVEtJz7UAFMQaBztCoPF9!*Dic(HF4@~QBbHpq2~gnDnQ4gYcbb&5pQr|hH=fTp-OeWKo?%k8m3 zatF2NsOyxUwcUZI|Bte_fQl!uTkB;dDpc%N&gx|zCMkh6^$Knr5VTp z7*1#1vLljRUS=kIkx_A*%G=vC%w1PEEz_`UeI#_gX^AHG{Ur8vdM>Z?Sr8l)T`%q2 zd(?0jA#?K9rb}Bck%TbF?K(X2P&u2+Q_=2~ZZ7tC&L1(oiMZUADf< z7=Z)Ryn3~mjY__{GT1M`7~S1HJhYg7D$~4e%J$?_wZdhpjAZQt?b+F6^G|)di*v3$ zCkkD;3O=jCUqHw#k5XIw(}E6=ADPm=&=hDF_sZuKd&wqkPGtKzDDk8ykyRdqBC;GT zLb{#lIKx^}97*lga7q2wO9sj~bMU?)g>)-60%HdA@@V5R?JJF>04s&~1!1f*$yLYW z;G`pPe|Tf+C7>2y;t&2}Mt~kso`E*eZi{jcWCb8UFPLo&x3uLwjn3XBF$ayDOvT>b zj0iIOqZsd0<2!i^Ec?p9nl0>wj1w0y`4aeVCn@XM59(WSBqPYeL1Xn_6xkKiAXb$IpqYVDr zBy&)&A38XtY5wKH2f;TsHm2Vr>dCTWIl(9P1mT$T39pf-v;8|(`7f}2kVOKj9_V`h z)N4#pdn0@UyZ-D27nzbQ7rQ_kCXf_h!gO4ROzTAJ6m4seskw^)e~(di+-;i zYtOROph3^#s7s}@pzyT>4PfsLW82s;J_B>Zk5Ym;>>uoTwY3oSbn4^ve8C;GCHr~@ zMh`GEh%=Xxw*FT9yV_{gXNS|x&5gw!Kl8uL#=rmgof87B%qr!>c#^nF1+=f}F?}aD zcc{FM>=#=>cUGNct*<*UH0NeSzKG#GFJK4HuB$<__Keuf4r-2wR(t)a{rT1l^^gNa z=Jn5`HfHPhS9b+zrwg2%YM?3=YHI3q-i+seBiH{gO)Q$6@^UjTMlQoYIXia zBS`ISJJ8ohO9)hGlrMJ(72js;v8DcAI73W*PH8wYPDSwEsb79*X=q-i+`OvgE=(1|(?7(3Qhy7X{hEgqAl1lJ2(F&ph<#}>XC{ZCtuFI?y2~mjwO#58u9mTPR+SrqD*yJQo7n(&ge<& zC8Mp@4A7lyW~kxWSKRrLO-&Xa9RmK-PQN}48{a>fb>&*HsVB&20P*~>)Ecoh^5+kk zk#Lz+Wt!sR^P9w##vD&zoa;o*xdi{2jG)eE9Qu<*x*a&nsgP4 zVIYH@d78lUisoW<)PSd{{w*B%>Af}s|5wkUbC)t}KfIUuITQ?q6ZIjH_9g%}9~zj= zjwX6(-O=H6zR8+JZ)rLB)6_7Ic!+I4JD)}z-O=8WC0=e}BP>Qj0_^!+l9gT`&^qIU z4Fq@{#%%*3TG?a=xh(TfcjD-OmNM{taxx-?hleL~(**LJ1bs})?iB1F8Dd`ol>t%G zQ;CEePbH=#SQFMs)xM04%wW8_Unuj?32wmUis?|6HY_ zQ@VRlFHmk%26TH(-0+B z^{%|18_n+K;v$i0use^X4x!#=P5$h~UtzJyW{NdHo0Pa@k3qc+o(OQmq~i;O)Z%YH>o@DV@hQK}BxpzENPpHT*o z&oQbRL6e~pZW3~CFPTcmqr)Oc>tNwB;mqD9XY;`V46?@@=>b<;Mk(VF+#{bf-`r;l zyM0VNk?j-7SXv~nNH8_ES`NI`7gS4g=?`aZ6UKA&I_jjQl^u;9Y^KLv%4T&!MEZI7 zf|k;q-DQ9n8YSv(jG-dluBU!@9ZL^KuBT1OSnA#q8>6UYu{%L`Xx3YFyXN)rDp|nD zFDZDKWS))PlBA{(>a27jG=3Q(4z)C;G_u###b%)CAvr5Plo~D!`CAm@osG66(aguL zF&MllX4J#2KsbUe>DfAH`8|p}Jy#X_#ju@_uWI=H3@p#^5x|GHe~j;0)k+gE9&hlX|H9z57a(gt@wI9mp#MgKlv{0%z$F3e zp+FZiSF-Om`C-A+B^T`V{w$2ci343zg}%c#(*qynWn)ZSnp>_U>N$>~{h849`H;L+ z`zfGl^i{`m>E&a?zs{EY7lS~i+7J4PB?P#4VlR z-2~JbpN=m2zBYy6G8qtA&!|NWjtg2apovSgLS^Ip#!d_m4s(fp1X`QefBOR**rA%G z0!qqy6*NhEf~A>2gILMO)_SivPCC`%Q475qF{flWKRyP>DZMKrPSt>^D>AvF6J;va z(#|mikk2b*{=YtPiJ*7=gvM{aMS_rL`A2tAMs-CW0syOz>q z9=NQUXx$}_W9qh~rii8vd3bx0#A~jW;P4h}Xc>W2yyUJ#nipB}qsE!DtO_A!m}7ao z$}H2&i4bHAwuWMaDzNxy3~hU6P6CM~y4=v>?L&mJ=0^dD_U6}aXYOc{{N`5-#z+*n zj-N6Y_j)TX$4Tq$k*~eQBl*{^Yss*N`9z4Y316qheTe|^WixlF3tSxY{fS`DVVgR4 zb#BVLITa<$(z)fa;9gH7lGMaWk{#?VweVA9{djwOL2DAxt~=Qd!_rjUv&Zn?#35D5 zJV+MjX5xKD=}Oq1-)Tj$)+NCD0CfT)IjyHfRFl2OUY;3Z-CLj=mT$m#czEPgR7{SJ z%w82=Jw{!ga;T=PmWX^`Sot0{@yt4!8XJEeXFYe1?T>rD<>BSE244F`F16k1H4YT; zgp$&5P*(*XR4mP{cqHNED*4P`kCaOH%`%kqxExzs=d6XmohMmse(4fg;4;yyNceqedq$V3Ji70E@!hc-P!e9~Q$B>rtbG8RYEzr@q| z(^9kmvpG_#1zYj$6bSGqCOsE8lSr9Tkea z^4a97YN+v$5fDXeD+?v=;^@64MgSMW#=?#@U~(HLwIv<9;;CZ=m6^$LpG>QYL>hH8 zJ~Qm8kmTa#(Miob!WoO>6)}H0G{BD2rO8zNQHsgISqHGgYCk<1=Q+cc?`=8oj?aq8 zg^19eP~j3jJVxc+tbk9T(evT$ceXOA*HP5>{n2y!OWCxb9a4RKk!9Po4941J-dk}m zNQb$lyk6RSr9AF#9>%gEjkwlXuXI>-Z(XXSxZENI?7X~H0{tFPSA%kkrIdm;-i&!3 zqLS%WTEWlu8LZJWgqtsEay*MvxnpSL&7!L6==0vAP`vo(ee@ZYc?+Vax^l%epQFa- z+}pY@?Xxcu?i)v^Q--r1ohtl5F(eK4JjVN_cXYRT|JLvS;~!AS2n=0X6d59w9wtG9 zzw-GeEX<+}t5hmM20!NI1q6_NFUX5en`Dukv0w^^W3nJWow`y@liC#b)jg_ouSNOL z2mR1JRs(g~{Olr0u^JT*N%d@qi+1uOZ)r?_BBrDBM-I90JWrn z`F-lZ(){vuD@^chXKHOgauwA8RNoo?c=xbil)7|HMV-56V~WF^ijh2X-VN8P!K^Ao z|6(natKo>)V^aK_5pFx}LS0!;S%wiJ@N}_mXBuN>Wx3^eTn+$C>y(K4RmGA3f^fF*+Tnk%#E+71l2zO=kdD|0GXQI zcf>H}L8<(+B($d)EJLSo3TRRHbwCE0IDKJ8*UC9Ys>}6(sct;G*hpIE%kbmv(%r^i z_{+y{F3IjocHD_#q;!!uEGp$70jc7JIBlsorQl<{3-06y%D`#sW*^(U{0Ktx)SeRX zx!%-r-;z#k?FN-0D9e)a1QLx`SbK*|lS7N!66hbS-K=ueMki?)P{>y|jK)!{9V>EE zz*#91>zjynS$tee4V#B0I?s<~wx)5`!0j=bywVz_g);PQnOJ8Z);PDLPAf-VjH%N7 zeK6e`lM3iBtcvsSqI0g=Ck1&8W%ieO4re|C#BGjxCupl)y~q!bh%}g z9JUF=y4r?nl9ClyH;DpX>Fz7nb+fVa)Y*GQVKb3?tRQf85>$pG>AF?ILVd6EZk33H zz>=<+u*n-XV~q7nDBJL`t1e9iG!I)JvQ6WRb#sDNGGeXB0#fe#Pwd+5bMhs&?YV8s zV*fD^1C9|AAKQlI6x!GBqi$I3@tOs=0t+vmf4t`hfSF&fdY&mmy@4{P{z6GTdv`n* zM9F3~|9q@1C?5^_3g9Y)k{yJGMjU;c!MstPW*ldwhpqrZuq|lrj%GZpIFDsG?xyGV zb`qkUs&{j)&3(rWDqrz;_aq4q-WQIr52Zqb5GJn7iqyu>_h@*)&A@DWnK7B*@jY6#0Lv;AJnP8ex5o|SWYbbDT`i*BFKn{b1&H8rX)>k$U^Frswt+-fp<)5$;^pN zt8ONMYUo;3GkL5<+=hBOnc<5Hz%C$x4E-CTEx5bqR}dcdQ`P~ zJDd8IF!m4NXvF{f;a^JredBu>PuuO?WM#Vwr5SMRcIm>PbUFuGNQa^ANmLKjwr}%My49mqF@NYvJZhn-zvy@47-wiKEnU;I4XFbb^vx^-k9b z05_<+u!LpZEyiPWlJ!%ak*#E}mM=5gz^q&|2R1c<&W3)f>WH6)rH)XEK@WH&E%M+9 z7#p_eo00WBsA;H)Y+Mu&y&2V!=vCFEH&Tk&`|JlL{vQwGxwn<<9*eGABwH1;b?kg5 zz4s+^I#Cy@6|Q-{8e<7LI{lIO<#paj0U3)=9?mG~IKg!)CTHejYt3ObZ24zger6Yk z%3JoW*da#=vnKz#_*yfJBP3qH{8)Hbl6W)%SJNNads73q*MZZa|_VWvx zav4@nUsx*icV8cgV0K>cMq|?yma(*1|A6+w-SAHKBInkHRnW->=N<4f1gXGZ9MSBs z?-KD_Pbqr{91vP2P0akVXl5Lg2v8VZhF;bG^JM*VHXXSAKMmN^;zXK5jGmyxK2!Yf zO!I%f_21q$Vg;j!gMy0&`@2Jcp5i;F-vM~~5YdMg*}oX#?`e=A=2RCFpqPS8UW5Km z9Q{w?`fs3kNS`Yxn8EfQ_xi#AmOwBb1tEd|KLfI%h)$1(I$P$0z2&z>qkOlI@(SjA z?*=3uS5;81`K!kY@DWGbT~KcFB5Y|Li~Q6I8CnkE<*8l;Th2Fj)M|Euvz)bM$*`Ph zg-c~M|H{w~>TNB<+gXd1JF$eJa|4dgmVCjoe9zfl?xwfuJ(h{V$-ZZ{r$te00*y`0 zpLSHg{Dp)Nuc4lcbD^Yb2JU;ecLKY(4EY@~gkrJ%RtO;;PgHeh6$imnz&F846}kWq7Y%g-Say> z!Cx`w58;lhNu8hgnIX|!BihJsP7V&>WL;AiYCDY{)}qJSdzXayddkmv_w79?SgfbP z#&{pFU+_7h0cVH)l1U7?LQU24wa(mJwJmLxFnJP*0z)R!6U`!y@kwmV)P!hvi*0Ja z?Dy|CGnSFLpcC=Lh!?u?V8%#S8o{2y=X8`gsi@%g z3=B{)aYF?ec!S@BC2g%lwS9d5-S_?m<@#s}Vajwwc%n<6!;J&2&O)g>{UNLzw8uj3 z48{uJR{wW1M+Gn1{YRw+-quI{vkZxE|IlEAiN*z3?(G}|3b=%03`8ou#C<1n8%ehj2w4N}M%YM{% z3@TFLOGowxlg86=_3*)g8^a4VpyGcTw4anCvF;Fr18|f?hkgKo}hRy54BoU_Xa&()~oBVi+2Pt1}Wq#fI%*J!G zSrQBB7Wmc5CS11tk+A*gqG2e*S;hXCIE|G z=cmZST+>#W$#)nsm@3Q>5wBCNa5tJh5SpGshWO*hx60J2Ou(Kn43QGdk?(U9k~!Nh z){0$9bEP^25qNBG4bK^sO669k?`|$ca-nasS9`c!&)Bj?(pcC|&SiJu(!&5D^~&%$ zX5YEJrvxUKVaF#x(aiVw| znrKtVSC3hqjwzE7kyvbn85)<>oV@NN7 z9e!Z<_xIQ&SEsz5Ro;8_w7N};E)kJ59sFQ#eunV4c77<;Xc={h#7iN0>-Zk|Ce(9t zVD`P4#v->PRt?)npmo5m+qcc0x438qkv)BVTLszO%u{Ph&XS3g`aE7qWm*Nu*xVaT7)1KxFHLeWw+HF?hUAuk=zSAvgVUZ$XK|!}SH$BGYVqxCp>qu#A zB%7L%SoDUNaWQnBD9{b*Ld+cmb1do-Pc4ALFJq7XFnA5-|C-^Ap!3Rr3Ac zK^wvO$DSk_eQvRehY}~YMUw;m2=Z1LTf?qOGp-K-pXcpmy$L6{e3-+&zI?6pIPu__ zJ|leM^CI;DIvN%8@y@nqx%HQe6>cvP@aj7hJEp7@(R2DBV9@$4Tww}qsTW`y$7N@B z9wxMChL5JkH8|vjhHyC#ZXeHh-n)5{=!{Z?7F0(?9kJP23kyrzuC%RqX*|!^O{9jo zjXu3WTWv04gqi}gnrNz*)gFjmm_eL#q0kZeD;9MR0QHPSxvNSTE;E?z_s`^0?{h-UA=>PaY#e zIPkyGE!9gBt|PmxuX@V0S*dTucu62O?u}*jK`P8ib_o$h@))$nv^XCeE9ipyHyH7J z0p0+_dKG(IMpnIHh@`8(6e2nJjhMb^#f*j@HApYD`V z&0>D9LDw737UkhxSr5&utPpt;og#dmWTL=*+cR3ZdMg*1(q&E_&G~Ez!zM^2*bQ;a zB$w!anK_zk+h;t9^MKLp`Rb79)rO0vA*nH^adX1Yiei^7^aho{d(;i1A=YlJTP(7= zu40e1lvUCKVhD3o+S||buQZhagbl^|-Vq65hpNMJbFMQ|XRFTTh6TNzuDKS#i`BF| zD2N^((*m*fJ=){Lh(CkgW2yUTW1hJ%Qfp+dE^qcl)k#GAEaT993D8`-WDj3b&5Y9K zY#J5daVv0?NJlz-)2wtdb$=ut4*W}c7DR~EkLzrPz^$A;CTrEleN`R|`t__<>mfS@ z1(P{l{W*Ee7p0Dz${(kfyGZ)Nh2`Yl|NQxri2+ayYcI8Q+xtlS(A5N)ZdD$H-=#z>2n>80$C5ukp1W&Jn4CirpXGkG4vY1xs z{MRaY4~oa=X%l={=&wZN*8roJ;7G|RCcx5{h&}-S-Z<|Vn%V{ge<>c!}TMXflVG#4d8S0@V^Y--){08EJy|V71l8J!X6(S94v~^Pir`Hz!}z4 zq^5#w=&de{(tkc=PK@|g2wz5|Z+&tHs!=iDnm;9+s%5$nA7!xi$Wyd6_4mVx4I$!9 zahMH|{tPc&mbhd)WzmPXrZ1Y!K4_U6>^A++HFK!2)Yu`F-yy`s z#RU`;Vl^L$$TAg}Z+~F>u1pf+z5fTXSg`L9uf?LD91EnTmQFc$2p{&l-l_%u!`v2d z?_~i0y|oDF5JxLt(o!*^ZgvTw+v(iLZWl0;l9GL3dr!}glwwHs`+oi4P(okeKfLE# z1zFvw)tv9CaH^6VJ38jy{2|>->^tbaEtUyfd+_ty zKO$oDe%;1BKV#BNHgjN|bc9YeB*F;K-{7Tl!G&Qp!;z9Uk3lxNfzx-@m=0P0 zdB!oxI!Tk?3e{BYo@tXPJCKn0?9*@F)q|DGya@YadqiWw8U^xPMcRhfRlokBO~UUod=2u?!$3T^a!Jd``0f4ot1hen z!~YW}=_iE18SO8!9% zeut8w$%&!i(O>Qrq|rA}EHIVq(X0=9hqJ}s_*|XEb|f(5Iar`jW5g=;r&Pl6xls$o z(gz+VE|CBFrctn@6cn9HEB=T8K*H% z1!a0WwrlrkmNI)AHdZidE``D2u={wcGVPyp+|vYv3_je7;E%)w{1|1FNT0b;ISYr9lunGVL z;lb4T@Q|&+-YHd*ZP#q-4_ zBAh(;#O08P*xysO4GRGW_bvIOB^74*ax%%|R=JE;qZ?{P+YPD@5g$CLlL!spOZ%BQ zlna(tz3?w?=8W$MvfjXEH)I_BR4_}lH-6f@Om~Q{E!q?MYmm_tu-hydh7Jm`K*WxQ z)#cu=ZuPzjKomkDU^zgp<})5+IZ`$M8&(Jucc5~in4Jz)TU(pEG3A2+zULjm&x<&D z@%V%wjnS-jKYv@KY*Y?i9&hAWkU`gurdH*{U93VIkbvj9jT7jwJz>b<@kBapC=uo% z>3#ET%L1`C$H3Vq4I+X8)@rc|OQict)!jO3IxkilMQ>^Enl}(-!^)}Fn#U1~HV<`P zGyv~6#dolUcW0^6EpmBY=B3M(ykm&bTzx!Ol$>vWiJD!CK;#r>=v4UfoxllP5||@ z1>i~JvVIys>zT49C*{C)y*ZBb@dk=ZS-V2d$lTTQxD!A*VXP`Lv_&cw$ZS0n5DA}g zw|pSwpq#_;mle(E%_8X)0`-vlBho*l1=d86MG5<2(B=1CBqgDs@AbQHc^mPwqHjf- z*26ts_+AI~o|s+GR^4_uKUYm)B+w*|C5Y;=YBA{6Ve&scQcS~k{80vWtl-aON&#nw z(%klrSC3|DE%1qDOQ8t%N#*6~VUhc1FP65x;h-P|+8F;0<>|f;o(8L}foSwnM290L zZRcH6gBdvj_zb5VdnHU7Aoi`M{Ze-u$NQ+!+;ePNE ztMI+QI6kjE{I|`j_%unK2~IA)0JP~s65snu2y)G92A0#1Xp#UiRZEMvxvELMasKaUnQb#$>o@34ukS-hz4u1VR!yW zh^$?dP=iInU?XMRL~dVX!9s(Da>@E+-q8J41*$k&AuVT`Ut0*$!?EVp1Gmj2;?u)b z(*Ep&%5;%3q^<9H(+KR-MN5EOuEq{NZ*rkx!TDaleW7jxro=m%)Lh`;Xbeb*IwFWa zT6ap|^Le0E?J?xYbqds4t_om=x@UWbK_@qvAz_x!!}|K_zJo*RCCFIpnCr8y5LM^X zVtr|2+2>8?#It^JZmy%tbi4L3JB;8%;WOV(_kggJ=S-R!8f3*gp9N;+S|9C-j|On2 zI&%5mMuCLE6kV;sn4Rr(o}~R^FJie^r;1kM9g?tJHbnAAPbwcmP7K1!_R46X{rUVV z&ZHW#Yuhv0t%}{x(@d7DA6g$br%ZtDubvo(B6*^Ebdb7k@Wq<_gVmrdn^va{^hUP3 z_vlT>9%rn!8w!OnxsJgu_4z8_hJnXNSsza;vc7wS zzB7~{va(nwv?8kKJuvCjB?2FuZE{1+fo+dnRXy0^GIFTaY!Jbz1F#?i-5v+zNhO>~B^Oo9 z9b6cQgv~!1wA()ps~^wT!C(v~2p%YqSrs_Fc?g|3iskg@I^~1P3-?~tt2?FBjnbBs_>AGc4#@&?*Izb;4 z?h2y06a(fvD0HjLl_?WY_qDz5piKGPEpg)-H{w<|*sKM9$-%DBYzfkgKi`FZ`mHRQ zq6H?p-d}3EgY$CNi|$|>993Zqw&B3_txM}8E%P5_ zV7-QhhMM>A;T6?O;B0H-%cr|huvjc4Y$bC@&gGe$0h08|^gc#tj#sObIKF-tdK01V zbbU{S)>-8BHw&4HhPNad3P)K*)*A%Ok6Fqzs+E+ zZp~nj8cseG&|I!q#AW@~IGEtR6YYmuB3QIH2aAL{I~*$Bxc! zQPGlg-FEvD>3q)1&okhszf@8O>)ZYV5VhS$%0(V0oqa2B`EFNNT~psS-=-@u&Z`UY z@IS75n=MziJyR{!mj{8G;wM=1GvnLV@=7*XZ;Z52gUCie zqB0YHXYX{)>2x%yPEIct!Psbgl|N6=1_O$D2m+d&?KGe=sw+{r3wu{UF+}Vhqovuh z@R<9KY~lQ84tWg=*p1Xp=TaIEcH0$`cRyBa&cbs)L^~VI?I1ESFx=%R@XLd)SA=#`+o2MXJ`x7)(vi7gYJP&sKQP_Z1lof#r5AsCnJY3;^>} zPRG*^Ucf0i^udP){+Af6Y%YU$^j2%7DPH^!T}Qi)KfXTFYB&FItHJ09UqCQIq#^zJ zXd;!$h0suMNf(B9yXgxFLqblDDg26S9lpd`qAEQ@==#f!C+2@kshCqJ|6xqCjc|rK zA?A%P)u1kcr0FP~7qM8E>`JmSpZ7{%vOa z?U)==UGehsK|}jRme}KGjvZ$v7wo``;yS#yn7AL#coL2oKD%m0RKL2re2?L-*<)bK zBCs_N+kiIhk8093eNFfQmIDsYK_UoeI_G$$PGPiAl(eo$=zVUV$cj!K$|fJ z8N~Vgk9_$|mlviws3O9o`TZ9e)%Qg!(sWZN46?t>QUM+E@0o4QZsvzi7`-j*q=kCL z8}wRDsHkWco2#b{{SXB0N_&%wO%qHbDHp0hkIii=Ei{y>H%xs^xt0{nt_p7|a0BlM zIE-=}>ue@i6dlw^1fC!J#FSZo5U;WgX`V>j})F*@s z^%ule1$BJ)45!hAFXYurffHD38>5677d`C!x-$(X(zxrA-!?B!WV?r-oTeM8ay&T< z<8N-q9M4`L$#MJd2bilwA0O`a;Y{?;yLyHg^#`Sv7u%59UN&5<7TVAW7VA1IxIFNRFwp=&k)kXQMg?9@r)_r#K*tav@D}69h?vn(02JidEQr=wQ47dV) z?9+bx7azVm79J)H8bobZuGO&%?V1@Y=1p;<69mXBV&zE3ahb`LNN9}w;PT1cNpT@> zaf(FWzI&Of@Oh_RBly9Dw)b-8Xj~X`{DmYe3EKDG*980|8BQ_V;WBruz2>=%-CBz% z0NR)K00Y->e$CLRYKoM{&X$1(c&TYp6(?HnrQ!3+ntmnz4IY&m+Gr+$@9`R}X5ZDn zA49vf)_ll?nZZ+K_p5!*oeIe%LIogzBWB5B-Fdd*!QsSjZY`TBWR(K+AA1!h@V^lipilxd4H^{8;P&AY=hXd9D#{~hsJ5arEl-oACSM&DB@{2q5V(Q^H*F1I;Lo_H2>;h^nlXMVn!KSw z$#rV=k8($$b8evKcG0>JQSR@N3zkS8FX#?3K#Hu`4T-nOuY{9Ziau*27Gl6^sLBi*aEQYBnF}u zfuhJWu7tnR>HeVC4Y&uSQflYGvYm;me>91(_x&la72Ctt&|sTMMON+n9C6=DdM3~H zZ6;x4G*o8USumRVi`eS}WG~|Dgd3fDW{1b0N(re=>7@~L?0Ue~h)Q=YFG16h>nw59`~Chxmx%DcFamj``BKbI=_Y;%+gd6ZC$ zZxSEx0fN1`YJKxB`6}2?iwOJUFXXB%BObqXtSMYPjWXg1X<+B2^O8aYql_=G;aF(R z=0uzACEsEl`o!z0=WcT&743GR3ZYm@B}y@Yi0d_l%cPLj$=a~Eh~m7my_Z90bMoa} zw~YzGPPwy;X-J+hgAd($lFH05D}-lqA8W%?WcRZYGo6JI5>`rqsMd&l5xgssJEz6` zU9YA^yc?XR$j*ZBvq##BT}D)BcB5w!@@|%L-^)~+ZF91g@ovD~O;>;R+F~*1pwiLL z+-IOc|GIy#xQ_0-YhADV#MUh>Hb0ac&7t zqvOH4jr`-3r>jKv68SdeOLj)?_?W}OblqRz3a-q6Z3j69GrAT(3S)%ZnddCuQPQQB zi|;(4v?F7!xTiP)_oOHet=ftmqyV|jr_C}s?q>0yBF3}BI`AKoWzJE44MmH>KfTlu zglAnn9aIC}71~#8Tw?}Esz$EPoXKAxa>l`D;eK8E_Tg+StBoy)ftTNK^hwE?sU|mw zTUM4~iEOq$$ez<>XYZfA048}l7^|uZ)!KM8*lfvlK9-=`k$J@jF-_)dq&O@55CMGA zrh@xM)fA{Gr0|TF|Fc;IBl(ci0>fHY*t~LC4I<50q~3>tiMrTo_M;*06d zh>|_|t9&Hp=H+zNH^Y0euR?&*>%3yC0|hi{o*fZURily%m#D9MsVvr!a0xl@KeMz? z%L>2tB2=`I=nn~BWJ|5G4yz`ArJu<;eZ+l{90IkRHQ}29&QbT|RUCrru7`@vdIQcn z;NW^Vx36qt-e&o!>bOl756b=uNAxE_ZSFLTUnWkTCzFw0`-sbdqeCnbWVSB6Q3m+3 zvZ#uRU=^zCGavPTRjrtEO}b%1RXHP74Q+ReHh%}F&4^96%I^W7PgZ>6Pp_b0ORH`6 zqhw_UPa;Y+z7{(2vLO6ZY2|jxE+@~2YUQv&-W!S8^Z3V66J4j+mwVwm{pIZg@>(@e zWCG{GTsL^>x1o6Sh0V zSKLW&liV-x_^4c_)wwLzZJ^Pk;GzkP%!>35LsPE^AC#PC2IioO9~&1tmlp^5v~50= zNw`=p*@L$QoexWKx{zt4<3i%^c$zHgYyS@Dkbd_l53^4q!&MYSPoX-zR*$DunIE2D zfpW5;UMO;`Z7kGMa?ckbh^-ygtsSP-P~fcv*l3Iur7NJUHX$S?ha?af`=99MlF#ES zD*3D|=Lf2=UNJ}H6zKpoPMb=CQyM;_=NFbMDK3~F3Y+$TME(0`5{vyW26WWI&i+Y> z(d&9771J@~xi6ek-||^?t>^OdzU>#vu;3oZlc<`1j*2Sk;QuwVb`ekx#iR0j)HYXO znJsqTzMiHTzCBrJ(CrfVQMzkyl>a`FRV+Q|)$@5JDtz3&bnEa1&RjCWICV}n!Ysjf zGFKncSz)Qjro-xbUAa*YS7LF7YTciHXMY8J$iGWmVl%2C z<8dP>?$Y-$Ouyoj#+X724liQckibXrVYuZ;#k0Gq@A^&mxUep zn>1J&TFWVYFH-uX-(dh`u#k(7C)3*>>0NElGSNoJmT8qHLoi3TqgLl)O{s@ir0Ot{ z_>1JJr|||A*pksN#ikgK=K(PMJ`e9b)+UrOrAOF8?T&c(e?`+lQArM}OR1|SnPkAb z+w~F1woU2?QX>cpejv-t^Y0zBzONw4`3F@_f8$hdxMLB~w zxN^aKfMkZwjFU2~U|2{eW$B1VCgOWf7*@~fGt5X1g|fjBkF=u30b`kBnZ#mD5(uZ1 zOTOaI+}&Wf#PFz8gm0n#6hxC*5?UdNL56S8Kjh9h%@VK)R}I}>?LVamEW zMIlbdhYwa5rk5*ReZO~-Mofq*Ciba7$qg8ii=qCaWDZlDmsAeb@W`ZW$A)DkklrZN zDoww|B$1Bp*LLtN6k`zARx82O;z>EmfbJQ% zV!iuKg~Lo+E?wo9t2vcgvFfFWP$QteSClF$zLUz#2YTJm?}&m8akQdyB$wE|Yoc8S zS!zjw>ufb%Vv7q?y@GT-m=?UmUpDMw zU6PG2FJV@t=X+<5&Fyj;`-IT2RZh5K{J~8icK(&y-l30oUUV0(Q1B%TWzER~s$8q8 z)xBy03*zSX`J$mc+%#@gB%+i~NgGhornL{>;Q7dJ9M9Zrq5TRRPng%Z4wORb z(74_amCOxtCL@}HZQA`DPo;?VSiPy1iOQgVbL}m031->IqMc!iag#rnuC$oB3ud$Y zolvD>CBy~*lmwtRmY8CdQ2*h~HB@m4iC&z6*Ba|P5YAN#PxkuA|Fog@=k0lb!B45g zWtSp=tRvU=23^lpsYL={QMF9BcF|pFBIupaTLo*cOjNK=UBHL8F##XMwCHXu=2$K- zim-IqH|TG;ZunG;8xx!BEc=U3;#+%CyHzn@%KM3KOgwJC1!r?&6S*H6*X$p(Qu-Z- zc?k7w7LUcI^6EGiud0GQDZ)NK=yw||= z2c>A7oTSQyCdG2N)!xURaF>T}999?h$C{QH>0I?Rv=e{03`t2s@_4|hZDq=;Zl|wV zG9Snly*UyfQ33fKX`;7kqS{C_<{}g0BC-}geY`syHwRiFWuMMiNdiu=4i@jk0Oc$* z{*edF-!Suvf+uN;3*24~#K^wxn6zH*!tCl~&NX_qJNCJd?^Q+isfU=}Xq6MA^02(y z%k+lnnQ99AOmlrIri%;Na65(Zkh$vS7WvE5!iOv5nOf@t#3D4RCe^zM8EJGcDp%4y z36#f~f^>`1b|H{D3L3>@0-Kk}XOk>E5x=9LCwKAUZe>!kSf;va|9zcWBo!CGB1K|f z7J(Tiu#NiQV3K+<(&No59pO(Ej)U66Lwjw+Qs{ltPsl}HH#g3NRLp>XXp))2NRk7O8btT`$TK!wkH17O07^0O&BW!S*Km zWqVs?u0spuzML+}z;N#gBA*PoCN#A9m*!!sqhhO=kGRI@f+Qa0^O_{}+vnHRlldvG%8lkRvqP^cRrfqfzG1+vXZ#lU6hZ4)h zG+@_m_Hep8#R|?Ri5am9AB-mt=T%Hnih1xp?yb2deTz!8Z-RRs@dXAjkfPX1$Or+> z13?)SReAcpL4$nmGwxf1L51{^cee%XFIO%GugJ`dcwz!@^I#Y{@V5zoG5+Ys4C|N%mIB4@f<>P_sa)C zHd&<`E9I>U!)e&1z`3do$|xn(@A^5AWId5IJr^FTREVX~*6vd-hO1r6g(~;X@O?=iK}5TU6a}BU`O}OisP(o}Y)@rKR{<%b34D>7SLY1oWcOpxQkIGc^ z(+KlU@dZ1D{q)+CtBx;Y0!5`vv2v#MTGNu3g>S{&eTlVJ7_I$HYOPV~s%luK>+Q0_ z=q1B1k71i1LPRKFx<~=+S*;YQf7ze9T8KI=*WWWwwgp{M=8t8 z;XJ`k%<33uscMc^Ivor8Z&cSF-!innWPXY(4%Bg7_8A#Dd}86Z7^K2d6&n?2)4UNIYVF(YRZu_?_++SS5*8R7uh_BH;Bu*-7CJ zZ=d!4!@Z=@ccAQRipd#VDS|zNWG#aBe#T7iTKbcb^F}lMY4s)A-gaBH?Qh#M7j&k( zXvWse8dQQY^;`yyT}Al_$#3r9hig>Kq!RSq0zZBW&*+=8Q+S03-lyyi0{=AFg7r!M zN&~wG_~lYml8u=?jo^(X<^F;n_&SVh0@>Ir*Ty)PNliC6<&Q1A-Rq4FzyAmGQ< zSCW$NQSoep=<57OmRthD0?OyL3>OzMq~q{q47LLrJl`_ap3Z--~e|ofGOIc{K{yCtZOmH#;7 z&w2mL&uB3?xBf;mu+dzUr0)M*VFOwxkk_dT6&nkxdT7sd<0G^N1>`^B^L+uuNn@g; z*U`R#@^&$#HkY&ul6t%8f#NbQgsCsr2Q~Me(@^-F9?N+$(pi9xJde`fXqcuz0j*^+ zk};+H41V9vJev|)#gyJy02G_mYLH={EC=9VXXqlC$s2jLbZPs#b*mFH7}*dwYS-7j z{5uY%B`L6Wr<{1e@7?EGhtF+4Ia6G^M-CwtmC+#@V{V?XQ!=YRS-W7KA>*vIW{63v zlX-~!Cq&+W{(Y43j;Hh5Dbn+nprgX~DK1}T(msBglM8JF1SA!ZAz9x!5J6c_zm=QM zDt-Uaqe}_#)s$zKR_%-6nU?_{bMsU)Qkp$IIa{v9?vpy>$0J^~@~n>Qg{vu+*4LLg z%KwkDuK73V){3Rd*xkVxw)>)!Mk7quim;~zQOnr@YCqlvrxlXV@S@XNWvN%k zlgQxtu#C|8^lhu*9BQngIa{5P$y5)yRKjOp+W^K#CxW@rY=q)iDk> z8!5|syg%xc_FIHFn{pJO!E}T+9K86zn$p%+iLC!oZ!Jm4X1${u-o1r?E3MGq9z^AH zwhsNp9)rc)))L4!V3p1pP7ZOhRL@%Ni><`m-1ua@&RE$CykF!$@5i5_Pv;`8(h#3I za-KvGm@b*?1Sr{l=VN0Y;4v9>;Q)h-E!F!r>=ewMA8cOexBg?&Se32EImnkT`_d-= zsO}KmS0AAuUQFM|#}$7ONfo`#U&kg3dAxvT0G{5OezAa_r}p#6G7akDgQK~CN^0S5 zcw8Lj!_9#Y;0r+q&{TnY#%yF9obT~-s?1ihZ0=CMpieS_y}f#*0s?cP-!#RdS*alU zAEm|ky$K$c<`I%=Py9-zF3-;eq64nW%!Z+8bei7-atE=9MJ~ObWJr_A3ha02G+#r{ z-NJrte#|)AaZeL$F7|wUQHY_{8o=8n_%p(bA*M{`T345I9)Sg+MXiY0{zR~A@Kh6k zdcu;EsNqs2;;4MM8kV7CWMnR7=;zQ|Fe`UyMW+YQ$LsaEE|a))FW`k!IDJ#KxPGkP zqP0BUBj&M7`8~o@ax;hos)0*D48FA3_>oRKDEjuNWgS540mK)gZ5-WvkM3(r`e@=Gt(+Z` zPliZ`#f1gHng50}9?@wEy1Rq8m%d`VEP+)p>Q5d-b#^=rU0Wk$0yeT-Bj`_0Pi2me zbemf0fS8}zdg9YG-q-oy>bhJN=eXF0`h>#9hmukKA49fq5KtYGD`ZvdEn)yfF0|R* z>^|x_RV0U=*(<|+u;|;{9 zPv8fZNTn|`C2|PeA@w_0Ka&y~3QtS+@IK0g~mViD#fx>(I2cy$?%>Fo&#hljh>Z$T1#QQTT6ZA(Nef+p7$gTD^7*3uz-eZ zbw@Mlm-KkiXYd#bZM5w!q@fK?pVgNqfO?M;h@ksW?z}2szn-fDw>eV#TOvF7&VGxX z*;Ek`n7KPpBUU2dE;Q4ouG5CZh=wLm z;45Jc2`giABYfb+iQu+DeE+#`c>0iagALYJNfe`JX?FMhH_DYRgu~f(T1$4I(jP-(9Nmvl~_vS~1tqlAzXXHbVPQ|sq-;r(z2cMh)CWN?F%SWpi z8nq>tK>WDw8rnqGj^5W>aL$zpsA3IfmT`KAyczeFO--VMDbpt&H!yq*?A!|5-jXv# zdl0Ry!vZlGI?;ogk~B^aQI{`Wowv8`>m*{sV8+I$$THk^tqetnnwI5QE*gN&2qVVh za#MB^^UyW_!sB`Yq2^`2)%vV{uFS?4Sj90>`+!3Dnkp6vMUw(Edf=s`F~k07yqd&J zyh}i1pIFTri;U_Glx)BK{e2LWOJX}0fNJnHEy zX7^g*?zg7qFaaP2N(CN+EbJ%Aw^7h!yFT1sG_WJZ8MCgBE=gq*%8xS~o_`ooE7ITi z%<77s&DRT;J&m;v{U~`_l?~@hmP^0izxzScP%OmN#Ck`vM3tJQu87^DJb??)%%75Z zoZq(#_;1bOb6?;2?mX=ZtuUuOB<1($$C?A;5{BNpUKX%LgLc)&B=vM}tZheuc(>bd zJSJFxZuUjrP%1_iW z7TH}B8Q(X^nj?WfYAB5qqRL&m4RnL{;%XBy>65@dg6GB*HsFABcT)m3^GgzI<ifaW{#CO1`_K$=>?L&jOKK;J|s} zl|sW|-d9arP!>fKI$;(glk?kKChAjd*Y(etQnDjA+csNr7AqCt07v(Z!1E=AL#d1( zRI?!lg;yONb1=az%yUy+V0;aob%|USB@Okt{Id+;UWm-TxxXig4yqo4Zlp{PV z+j-!T{Z3oj`FH+%)shr`6Y(!te$Q!d}`gSi#x^|ZD}|40E{fAS9>D4i`uM`Yn7()CT-bUx(k`d<Qc`|y zvPKyK!zwB!aQ0!5|Pp>_Eb12 z=bA%F1xV$n0@hr2)Ctt;vdB0&p$-j@asKL-l)$Dqc9uRSsikZt-%A?ce+YnnxM8$s z{($Wm&T)HRVGQ(@DXQ#qbnP8_mewZDh{Nd;?4W~JmDK#!YBCl5GHrBGXKnqh78$= zL&kzw=9XK)P&rv_S?LI0e{b~su=L}W?|#8WsZdbNE7>XzY0)?En=aZq4zW~%hRCJY z2aN7;Uhv^-C7q_S#ahcB1fIB79cc%HDIRhpFSC440S)pID;75fb#F4%Ss2Aad1U*Y z$DpMLKnb17GLBxvpd9z<;Q-v}yq9T#MdWu0pgtoX7B`!Lka6lBS|RIKPY=@2{aXs6^;DLK4KMtb(V1X3eso%S ztkVG&oR+Hith}*jVFssfH`g*d%YufGyn-(tnkXf);=L0RHkncD*%i<;xZ<+0f96%-@Y5Gq`JOtRvVuQM6xMc=xjd#P}~ zOMR?)NUkC!H4$foJ*1d5Cn(Iv=U2I*W^%W3lM2b^x3LiB@%8X{^&o$MpAuSB zt0$F!)?pY~tUWElk9bp&^1DIISk3?N_U5mBO3f?6`t-5gsqsfd1N~A(GdC9+@((Xk zgbslIH$5@*8fmECWYB^~Uptj?H3A*j$|aF$UYchpR1gZW8&C`!E7LTqeCqm}Qfa^1f$jGT28Kh z;x1_i#C{KU@UGFO|D6fDG~l#au4a2VXnB_OK3T-Yr#y=l-w9?SDXrAMDPF+!x~_X> zDl5%h^z`%$53Kv+4RZbEO*ZPV@zCgJ&{XJdyr^1iUjtsT?jjY|s!lNqA0YK=yOoc? z&Sk`ML@_FV@LFh5c98A7)bhUe5A#!sRF;Ma4-{q}jxxaJ(<*xo*^M{x+KBBf(?$&9 zD)p5Q!=J8Q{@eYGjDuBeO`9#W9~um;FtZq0!A|~Ad2)KB>2?u%ii`1Qsox?6Ylswq zZ#zK%Ee2QIw0_%bi;HV2adTdRIAOvdoK|^ml22-e0M_gpY3k54=581GnG&T*zi}=y zqH7?+PIu5*1(4Cj{FKE8w4Gf!!y2J>m1@yFs?Ao-e!@*bfU#AwCH)db?}kF1+46f( z=T8cinoz%Mo)B;pvrOe-oSs7)Ksr74=pHut5eX;nu)_&R_0mE#`2ov>(st~FM`K<@1zJ@ zh*&J||E_1vL*SuKf7Yt2t0%*)x?loP=3L@}&C&U2!9TNig7s5U&;3VfuwyGT*yrAV z%}VtN-d{AKTWn6D^jm-|5gvj26y z#!-Ui^+2icw`sH}J8{qQ@85zZUiKQB^J>|4WVS!Pz>;Y+C{muc&{}Y_T-D(?M+Dbe z`MNpUs*Iilp^#Xoe}^3AariXceO7c$OxmS4EfG6b=E>tCso^FS)-M+RK?s3iOu0}V zJMcXL>n0yXz&bl@=^X#T!TGZLO-8&Qjuqvza^^*SJeI=g-49flm~XZVzkUrc=FI2` zE1#~nfkBSdD?m?YCTDu0#Dlyst-evRfTSp3sl?j z7Rh?!Xmx%xKQgh3mufd+G&%+1%4l38ycfQRgcu^3g1C{(fe%HO6$6k-|nwgdPB`~5FJ2}hC7yrG+TDoj*y zYb`3PHO1}n3(kRb9>_WRfZ-x4UETDttixS;P48MPcNVrX3>Nsp#>$<{bM(`%gP)35 zD$EGqQz=p?#ZCQodl)iZEC)8&mBp6k>?X8kMb~F&-cge(k|riWSPq+!k@4Q*=X5o} zC*};-WWtsDauM3Yw~dayUnc2~b=*Yk-)Xim(p^`~KwK%O`aDCwPn5jYR*$b8{rY!@ z^)`6_$44I~1a}v?TeS!yTmUbi0og)N2hKt2LgbD|Cb=v+U;w zk<9DS<0q+Hx;lVP9bYzpV|U-xPm!-NfMvykX;VvAi|uRMq?svd=EYE^E~=%%S$Hdd zs*M2GY&Oh!K7`QwGxHV4`@^DaW-*RQvCX6sslgTUdWNg7kdUCIsK#*S(Pp%sW{oxQ z-#B`e8LUbc`tsT-4oZE!e@%Y>+oxHDi0bZ%r-fRZG#=zwzS;aj!43<`FBl;nmOb3-o$;tH= zZ$8wCT!7vl&7hZ;mnR%-0GYp`&}V?os{1XJ0I$o_RJ#=F>#*d~`OFOkNqqo|fxT*{i{z`}0kn*Tjaprv};C@anCYKdHvbd2ui8TM-|C@#)%)Nj#eVdgXosfoi$-Ox#^|6l`{#mYgkT z=PChO0pBvSQ8Zk&Fi|#)Bd;EU^X+;i{CP^>ZREU8+rHUjnzf>kMuUB}xKTLjf zmIrK*^;hlXe4S_d9u;5R zmR2aI*DB0<6V4S+w~y|wvAibDn2K%H026fSyS1u^&7*(Xe4+3ooHwwp4$aqHeRHMR znD6F9vOhCf`2!EH$IbVvFPC`#%v=W8pa35+IYB+Xt5;^Ry>L|!Z(lfL6%&JP%v99- zhrR+nY@v)_4W7smovayRNTNW~2CXU_w(r zp=;C}OWh`Ft-B+)SbH@%UTsw+Trw7Q{@apBbO=`KMIQ##-0a0G4h9f(OF{WI<$z~kL|Z%bh|ofoXJSk2I@xL*c{ZkLvFAP;7X4G#}a&OG87 zj6s7jC_@2hM<=h^6cV7^#%0ShFYNs5`~{k226lkW;`W4=95>qN=D<+P<$QwYiO&fJ z2?lN>Q;H8nMSQ1oNaV3*C{Qj50(L9&4Y84tLFIaiyq+7RoXZOsfLOhoPygef zI}H)5e^w}d3>4YYwa=4AgG~oW3pG~ZCe(esX61+Jzo;(D6yZ$ zc1Nu8-LmX!fo~=Nm@T|vqVe>p?bYUrLV$NhetPp?Kd1rO8y0}GYmq(K)w?iPA{xEP7Wi-1i-Y@vP=_IYaSbuiUqcEBAZ$i>;%r z_sC-5_>DO?uE?-;zHR;?nCaSz0OCim=zP)xctx2p*k*z&)_y{tv|lJ4mg&#p1FoL= zWG}$AgvH^6!d#T2)N*f1apeM(21Aa(EGAOVm|Z;iho1+gdcZSY<}y!JWwrvr)9Az) z5`UG0vr-Ru2iA6XLu@lhRqk0F?o3EF-+$l*k`GuH9%@|XB}mE1Hqt$t8C)MU?VC(9 zADCYjU~K=chJ;)&M@!AnFP@$*8$Tb<2g{J7$SIKnNg@(QA5ZJyR$sW`O0v z=hfZkDZoVp`3@qOxVij1MxLxIJjLDSM3Y8OZv+?8k0tfSQe!=u-s_WgdjsdPhG2?s z10)>cA8Cs>M8yw&p>w-LR&(B~<-@OCy7np_{gZptK-Ku;6gUVPaBQ)L#I`(}VvPhl z+t`e?f)!dFC(4psj74X5`_@&~x7cN0%ul(m{I%G7L&(DbC187B0g%qrJ)Zf|=A{Gj zXxAWR*-{yJU%Ka%EANK&jx(|XQG}J=O>YNX)&N24_Iejs7l5_rKc4T17LxS{vmAw}89|O4kFH&>W0dW1@C}3;KT% zBzUUMsyqf&$*Ft=mz`Eb0ej!3n;Wsa^FqY^XY;(d zKqjA@sIAIGb*M8+?7&y1wSDHeQMUDq^xfm*`q8}o-QBBa+x(u%()?z343hrFcBAK} za+g+vUtHQ_ME%O(pk#-Tb;rQ;A+qBG^TX=wVv+D{(}zU9pZ{hJR0OwencCY(Whk5M zkF|tt9Tk6n12X_4UADkDvF_c|VVdopO;%sYYhoT|m0 zV}wgm{8DgxhV(O{wD}1hp`BF%ND_N1er(u4rRy(WfCJ`?Sok~uR$D557a^%e;woFU zt0z^gxuIIh6EHm+XGrBo=@GtX5nX8)YWegt3CIQ0A4JE)!lEEz4$Lqa@2GRwXOxJg z@>zck6>4Oqm8+4WbaqZVeuEBxhj&<8^&XyNEDaH7->IBn2R|kf_Dx#gDbumsu1V*> zKlu%Zq-*2lVrg5_8nrscOijrU6XazUx)QwYRptgKp>8Y)3*1Xi2N4F*NKOeIX_urc12h6(ra_cP> z6V%Y>GM|Jy6YNDNBR0h^^MfSK+A86N0(07Q9MQHj%}0jB+X*86e+_ZYuv@|jIqURF zQ27^&E$jpI)KWRsXTUk@*@JQGAMMoPwi9@;|7yloNF!0V6QXL;bkw#!Y$xD5J4r=w zjhs_V#=QHBZ)t&@#28ET%Vf+;Ovpp|j5O+ChtrUNDT^CFx{=3~K7-CX89hx?h4&uE z9=(^giTh_0Q4#75m@lNF$pelCd4Cp~Ekzi`~>` z8D9=qw-e<5A^U+v_h?sT=K25Co=ES)x0DdVen@${u6j4*!+*N$Z^!+QKexESQt?yu zKw3~5Sa%&Gc>Y7O{%zRBdN;hdNyy`=99H80?_0uu*W&kNk;w*v{7ld`?>%^rF#gjQ z)=9Mu4u$~1Svh|*0RAm_{-^pl(ifIA#|30}4Zp(tzdbJ!4%XyXY@JJvdcS`}xYpnC zRYt<^Y6R_@^!J(di6DmRC7$1oxB=XZ9M_*6=}^Wg1|k*Bc2It|{ma?rM0Po`7v zs6!5Pv)4EGV^8am{@m*=XE3u9rA%unP2zX=?z+e6>}XfF{ME!Q6l=Q)AI~)DN>Y^%Rdx%fig}jH#Xz8oW@M*u_{%N25 zrvb*WTVe^K&4#JZoc3QyBPqc?_kU&vF%HI}-ZcYbDBpe4$_>QPB3N6Oc=UZnyu9xc2P^>L5iZ72vYd80okygXkx?vBTydgl4$J<@bvE2%DhuhOq{wjgWw3<;2g%?A zW7_aOs>9B=+25CeBD5PSuekvY1qDOL2-m~6}WNwhhMRPyzUXbQ@_ zI_YE=Vz=;m3VNo^$pz$MH7_8L%-;WCB3qSl9B*aNdwa82j>=AV3JLg%zc^aIZ9=xB zz#bk^ZSP>-p}`2RmyIRQ_kUX`9O`582nVVc9C@J7M8?3d>x^}rAifMlXopeNVt zc`WStT1#V8LGMveEqZEJe1w=_8Jh&K%ALVzL#N)-l>1^Wsq4-Q5 zsJf0wBR@Q(9pTAnH>emHBp%?=>$JcV5LlyOYGr1-4_V;@l{peYnm6DsX7V^p&Z$l2 zFeb7(w7EpkPcZdv0&W@-D7~%fU{VQ;>mHFU#R(CRV8;1oDN=2ABGLAEBmtB!cxLPx zl~xLgY3Xw?NDU2mnI$nT7H)*qVn<^Y8`aF*y&?_A`f}TPc<1 zE=LP6794J{j~R5Cie-pTP2A^-NwN<435=#`m51CTT2d7(MRq2Dwt&I79R=QP7xmJV zDmR#NtW@a!D97xpe|5l7&621qY1Ez|sI z2^g3eGv3umuWFqEBoI6K{*977VFb*MYB6>T`yY;1*He_&1Zt7{Lb)692qHkOzj@z> z`~on$A6R!gNR@jtmMQ(hLb$b7y_5`K#x?M2Ja3n-VQ{-;lv+(w7kzo?<$IGH4e8Y3 zSvNhxqM)EKx;Mff7J|C;9Qk9<=mT3UpJaiygqM$*99WLw^S4KgB$JA}d2lvB$sdBL z?E(Gx9&ex}d|~-Brtfz+P+GJjLUd57fV2mzdfjRg#F)X>xT9B97oZDTU zInVZIjvh*m%m z^GD4gdUqhP3ItW?9(9@a3}yiolw75o5|G)!c(Om8LY>HXZ-LmG+);jMJkfMur}v!! zqLxC9c9_$Rn1Mkqwci0qjtIlAbruE!2g?6T^}^O$1l96%qC%rp4z)cP!x{+Suk8S% z0t7W|kEd(lN+olmdYl}D17Q{-Kbz121F!i3ZO~lQz@-YJ6H>&srCt#)mHt@S*sw&F z3nrK6F7ej3<$myVU8aWd-NVDzuDu_5D!Dln-7O4U@-s+_(_(PopM;{e6RYps3t}20LW65EwKHv$_4C6=vrM zh;(*t+aILOXR<)arIRel3oJF*#O5cm-=sPVTPZjXZ#tjdQJuC7!@}RKTyF_DfDkKe zHfx~ao|wrzB$-;9<`O^sejPiDrKfKXqKMlD86EENx0xKl54xZ7mFjXjnqS0w>^Pjy zq___t@8%Qt=3dB|Ej4vwdVjc_#MX{w>GH_^21K&!=Yc1vokyIm(DUIrVF#H?mpwZT zl#{#hyBg(!9DAp)H(KjPx{6=D`K+uaH>lL|g2A6t0X9R`K!YvMxdVvUF+fSRxB=hd zq%VzOhT0?_{#3YpJNP}{{q#h)>HBQIU}pD)*0?jp_1NTT+OO)~+pLSHX?>Q?0r5Ly zK8{*UN#fc}3GvT7G^?eBt+fv+vQXAx?M-aE^$qN&Sa`dJr#pVr zsnw(!3)tc5uruE)QApCvdh{(+Lid-O(~FzaGk~WQN%}P`oA(uz$$D}Xymz(H1Dbe? zjUFQmcWU^a-%b{>R3sIh7R6|1kuwyiN10j&#SF&Pn}lgiSE`5vg}wt0W}2UDsdWIW z0s#%3uxm-LW2RX6Ef#)>lu_1Dt&(APakK8)?wL*P#*u3ToMHyAK6wg6RigFoD+?Zd*mcr_inWL+*swl z+soNgUWUsT4n5K_e=*KL^T`Uzs#dDTEA8;x**+zzPNkijn59XV$raa@4^gr2kxH%LtNs0Fd2BKprHJ)PE%7;=>#QLODBZ@AbGPu zf3D748JmCRE*=bzSCI1~cHGd-uDwZ=6Lh&TVJ+xiTO-*|W%eHaro@KG<{ z_!YnySpqCn5N<(4$AhPys3gavsJwNPv~27BVgzk~ed=om>oEFh(#!3NC0fd6_ub5O zjfRx?8drm3)LAHlDm|(gE$7e0#@F8zf-1pZWx4<>s{X}Q{Z2BKv9+QJ(xi{(L#{*n zCp1JUE~)s^Cah^X%^B8)x;ja$n^_b1NB7WsXUz^b^YP~>vfy-;QM$W_z|~>6xgATc zv#ZT#AfnXj{AtCxa!9!%w8 zl`#6`?n!K7hCq2RI}v`HdX4aNBjeaP*R19VIyF>WH7;wyX{1~F;ox)-1L|lR2AFb2GUo04^AQZ4*htGk8ghb2BT9Uo;ua`;7 zu0u&FwU4Z0VN3%nk0&WyOt)kI=?A~~zIzi!aw6q*c8E}nw7lA4;S!~+ah&2a z?p6W0MHD`p_AFb-Ugv zUm@z^yfdVNIDqMO+?If#4Z8q2d7b=JAX@@Iy=i*`hfCFt^iKQBZvoi~I^78S6jtH! zrVcfGxo%C7JUnWz#u#g1E*SzjwJVP&n=lQE)b#iO7Ip|~(4lz#`pJNOb_gulz3)B8 zOU;3)9tUz{NMi180QYlSVY;7RzhNX@;SkP;u!b zY(4D~66KVxkcuIjg-QTb_0E7zgmVzoGg2qbovv`OUFrErnxp zNl%wh(H!8Dz64-p$#z029ysTIeYA&D4NQsL%q@0<0-c`vLaX1W8VnT{>di~7Qr~X+ z8i6QCgARtsXA}r|UNGk}mCtF~ro6{Xt*r#BSW`S1#v?} z_!PNJ;WOY6(Lf-T6bar}RHfX8L!vI^IKnwDuJRZ-s@a7KgWT0pYxP!s%m|Y#)N_`f z`K{ru+2LMe+u1OnfVt8@WWf{8Y&S$oMuE?5f!5xDgE0#Uw?#t3S#vtKiT)1WBcerQ zS!FI0SMV$po3SicllMCtXj2Q3Lx32b-wfyW5LP}ChwVO1vqm9p?E7IEol*m8z8gis z0dz-#-JAHuCdWXt%kLtCihVNy&a2r60b?MJopA3QNeGI7^Ca&tCY|qZoygESJM3;( zJS-ghbc@umCdyW{%vzaYYLJ|yqX-FMvfe{{+Vxnz^ypYRzbw^f*K3y&8NEtY&QL|> z^t>Ql#gz<1Bv9!2MBdCK_Z^yo+s>}icu!^ZC6C+N-fjyHW_w+UlF|xht1tQ2O`uL@ zWa!V8s-4MwSB16abz}*Kp+OiK92P4^gv}Ptp5Ucf9oH{ffq}FSAU(#?@hiG%yMSSh z!vE9GE0k1bnGX33v+Rl4RR;4MCAL7}!hb9@Xh5;g~@)gq|?G+N&Veo8w$r}(h z)Q4u1pMnF_Y}`-GKuWln>xd!TNI}oi%!k^J$nDUD+n23v`>E7h@uQ)E)zraWxw}}n z0&x!lKFv*xV0?=~kiwe+xfR!J2{f`P$ac1aKth63IL`Zick4WH+PN0}uD)|B3!0nv z%RPq-F1NZ~MGwx?Ypb;jCM`+HaA4p0Cj>zb&)9U_1+e4WBrl^^o~ITxCuxL@fbZ`P z&dRr>Bsp_i_m<8kkWt0YwnBuwSMZ_}pC+C?dNd9D-bLhrN#z3HqhUF!5~|XrGC@e# zOYTrT1{M92n*ER*A0V7hZ!>Eeui6|Yh09ssn89Ka$gdJ@HSL;)5#i(a;&;}SjpX$} zLZ?&Y7df4{x0$-4+IUc+!jFpMExe?7ZnDL<jYVWTHAg~vX8e;x~m#gh>ofw~CZ{+xAf zkTBO*j&26`H~nv0fGbc&7AF*gM`+$&$}lCYfi1hojFXzz?lgfQs;t@$IK!wpoV-V+ zs;$~iFL`+WsgV)3wkaj$fcxiiuXtc=?0Qsz-Wcd-e9-k%SJp&lK)B49SE`pFvUC-w zcKp|g$lqiB#I*nwL*<9o0hI(RI0J)GMl3)HGn$7=qv^FjPbVR_lyZxPe>Q9k;CSCL zdM_U!2Ph*BKo5=F{8c@F3rio!Hl(VU%8rm^QKGere~tRNru%q=nliNkhOu{62512) zQD{7Fwwd>?Vcp)U+r|PJ0#qa&kX2n z`9Q~fDPaiw#8{@-+xq1gU(vGaa3qQ>x*|2O?Hy73#ny;Ou{yJ|AedjhC{AxWeMB02 z{s_?%;>IWaCxGn85z$s;2Odx@}v#FecNj+MnU zG}qAwi647g#wHjSzy5M$Z&rQ4F*{ODPR9>-*j>u21uOpCbZyhaJVKf zUhqn=GkRyRZ39w@X@x#>eg2-RsJz84sw071>A|b^2{{WToZovHt~zOrJ+w{9)G_p(scIzq zUazj|h6vP};(Qt5NEKh{$0o=^P;ca@2i7x?iLb&`(o_!V#@w;QXmyZ3qo#{PfBAE> z_DQ^9paBua??;zI48JC?lh)i8?2s?a3uNDzkxz}l*6^4V4l_-+fKGxf2Lj%_V0;XB4WW}~C4@ZZv_+dl+9#o38O3~nUxb(uHO6k@}p<3Qi z-IKtO;zDR%5d7pI4?Cg8LSe(FB_&lIXQH*AvQ9Tux0xxj$a53z$Bz&_H&Ijb%q^Y2 zXEG2c2=fx~BT>kCT2>(t)95s586>N8n9e}nDNfS2To))kmr#? zpiVn$dn8uKpGQ~~@o3ZAMhZz-apTFy$LD~}4?bI!2p;Z6X5OsXy*6&@p6^PXvLG?Q zp9-m{uvLvmiD5NmU?vwjc=qb?1DCUzSaZuu>V+Ma$q{H!b26e$ADRnlTO-Uf)H*my zNA+)dB4Hacnw+)Rs}Wp!_hsPQim=f3_+-WBp$gkHoMb7vEF$vV-kq+#2hR#)lLwd0 zhTTl6tz@cRU*q6<+8GTGuL$D2?5Y!A`D~>Dpb{K+JSG77mpH+>iwK2`&UW3vFY~t| z_X#;q;ql8sukMLZrLl^3CL*6r#B6fFvm){g(u(3PI~qv`rC?COz839DAmY`#$9a{y zO!(1|gu4}_fo^8R?^jPpuEEWZWuwSwLs1(6{KXyECd28}%=Uwh9V@N6O$28no5)qM zud&0p!rgYlIJEZ)R5h7*Pf4Iv!MCISZP*erSI9HeFum|Z4VJy8lO4vXdwoRYvjs?_ zAHSsN;Y=t~O6jtLJouE2Gh{~{zZ2U-eLrifIfc zncoFmn=8BubW+BL8I?JSkpdnp-(Cd2h+&*rneQ;9$K+(>Jm96@2m-w!j(sv(XsLJf zGwAuDbXnb*UMm6krrKy9kl9jowSI2ZG{@4(%RrDFX#tYN1`g*mHx0hxUv*2+=C3B^ z4`9tJzU*w6U}_C42mkU3Li>~j_5sWqR-Yzxo4JV4bGLJ)==aHv1(=D|WWY6=^*i+! z!c)E@H?gtqTizYk3S(?yJ8={U2%jw?QFpR+5-guZiLlJYa|^%z$8|+nkca!Sjf}0ndh-cK z#4$uV_1ESsR8Q#gTX-x%OSPiYbo@`oiLzRaJ#?2CL~LmSo-rhpx-b^&vKq$3bn?SZ z*&9koGy~->6a7ad9tdJrA0NHkw^>e|PsgKqT^_ygW%NBxH>#*y`X>cab@$iByleKK|OU*ie9crh5-2CY^Q`$&o9-5L7X22`}Zgn_;X%Ow5gp?w5MMj4;lq zy;-)fs5aji9^3ZOMP^M}tVGr(0K`m|m`RB`#>dVsey@()z9jH=Ol<`kIl%nXE-P3? z=Rm^sa*2_5!Z2xc4X4BUJL`etyB-)p@5Mu$=O$h~8bYa$g03&DPK?`IvIVq>%wx5& z*!wXlgQorK#Z-c#FEr_jEY7xWezi{Gm^L98htNm1jZx_HbG0~N-{Vmh(nf-u(qmSi zyiFktTkWN``*{TTk%EK?DaV+LrWr63R+&Pzf13#X*?`%PGr6~*CSuIb+CIB;?P2d& zjS)SCgthG5H0*dKc_x@+JtrRqMHY33U??vNNGDxGH}?l989hAqy*Hey+Mn_qNTvf< zG8omdu#KcvU}Z|;>}~w|WQxde|LfAN)-r<0q4QS9}BExq0fTAFG1p!pFok3!RA*WB8JAN=t;c*X$pRWAb8 z9RamCG7qkNI!?Aeo8C2V86OpVsnRRSVbxRF6R1_aaecb7jPrbA`! zgxeHfHlBx86I~}y5pssmN*evLc9_Ufa|a{!myjt?4_d*xlp6A%x;-i*hMr1Zrf%JB zv8F_u#4--&rF{1H^SN*ei(pU#fH+w)`_QoMvm{ld`^wztXZ#!Msr?2F1&yuLx7`IE zgw%syE#oe;LQu)-Tyn@zQ|hDK`qjC{UVqb42sI8qc}~Yr`KmBR3xRv~RU5D(I}61~s6e zdu;jKIb{Frbn}gfS(*g3cJuhR@y!D%`9p?=`E729=RU2^6;n(2+>IW$bv9}wt9R=D z1f_b-fvY6ai*5+PhYJq%oIE%aFICDz_}>X3Ivm~gEhWXYz z%Smue`|P;`$)F`ha-dr={&Z{O$~>LWTer3yi|}K221B16L#j#ade}!tLYG|ucngR2J&(1!FT#3~NOD3DqTgh&Z};Foza}9JsWXJt${NyG|1(+E^Z(~-VZcK9 zao&T{^8dN>|36pF0s*dy25IwdPt+BX4~gA=hvt~iCg?o@hbY_JHBb3uzC!N)Lbdr1 zEsFw5>vf&M{U`sB+G7uy)ORX|I%}MCIxQPTyM5)G<0Udl*J4%XH~v7*`s9d~`|;q9 zm&b+bLqFRinT5(z5X*&%<+LJgSHL|iWjva4+wcpEkNM??+{`i6Tez#k)*ge*Ia>lh zpgZokT;!cEH%ENFy0rHDb}r3m-Gx~{^^CXkX{-_)8gViU3+vomd>%qh}QdVCr+XIs9q{5pVx7=SebT)Zq0(t z!}N{r%;zN%nH_(*QjYA_f}gUp3F3=8!2}I4m^#lN=fEb`A}8n%ZLT?;!{fZho4-1r z1`nODcJr~!dJ3Vn<<~zd%r-`(o^c_(S~j8BJm)UDSGgXB3qyVdLfjmp{3~+R1OgjK zyAgaKLw-%%U#PNyl}V0nUG-=D6|6ZDdsn z$EycmA(`z7r&Pf?2kSgoJiXUMVW%qC4v~qP|r%?pr)xjIjzaE zobJ_6D+QQa*^a9{&Q}(JXjg4nKiXxyhUXtU8Dd2M9aBU9_*8t((px?{bY7Q!K6`zq z(ZzR9`+@tLpK|H7GRPrIcwsXf27UI~q;a0$J!8qDrxw#J;_Fu9!!&z;i~({`om(uB z8jG4(yGDO-xhykvS$K-lT?%06%z@ksfaQLJ$J#mvK6VIYkj5e^4Kq4+=vU7tOH zRv6f^$+%u_WdO&gQIp{obq4#PXGtY6R_L)Mdw$^gF;fv$tEx&Aiz!iooY^y3viz6_ zY+46UshvykrmWkLI*2nI`ZqMSt#G(ZafWs3{OtUUa>tH{Bskzp;gJ%bi!7)$^FWHV zT7P`;PgA}&8i@Z1V9;BKEb>R!e3(4qI$b;QX2C$6QZhI7r!)g4CrcE17$jl@`A~1 zCQBLMeBWcgQ!6!Nfg@FPaN@t4HS{VqBD;YcQJE`D+j%MvB9}=LF`dlG2D`s3 zxLMfLJL<|G(03l!@hU851n58YYNuwmz_vDirIkL0=)IqT>XS-0%mjm?P?wl=+2364 z3!MW9A#OW6M`eF(h``{tVbW{MkLstJhugXzd9LKP8V`Xznm@I;+V9H$nsj+}*q_L^ zhGJhlPUUb0wbWfI3xbov6bDY$%IrK6Y4k^+88~zH$vS`4J%f?q0RY~Ro8#`OlCiN! zlV=w5m9*3b-}=>IYqV3VTLoF!NO@u!?FvC%jdV@Z>5mFIqv;%8G>Sb?WFeS_QkQv7 z;JeL_?+eZ$wterHT<{gJd?7D)6`p^z1lmD&*sCTd)#Z0&ytH+i^&H(FseOw&EAEt- zOJ(y8%gPFCOV|H6LzkZpvw{ArJ4#>!+>oG_=o5>P4>z4_0-)r33yqER{q0!=bp$bY8@3|d} zF@@+6ySG#S+=i2C5RzQgUbwM?_jb;xb0&5JXNTX~7)WaqWZfqt& zitYUUnJ8o!+&evW6c+rUnnh+5OFSms0(oKm=794$zh+mQWv!13Shf3nO)xA7*0QCn zfTT52B9^L@a~>`5N>P!12$sq)UM?V;FOP?QXvzeeW6zh!zGyf!L7g=)?{Gs8ai*yX?>myAKh+owKS#IGTg;Ex zEA02x_rHW6bAWP#_Ovv&Kzo3fg1D*YDB5<%Nhfj0h~U<_oHIJlNWWXJN1tUh?$4S5 z4#>lhY)ElQQip;(lh3(yzrjB%N^!r%UQ-*5__}E}d~3B-5$42{;`5z}ojpXIZ_)hm z10Dk`cW@45Vy(|O)sEX3tGg9wam6gDs#k&o#-Z~p_m6WhPCrM_nWCST6S73#gNJ2& za?5)mP3DPrK71O@l_BVQ>JjD~IOLa>WY*$e^cWY9M(m7lsC6X~i5O-Gze%{_Q^=7@ ztp=ZQgPs>hw`V6d5`Y5Lr#&NTe@dn;Y~wmS*Q^(GRqte8OfH2JImc;hMZWASb$sP7 zmXjTI+`0YA+-iErO4WIV@-1(kZj3GZmVTC6%T5?S3-)va~$nx>Amx)oIN zU7#)Jnx;FcbJnZjZ8{gwxy>3VUU2+R_+0u9KiI6YSd;c7u3n2PQZSy8}m}%**je#X6#)gvG*&Q%Bj3*=!)^rF-Xwo=f^o% z$;=I$@hy#wsX`XFaKq>`H^zsGg7BI00g378ufcLDE6GgPL+|3leXhE=tF@tW?G zZV+jtyBj0~q#Hy+Lb|(=MnJl|yBh?gySp~A>2B`goO7S&e!Tzh`oLa{J?B_+yz~9( z@w;Tj8q?58uH6cw<+|5fR@pZp0datukkerB%S@fJUWrO+%SB61pSLiI^0>+9uVnQy z{lY+#&BE~#OD9W>LP8hy0t!kg<`7r@%TEL7Ep{H+hyRhhj=1E9} z?pd}rhV6sSVOl5!xzv6m6ENQ-3waq58|KFrfAE|i*k8kqEOxDCanDj?UuyPXj!YGZ z%y2DhKa{2wc=Nl=A&B5riOrVb$&(IM0z=#ApNCh->`y@ZY7F@oP7My%(HwnXOVxPe(+@1M&&g}M z2>H!`?!x`1So-_cOtLKR79k!qTqJo!Dv$%AJfFCh1$cjy=NPrjaXs!RC^+nxOyi-U zr>Dr#_;ss9ybl*nAoFE>q1^wTO$I*PR%5u`4{;t6JlRD8+rrl;Fe@_1V_-~B_AHaa z%iaBCP#Q-Fqo{ca6%Gncf~DYYE7fl%B|(X*snQTJ`rDSx9M-&Z^TTl2@{5$-Gt90) zo7~dM`WHG5+!~Run8?I8C&F3%Q6x9mL*BFvy5rq3Fi_$Em7@NlXks)b$T;HH^QT_{ zvfJl9n6C&Fdg}9(990v<7kG5jmVG&_j1h_mNvoaB;g4sF$t7<;3SDtObxnd!z^XmT zjj*0AF=I>yB`bK#q*EOnWR-gGJ%brAvQ%IjxgUw&qd>AQ`MTjE_O$lEBLBHUqI)1k zQOe2+F?xr)4CEW(`(*Q=NCAov%C)vu^~DPqPINi5~+y(uO6w~1^lI|d4A&F6!T48O{qL4)Pl43 zn1n3mOP3#i3^slvL$oiLFw4we9h@` zB$6I5>22u)2 zU^E2&L`I$Lz*|k>q|=iB9&_1ibwU3UhAc1s=)xt$+*b@&7jutJdW)y{gR#?v46K+% z{qh|)jmq`;MJ7x4Y|Cm-zUJOMo>~`lAjX?qir(p8xe7VlSk>JV0vbEk)70y$lVQgW zNlMUO*c1!bQT;M3vhh#XDkd+4)g&&dbm+H z{2Rb^sn83&BAE+#zdyftm)1!^(tR|g4fSJ?vZ^2^{sE(lkDC0Y6|X?3YbP>}8>&i#x|z8{85GP<`lvpkIMn^bzI6@)|L1HOA`SUw5csl6%cg?1>TQ>iMjHnAZYL!8Y2s8>3=7Ndc~gVyZJ z8oOrKL7*gGTg-;n6EL#IC`;GdhMJo_JMl9TTb!aw;N(7>b?WIYbenEft5#wWH$slS zU>5dD#2H&ofFgtnTlKi5qlDM%w2W9bo+qw-Xa=m;6M$dk!0ZM|!1E3t*^#Mbc{D5% z*D5wWxD!*euQ!+Ecz*EDdKpc{vHdk(Kir`dnZ%=%mU&^^3C&YKH;N@U0+#Q`HXOpa zR=I1Ez%OPLE1PEl7#m50-b^2vv!*VRW-{!6>=3o*%^rosfEE^9md_WL0uX+`KT~r( zq!7dUC1EU7eNSD&LX|>&eH@=2bL-+XXaQyRA8GOaf;Gn|Xt0`_Q2E&3nBkGV)6El- zXpsBh#M5S#M-f5#n539JqB97m?Bwe0FD+e>mYq@aWg1anG)sP<4V^jljNMGYQPPM| za+!k~lmflh*1TwZTx#Q^T8GR59ZM+^p7poQ`8uU%E=}h-UB`;A6g)pb#QmNZYC`mi z48(w6zYjyD0GF}1nSks!#lY;WmVtKD)Roa*zZi#oif*Hhps}%-*KR%1&}#lc;1|np zICP@?w&JWnQTs{0)R$!r;_+?|!dM*gh4l_&bmUq^{^iIvcY#(G&U0?CUH$NX3z-(M zjP-e?|5%Io)2p9ps5oMgz);jtIu53EEj3z~nBZ{WhmL^0`Z$+;8 z&FTWMe)zIHBi6GItgy^S1vDtL$ zOc3PMa~#aTa%^8)KSCqzjJ^DFNsZc+CgZN&Dy7D`{?)3N5+}o7?Vi|^WG=FpbJ5b! zVbmg{H|rhTOx81%mYeJT-%0bQnr5zvw#ff>gPWP}HlzyUakS_T*mv#kBzsi@=~kmB z@9A!vAQXSrqq}yuxTtAd@s!y@Qqiz1pX9_cc}vgRMB9xNQa^{>@}0qU2Ar3w;D9VC zrhU)pSZ=9GC$?{h$~Ubvv(TisZ3i91wdP`j;kyw4f6;O zg0?Wx!QZB-Ho`*jf;1N+5IAJA>a)zwzRe+SLp+CHONr_;7!gZ+#l@$S3*Ew*^of}l z(L$kQqz!YrxJGNT`I`IBdXM_H4t?$5*n>HP;Y)U?v|KHkw`WSu{T+7zl++}!4%w{e zcDA^h&p(J6f4|_;2H__*_vufkdgYjCy?2Oc9hoes-Jmo6_141W^3K^Y(OpwO2&!i} zdL4rAW*+*Fmsg}`OVx6?*Pm7CJi>!<3f|_*5)ioE2#NrvFfpBnsDwg>&p?|j!ctM) zgGYxR^7r`s#*&_6%@r+oPqxZXT1KBGy-KJr=OZ#QlOvS))pet=@-|#`NY&hU-V7!7 z(aQt{tso{J$^m|eec<&JqPl9wl@(roM6Y)ebjQ6stJPojXhGFYFXYX4*kzgv6r%X$ zk5YPP=L5KRB5rt?`Aq@7gc(|f5#7F&M&n;;-#LkS*r6KrRPBk`;R-eN!&ZO;FGd2Z z;o*q!DZ1GqNu>>uFmMr!t8qL&iQ`9qB0+BURZbzW zQThv4JAbWH*O4aSaFI^icQx3*uqwHU$ZW9cym!ZnF9L&+m8iDieMMIKA%i|SdAVu z#ZPvvPC=G!D>sVhm1B2~2j33zb$S&%GSZ2sRK0pS^q{(``1Su4sCd4&aup z0Q$i|3rYJ(XeY|U=pP}--09359|C`8Xy$v93Ob8Ls(1nO2@zdP5s8GcbW(MSVVW~_ zfjjkMx2}~6Brq54upiq=u| zaq{yKR#f11As5?AaO0H7f6oJiql{(?o-Pg9vaH_q|J<=Ax(-NuF*IP_$$=2G0)-v? z9ae$)n^|LD1Kb=sRqzNalS6oW0bX)$BZ}VmZU+meo7`#$&ypiOr`RchsSA8+iZBQE-BRUds4B* zd>^&B@}u3oa^o$nlkH$t>c7rv<7l7#DOMpCEBom2iBxU&{SJGeY1Na*#rX}rCnikx zQt!Oga=IG*BtO}WDTNWYlN-;e!*tdUYm;LOkX-rRS#V!RHCe~B!trH*ZGhnlX8$9D z`WN3=n@16bd{4ky&GIv3%G@k>Se{f&hBAcavB||?cQCMO z$dnDtS-DNDKCa}RC3NLLb>r`Ud-5IaHS$cSaB&*kJy%(Lwsz(YumnFZs3pxuOOKji zgZUcljOLr|uiZ_W%PWoBuD>GhZ)_M+LG_YsGJ@7KDc(sc-80FOaD(Kx%_g5WczpIQnQ$agx4f*HY2jE>QZ+%pt!F(x3 z=x|ZmP04Z_;?i3e;DtxOp)*FnB5Vrpv}{!82rQw18L)_4UF6=uJV|C92i#7T+IV+~ zmESI633pkN+KwjPaj^mi7Kod6qT6Sv~DZcG}K2FN;R&?Vflr!yzHYP@Vv{|Y{yP7X}C0y5N zZkM@3Lpw`BI}Vm6+vn=RmbPqU&Cp8%f+ez_q~|ZYU!R$m8bg$_Z@BY2sRHG1C6SdH z!soblya*m0c&zd1Hgz1R%6zC_gyfD#5nyfe`G=Zo?!88Xh{W7_7@(e=O=JU0kl~!O z>`bRl!#*kjEA7jB-@j0~t{%j$)=QTVH?M&Z7)Dukoy$I%`NWI0(vMNq!a0I8`w!RL z$Q@9@bfGs;(V~GNAxabS2=&ok+f^0(y%f12muuO~g42B?zbwW_IT_4DK^~`o%J!742V?*LUoVFtw1r>AIKSB-gaffJ;Nf7cRkUgu$C; z%v=@zC|#gJjAr8-fm6HeIk2;2zfJA2rg%61YTjF?A^aoS&g$5B+Q-yF&=UTc_=fQ0 za+TZ!A%aiOGiEG27g;5>kz<}#iRTS?qQdH4(QlQiH$P?rj{IKP!OoM~_}$(kmYBd=mk1G#^ZvqLIly zto@czw*}y(=H9l>Buewv<@DxP3jc^_@%746|MkOcpLp+PufD;b4-OG&18uv;)cvOI zAg8p}Z*MIH$3*R(CGg92qNa|0MTYm#`wCJVV;g{Mi$t8`Vj@+KW{#U%mV)PtCRX8s z`kn%KRveCl73dy6?jFCW!QhT(#uZ%$eDZn2O_Kk`O~w&#Xl3zsvoE$RPbXXqyvlBn z*>D03Aw>V2E7AzSi&^RS3LbI(X&61!-#N9+aCn+S)K5dWT*!SaJ$$z|%p|v`P%ceN zXb6j9MQ)oh`gq%AT#Sfp>f4@dm|ym>G}eb1*>3|AdD!}>7r`j%vnG(3iE~ijD3s;* z?0%qgSOmUqF@F*vOWlTU+#sn~{&t(s+waI~>g_e^6i9;!Gl$`sJm|glVhySXHx5(? znw~TE3#|2;I(yAROLe$T_Jn1Ja0=fTOx=MS7OwA>nU3`5^p(=kk{7~3m*-D{N?!aZ- zM!gaUc6cPM=;L6rHK1uderQoZwq4aot^iX%57ZJGq=CQfbvtQjHT^Jt*kX^a6?>}c zA-i>A16L2%bkaS4Gy=yYnfoyROkWwM|4?XC2qp8CO@T9#FM6Y(e8*Gxjd zpz$5J%w7g}Fb(AW6tU0x9vXPbAYJcUUBSx@sXU+5I8ZJnRA8gN*KJ2d1g3!TE_9BS zLIDw*zil9(6tzrB3m}K=kwiZ;g$zoFYaEg>u?C*nl}Lhmst(o7j+w$CywYuon9|uG z<#qz@k-MV#oq5ZmbNYZJr5?xUy+$zmf$&8V@Lh|Dq^k%3EqG#7w6cu$Oa0-p`92p= z6LVI6@U2nKM~>_DmYN7*Y0z7`4$)~rz|?Ad@pk)RIRo!XGxV6`ss-u^32 z_RbCv-&->t>8Bum-55{!EsDC_B@(*%whyNHvxHq z-)qwNfhEt<_@QKSGJJ;e*}~C-EBv?#X1qt%S(?awIE)p!%t-g#&Vt8mP_g>U;ZuQH z96;DSroVz;EWYaKsA~&zedbzxt+F2d_i(zqmxpUO) z=;3(1;?^$GE&iaCR8uuNlOqyJGwD)3@$Xr>Dbf4P>xQ(3Wuw=*2e(#| z8ykbuZ+n+}jjL#axVFUO*{J8o>!U`xn7>x!Um9jTrN&?(i69*q2RIB;?v>#B%PICQ zu}k;#y}G?W8-z$JP_D@>WQ1IM@k#~L&+z1M%G7eo=G=47I&ZT=tB27?=nUDk@r&HbIf|>j1+^1!hLN2yi#-t_>B?JA~lTt_D=8Y3~OZAFHLFQBLMBMl3 zHJD9qaj39Zz0gfo2W&^NRf)aP`e8WL5pCSMu{}lrxbR!^GKJtGups+>-6T72_D(;C;-Dpl=+L{S@23$cqN5HTG_+oL7W(oLyWcYh0g7KHs?S0{?SmCemIvjsP`E3ezSJo*HB>LS8GD)YrUL>va~cqZL88JBowY1GpSypLAO{`9dS zz`Z=@nQ>x2) zR$@nsMB(_o`nO8`#wh2LrDHXj$to85cj`$M2A!XYxU36OpA2BtYnb$FB*tNp+1 zZ#93F3{%?d3EJNZLMCo;Pi@gJ`G&YA!5dohZBXp-=Cp95Ol#Jm*sm)D`lM+xb4 zwK3Da2HqZ!xi`b*lrJ|-|BR!da{!H~7K^#96*5wi2snl|oH?;4Fe<09eH#%m9#G6H zov)ajJLWIaONFQP>1aEG^-S+^Wv&4<#uwf)uy^;pPfuLq+e)OpmjtJ<<9l0uVP6_H z4A&|x%W0z1{wJnNWJrwsBZE4$Ofgm5TDabpwRiYVc*PT^4&g5<)ZL+s;EF$6Gdkox zLiB$(W6#mU<~`jmCswZyX?GdpYerK!3LN&vu;P1^s-W;`eE}arPJE%0#e5E>bo3W$ z?*_?QY76lTfo6N zvw4I`l<1v))I${!5p6VZTdIX>rwU#2sLODl>^2t7XZl^{3IoGk_9X-pds|!e5?g7v zp`K$=LExvVBI!Rr&6aA=efU6>K&vmW=l^wi9(4FXz>D<1z-=NIQX7085VcS|-wP0t zE2cbz^`ltDuA`v2i&Ub+9S_26P7T`h2Dkc(wP%M84NlT{56)t9ufY4P7Q(9#YtXzZ z1rf5v$Tq9Mu;`1Q=RJb#Om7If3gS)1YKs@YwbA&xFy2RX+4+@bd-{nOv&moiL%ABc zq3!O`wOh{PYhI2IBLwuSRS>Y}pgx{~P6&0TJ=5KD&YQ#0{xDsM4N~?WP{$_BU)f=+ zU08>>r)MoS=mZ&}o&2SlZ`k|2dlze9!NZ450fLFW3zh^(HnauS=;oCp7vlfZcl(zh z)Pe*z3teX)GkfJ1KV8VHBj~pC%7m<|;wjSN#e)v0^GuhvNQ0ZG>(Z$tkVrTo7@fg-9s3 zZ?+fUS*;n%(qem_vxJ4`0U=$$;plt2g}U?XFdR?cH?*;Ocrj@_5Kr^*X7=OA;|W{= z@>QT)lsn7PW`86&@JYs%bgg5#;WQ86aK71Ws?b|G)b^tu>`xkVm@J{p;<}QgBCh=f z4xbpPSjD6?O225W3OVkCCXep%ihb^#COG}I)b6uM(f-;+aZ@8C!Z2T^u{WT&@}=Ps zhLx1tQLpH93ez@nZ%=lXS-Q0i z>HHb*Ga>mkakF&~us)M1PDJftDO_Om%Jn-;?}rHOI3s~SRFV?9yH1x@Jqw8+fX#1^ z4ALqQCgb~R@Mz#Q{n&Z>dQ={`M*s&eWBVPEp0V|bY=?O|gN=O~T-v*EGa)znG(hk< zYX7s{E*NUu$nz-<@l?oo%(IeS&k?644*;~Hr zp_|qAS_>T0Z|GAGk*P&grzB6BAvf6SMKSa)w;Am-dCJg-6DLM$r=M>YINx@IggyHp z``w~x%41dYL*64!&sd!KmjXcaT2$70b98a81Nv(`~@WktS}_@mS1fP`joh zLa>fU#&d^PDNV^^bqhZA%sJfuj|pr$d3=D}?D@4@n$|dPwID&GRh|568t|EUB57=0 ztgWLFm)r~ieBx=7K*0q7p#FwE9^QJ=6(a*=h8c6HhP+(@u7MC|(PC{zU0b zqzEu36Q^$OFPT2P%h*`u4ni>~%qNSjnT?R=Yjc!Bm8|9qHjej0INDX_bm_vU>Oy~1 zPyG9au>~XO6u++X|FztowQQf)P%#_1E`b|buIzkPj$CE#xbFsMfJ1nh@EG+=;s=1_ z6+4rJe=Z}9U@TO7RHh({7yx(fllX)kZgll-|p=TJsnx*Biy)mJs3sF;d?8lvxX4Pl$-4S$^_u( z7-CEepaM@hfeN&50`S=>vdSqjQSxd`nJJ2_^O}2mI1KDg;m=W7qAC?%7T-h8lpDL3 zC^LQBMZ8AyID#@etHPXQ$`o{olDh0@J)5e(T}SaCsyFAe601EnmetuI{>D{&g`JO1h<5EXJGR;fj zG{!aw{Ih_r0rgBs)zqG+=0^{7@!DT(A9t5ow<5id6{txjY{KWUn3)H(yGrLdMo6aspHubgs-TBb+T58+B#HA@ zac#Y|XeeY}r_&CZNM8aeXpb&Ox0_?4RZ-I1)QhEPato< z>j{rcZKSs;hAO{EksNM*Tc}lFPE6eWuI4XrNVTDtZupVkv7a|!ku0rM3P|sL)GPyK z6t+6%B_H+q-&dN)_NeB6b>s3LuG}BgE}(I#J;St&JB53ZKa;wD%57{%ks|fjp+G{C zl@|9Q=#5pGQ)zFti1~!f>e735T#@2Z(oCRV3aNaBl92&=N{>A#EpP(R!VeBFGm}It zui$2n+cjwU)^U~kA)H?dHu(zS_4*$Itf#Fent(#EcTC!zDYNsU*J;U1nIFA|Y#@LP zPP}_#1g_VVQck`}e(HU2Q3h;7e@{+!os+X~CeBG*RBKsKn>4Yl&9h}yY9TZ=5;CaEoE(5cWWP8Vhe&j0P zlU084bWs=nML^%#_ zwJeyL2HU!!Z#kp;@wyQ396oDyLs_&}U|I}0z=H;6iHi+!FTIjNvf^h1B&VgWv3>3) zo^Oksp9X_D0GO5H3_zuIrukeOKPR!p9q2&RH&?=%Y&VZ(FtDFL-N9d)-_~2@q{i(; zZy24S4;C!46u?IYh_tg+GITu}el)5;zY-917=yA{+xY?d)G*oc9D)GX8u}Nm0|%46 z>NCqDQO2y&jB)$fU-5E59=$f#pVUu6;(E}+#EZg`h#a}{#amV)ww9~4-|l6ZGYIrS z`26{D_ChBivI2AJw#%5`t$`>d(g&oV*FVMVi+3Hqh5C;Y(`pma8xRl4{4Pw|z+G?m zv4YD7HwRrbd;u}Hw%*poyhXmse=Htu6fv4?1Ji}$?t`diqdNPs#~%ttCNnxELHnb% z{uL0EqYMGBV(*f3{}RW|k9;V{fq9SF<<9AQ{pAy%;QHgc)#o1ALvCt+CjE|242`;l zXs3F4k1cvx_oYARB)2c$HC%8Pu+?pG6$pR?<573RKV!tS44vga&5m!H)BB8qS<~l> z)#3p0mPF|s^@;^|f%`9T`(@BhCp zfUutfURP9s>-QHR^`W(nXZY#%Q6lB@uSR8e#=F_E81`@BBz0eB`RdmlSWZ`0T2CQb z8YL;|mCI;RiFl(bs=nIx9bVJtouZinmgS#W`O2hS&zjxy^$zepJQY1s2={8>Z}9Ma@D%@fwV$2TWbO}Iylvz%x(9p}r zNXQCAaFw#i40UV5f)Hc?+I*q)0bv|x^L!Sthh%0?j5x3K8XphhwM@RgoaV}ya3tNfNwAM?|AWkj@7u^kQRe#%Q&Z18ZJlJWCI!~^! zfg?Hh7x^L>S~o!H5x^Goh{n^EuDRgAE6|ExdDJ=!iILpkCej^uIedk!BA>sokWB1e~7&iwcyqWHRpTGAJ+Ds#g#8LeLI7-FUi8c&s`%VL$P z#*KSxlIgG@M{rnZ{q0OUA;o9p4(qw&?F~HHW2!1Xb_?KR|70<&zK1$fc5;gC%hZIT13)X4PsKIrv?!gh&!+J`4UP0Fo&XBAMBF-L~)_>e^T7*?md$ zOHjjfBxcRHsAM^ACOT9A4*CP$X#Wa2Wj0>M$q4x8u|+CFsz(2#C>&I8?pG_{U>M!* z#xau)RL|1VPLEroj%GJwlPp17nQwI;1L0otTntKv93hYIO=e*Fkb9KO5ih+!E$~Qs zWm@tX1lADip>@Hhxu-OW=?Xxw#dS{Qe&^%qcPU9EJXNW>L&)vuPa3t~#{PI((h5Fa z=x}7927=erR(U^PZG)NuQF5NeZ|(jxg*`Oqo=12-vVNUH-vBHy>~sTeMaM03ZD!Hagc-TO@Hocc&r zwoN45)`TgIT!zMOCv-vfIMUK%{=lC*(EeVW84@zS=k>raZtF@7hJkDR)Z0?x6?r(X zJU68Dp*I(2O&E3&`FiGHdgu!&sl*^_cMjSqI&(HqQ>C^BXMacrhtv>Iv}^M3K;_E` zeO%aoT6xp>VR#oV9|%2c;LS#J^jlSb=EQ&GDgCe%&OI_F5+(Re#;#uJi&NPl<$yL)1Y@(JEbt zW`(JyQfiU&L1MQj*|61Su?deGDD{jbcg93UsWWUoG&|tM`|`)M^NGTb!ce9)A1Pld zS9YkD%ffONF4bY+o+|zIyAjP&3PrVatl6{iBDt)ze5u>}F>(-E5=$HxZmL0R@SC#f zNF8bX4I!kYL3ryw)^LIqI7=kynci1Ox$eJ@!%jGQ37&xSim30RIc*o^<%NO~IIKE` zZL8UcY?x$4Q^KFa$qHb4Akxy=Sx+TahDHR3=SU*a8AD8HxH5MKpnIuh3f^IqV8O{} zeNt-Jxb-0O>Gpqc9%_OZ!f_SuEQ*I)R!zR)>Fx-3^*&fH{S+;yS)Q_YDEd!-L5Bt@ zQ4HY&vyCm0C^BVfgFn#~UJIPM)B8i6G~Ypu;GB{6wHZ;z^Is)S%4Q??;?36r>5T<_ zWlFmJm*^3SPXB!HjMC5<;3bpXFSt7wM{4~>_vejBanjo(Fhd`dYF>mmVn91UM>m`x z@$iCxf#ML|mE*&N#>D5d^d=Z@HZP@^3npfbg1n&sRw#E4iD-Pgw#UYm=8TQX0@Ds3{`P%x|*4s6_SOLJ;Yq3JCiILg*LqlEbrl|1VG=|i*LYPQXh4mj^g+2QAH684xuO{ zec}6aBM{&Dm-*o4J;qp=$fpj>q|t0u`>H-BF`f5y@b9oY5xo3i@=1H{d*VJ7rdUa$ zQ4v>kBP}L=L8U0dux|nGp3wg}16K+Zwy~_cUc2?}Y-$Bg2X;Z=4prF>kjlwm;O}f= zd=Xspj_kKh`g>j$F@qR!ZG?7<(vOE^{sc#v79_4(7!PCfJ(6GS+ZM@Y_SIx-Wi9*? z(*uA|&6*)3qx$*Ylw=NWW_5RgvDKSmgw&)pe9Q)aecD9J=Ij>%l0OqLo>LzzyyrEGyTWsan06hwR5k!Gp(2;hzmcz9KX>87mo_ zaPde5XuY%RO2Nd!cCtYY66PvanGom=q^kyMmWWr$mc$S$3YofB+;X6iAw!7!H%JQy z{|(YI`m%G^lM(D{6G$?Tc-)5z=hZ{QQ@Xhdx9vXg%DUfr|5NpyuQ%uqjX^j!9F;USuNb>BqTZ0-*SZZum3|fX2 z3}BhRW^90dDAFLS=NjM=;aj}C^|cXFY3vd{%GECuYQXp zZ;rAe+BmvNm8BO;G^G^nQ^O$R5yPPci7L2Dv zrN%JbzO;dxphqK#6#Q1r@+U*hneb(q$3m&Gp+>*qlw_XVh3I~FlW3n0Sf=u#g`biBxhkB; zt?#y0rX2*$1#Df@oIYY37BnR8??`h8KQ^1+2%({)K0t_9ez77CY+lhPBUeOTW0=&i zzvkYDdM`sR1{05`Nw%{cGl+2foVk~wbi4Mj9v*g*wB1$KE9tX|IPWuITfYm5lBkP# z${KsS{d5&>Guwcha1y#gDYw_>9e$Qqq5}T>aGzM+W?+U~|Jxse%#x8-vueE7GA%0t z$mRY;n%E-$Ha&Uk*p;mfS)ia5mKSg@4YuK&_E)1jl-4pA#@G7ru)m#tDeK)l4>uQf z7L$3-^d|juS{pkezlOZ9L)XP_WrjYzm*GAFt`=D%?N95hcVUgXJ^#&=)N7cu^!rxU zC7<1FZ0byT`yk(}rRfLFfhAK)q+sOZ4VhST5}wqz*r$=&#WuExftadb0PuQ|$_!A9 z+VOb5FS@o(fY>&<>SUpOODcG<%Kch;2(#)piS261-n`F@vLOT#>e25{W;11}^%$cZ z(9pxFhl*v#IPQ_>(tEyZpHCp(WZ+T~a(?=~A`H#5n zLJHWU*w+94<}`Y#y@whJ(Qb$X!b|6$%9nRjgjI4!FvIjyHE@%Sbp+*Sl%wThdEL9; z3&Z$*_$fjdr5<1K9qi0K|6?Kk0}*YcECB*zhA@q?0gTvI<2>qZJa54)IODP@%~w=h zs{l$}I6B7o3`5HOxE$Twuv7oxcg;~S;GiE5IB=pn7eJRtr|ss?T<$^PC-XK&|2~r) zgltNx`5XK&3VtBAwihfsrRk#tj{j4Pj-$VIQdYIvZ5K!r*ns@LT-h|hxgff#8Y%mD zUN`C`-pk8xi|jcF`WzF6XoRv6Puw3-E(LgBG*&86i;X0256tuY9B!&xrs{lCaGJi6 zKRwI(-?n7XM7@1FJ7oVfK0F**W-Mc6hWWV491nu&7!P4GILzGE-_!nFe=g!~fB2%Y zTkXG`BtDfdx90}aFL~yBPSXq$ns)&9_r1%)h#jfv0{K)0QR0pow{01!3NXoW2b7dd z(t0&>PCI8{|FFQ4$wtO%rs+#kzhCnBk4oVyl=0>9WRs=6eU%;V0e$)H4@}!r6>@KG z50Mds50WV$ORsQlKd4F`BI_*#n>6ml4cNhq02UY4LR`x6JW@x2(#qDTie|j~C#`uJ zZg;qVy!C#C8+?~V{O$J2Gj-`u#;?@rHsc-8vUE{J*sN3GHg8r3fUPJ!R63;s(7A`!sl`m6sdl!e(m@>8<^NVC`>4ic7Q$56WG1x5XzeA#xq{N$Yt1@lC^K!kSR2d3Bfj zBninnyQc>E$`X`wBUtPGNa{_~7_{&dB&W*5QvTG(zi^6hbmvoJmjR)Gf#(nsnoeqkvr_v+vpc8W^`P7Gd0IU6Kjb&>5d z4>b&TakdKEDHH;K(y@TtReii;)=RJhK&Y=eN$4Ck^mBaPVq?;~9^`!6z4Z|mE&~?b z11pGyzZ1gdp=O!2I9{yO?2-HIF<~1DgXw1Mwo*J|N7DvKqKoAQe`bh6vL^M~HEtH6 z@-^Gm0JKf%<)RMUzWq^dX{X-MyGvdPmToR6H=e^!ad6ioIvEbDXkL(a1L{l{0D-dS zMWSci-O)K7w*UdfAtqsnE1&kSBO`liNXrLxDt5V!$Vj%oqrhyzds`6tmWrb7;=p0 z8-HB6>HZtbk~>k5b9{8VZfq<62N9vefy_3et2f(i|HtPdqlfSTaDmw$ENTUM9lEmf zeEOe_D}#i7soXY7T?%VN(r@az{|ytkkRnIre}xNuwYfNv_}>0M{(lj21rZ1_5r8cI zpX@Y137jCTv~>fu|0pN(_&;$VZ#W3P9|tj%w(tMw!~uRPB=A$|kT$RN}bP* zeno4hr}OvpJu+VU?M}`=k4$9aVwWK-!>>T^%%S5gq~Um_UIRj&8bP1MbH(Q*H5Qo^#GnGib@X`&-_M5$>iO3!1zFw^JKZT&hz7YmTF%>+Pz&?eZPdUdxH z`5bPDKkB#iWS0G+`AKfOvtT#-D>Sqem^NEIcb=7gp2%xRDa`2P#ZDfRLj)?}S;~$3kY^|0OgN>X zw)y^>Vef6A#e=Yd0c*PjL6`r*3r0%uE@`0C>y941q(D=&H7N{S^m;Gpo>7S!CRg+1IHJ{qcRL9{;LeRScYy20pK9P*+Q2>H*+vo@j&hQSKxQp5R_bx9{lWZM0tPh-)}qIv%0H ziz9_%QD1KGO0-zmz|1XjEZ%w>19TGd*`6`8g2I2o_1$Gfrvd}eAV#<&>HFSJzS;Mzq<`RZuC8tVY?-RFnnG4rnb0K-kkr$2Oj!$=#Faxc!H z*ziwSe)q~q!$5-Im%9UY)of3KPQsE$0l;)JN&d(Sq*9;WKYA=j>D74^-nz?nW_Fs8 z8+f|O;fIck@D&qq&B;cwf?#JaKl&6>%Zw(|BtW3D7yF+; zrA)gv<8Y=e?!FRqtOBTuUS)V9oQCeW}ib zUk%U#?2g8T_sb`hJh1t62*f80`zv|u68KeT_68%FGh|dRhyT?1 zPw`32=iDRC?n&3NHPy1%`opzG_zUNP!wGn2W0Uf5E-t#h5Fh|17Jg0_sYjML-tDt; z3LAMd_Bt#lmVKCWScb(dA`@^7{QcBE1@yVbevF_dMuF2$vH~!^ud@0MBRlobpi2Ol zup?8*&&qO)7q%e<1XR8n!s)_3qKFQ7%vz$rJ7gi!t}Isk?Y!%=-A0$u1E5YB`upst zF=_!LDYUQbdPhL7+!p#SFO>sXY_q#pn)TzbJm6{$kA(~RhJ2$9WTX z|6|1&VIU{F^B43$cLj~4w-pSmgz>6>*`k0j`3U$e3KW$*3lzyDI-@dVq}@nhIuJ=$ zLYwm3zMpAl5uC+v-tQ^sxM196{IKznC~T6Ss4dl~belw3OB)IQ+{K`K>jtQB%=tam zQl5zJDB|Fn4>`Y6-_eimt?O0dUTTK}uYTnt4E}oq?#OMc{iDk>NdbHI!6*c@q4<6@c>J&WG|LFo+9MY5dzmmTMVH3fe3WMXwM06 zL*$)IQTbX?rL`+kd6@O6K1j*xEd~QX>W7f;T z>{L+lWIBWy4`r+K#A?f_be7ORP_6OMI?rBzQVz zGxz?1%70)vT+zj}?}W>=r*Z0}{qbO)73-jQfK1unf-$rz6j9hq{o3p?e>YM<2wORi}Q(yR_(U}oYMK?%a)Y^&XY2EU=CKmS6?!{dL$?ifTt!S zty%mloZt{Z7y>0n^{ixUU)x8E)_p0kQv`_=l7(~-ML^8DIf0r=LYbzPEy9aK&kvE{ znS2$U6anifdSw512$fm7`qdWc?qqc)I}4vItfYczJ3mJQxy4oH(nTDt?Iq;@9;*>T zy2?;kVI@^Mcm}@13#Lrx7bp{u98k7(QKT zPb2`8EvkFc=3QBvvQ^6oc4Rysrr76CXlQ*OJ#Vqkciit7dpI2E;Ow*4+H1}E zT64bN6if*1sBMl6#P658x!$6hT*fVV7coT16GX2)MDO$XX#$Ntqqgrt5|0}tIot=5 z-BxP0hO1(_fjIv$@H@&L8d$*tYYBG`K94)yuoV(mFg4Pz3dD$Z%*l(pyWZn2YFI6s8 z8l^xZ>8jJwVq!b|OcR6}BrR{?5hlhW(s;L#&=Oem0jmHGK3$$>rxPk&Zv%~+fWXy* z*q*cWxy17wP8FVE-`z>sHz4wnE;Sk&nt1w<8Yl1`rl_-=qJaC^M!_01^0aPmz27zR z4M)l>_20pr-iKhIrn%oX)aYk2qtz-|Dm6WLi-WbQT2_q)en*G@M;pY0xw;}&kR`WA z(Y&)uGr;-u8J~xey!I2Vfo`pKL*sMUqlI-o*s!&oJW9@;@#?y1re5DET-;)p%E@^l zmUC`$>%je#MDEgq-(tkUk=sVcstd4T#xVD~ixlqnAX>J45a51Kt!s=wn3+`8XMaE( zB7&|!TzHE;J%z2*twwamnQgxchgq=<$kmJi&7I?@^}R@Eks|h9uB4acsWN|lgp38Y zREh<>^gPRVEjeQVabqTEcI%SVsJUD=2YtJu08umcUO0rKZ;bqd9= zNjzY6+;h=T^@sQ%NKT~*B>HB>9mg>{1I7mGs(k54*P;ykWgOW3sQn&ptZH2GDp{Zz zV%T50&rrg-V;7YhHH#%wH>T9{zKTg+IaFQ|<8I?-1Y&!fO&|1IpmwZAwU^R`a?zi28BaHGmQ> z*b=v5i5Q6JCKW-zpE2{mJM~$>_QPj8=`5nO^7=yj_(1uqIX?x$5As*fBCyL3a_hF2|ZFkn` z_aD(*V{>wYFS&q&snBR~F#Zt7*_wMAIfo_oAq?nbGnbL6{62CedBjuG;852e;{Xt3t+FN+{BS>G3%zUZyC%CfPms7h7V-2y<&=d__ls%pvqvAdejA>lwQ>v3{g|Lf4mAV*oOFv=l02W zJ^@hxqJJZ`pS|eubitl4T(Zr)vs-vV;Ea1UO3p+YZ#84=okdbSIWrBI69xj<1(3lJ>O_ds!`Dy z_bb8v&*`tp+8g2X39RKJfmAMQF^J1ymbd5CX!n62!^XGr1@~?H;RcrrIb}h$J=Ct_ zw=CM)?6Q!{!~6k~9=4`}@+jA=-6;ll1BBHO$DojLlPICQkMeGI65!sh0jc%NhY)F| z3Yt$ZvANPv4XXC0?ehWe4}Xz6j?=rBf#6_aHxqdk2GKhS`d=!+bvy)HLQRgUQ4NOb zl)s>k+Hcs_1;8fCrBO@X;BK44f6fR|Hzbt3=Q*fkU0^|B1^1?Kpz@*^YV;s%G>YO= zp7tuc+6<=TQQ9ru-+liJ1bMgWdUfSF4jkX)x|3v_>0GXVWFXQ^PFT{2|A1gC?xuoL zsTBO3k@05DhCA|R%#O8m>sbT$!>&}8*#uI5w^puS6ULc(rh)KlVR1JF&Kp5c>PWBq zJ$CQ!esz?Tm=6U3Za-1MUKi@xs~-h^j+meji`DP>)r4Q@OPzAW zDD=^l5pZ3m&n^3M?_L1vuUfhatNtlkK;Ic)a6##Jpl#suC|i-_Qr0kp{y3Oo?!zA? z3a`>jE%qMn#xF(B={Nb~oFu2q;Syei`A#!MZS_qylg(^}CHCnY@DkDqg*cx|9e!^z!1@dtM`-ouNnS&nRWwnVz-tP8Qvm0yd(7QVDK zKKi|+yQMUKA1XNcVufr%;10Qb;wAy`dwzqevX%UUO}(iwXdY%X&j0ahyRwQ-V7Q9_ z-JnG~5mTOo;u5Tb~!}Qs{5K6|h(=wWkYS*_gbcRmFfZ-~yS4!xkqM>msLc2*1mmCh7}! zv%mI~LdxrWSS+>7S#A9A*>bcpY=5>*!*HIY+WbP!ZA>D(^4%MmCAKt*2echNEHA*Jv9gh!Qs( z8#Jv1{$Q-voGCcw_)9D(--6k~u{i=oiRtRBG#$+^Y}>hyscV$pyh#$F(ekE_8uo&f zSR1tz?N`-*u@xQZDWQ6}y_~G1&)lVn@5-n4uQZcSs^H*UkF+}X^YQkO5hI<=MF?GZU%MlvepK;|L{HvCLK4`*7J3nfFl- zzIy#u`zMYyH|5*~4~~>gmIoE9VUy`_eh*k38AOQ0i}UF5oERZV7~ z=d;?MsqXs%_O*=-{0yF{??OTT*FYck=*9OM;)+!s=Cm8TA!e+(@2ANeKm4E++}F;k~Uowi8G&OFX>EVaMhqP%&B+itv0G~Q)O`{SwONsoQ5Gln>0 zU`zQ9Ehl)*OXB7<*7cqBZRwzZ6!DjwUmY;ZU>44Gg^Nk?>z+-;a96helOzU9r>L_v z*5tbIszfR(t43(zW!3QwPZkIWTFh z`6;_u(7QP<~~8N{=!B_*PQ)H2B$`iPtF;i#;>xC@12#gzKs z%>va)Xe_U$zV z>!WP_C7LWr7rQ&2s@-*)D)x1rN1mfWItFbvVKxhuN&?FZLvDy#B?js~x8H&m97;TN z0h+$UVMhCjID)_VAfZ2@f&vV2iYwFQO|vEuaO+6IKw23S^*-W!apimcjga>ezbW@l z|8xy&`L;_)@T~KNmtzCWi>#wzaje_z#nD2QqQ_-3G$_TO)vCz`)=aD9M!GD?*&8x( zz1IRL9jp`i;}4$r^bDWIRYwpiu$Gan3Yt01YCP*4^^y<_vx(o~k_1s*Ncr}kWlA$>fxAzU{Hl$T8 z1BdA9pzrrYm;w##>jKcku!QezBt5?l*W<5dk(3LR5S9;;r@dL)<34!MV3(c_mVgxs znRA37KJ7{k=yGz33l81=rR&=yt>t@(DN2MZ3CoAw?uCJ6-xo6{m6`m%Z>!3TJno5} zj-M&J@A5t*?3Ih2zU+d@8}1$vEhxk91b?vV=Va^4!I7S#ianc>PN^-lC zwiM30lM~n@hW(Zo8L&#bvUp~#B?4dOYfg~TK){XTFL5YKTS<0!r9zeQ$tXcKT#Y64 z(w4((7Q5(kS^k2Dr)!F!XTHrm!Lmxk*HIu!H!5L?^A|ga4Fl$Kh*v-Y4AI{6mri_+SvGVM^lEGd^ z|42j6%OBDHuG{vjGazW3R@uA<YH)W!&jtY~y9-;Z}S+QR7CSx_LHJiunfKIV}0*9KV_w3>v~-J$Bw?ddh7e-&}OOFO0qR;!J~ z6x4__@D>CZYx#a2D&YtsPO%>rriqtjx*kLhc#I>4hQsfVst&jwa}GaJn z7jQF&%b*3nn^C*aO~cmYi!}M}?TpFfV;u7beb*EeWc}7n47Cs4Le!L|jwRE!iqO|! z4Y!lX#~oyTm-A5mrB)ERPVHC!QtdX0H##~~Y`Vh}FZ3XG-f`qe3qPB+Jr+GMgfTjx z$`V%E&FRIp{Z^+N14+x1b?!fjokO-HTc^pJ8*eSe`1Zn60jJ-$-POHu^yBv5;PGJ|&lVf-jiJ^r$Yd zFiA^yuVj&|kj^%}?)ccLfE~|nvt8hD@0Q1ZzS6Cn>+Bz8TIcxO)PA@Etcb}jt((S+a^W}l^=ULHTn`T-%KH+Ghg zJo!kBtJw3CBm_UN1C{7lXo^cj9hYK_5rrm@{fDKfx){%_Z~fWLwV0@sUz zi-O+U|MjE%a2s(Iv5X$EIZ*tmrKwHNrb!+CntrN2BG{8A7jfXy*^-tZNs>=uq1{^u z|ElJ2qmarYNWHQu91DCGzo?ET$1O9U6SFT-x;`uLd(`X@jdzBytCErwSZZe)|EurtxQ>3mCM3Wc1Z$s5`9 zEb%~AfS1}#i6t}mXHwr-Xs@5@+!oF;>Y&W?R1%X~>H70iDnl{t2Ga6)NkJ&;F4e&; zyU z8v_4(hQD6ET!}e3CEt88cHp}Jbkge;(@#X16%QN~hynhlv+T-e+}mJcFv90U?_Ti(ccq zZV}RVK!qYMZfsw?sQJCABTKm!(8!P?;8<=t6e|f>)iT0yB6_%9RU6hYr?Cf%yf zy9+AffV(&Dmiueg&k~tuG-pf@=N*c85g*R~($tO+*5E?HYi$sLA|a9V?4+)w=4} z@AcTW`3%Zl`A6ZSZC!lzs;_>pUmWxSbFS9o0I4sQRnOcp|Gj9EOiC;*jqhWN6pMQuq`=9> z?-I`>2ypA>F%i0&#l#oQ$NTCt6{eA$P3Zt-20Ml0liIhq8o)GaVGT}pE>uUTF*PT;7T-skO@r!VXc+qD&Uz^pb>oyRYGHqHOeHl1z# zYQ99ikEmNFJ&u*ypJy4a;?LN@BsHY|F|*iuc$_eV@%&Pw@WA&?@!Ku74e$={E>v+l zM{4bK8d!r(B5me2wwCFN%GrO8X=JD_ayU661(;O?7P95E-kEI`of@xgmi2_*Z6ml2;2S_yK5Et%dAzsIZ6W9V*1RwJRpt-^Dd5NdovkqPuE%hpVq@(OJT%70X}R8{ zKxq8Qj!^JhF^TY79LEn&!{CNd8(zrPs6agpV6pya!tkI|jihhAicCG>7U8TZQ{ zXPueeW*Ivm!^mh@^O9*iaa%yszfDO@fx`3BCYIIS0Pk1VV!2ofSVhnRrdZ+P+OzLI zUg7GRyF$uoSH0k+u3Mzi7G57is#rK&26K7!@ol&EO*B!?)>D)AbScc@b+>3Z86#jZ zZHCmS@~By)ZEy;-IQ@RD-e84d_DV!076MqUiurhp?rL#+EYPYFDR~aJh#pXK#-MxS z<|PQQqstwwE=O?SWg4x3IlPt9x2l|OtP&j4hJyTJM4!x0yMJa4V@&<3C6Y>h-mxgk zusa&jvIAS^>!L)+?dF>Rd6Ql>@Jg7h3Uew?ycEXn^?IxpzT9ro8&6iSIRM@?hrMLOCLIUfZBcmCuV z0$OFG(ePPi_7W8}#4>bit?0zYGH~M3US{A3l?)`jw?%I|fYkyTf{SVDPh zch!|1C|<`W_&wCbOYp;6><;JlLr~DFa8RyZl0{T@U8=`0E2*+30>IzSkIE z@td|Gp`O=bMSv1gb^1w)bZQk4BgS+2FnlNYiJp z+d`miTH;%wGO^m2$B-Bk1&NrNVsfdRVRrYCa8L{|xxzc%kj_}3Sfu)?hs1yixZ%+z zP4HI>fLD)NU$8E&UGyZ2aD7b#+hw;;`jE4K3jgV$9{uLHJ)=P@->Hc2u3#jU9uQqh z^7L4={K($JzkMAmsKpW%E@b>MfpXAE45I+mOdKife4k>qb;n$_OpdnGmxh41@-3&i z`j;}ippkmieyTh?`W%H>keinCSjzTBe|&s{==CIln2muKkEXr9^dX=>h|uMB_wo z2t+*Owp?`MD|cpKg*f%rnq-M?K2TUUygbJJ-W)wTky|5W?R#9IkP)z;efuqc$|;#Rr|nxN;jgTSu>+u)RKOL zL*yLUdFN>~Btt=s&4w4@#9}VhsI_E*MDNEIkFfMG8l*W4W-JInYMUvA{j9@cPb1;1 zy}j76yQJLL!qLF}G%dyu#66Atcwcg~oMPj3V0toun5{m{ih6X8QmNqIvswh79K*U4YniIi%E)NO(9 zMo8i^%=(!eo|pUmK8>##K7DwzgP+GIA=1+SB$#=(bVNMvErRKhqMd}8WNXtEM`h7{ z#-GIhrs_T-5T{$s^D0mTEOdFapXqt!)St zA*3UC%Epj%h&rISvfEjp%qJ5NZ-XA_3;z_#mfFSDAaSwAw!t0#K|=TG{#?Fo-{h`9s5TFxFJDS}uD?*XniyEf7!OGv z+gCW;dW?5A+i0P!2@6h7)|=2<#uj@yRFn?yW%rA7k8UE35_1+%Yo1P|j1A z&B1S(5oX2-e=h|^z&(a%ix)z;o|wbcjf&5ioTb0te^0?x*^VEjX`Scsuin+g!iH)_ zQ`m2xEBcX+LCgC;gxepR^EU*gbzyAMd0kFd#x(q!mmK+^9|2#a4 zhFaTfaVI(^NvCYm;hrrr*Q7VKWW+(DR=vo=E*R6z9Mk^2d+=}-E<8sxgBtfc;rF&;= zz}^OY$N%N}jU5`KjoQ@LQl;4OQK9z(rXH}EH~ zrIx14KKC)Qw!Q=E-D0nhy+id+r+AkOtK1ui0?6aqT0K&{A0In(8!gwQp0`)SL?SC8 z7Ond5BZwz8uw0I}FAK+t`ifj-qn`*%w3f*yFd09BLWwsOpDD=y-|K?_WwfmLhdrW> z&Xm*5A;u`t+fW6thsJ`q7C$dL8j2m9VxGIKsukhnqAY)rm65ysZMoo1aqo9Ava2c-P}ha?m|}W z*K;XGe0EbUDeMGVMS8M8K1y}5JDFI|TmQpEg2k}D)Pu1hUg}~`wmhBxrS;neo9RzL zoMZ9>0=d$URmPZ?i6EZl?U@Sd=OnD?&YS=Z_fHaXZj7!{&#dmY(EJY%5_r-dexMYB zIm-WVO8)wF*taoR*lhQJq|bQW2t|2uP0DvPk&4$^7S~H$QDhuhAv_~Jft!*GLi?$(6n5oEuG7LUIGYz zXyzQ(qm@Wu@&FcD?~BuwOMFMkKShyFE3`1lsV8c!S;4Y33_jD%sDPg_MlhI$Qc;z6 zC%DW0PiTpx=@2t;Que-?}s3Qk(3AITQvZeO^%%hS?8M zDwM{(lP39_!;S@kiBXBuh&zb1@}@+yNXyP_&3_L|Z-{8h>`!sRBc}ECw*02Hg+(ct z7zUV}T!I~UDuf99%==%bvFc8j_QdhJmg}{(09g+QtoLV-%7_kE{CO}M{-CQHZLZBe z%y-3)$P10Q`>1+@kydxwX7{C@=4FZTEM3LvKD$poi@$2SQZ(-}={}>=b*L)58H4&U zPBNEm;)?#MoT2ws+@~VFPKEqqfx{)Us%VkGx7RZd+WOb zrA+pzfN|&k@TSgBlG7Wk@B>=qCxXV7hXKJo(S{{bQ+{qh|A+Ln?JN`2`6KZy*MG^$ zG^$>so9dza_K`r7fVqRb#5(HD@r1GEL;bkf!xbrCp$EP_!tpA#wAtTyLLd%{^E-Sf zl@d?mr#Fv%0BGsxHaRx8IdJBR;sOsuw|96y$_zgWK%+vun8AYeJhzXtocH#?_LGSb z1uS=n$Z$;V#BWXd(Ia2+TUS|KSalM$cza&N0y=ZU?!7h*))P|2yQwXPUfe=U>=1;d zHz<@5Y(v^JZ}*NjN0O8^%XF-3t{>9*%u+vo0oi}p-#=#agLBoMBV=XvW+}q18OzmfE4#RSL^$s`%}0_hZ9iYL zAilf>$l6A&&(c^~jYvh#>Dg@9CSQx9V7)ah=im$O&xbqC-`NjG!0jr&;(N{*J9KIg zP3PPWNkXl13HcGh=G3}8U&BfnUq@+L=|~%g8PY{34zbXbH?08r)YodRdWH+>Ghqnr zk!hu5e`6f>hDd*X>ifH^*rw>q+Aa^sQRS(BpRRV_!@LL*rM=jjJe}efw)=o{yte%< z_B~D9YJex=Q>Jek4Ztm9G8$YHrmT5ineTR&kw(qC^-yRt?$iW5dU(I3mLEUu>}XY>I|fsn$pZ$NND&& zN0`AdXpy7GJ%`UJ`oYx!=as=#2+@i5O2+cdiaPbxiJlSSw^Jd{&Gb^e8m0NAS-spz znQFwDX4@#2!c|f z8s5of*PlWSnXJ$q2uA~CKP`LU>{8TQv~ebUzIdq=8o_Z&rSZG^2W=xBVx-l}eXnPZ z2tOl2Bl|=cih`j%T{Ht;{6rr!qzB`FygS+Y<}d?pKrE$*bab@bMx)~xQcsB!zu?$b zrf7ne0Iaw@pzUQ(*;KLS58nC^Rm3%p@P@5jf+hf<|6Q9;-EjEbUT# zpYtSc;HV<->I5mrk%ZSWWcQMna4Y67FMe1-H$au#GFh#qfTfcSj=KR7B;d{=XW)Cr zyfgo+rsX&aG)15*iy_8tQ2THMQO@w%S0%jfE2wsc)g#&)4IiCp=;(YnJeZINozy1P zHJ2X)VyU=7$jGdEEffGDVwYM#a^AZhYPb2rfEUtyq%S;qJ^5+#>7>A8BmHfo7PSQT zGQ2t&om=e2t#ZLne#+cSob^a-lRVZ79Rx~#BqHBm`LJ4G;8uw~031A?SgPjcS;L$(5;1}!$fSi>93 z>^%Udh=VA#`f!G?0&YKISK@Z{7dZROdIz&47u3}WWFONhH!a*QaNU~|h;j2HjMkOSb@#tRI+7@3_N z6!ExuFdz3BYwZ7zy5dE^A^in}qx$Pf1vhYPGoi)l!X8W?+a7&tmBJaBT$kqO(vuYF z|GeBiuwsdodK!E(sk*$mBnnvxST~1bQ3(-eF@_DDnSP3AdfJ;ioNF+g>*D=0-KIh*%sV|QO|h%V86=WCH8`qI`wDv&5#)BE|j zjC#DtowuGYi+gIh$zPZ#h-x|8BsPD`(#1;cbea_M+ z27xenIlCRV`riYbaNIegA$87(n8z+5j$U~u^=E%!6nU9Q{p=J(8v4*%|AFFq6VOwq^D*3HzT_KP1(desZ-Vh@p- zLhs!tb&FNKHI93vD0$r{*2vxXIsx;BC+!Jb%CvT3{vL&VzLF3Ol7Z3@It9=g2o+0f z&7ED0-fmzT?4Te#k^WvlfG4Qda(j}K9FN`RX1Ha6?m6kLm3eWx_VjobDfQFBq!`Oy z&lTdP18eQEdN-IUTERZ8`OdIMLMJp-s60mQZ$nLk^MPr6!!w{RBmjG98|D3eVr?h1 zyrg{9jm2=~vcF(L1tJmk8JC#xv++Kg!dJ+M$@PtMDbwxE+?_+PXQ*vAn^I?NEpQf) zEma&aCxWAar2Kw)WWbAdZid5-bc(LGAOsF)%N}JB4tU_hGO&yXDak$GsmuVPfU6BP z>*JkmQMoF;&8ZbWW%o*iZoR^LPJ`~*a)ljS_>@gM(AbfEZqFZAWN&l2i=37uLvjl` z&NmFnh76WLl@G{Re~8hfb|jWcb-G@X{l4}UAF?{VM9vKLyw}S8Y^FQqbcY&cAd%gr z-KbF`MMs6w<=mZo2%Kq^s%UQLB*(}bm{`+ONDNk^It=!Ighc&2UIT(!l`Vo6Wmask ziG@sIWKEaCC+Lrjfu4y~ful%(^VJR39exg`3nGZz8cfkw+O=^(FUA=!4V})m zE^s-$q9@~8rLrX?T;)vS2o>`AC|@j(OlCp*xo*>Fu~PBpB5Ti=5?^3FtWrUX4t#tGa*oPDYgC6Bz&26YR-`v0(!vaf3{k!39N^xxNO!h=#!~C z0+F#u)hZ%6%r`+_3OcTT-peYY*fv@XrRHVy+|yu3y?8yYqD>71VX8>p9D@YL2&zi1 zO+{tslPT*!)X!~wDqOI~N<}j5=PJV0e?W&}GR@Vs?-b;})a+d#bk!go-f#>RZ+{!K zCxOBl6m~dM@Vy-FI;{L>GR#fV*meadc27;&1#z2#+C>&4K9Yv0mi|f3!0U=@J2^Ds zoFMBb6#B_Rph#q3W*5FM{POyerGy;~|%x{rbeEpB}ZOFJ_S1;E*# z@~=Vm!gRaB~f0*kK@4sYRV8X7Sq+`ukK|8V)LL<1OT(e`(c9I93Aw6SNFeYYdEqqHrD-7p#TZ`(8^{ z7k;#0zi>P{T{tESVm7 z|M6e;8~?mpUai27@l*YXQw-(#JH4v5WwCP%z#ED(oZiW7-H~k#YZH)dejTxqcVK2m z0DskLoqBG4^4P=p?(FemE<_|V=L^mwp>bw*5C+$3`rz)A2_B%Hm#3kvU1&sFOwM&q z8{Vx#HR}+A>d-ddwUy!5FZ%AphH#I9 zRuYCw#wqp0r3DAB`k~(ca(!(7U)DTUlX3e+;~81rzCO3K%NsqH%I=}2u8?iUg^>^~ z!9Aw6Fpy!QvWQ!Ess53TA=uq!%x*69K`A9sbv@;-jhUPt4Km{6adJ*SQE74LyN-#Q1@h;TR_96CvIxaN8oJs6lZaOKeF4WAUs#-r=U;W z0u%k!?0lw>xKAmDO%litvhF21|%i?y;M0n}{$cb9h zFTZnJJFEncdCmDn;0<=~)?%cXyeB1-3hg3?-$qiB5<*?%BE&lLc?>{WLc zRv@t1r^&j9=!Jt4&ZxdD--T-S%@%EHGpQKR;a$Dg!s+56v;be3*D-B**~c^ezvn=P z;#T)FY%vJ}cJ8NT(!$Q*7h2LDNP#9EkwJ~yImtJ4N^Kej_Y30U7L%eEa!zK{{sIMp zA3SJRrda4)Dh?gl%yWm?*{x8?uZXTZ4r+e6k;=ll))r(;v5yWDSnQ!ex{Ce=G5OE@{z(4}B;l z>-Yy_{uk^H94ml35-e(vzJ6}Un9f9mr>_JA68H(78IJK{Qyc-12Eg}KH_(ybSCwJI zMr?ag*U)r6LR!zu!=&x)S(ub<#mTaOqR z^1Jv-{Qg#R1k{{XJK_&FnPT<5!T9dy4FHvk13}lrmiN#Gao$KtiJqd|vt%v(NNeGf z!PZy^)52AT_ua;J6-vc^etMyL!5Rsy;VFv*s0ADrxx!CBn_yiVva9fP*t)vKbWt)e za4+9#Blji4-)~5hnD6JgTw&>e@NeW$PM~F@PySHM&JI>3uaYR} zKDBXwb0(!EuxB?CxG$lj#J^fMvUwFN`Dof84E0e;VTbn0d0jQTrj0Cb%{gwB%ZiRJ zU*N$2p{2~t6#B#{s4Rsr&Iduj%V>$uc5SLrwARotU4F=EmhDxAtAcOX+TS`t;P#?T+n8z zS=`64viDP`xALKfySKDH9x<{9*tP6(6`aBC+F933QXvj6ChjKy&2oQz!tgio{fyey zHW_)2gMq_HKThq*;d;4fq#-kyy2n{FF-3Vw<5qr>|CYM@fd-od@uT=h+S-o^L=~iR z+3@U!OI~W_3OesK^9o z(@V7VB*13!W+)H=P=dYHp%8~rLzV~e;Ho3<= z^X_4>fEu=3oYaNsX8KEb5g_~-6MMud(rGPjI-6^MqJ&aK$(QT#Z%pwn@!qxF?Pys} z6xG($J*YJv~`Ncx%+ywZ~IN#j@JX* zEpL(LuN*>!r=vC2aNH*txlas0{{Ud-Xj;q=fk`UBaQ3Lp1e`ToBt(coAS`FxBZcgE zrUUh))wXvmP1)w$hL6uYn{n@XZ|{vU3Alq4+L|_T{PMb4COuDCWwYHplIjFnN-POQ zzQmUPXSeVBT$$vIFSR1)Nu1>PoUL@^Gs8@$ zQZdvTTq5{xM4om9#qQj6XVM0J&_$q%87$nz8j*3Vx!xzS$ZXT?K>ewSTTmB%HF7&8 zO>aK2E>{@+_tW)ZTGodx(h1O`s+szqS7x+^OG2CJl-*J?7|bTe(G`FDLXK*g^?=Hs zN0vkach$VF*=`Q3j+<=mHy8f6xVTi&fM3EVY&|U0cGd<3iRGQ4E0ILUE!N&`LwXs= zG_!)i{TvXfqTf0t+nK4|+|`g-+o2~PdN;nsHO~j@?+ueQpzO6!WC+Vkh@FeS4_C zwwld1%1)RKHujl+9Fwt)q8RDwHkpU!2}OLk%@`H);yBmcS;%%y&tcClvVSTVtUKEp zu#SAGW|01Id=;P7{e=#@HtZc|+guf_H=N^_d@7`&dS|N&Z>=}m0X1BAzY+7tw+0*S zI&<%q6Uv_1xcniY-0*S$P$U?1W6EZa*QLv0A;%zO5&ZNvd`1MV2m8;Dx0k}gt~!!G z3(+sAS=q->Z(nXTItOEWipN3e(@R z?=o!Kf<)M}EdlWvwK|93pz51Q{XL)0OF~U%IU*Ifv_ItRNEPTr4y=^mT&)N8Z|7D!BxrA#C`OAuW*m&O&*B?K#l!4()^D&2 z9nXJvfoY;`;ER|?x<0uQChpJkDKUM{3mu+cqy+-zUZ4Mt^2)|H}*-jUF z^FwB?+?m#O?qlqGiHJ7>sGE~*D070=U=o9R>TGrQa6~UeA2LB%yVj=MT8nuV7K4^| z_C-`RmTR}>nhfaSK);(bascEd5)Z++K+%^M{eG$@J;X=J&xNT{GU$^=Up-wrLdmu=AhJn?D`6^pRox+}4LHh$6B%r%$<(E4^aPxkMb-J!unws2} zx}Q$wUdfM91#jFud3K}cv7ePe4|wGdK2NPJ{OO`)T+bX4vdhqix|HRq-%Zb7rpGZC ze=@B0N&rB28v9?7i-nT5GdAoBjojjs#J%ZXfweZ%z37)GOhD%o4gb`;Wq@Fszlcbg zRY8A7J8I3JP%nBTxcY4f9x}a8jY)gI_v4MU?K2V+%O{rMy6PLP1L$NcBKGM7WDpDS zT%E%oLRsrq^HR77dcA}HGTHC3?b*TL>KqqJXxt-4VE-)&DXyfk3l(pG$ z$IQzmlR#1O^8QkuZ@WRP$8>F3?N=XPo(%v*??xb zJg0Y5cB8)*pSi?m>Ja9*Oq%E=l$D2W#9Vr^Ly+A80=P85#i(cD#zB>y*VsYsvBmwl zz*n~3MrZIYMRFM?Iq&jV<(Bg4ot6|4Zy{qv;W%zwb_j65@ng@FwpX?#(}H>osVkIE zIUVdsihR6mV^LKn?Df@^u;#HDysBWd8a<<25aPk95*6g#}lqB~a! zJbo!qBJ{A0kWac)ZqRXuMaD7XauldW(jL}&hxt41IXsDbQ}3I`rm-Q1Z!&uc1SsS^ z$TThZ(LX1cDV7_BQnN1Qui|6)DNs4zp{L<*{4J?@w?y zr$VArH~JzAyau0;D%egj%-CaqwBbeWGN7oov37-=d};SLKBdPk#0B+->iOO%xzhl> zt1r$gInXlRg9HKA*mQQq4;R1(x=8KwMx0h^nZXYWYx+Ck5ktgXMd`kdd8CXVE!4 zd{5k*;^7o#O6nLw*zIz3xJaY0G11SxH#qA6baDvz@3o{@`0wRpPG>D)oGdY}rwMu2 zM-rnYvlE5?Ka72KR8(L4uSz2bNH-|mNH>CnbVx}{cgK)|2uMhmlt_1X3?W0$Al--z z9YYVzJ$~Q!ckf;6-nHI$ty%oVIkWe3_Sy67=lR5}B3bdhpxZa{ICIB2Y38e5hr{CB zfv49PzNWm^3ShHQ-|W%6K#Fax@ztMOEng-#YB3PJR>yc)1vYf9|%gXY{iDOyIf8mzwa#-$vX$qbodX9WR4;2g^od^^lK%it0zd8@T>H2|!{k0l13Z4?9Fls=zqjK=8rb5sTc zc>F#gF%s~XVDM$kG5Y=f2m!<&>l2ypJ4siVah+iwAD2nfM3|*L0M+KDvM6$0UaD?xfe?F(eVB?u^`(gmuk?mhPj_wW8rPGgFhYnNNw&LJMmq5x_il z$?=}@r2jCPhN3e1f>)LcUGqYDm9S6M4w5i#&QsOFm-5f3McATKtN4|!BRpxvY0afy zhw@S;JxW)k>m>YmJhpRiyq|<;x0o{PyE5CF*QV);M3)N)vheX0(Wf;;<6Woe>N%isLnc?8#2%9c zSYvARTWe=jAvm9}XsguOPksOTPSeVMMVyzW5PJHmw<%5hBZq#CZRzda^uzGOl*Y5x zkCs+RX;h&h&EMt6K=7U5Au@r@W?Pbc4G2RarNAFFqI&1~kM2%n)`MRG=nq+bt>Gbk zBGwNT@tZ$;ubgHO5Mig>SUux9>#-LE3B2Z#^kVe+Ys6?4*fBXRHpyOZnIEko;f=6+ zrVh&lsd=Y8ODUh7f=`KzQJ|Ifu}?67?8FC_BMYJlI6=!F4F6;GO-)c1bE48dCLB47 zmp`BsoZ^M4iDkjdM45*d*`S5HqfsQbN(i9oZbCno-F7)%2LkL{o~?N)Up1&C!vz?* zreVQRQu69#JUEAMYdxt z&@C{{BJC@+CKB0IWhhCX+Ya8|_kZ!@5uD0$VlX$DFYp)A)6vx-?W-tOXF@T!qyg$tFDZEwa1&U0YQ-{#g~_fvhffmIE`-(u zs(yMUr0y`Z?TYRUVk?|u;m9EcvOSR-I}^mTf@mHY$ZXuW>=%HG9N7J+%sM=Zah0j?@PxQj|fI4T^OEZbPWyU20KB1@jU4>%=&`*>dD{k z`3`I`%csSeh}&QD>JXE456y4RDODm z*`ib!HBqkMLoMDQpt4`NGFR8?cbc3Mxjx`!Z9{#I^JTBc{d;uhVl+NDh_c;m?R~*QlGAVaImW05c0n>seKgpg!hv8qR363Ijkl!Tq zOl#E7=slJe93%#qg-xL(-zU?z#l&?jRs(At;{6 zC)IJx6lv%~13@x2N8{K3~3%od}cwj?{U@$0Hn)V{s` z`P3#9XCzas*-s1jdFT6@C*FZ%!E+~eGLcsS-lXtj*gZ0}S$t{;&&gc%y=E}HJ!fp0Wk{4G&{oFE9CkP*(k#<6CS`^OC z@xRV<=9<~MBT2IZVF0(LRMYV@)+r^i*ZVZ{c~%4dI@}k7bBw-OIP{2%k_a9qBAIYi z4Q@*t5VQ#jGLfIVNW^GC#H|L)n)vS+fDT>oM%2MEd@NaYL#+P$O|vciB2}{V&hx(o zB;QDKlppmU(FP=F)W9UddnFnC1KVQ}%?_5_93w}ST!tT;C}w-$k2066S~m)Bgw|54 zS+-`~wBfqhID6c!1GXl}Fpo9!MSbQO^B=TKgC89AO=8OqdGW4|;yog-*OI+hP)=|l zt~OF*d!3=YPb^lWDRL+f>8k0L{%^eO^f%XlHaq0w*B}(8$3IL2*QjO4$(VmgVfS(S znohY16aK{k-$QJH>}y~W31naIwl342a9)WHT#8PXS9<(R>ia1o{8O65kFi63ov=|h11#@ovusS! zn@+-^A>G+OW1rus*}1h@>a%J-l%K8Ti)tb9fx)+Zrd)fGOqO!GnmL7C=bcLrr5AdT zm1zQ|2QEziwt1^3x=A;7ccRIQy|fUbCD#3s)O=)f?A>cU^bR$?qjhED08?hq1<=9#dBqy0VU(YWv7gIuy#R!96=dHx z0Ic7emX71$(}}w#F41VOWQvmaBYjAuCZc$fDfezYg*Mzc>{vQ94_n;neF3^C3k{EN z=qnU+bOOI5)^KSoT%k6-PWN~oTQubgtH)@ZM6E}c{oTajF|UH}xXkzGigWA~F-5+_ z7qSa*xDR*>l{E+BT73WMk(p0Z#`9tU)#VR+TK)zJu16GV-;Z`GBZu)~({DULBvOQ~ zx@Nt?HMxDyx>2aGSu#6zLwimrDW0@7hbPzbbvOn27^HI?-IbWeejqdNZElI-Oy7`H z>x+I^--!r37=7Gn6Xqu;%oCg=Q&N3*;J2Jqi6`T`a9-`cTj8?hGP~KsJm;WIz}zE? zz4P-6OTA$J>GxUx&}*D=9~>%ks%EV7J~3nsMdKPSI$|&2x{B(4FIy&p09$54IfuS2iwCD1KTB7Hax>$vS*Rk>6S z!mjv^lZ766sf;QAngW%+x&2a!I$ubV@y3-iCpS)&SrzX9|IvpqeG`lMwDqK0Tx zcu_mxkC~F_Em@z(9P{`J|C2Zsv&E$IkTlH=m0wCG5H#k2Q%{ zu{rf0dci&KbSfrwV9BxXvD(f@EB@8-gqid^N5t`BM$OC}+5wn;EUUg7GNc*r{0A18S8^2?X7P=U15jePNXj;o9yEVBcB zU#5jYH2r57IR%QkMXeiz3nD(jzuuM@WlT_%7DXi3Jow+NT5G-hL7(me6ZnF9cL<0)x~2^fpNjc{ z&x#Ud9c;l8?6vdVf|;y_cZZnJ%;|)UogVb%bm@fnhAx-R-wg9S&6UbYBJn0CYpZ)? z?pfFE5Fgx!qUM-7$)-~LQ%8hzZ^jaW_|hVjw1w{5K>D$Ez&wFB(KYuOedXgvgr5(> zxHEbGf`S8&Xqp2R#B>|3F5f=cN%`Boy|e7J9B!Nt@r%Xlgy$Hllj&`dncDa@dVTf% zkuUHZjY;iFIbId0-{=3Uem@HAdw672jc(H~OsD^JNhD-GQ_OMy44;^PTDO~Tr&VFc zxXKG(6ab|l+CA1oDXuR{($HdR_W)EkXirc_sh`F3#QvpDtqcg)g48^w#b1E`nfRdD zpmD8zBb)Y10Z$JL9d{ajC1~bCv(EYiMZMtWShlFW?Luy09))jSmSMR$u{maV^U^M$ z*U5BVAyfonvy*!ah+Bkq z{XHX2uSO8}qQ>$m}`Z~qk^j`{6G`6shOp7Cr_%EibYtW{M#1 zo0eVL=ci^jgLkA5ie3UwI;4alX4f&{0iPeAt8(JWpHyWpX5Jo3T^jc}JRRE0!JiN> zEd4@}r>S1SsZ(yPqiqBL6j8JFD#-r?Ro8r_=!L9N*REB#Jzt=%5cy1g#Y1J5DmLQ- z@$m&^K&vTo3H&yf@gfG@?Y)UT20hsuMiU`!QVKlZ!ivZ2H#pNkVne;N7A~#nsJt zV5b25j(?J==TS`jCqeVd{$j5&@AgCC-bd;)u7`6GUL@xGZx~`{D0m+e=DvD=l8B*^ zeh+k9#=h0*@h_nDa_7f3)M26648AO;3N@EdVx}}7Gp^{nj8~KId;c+eVRc(}myxoD zhJHa+#rqR!Qrc4A`!mxZu?b&7i^?vW9KS)mA|T*L3b=()L-ITo-S|&k^ef1XTR(_g z!B49T(JHlxk;>_jY zon1D4W3q_c_x4Z-ESBV>Ip5BdEKXwP{px#;8;G2u!{C0KX?FC9dEMsI%1I}9PtYs_ z^-R{{y3V)x*Npi=S4=9f_luYCWB|h=B8o)zwQ2oS!UG`D_* zD?|6vd=xGa{tDwfPS2Q5q3hPVfLG(gA6AG70E)4g3HDC=%$sRw^8QM%&>aMG+hEBa z9N((m@OPA0%1tk61}%E8|Jw6Xhq7oKdZ1i=^+8?fi`U%F%v+5(8g0tmJU0PaFvDTV z@@ENvKHnvmw(bpwe$yk(v~lhp4~2(6u_2#g#UcJNXWgAX8G9RJnT8u5UgsmFX$v~S zK{hW!Vx)m~=I@|klUEz>cc}c&w@K7}b{LUMelo=-^X2z+L4dMDjf3>z%O82*H`)18 zuK=a39$7kVMW&&DgIs2dOLQB{W|IZ>EP7MXv^40h*pbdma+&Y=`<*E3->4l#$3JC; zQmnfSc?$%jiL{Wt4vjn-JO1C_g;rdErY4E zZ8dMNJT@(KLDRVq^2dBNhNoZL6|vj~8Ks0F7PkS*$B@~xHyeQN(#e(!w zU1BMIJfifLbcteMa3E8Cu2hrwQrf9#P+jrfvU5(lPL+NP#!+q?gaEzTd$Sg8EvV>s zr2A2pIo0Pp)HNYO8^SgXHeV{|?pL6Fq%tDq$P}q}^b(00k z2JH)eHJo=swo0wBM%HZVHzVyCr%({jlBoqnR+FSx*!^k9L~r7GI4nA~19~0ybAwgQ z+`DXABylbukhhaLuO?E*)SoMIKIwTenS}_M?QD$`TaL=!2Qkr08<-e`ql|J;xy%257ap~Ik#FN(CYAzj#e){%>~(3>L0G-aPGFLGXsK?j%uMKQB@ae#K&EP+paa=?CXQTck%Ibck&;DW z6SLUj)bRACa%jp({9LdvDWwLiSk!R+^-N{^XQqG(vUSFN#N*2>3Z$bJxf!~@#@8P` z%~)#*PAF0Mg{jLs`ba6(V7;ukwH5PfW zrrwQ_Q)lOi;d-@RQ1Fc-$RxL^qg>?jw9T34(BssnOH5y^pS?AgKzJcg!x7`dtz7E` zRkg!hAWwmCEUP#8PNf28i|>MMghJ8@1ikgnJ)Io+xGVnE$@ip3tXdE0d?;Ibv22Vt zwMY@{M)1$dM~sM(f6wAwXQG5fQwqcgiNhV;Q$>KbzdiiW2@k+b0kmP8n{BT?6Q`8Z zyIv9J^;{q~c0>B?E3>pNhK8LKAuXIiA1_7$WNqfs*2e@1jKvVL-~{g4RrC|O*4*IH zm2|L)mmAGMleRpdo+o^`v^5%SaTvD##h5TJTWGhf6;sImWm(I}+Z~^&zA@MkYdmsz zrYBG;(wTR&`7Kl3cpvh-asE$-4Q6R<;mb>|t0UK!pYRn`R$Bh9TDso$Jz&+YN+GYl zn0uv09K>WTarE>Ktttz)E^212JI>z5?#08hpMbof*DWw)S+rQ(DY<{jMly=cx&9Kz1bGs2`yTuJ1@Ar#a zn>)z_f0rtGNdEGci_unpf6lts=wbA)lztAR1yV$6`P_%5uj!w$rMf|H59^-XsbyPy zFG!C=t-^b(ayCvwJDSy#_SIo-i=C9~t)j5wR7SxPCk=$QGbGZ>|4x@C*5Rh9(&}Of zL_y>uuu=e(E5Wtle-N9d?(AVwGPt?oD7{i#Bm*4!QtfY-`A_h^ECh}Q{zOvmwcbiT;Mpyegfuu$_lDWl?A^SW z%<1&^T2A}{5#ARYval$wveK)t)oeW#JpZn}en?@wMT)LPmb%#FT4wn9Kbx$^73#kcjS(uuy8?NPue#o!sI zUiH(M36RttUN+KmPvKjL?0LJTj7h)T#c0g#Y8mwG-=&!sMfX&fLe<>i@<6|8v9A*T1oG zyGuh7to}2>{$CO8K4-=UU`h19X2CQSkN>;*`~{e@;bRUZV}Gu}{7=i}f3K&IJ!OE{ zpmO%8s|ye-{U3r7ELV_iP7Ze^5t3tH(yNR)4h8IU8|-X6O4vK81@poVRn> z7yhcfl|M(6)M^#AlH?pqm;S}|?OA~oTQ3srBfllCVvF=oD65jx7P6OsQ#3> z3r^ZKt1j-r+@(XT5lV<@Kc0R2XX;06Fjt0N5>d{}uxkx8wD~tVtj8|S4q2FI! zGU*0mE@Uq7r!BTMD04Vut=k=&tS?DF2ghr3)WGW^IYa(xil{wNo^?!ioUc>%^=&cZ zj7?@3tpA&V67x0%zg_j#5}f!znY+})%DLj8l~X9jEzP|OBJ0<5R1OL>?7M=|4zHj0`^opmC`hK$w7 z%E)taV(KFhpbP2z1mhWX!kW}A8(@myLyLWlI8w_I_sBWc?*hXPv?>(yQ1`M@5duAw z%5XZ7{wGRo(Q|yQz}WHkQ77CfbEklrtvVa$u;j|1_ltn0JyXmmX(^ws5JP?LII`C`fk|#WJY#=!!)pdq;jmU za8=cyP<46K{W@!KCNK?!ZT-FFFaT`qPKsca47o=xQwG~$BS*y>7W8Cb!()MqU!1;s zyxKk`aNkxm1A~7Kx0?XoeKCnC4nl9o%7_{Dpj0{2irmOukjLXghGKn|8%lLem!>lhBm0*8=2>7j#ZB%MtJp-l)D zrunN5iM>&)J^SCcxW_DL5+9)=VWKbVJoD-Ni0nH>0um;VHx#<0#XoE6-?5pW9ItvP zE-RTor+BU{US`wH|BmjJCwy(ma$IP(#}axJjm7>4-Pn@?H~C5bg3 z&n?$Fq)n9>^ua&Vdrp6A{8Jr+_LNj8!I_^tLO@ z833KYQ{PXQJ5UP_)<$Bkn@8^*ECx$surY324kc+R^?g{sfsAFUg7mdp6Dg__W?Ysm zp?s!Qe{$`0*pxSGMo$*Q-nw7a1sCbnV(V3I|NWtwntz)3E~u%(0KiwTSlM#6vvP^xRYic(V%uJn=}E{d8JZ*{l#wa+5G{E4$f8j~R6VJWo1uw}w)>pWAE z!-^68iB^(=-hyc~K2>#hXaU!df9XtK+uOO4w#8@$^76$=m|10nh8K)mvm!D}D!2#U zQwu15766ud?rXhr^@ty8EP#_%*6)C&c{eN9wLfw&k2soW{*|rIQ{z!zFk7B!^*p(P z1C6HF21nxx#h4XOR9QE|wo#Vby|mT=TVuv{FOvBhP-ejKfo-M3z8Fd&N)X|1+uUj? zHC}KErfG&*vXD9#t@^F=ks%iD)0A7#=^z2;X<}C{z|tV*sapb+VQ9C6cG$-)`o?-+ zKJ@{})Tpc><@1pBRQl-gaO#`{hk-5bs9IkFx9 zF5&Hd*4$wh_Tq#Afaqb}GmgVV+DPjve}g`7jtoOMAEt2+gexe3OoqmK+3}EnXV3i4 z?7#LkAdCt$UK8H!G&G&s+VDX7y%KmN#i|c9(uX->Oe?5dn})U@@?(+~Xcbn)#^KX~ zj9Wn{)NvoOh{`N926qR+KaAImkou3$VM$`e>P57(wK`QsqEO7=MuYr9+6|}B+L(5! zcrtFGVmQz6H^yl36>u{kSM>tA!BoDVD{hhFLc9CE%aQd?;!UYDqt-wz@48cPR+c-( z=a!8e>ioRbFv%xyT1yH@DYQ20;OQk>Vt06^!V?2fB#>MUVloupVK>AMREiD|PKiJU z9S0@FOT;}o^8j`X z*94i*V{PD&7uXCB64dqIG+^6lNOF8ejF%shrE6+O_5C!R9WvzV%6(j0zSRV0lUn$? zn@Z_HvmlvXJ5j`4_|y8$)%F`ob$$n;07>H;d`eu*=U_(7kA{$m;zpT>o5{eFuJ*&U z{bWc4{ak&Fz}^%~ifJsLh*kgYwEMn(*t0rprN%OyQR$P*FNF4}E7mxsm>ASm*b{*_ z$jqa0&EgL+5wR3gX_(X>>G-|Z!_Td5zGj!b8DkP!oznC^n74?r8vDloD%=)%S9pO8 z-@VF`!-xVD21v+0<*UgJVF{=e^P}}qbf1!P3WbkKBha`9q;fh|aoh*gGdKZUoEGfx zkGsw6GYDC;*)cX>w=8H`v^gbHY^4Np`)8udi|>x}R?6SzF8&8 zM!k|>;80J;SsC4uo@u{?p%NGmiEM~wsf>a4^kVHF1~NgwSCKQxW;7|dXjCxx)9j8u z@2}+!r{evEdRW8q&9}n|gV-^FGTmf{s%;Mc3u>OIWKtTi-X2Nnfl0C2?zMFO6G;K7=2|RQD zGG8bAz!aC!bdS7!d4iY{p`je9A1<|Nem3zI5$nNP1bf6Xk+vJyI4ps3O%}&SM*p6_ z)aCx-yQL2eyRZauZuunF$?XXlva7BA6LZWH{YY1Xz0C<-+>_rppTsVg{S9dqBv_1^ zoL8;9=$F-PNOp8LUjw-oKWadBT;C#9cLoT8>^xMm(`XJ?8Tw6b3Rc5yEq;jAQemEq zcQR~32)-)S`uwU*+9{g1Vg)pEUgv%w584^J8RZK=uH0!Uoc=N`e`V1%JY1GCGGcL! z6H(i!7x#6uHB3VzhiJ=2+ZwWB0B=eJ@(tojYH%l{jJyWVc^XY+O0OBs?D%4;)hdj-M1Jjmh4iNrwBv zT7T?WO_UkPzU(tI3WBc~a@!(C4Q3HY+eJ2LFyBm25Rg0g%K!c_+%PJ>o+((yE@T{R z6%O-d`J5YY-S@_ zAehp|Tc+2UvYnG%?I&!AJ-EI_7PR;Yg2YTX#sg167FDB4V>a{Cna!c7lKNyFet|K= zHD#ou#ICpU*$(=rO`WwE8R03eFRFz8l3Z`(K@~HsO8wFgV_q* z*3gVTorsIBXQ1#c z8?0$l6E@KSArs$s6tmTpr^Rh$0XCKgy_`7Y7FS_?3GbXrc^FY(;2EhlGk7|WKZ7twt;;5JB2<3W{tYi(e0U0-G+Q(*?ueg}m zZGdv_ajH*JyG_6;%O67&(i?Y{hCa zx?-+_-(Zatgpyl}<6}?#+S~nlXQ|p=*s6h@R z15>&U8+{<8w-?v0m=qXe$~~4g|YisNDiwekbA((t0ERz_eqY6u(-f4rk5o~PF5H% z`a|s_`dsj~A zrC{gQ+F3OB;Ykl}hspgXMY`<Tbi7Q+^ckJ+sCQb_FbNduHbD)p3+rVW2A9>Im z$WlLp7&yb-h6B%;dzY%d$iuS`uRYoGYb3{)s~tdXa0)*tY1fmod~{VMJAYN2V2PJokKvmZ+HbCvG3vFMskeF>WUq*0@? zNardgxAN|q0O`&F&h<4!{zL6E(5)S9Zs1}Py4j;}cbc`R%2$W0cCRcBU*!M(=#$pj z-ABq^=CJ#Qlt!6l@Z}t`jeS&*h!JxvCeRqr_`4BwUa*#2wfjT**s-HVv>2mH#SP1%5S`ZPPt&TB zA5-+u2X~3r+0L`ow~?Wbqebgcd}7b>VDW{#e%qxZ5ur)4{et$^S9^cG-o<|c-4X>S zGk#6ipyF=1i?2`$BTL+M+;p&@*M7YJ57(e~fqji7t|gVFc=BOdx%j7B5rDg{)`0w0 zE1DbaASwkm?tDp?@<@E*C`+NIm)RvcI~ItwH@?$PHhsDy2*zp&skXmE+A*p}K&{hn zR*)&X%R*smI=zQyB3wGAeqOe)6^pAp=S2$tkg9VWctTe+u+ zxl)V}9Eo4!#GJH&0)xDMWZndJHak%QUjrj8dF`%^PN_As zSFYgx1H9^jSp!?Q68VBkdsJ>ciNui0=r+5}{M~HF+&CFE08*ffo7t*TK=-ESqqt`d z`*&)NQS52g!7rlV?6T-Cafw-CZ8C!c7>EoWmo?k|`+W|P80k$D^OkP&5Z{}!?jL*Z z-pgIFQ z%E6!~EBA|l0G@-W!jk5jrFze=r=*3#y2kdR3W1HhD_|WqK+@W*U8TvhzUuy3B1Ic| z^6S28qs;h5(&a&Q4z>a4%X~y&(ot8f`aOb~`m3qss-_wx)M!WZmy>fXXDTQaLrTNn zhehPHzL;*GCC4?{^f_E&jinSE&*C_SoHpl#2d-2=^ohc3xlF6FG_7?i6BAuXZxT>2 z*BC~<(Vfzjwt{;imkPBpG6XGB{RABOAe0`8Hy|xpal|sI!^tylleW>I1+>K*JgRY> z0v)d>X$rlKw9Dce#iJhr$p6${W-4z&=#UL_12(G{@Bp|_XfI8UcB&5%5w>?>XlJ(kxkG7$ zKchuhAYk66`t>E1Y1<1%)K@2rALvoAWn6yHg;8K9FYB4C^n?M1?|0RslDBoaPic-i zAYmPW>p%yqtB@Z-Kx}y6P;2PWOCoiFI_!8v1;6^m{Y9W@EgfIZg&p(E8@+C z{@CT=z}t3FrcAHYwoO!<^nwn|o`dk^q9;N;!`*QpCfi!M-Zr#zm)u9BV`;1S-aPOx z-S5GqIArB-H#FX?X9X8S1@mZO+r7L*9r`joUu_C?jdEI;3az zS129SUQ4pH3zd6_9e9SQC82x4o28i3ebC@vLsc-+2(J&ViCLoQC)FI(Ov{CHqwkYV@cUhkh!yEj_Zqf_Kr=O4%PT@4a%RM9mPk+rVFcg2;}*6zBE zkBcO>1Usm~r&oWKZC%|keQ#Vyqt&^!^SQ7|rq!kaYdMI;+O~k6XTle@q>9k3rA#*1 z?w-HskK=BhFC}ss;V+a3Nwy)>YJoHK9lS4u>9vxwNX>m$o5AI>O$H!~%Dn>Qj=1C& zVkbM=f@eIHE5kb^exr6jfGZ~HU}_udD#BCrvS(>(+URj~7d@;3lv}*@qPUZ9vb0|7 z1&P544uWKm5}{h_Q44H;;l??yt3H^MkocSTNX6W4pR(Rjy6NB(JgdV%&Jie=03~rwiu~yxR6mN>0u-u`_YnN*!^7=ZyAtP_lFNv%J z59kjZ@04bpypdd3Y;^Izap?`>Y6hNqfC3!{zlv6+T9?BtGbgo34hLSKUoTn6FSCsa z^OM6)P=Q<}Q6EPPxEgn&Mt+f>)m0QNFI7EDn11$RmsQUjY~*d{i#qZsI-s5B>;f@6 z%kaAS+v{w==|~AivpiOHwon7@VU8K)vw@|MjAaf}-?;$PS#Ca17y*>rYOBh4lMu3Y zhTW^=vQnp01ujZ6T0T{pZy9Po>fdS1Ua4gq(UN(F*K);_gyOp}I`!7mZwqamH%#iM z5;F@rpw?#>9%-c2e0G?^B*fj(Wg4^_w2m85duHTuN6B~o?&vU&*S^YO z!C{(I>|>fd(Ado83|i%k(7+?6K0KM4Jh~2&UelLvNKano3FkgN^l^xM8x~Gg0YUHR zmfM24#8mMZRmAP4x2 zk1G~8-P&~?xcGHBypVVWx`4RFf=Nb;T@U^vNd-l-~8;7$y4AM*Ihc^o|2J z-qJkGH@{ay+-J?behy=RK>m5Sx-WR)>5C6^N+FE(-!DI_lp9D`X;R~iHanE8X0b7! z8{oLqjOet+b!@Zn=zr+J??}#inll8x6@-wX#e2}8S$sIe&*<(1+z6g-cd5xae zB1(F507IOu5Y&7e9OQF?Rdvf#&KvK)B5G}pFF$flvD1CMVxfm7QuJp%F4O0wUY1ez zdg;<>gM{AfZ$JzvMETRBANYWCi&~QdQj*VBe7(Iagfvu66RZc!8w{`5d$^0 z&rVYo#zL2|0fpb4l5z{x)ma{YxMW-8_Y3o)=}<~=fqi4oz@EQ#n7L*2yJNFES^ThH zMS@E$6{HnJE;-}N+J|W`r|d4E_pz4bWHf4gZKw4x{f~&t5Z@p)izA)gAbdaE%VGF{ zB5<$v=34B9P`_<3NB+l7)5wpXd4B6Fe*6=a)G682nV@~)^LWeqU&_ya5tJ2sRLuu- zR&a95Z^W9%T9YC#jA=G)9ZEgmpFRVnLh2{1GgF>yMx(OPLJWhS>`{X*+l;iT-CMSA zp}um?OQmbR%sC0cBw1rM{u*j(If8vEo4-RoAZ2=n$E5~tQy`}VxRAqD?oYut%e+0x z)v=9xLfgX`xL<>QI<8cRd=bp#fnL02VUxb{Q|!b@upwiV$8Eo?9sX__p`ZE6rANk~ zhJ}&)Tlw4Q{2zEiH59(RKObU{F@fjtc-(HkzV+_%gjrt{sZ0`&Vbbc<2H4fNb@-(y zF>(KyNO7PZlb(-Bi?yNcTx9uFd+ivDaTUHMFumRJn2ddPjrjIxxj8+4!+%;o%GGp* zC?s?7px<&wzYEgVK#hgm7l-K z4-CH8mn|B0Gj`=w?6WG?yPqMxP#5=Y#{xLvhs>+DY^vvIn<>S#JP+ukc9?uFi0pO; zo6nQ3=g<5va}23Kgl@4lH7tF9^mJP0ayjMr@%Y{C{va@-KL0)XL(R5oLz4n&WfIm_ zZ>GPkR%H80<;TprE&*A4cs`#P>>;#v>Ce}!zQMZ;!Q0*s0Z;$dKzFYX4X2<}pxOHXLW~AR^G>#t< z-F(6?LBE%}?&gy-YQIkNF6V(LR(+2N&Oa^`tiXt4Fvg|FPqREDDsvcRhqY9(vOvDdT?`V$2d ziI2y=uWiP&t6c=fe=)rm=8ui(yXlEAeO4!QFxQkDo&7tAT{zA!t>8tFO zb<6rzN)DJ4#ZAi%%$!ns@#-m7TY4T6byhX2Ot$BEjY5YEkNtrf!WxX)%KN z{L9a=;g>8MHrz8EPfswn4C+>hncKx5?6z5s<<;h4sxDkMh|ET(E%SqQ0G{*lf$-?! z>NA1nRyrtqegDdCiyazF-&5tm@s4L<)L_l5K|`CXXOHyMO&J;zQJC3?$cO&A;{Lq) zmhKYE?6HGlnu!oowNbYH`0!|r!*oe04c?;a;O`hHU~tq-F~v66hRbS>cva>?4a*dJ zv#X1fl)n7sIcar${Ug8-u_+!X6%7LB2UY`6T)k?-@cbP)|QrG$0p&=FF%mfba!5CJc+Du@3lX6&R!^^ zFZ^M^@_3!=L~QfS-p6!_RP3o~!+@aTZ!P*7eF+ zH*-_q`ByW9gRaP0XOVV$YG}hVz#;zCih3mG6DuqdNp!8+ddz~L`f7&Xc&J=ODyJ9Z zajt`{0-269UXg5V%bX@|IC<6Kfo~-ZX(SZ!D`2WDcR9Dq%pQ?{N1kIK zf9c=-qo^!pICIJnpiNGCs_;>vo7|k=1};MTR&R#y6Fy22w4|TlP;~?bX>O6(Xb~EZ zg6V-WLYm};!NrNyUXOBj+6hjw`_7-iqf=Jrx;IXyjupJS;Sry0=G0_9goY2PVwI+u zTN!>PEKaoQY3tTfoNsgK2-B-FSUTkvRb(ktLGkLs%K6Ac?t@b^c6EH zU(JCkR9?^D?K~~IkX*=J%diRqW8Bg)T=p9sS<&iQHy?<PMKamW8*BoyfH+naA`4&H*z)YvWQ^SA{ zf$eQ)q!9I*FF~jn*;B~pV*|$Y0sB~9zT7T2YCnjMGC147GMpLqw;CXf8hS;JxOK$d zthL^#9+=Illf6nvIR<%D@jZ2cS{Va5K(sPldhT&mj%tti%IH0#h{rHnzO>N2=;=NN3I%nRltU;Uf}2o}2OVqm6cnb^qAXym&O~ zlSt{V4p2enrVKVD)D3u0Sev!)Ek;L~MGCb%tP}jg{vs*v)&71d6jo$Lw4x7Bl`RjQgK)2(Sl>z>}3= z8FIS>&Omz;&L0v#j{obyK=Jnko$DBJcA?Dt#A^Y75U}pf)g!FC#p`#Pouy^hYLW2% z)4vBnmV0~%%v^Y;;x;frn@!?>G4>Wfac$kUFd7^JBtUS2yG!H2-GT;(;O?$LlHl$T z+})i(a1ZVfq;YqB8*;wCZryv|`%e|s1$3|8d#^Q@%sIy#!-Y=C!WMvVb*v!3th4gB z0?!-HS?yzZ$faErx_Ht@r~6Nr_@CTt9?rc9H5?w$MF!>mcIbUy`|24#Vjm6he&d=9 z#!=2WdFH)e$?JyJm0AG1;;5Q@oZ`TY8rrLkkXPA+mrZ%sPaUTu23+tNGxmJJ30D4W z;DTYmgQ)=z#z=})$o`CAT0MgC-+6b`Urn9ZlQP#{8p@?@9?J43N@L_1PDTs{TAtDV zrzQ1&Z5jCyCYn9ZQ#}sXNhf5!}+&MRU7_Zoaa318-)}mBz ztO9V3)#HZ=Y?+ddaa3gOJ}%rz6k>EQI`gUb>^l1r;uGg9OQcswl)k`F9MOFFcPnL2C8%lXTt`f1{FcytHp9_ELdc2qBa6&jTzG;VaNr zqvzJ+achYff0B3Q61U7O&!sID%DWatC{GB?5DAi1v{Ph0gm`cQC14}(02@Jh9v*r1 zoarq9x8=(N)fR7pzP82$F_Q~9OJH|+g#hWp0%0iVDrq)T^o@*yVj%vY8HSbwDwo*K z*sCcQw|#5;69eAqXO@^!=FJM(EB>Bqj%T*G5F94#l=fp{%8PsYdL#wrnPNREi@tz$K!gWAL z>Q0Q;@9-&jP;Id$rmQsYwS=0AAYEZmItU`90Cj)fNcGY38_24>d367oD*RDQFo5+-{V$M zF(jrX>e(Q`#I`!$|DE)cFHBXKpaeWh$K0y~lz%G`E+~f~A(scqMd^0-2^fWc^%JQL z6Kx7H!cb$G@Ne(TK3))zSqh*4@3XrBn+S{FLw0PF_qCANl07!a8{w8| z(-2{ExH+Jy|9v9jJs7Ids*yGH255L06W(-{q@eLqyc-~iqCj|Kb4Njh2bNE`itPQP^N2& z7W=#ft>i`F4R~9FT+l6E#h*9hcekbdL;e&H`< z{x}xGC^MG@XI`mPgD|A5 z{J*`nki5ptGE*kU%tS^nsu29?Z_mXrqQ&^0MadpKKz?#^qSRmuNc^&sF#p8ovA?b$ zBPW+UTx=wFN$16I*^Zn$S`>5igW78GheaB!G(3qe02A@qi&U&kIqgFaCor~)>|Cs) z60kKRy=9=&&(x;|6Nbn93J?LgpMULNP-_v9oT~@Q8F@ssrg$8-LIDcO{%@GQALnV5 zXH!F%MMO8Fq2@Ch8ib=GBU$)81pKXdXlSIOF=95fIve%yaC`rb)qker-+up}uOGxA z%G%eW_gmp?AMeNQ_k5c-&g7Rl6=+lo$Yhi2W7mXzqBL(;NWPiVE!EpZfH>?rxeR3h zQLE<30u4=m_b2-56+i%LSX1(+P+*Qh_kjKVH8aTd*mo$IU*&e0w`0>uhV-_0$tW#H zDz3=flb_}$8>w$wAV|7*B%zq3sX#2E{f@ zX6a_wvQp($oXYRTz1`CX`PVA`F~^r>{D1jfkO^Hvm8W#n0U|?5c&(w*2bWP3NmWE7 zAeL75!{t!~dj^k-Wj!j0KVK%2~nEA%xmrW0Z)(^oCy>>2m0M|!!R9A3foo0eP9AxOD1ixUxwH!n-A(=9{T?|aR2Q;-r^8-eC!Y&N06aW9R&-u3vZ1T z-qNZ}R}A+Kadf^i6zHTQB|D4mHfJi;`RIH!PrXoY702zg7eWhN0h;S(beX9vVwk!t zZ_SJ-(Ww3Wc2Y=FL7(p^cY}pmmoVH|YM{ zeVU7bSIwFI5$d#I>DmItS4=MiplA8trqq9*a~(bi#7YjPs!iC{`mL)bXL&f{tlyIH zT)LXrtMnf}1!vX^=QH>(HUYAItgrMokT3HZn!=qPTN^npjffymcrvg zXsAr5omWz*mMumv{t(Kss0|0+9Ej6AY`F6Vxt#_KC4*#93w!PQf9Gm*d{Y<)9xRSV zr8s5XX^-7j@?~eqe?ziYSAc{Y@FT;E+>`T z{6Om_h|W%ypPQ0T-4ZtjT(7@d!T?;*B9%o+@ zd$G}C#Z6Y%gs zPP?&|OD#A46NB-zho=g)M>0mkitgyQTbJtm!w&>%5Ut5^ayA z#1?~#dB>ugCz1d0zpq2An^+zV1j5>Xbi9P(|Jxk=U;bRRfm!9kOWwC>HF%aC`Uf1KlGHSrUed__ja$$6+IfqX@UT6n{z{^ghh!*d@dXkQiNE0+gX(Y}Oz8T{{u>~+>id*=Dm zMKYfJ|9cMpmv`GqUt{3Xta;DSI)8;}A@&uSru)Z;06$ttsLEsqe^Wzl>4VP8eK1%- zWj4X^Qd1kDZdxQ^`G}EmOOdhg%8aB+H`IJ^IRD#pPKuakQ$ym9z9cHGh+BNAOJvwz z*TGlNd7yeDhYZ*E#sn)7W{Y5v*ylh;k6WM$JmCj|zfAsn+9ZZ2BO@C)`Ix`CeG+58 zW$E9R8hBf>f-X)ewegck)+3jD=v1TG5&%ZO*s~Topn)|9AZLnXc*yh5ClXk!c9zNB zwKFR+ddT%GYD`aUHvT=90x3qJ=Ucn3)WC4tR+|_-P0ha9@UBGvYvVI8RUgy8aaUf; zec5SxT9k(3r+n>m7~6Hdch)mU{ExA4p~I`m=lS29Nv&;r0wO4N#nZt=G}w7x%IFfdKao%;P| z;$I74fb!dqHOU%xfbM}v<#Dkt)Gbec1Lz^c*ZU=$Q$kT<$lZN(nZ^F=t4Ie30y`1; z+9Jph;!?72>lZ*Tg&+L)><>&2dvVcQ<-89Jg#6v-ypPTqb41V00L9K{$76&5-q{lW zpYv`8`f)99*KxFC_*M*lM<60rG*H4N;i}#<41m@8+Gf>3Kv+)v>)vD;ef{MgC3}%t zOhp9~`zOl5X{&|LlZllq4wse@^Ln|f?-f|oey-xa7XN$rd(8xMOo*BOMt@ndYJMlD z?^MIeAN-FWDVPT@@)KfTI4MXDpF6Iw?3nDIrKXR~Z$Hu`6QMHc5KaDBr2E+~8+zS` z<(5ai-|1B;hTXGb#Zxy}`0?H7^=iLnr6+6za)%Q@uA!#msc0`M3Q7z#UECdrqv?-{ z;G+SUu34OJ9ebIwm1s1}{fQu84F(>F901@clb$+*cegy!dEGYhmS+aH z?bgW|Mp#K}2ZNv-O@?=;a@8xdxG%Xhn!Ud;H^6Nx%>Jid_!#83LFVP~yW8|X)8_Ry z4AlhG18UXN-^oBM=+@H?{r4I4hymY{j_nuR=hd6NZztFZLVWVPf!Wz-JfZE!w*)Nv zoNiCq5t~<0K8aeQOfUksV_%eNrQX()`ua6E592~*mjUn>M#fyJ_=+~2;~gYIY&KkC zp52*JQk2~p_MVkb?B)_c9ioPg74tt=X1YgblAJ&xA_NL*4U@PjY*rdxd_QJsSmfFRp{S=RoqzaRSx8W?2F z6Ss1aP!@BT&vw`~-rt<|YuB$(SWGdw0TnQYi8ZJk=8+)ADO(gYZ55RMWA~+Mn~cdx z{I03uJa{4%;nZ6^5vVYMsa7luLGj)Hu__kmMPCMb!M5W zs{iHIc`e|lFbs*m`?Mr;pbvXd^>E&W7D~eIWHVj2MXyeUXUNh2Q#cq#OtaW=cWfC) z&}1mx@PH*_Z>kGhr_GHmf!Q8eO$GWmC=uu9W6~$i3|zku3)Ce*tw5?DqdUiQjwSEl zbaNp2;TV-4m*~iJqLHuHFQfuJ&*!zOo-|IQ{(Uaw_4`G5o#=nHN3lqD*r0R_s+LHw zlHriK94(WYu(MUR(rH#MKC%xx8yNWC5Msr1%UDv9k#X;mrF8~b^_r;X3!s_18ZL&tIR=Ns?)D-ziEIQr_LgZoCbJ^Z~2{#gfsXxwc5ldq?M zX(F9Y9m`ko{>;Cg8w?NRkZ^Dq4(U3j9J<9J!M@?v_213MKjxX>%}p0X^K5ePZ=lRi z&PQLj)2f!mvL;{6=+GT0TM$uv3Bh8l1g0!tU!WKlZ2jb}uC845@hqhV1(I(|?N9zq zBRmu`_oEw+2V-TW9V+wMA2g3;f;gE%Ti4LnRFhpj{=)@ezuNKP3^R^a3Q)AQ5|TNZ zZJY>53^J)`j~BMHqp)P}2|1p!yXBi544pgM{03f$6zGwO3{H2~n6^3R&v2|@{qu6; z#Ix|0^KF{gT)A{xi1C#NvyGaB-lsdNLsxp(Xp|4p*kDnx6-K9DN{Y7C@^KRCRxp4(}^Rg&*r(~960 zH+lb8$or>!M}}i~f4?L=5Us;Ub=VCKjbKX!(dVCGK6!O zYcXp6Ra8mXU!f?M%0)fecFGaf@zZkx!~yI&(&-&-&~_~5e*j5oJBzW3*;bc~{V+bg z_bz%0uHv`*dwy(CaKi*!g3n#GDg^b_oc(rXj>W6(?yEX>`sRv<-u+UQQ!qZreR6!f zf5LbIe*ec$lowNxgBqOv)tB)!6x$^p0Xqx<$e$)__WM0gecxV~ds#3JE(Q&WcKKmU zi@_9b1l(GYLF>Ge(bW7@8EWGqSoWdG$$YT%;a2j(d`d>bn2H^p+wFX|lErYEF5#eg z9szANARsn!tSsEbNmpGxu`4ohs*d6Erp=h$AE$cU5?;!Y}^X1XKV->?XXoJ=0;q;uILM z++m4Pr&b3@{$>|qyl>^7d56BC(MI1&&@AnAeBau`jga0h-*y}%c1T6;Rn{&QU#>+9 zEChgDeSOL!-)%?Q?`jW|V(x!jcK?8mXcauz;z2_+!uZD-Z3fuvk62P3th~B|H9$^z zq`nt8)TNXq22PB&-=;H|evboZxGHSXsngwE5&ahJ50T~3=B|q~A@U&}tePp7t7LiE z3;QDF>Wyk9F=@+d4Yv4Z8;?!Y)r#kMr6~gsuisLC+&;K)>)&Ol18M>@^;F1jHWm!k>V_@MjP7*ReF^k?4dF4^ zl0jQ?NVz}CrOcJuM9dqVTSM5FE8Bn}Q*#h*#-D32D~@uL2r(L7u+P$+X5V)Zu^@V} zfQ}iM$R?YU0Te)goP_c+ig_<2QC9Jv$8a#H%5#|F<}_ja^dhp7#h~-tm*v$$ zSo`$SVl$&?x~ND1lD7hr;zcdG>&2%T`|e$FXD>ot$G*N|brcsXM&~(pixH?Bi%uVk zZs?77$m?*Zge<4XL%j5)y`c$!d}U)PztOGbjm^LiTW^$VXOnR<^xv2sKQTnaiXZ7? zy!@+2lCH<(_fxcSanap~(i86N8{3tbrw=yko7Eq(KOb*P`i~j@7LVSYgV1gQ zqkHpAj%(3p)3hpYqtY_Uwt}2N7}aJWQEtF_@vX&j(4*(sx}LUW4cVYJUVccfSSaC9 z9l;&`xD`3!%Yk0zb%%fM+IrrAT6=gDBV0t(MT(2Wo$%QYAmhnlQ6Wcu^W&z zhZlo-EOQW%(-h@~-`?=1H+-(JqV!<#H6S5#EAuMKJsnu%>?D^Szaao<9} zFqMy9dAyd6XV4`tP3D~TSZ?gWug0vt-XDT+uQem8*%gIUG}Smys%xoF8VCtH}p#C=2Hdc^PP-+L2gJ1 z$;v2zKl@tR`9qxs?CDh;m@o#QA4uv}Q*Z=M@_lcYxRWEZF7~FPE5?{G8JZn;Fw8b` zr-Ck$2%#r6s;#q6L58!MGc_ZdFC0zR-vuk*!(QCDD3u+~ixnwT9hh>|50rbiDhWLp zW=4X&5`X~J{rf&k?TGj_1X&S$vvzdTOjSi|L1UN6M!-}Mj$|XNe z`#9w7<7rKOMwiv+YK&5%G*xJFT}eobV6d@FjFkqU=HWA~t{6m{CVUb;$-EeR;s zfv}G}2kW~cqfXyuWaMiHokRvfi-KOJ;L7h_)}{&CaE@(8z#PP?b}JUExLp{k2W*Ta zF-G6}6$l|BhiR&}LCSE0*IMNF_3521@+bw$?mI>c6Adfvl#J)$UbTQ~bmaHzE(%22 zO28}MFBmO*Lx58@(ICfu=CEzsf=s}p@g3{?<;7-oDWA_6s52AFz7SBPNg7ZF$eTy9ZRQ@h6CfMNc zQ>K=NADyT`cm|jLG6ItgeSw4c8xsvKqg(vStte>gwx5%dGE!^~eE@e6Xse)X84ZhG zBlGGh>J4A})3haj^`fz9_nXu_nP;=4{ zBsGy|Gb#nvRL+^6f?V4RHm!<1kNgU{$f$h}5v{AoFpn2^^25?!C|eiH2?fyvF1|Nn z#V%TDvd3U&c+aLHW}+Sa_!9gl=ZalT>S)h>fDcW8HrENN-C~r#FrSgIy!AK$S#5o< zLpM03pEnj>*Yi5)o^8WLqs1}T419r|5gXBdtrLl~R|d)1&&t?pWiL>s)vD@^d*95A zY%|v+yzH|ME1p2)bLb<6uBg&g;C1=9$0Ee%t54T+Q37R?FW+{n$p5tJ161rhY$G9; zE8yQBci(MsadG4Mj;8X_M)02%DQ>?BOzLVAwU*Pft`rM<$oHqvH{N3kknKw5$~_N_w*!NlnI1MVYAd%? zvK|E~x+py_p#%shdc0|&0o@y~d$z_xHOhXUWvXw6;PV>+^FJ~wbZ;VQx!p4?5Un;G ze^y`aBrLP6rE6>B1$~majhS>X|5d)S{U=!jQy)^y5XX;MKgez>#@`{s=Yl|^-Y&*n zHdy)td!2XLX&23i{br-u3i~Mv5)?DS<4N(A(ZrV3=|1gIq9pr9pMyMq!S_>x(|e1f zWMjP_^r|&q-S-J2kN7>*+OC}=5pG_6dG#6Z&4NP5inD_at75NYdFzASfP+r&^#k@% zVg8{#Y04x1ZR->}J7I5CW1L#mUMU^d9#s)>3&6~Il>Onk^T%HM>XOCzH~!94Zmlxa z$M>?q`J=jJW~iwBvT9Dhul6TlnayZ!<`S;>ALp&8x+j_{UWv`R@ZU| zzrXkR0P_mRPx=k5@s@QuD#G{Kgyrjw8fdrkKCk)|*lMm7jWRvvgOI1Y4!TtB%(IAQ zn`X%hg$wkw^|asK`r{UAX~BG#tJ}u(j=;SCzb{ z=f%c{m@XS%nep52ZqM@C&K8fqTc_dM|5E098ceQdw_5Dj_tW~ti+WZ!$Wx!te9`2} z+3>ha%fJNDj0+^08wt+-m{o7}1H9mjkyBGD{4aqELFd=-C@4`D3^o=3NPT`Pp?*9t zsZeaSYgfc~Tjn5TCLyS{*jOCfb2+bl8{Y>Hq)H*1X0-L$rtP$5Ae9DN7JU5^ydu@C zI=new+{EK*A`0d*yyTU|<6ImfsO&3B=5F=8UA8cD%pC&(tHm5$C_|3JeGH|1f3HpZ zdug>27H~ZMN2!VqM5H)UCz4J@tNa4oP&lu%sBup2QO$!hiEU!VS15$~TV7)ik4L%r zp^Hx50v(I5&$L^x&s#GSTAunje>W)dH9o}SX)mL}5nBVt$-j$=5K%IXJ~S#Lj|@~g zrmy~F+{sKM?1sEZBC{(=9G%V(KESTyJ{-4a_D;a=yCM`a!FA(FM{-%rEtk|+H74mP|<1ryZ$u~GrJhe8Y zn)L^4H*RY?0Y^**!X#fZ+Mm*BXw^z%0EY|d4X%Lq%_Hn^_KNI9V1Dl=!Dgvi26KOZ6v{s_wgXnI1VHI>p=+S(wLrvCStI)ZvOK%5{6}w?*FLUoh_TV3@{W z^@)T1<~vV~1+{ujz&5JuvToifGSW94_e({$t;##6Tt`q6vxnb0uS}}X!y%@2i_<4# z^FDCsK<`nTK~cvT(hcMdzb-cKFmT{2xwKyVQN|oDoK8}c*P7D{k*>}3s56xva#NYn z`x4mgLxp03av6K`$|0dda)P%L25@da++ObEF!kv6k;&V?H7J8&Oo0#kGu9&c9rDc= z-JJ217HVN{B63;W>6kSEAM%;=eF^~!LV;qg!M8&R6%bgjrZ|-wL%OA&WRfY4Q@y~o z;Ln2Xdq(~vK6>zC2x_A`bWOX9nPE4;sZfNJT7M)6RvB%XRZz$KP?7YC$>)pEJk4ce zX6@(_<(5h00p~^A{LtG>b#dGPV1Wm~I0oSy?u#P5_$Y3k+F{KO!F3b^m@P3!xMQ5! z8V6;J0FySbI%$wVd^uvr9sd)_zwUCxlbBaXUTl~nwrLM(crOZm_yGReE%CJt4g}W- z%co7ehmRgL_8at`<*SPKbfC%@N!ip_rLm8nV7Yr=Ia%ORV&UlcJl$AS_W?2?>m@X& zOW-4^CvQ$1`4#zPFO|c_bz{sA=s|jFXR3F+7$a0vOz(wPLL|ZlRhILwyjL5Jfr{;` zu13^$AR7Qf21_NY=qSmJ8&>6jFS@2sglw(rp}3hl)mr6OW#EbGUHZpW*Xh)$++hcVQyHt_Sqd zWQaLaGK&ojk>Z?c8%Zenhob^4NM{>Zcg#8yiDUb?w9Os|$D&Xf3#);!p%lXfjPJ}_ z#Li^bk3VX95ziRgpXO4h+pLf<2TCf+3WR^|HbR@f7AySm&{4bJy?iuxhV{+}O(-XG z_U`KuQ)ff*m#Gk{W5S>WO?M|*(Z}b|v>%|Mj@YQ@DCGzd&`VcH>xCo+W+(Xp-F+RB zsmBr_fdFvU$X+cQupk>!bPCC3u&+!8UMVx8JRO-;+8-E1hp@rkOd4XbFx+}lSv6~Y zpejyCcT-l=jNnL@gD(#BHW+_BFlgg#%712or}~q|+|#D8@YU;AhY9+m!+*$e%+LtA z>4?mh3wBj#Q$(dV2V(2wVlZg((RWD7-I+e5$1x;LBaJAI4~z82%5VQ*2K?#f zF+cO2^G$0+yVAND>pPIUe<2d=RjF6`=rOMcY#$>Xlh6ls87hy_RLLYm&|sW*YkAZ| za`@^&L>rqKO1j&<_gtG-0_cihDHv+bb=!8pzl4A0BDymWG5JB_>`NmumMGuThOk&2 zGA>JJ4i6VDkqwq1AhxJU!C+zSdf7r*Y1a2eS1esNDO0N_r6QWbyl2{KJ|Px_yOC0C zrx|3ihq2RdUX1#mSi^BBx6^IOo8KVZ_pAiUaDD(CW+DhzNDxlx1Su^WidM0u;ODDG z|NB~aKl}o2@c^s3jdv|hrT(!ZU$ed%>~H`g+{h)ikA9&&9%S+X-8&hXzw2gz87CXI|%ydL%#c7ddO0YfKXG9W7f-o>h3YuS z*|@@wx1}0P&(>HB=p^gwAis8=`$atE5ETMNJ7>$+qDNe_RiDvjD5M;ocATLRh(hk= zG)x<)vV2@(HC_@G#CyjeYaJVOv6uMPC;!)a=p;9 zVs_3qQggqlgUrslhZ$)(r<^Z1ph?g5FBb!U7ktnT_J<}(FZGW*rFtsD$clEimfj0BX<4M7Wvzao!MQp8NB(5L%4mWVM{KdBIa?Y ziW5ci;A2_@g&)@8V5`;w^9fl9OD-K)=7a7#@hulF(~`x#za>41istr_nKVM#q-zz; z65Q<7pI$qZ4+=cVvd*NGozVt-5xd?=q}L?afi7u8z4a}?(lzterh8>wct*KV(#zid zcujw!m~B8T>KA1ShK`s82RXmnvF6qLB0;P!6AF(N`Oqhg%xL@uhlH&qcHL2JJ*X4U zfv-=oZ|nSXZ8pjC*(lOC0s()<6h2a)dBJ7yKG%YrjO|y>USR<>Dw7LR&L3=UaoC_q zgh-G?;q(t{dK9k~Zat^)L`(x6i@4muk4k^mS|JoU6b2G#4f*Lo$X!fW45P`-9U|BP zVB0+`S(&s@@5bmrfS$BQ|6OG1lpZ(`@RPk_2!{`0Ow{fpW+{BaP02B z+s!F^nBS`mvZM|i56;(d?FwKFvco1U&1VnTM5$kB7=aI@xJ?-(1l-yisD*91o_b_R&d;)G+vm% z1r`z3^^*U7axm|^Q>ol8XtQXtNO5Z8@>=pOp6=6iaiRYr!zC272=Lfbv+<;mKSH6RSpktUrIom1l3M*XOY@^wV}1fwvnJ zH7)t6n~0qfof7wxAvXOM$%a4;*+~{--xTU-5geh9V&on=ZH4yRV(mT5My%0-%(hWq zxX?^!ByYA)1=M!Cc7dM2&}pu5XxEw@bPXH`_Tr7Yz2<&J_Kly{crNGa&brIs zgO%S`+%e##W!t-H`W#L0;KsKAnQ zMCztgOwfg__f2{$T%xG_jP{an*(8|n_Iw&n*E}ag?Q>VpaQN8)o+_#fm?2XuWk=4-#kJN+I&REbXnD zSGik!f%+zqL%2CJkbK~xmqpPM2;9HapAxuw@FPJpLH5Y(a6>cg1uc(Jq6T#Qr#69#Jt2Z?;1>6cZgSv&}eoh5!~c z3w^sX;*@zG^rs`mf6w3|rPXWC)0KZMcWeTxoF?E?P!un=g14W%OYG3PaU`w_w9*7x~@Kd1X#7yDtMhl*bhjK<5NLd8ylzI2CP*hPw$dgjaLmWy>m}yn0!Y;tAOR7&Zzg9u4MuiqdNtGI`%XE+ zd}X>F?dchogB?kP@WHUg>Jr0+m@twq@o$?5T0rdT=5(ENBbFhy_2H2+tlmO3wEc3g zIYzu}15xOKRNJ?exSbD?*RGYiY`*g*lHQfbmfgi+hTk6EkikjmrsZr6yGmME=kauj z@~|l7p{ys&s|drs&k~3iSjECULxYHsV2*kD9LaRR2Gl!0OY$%-8XKM>gT72RrsQ|e zoYK)c)P$5eIFgSZd!jj}k!Hz0To2kN@g8zwPEnIC85?#R4WUtThm9oR76(?^eX7gg7wZqi&}CtZeE74%@1+Z+ewW2H`Ca&VVR|Ib$L}x*Rtkr=6tw2M!Y{9H>`W{Y|lIkre+HO>a+bmd~$p zG@(a#5BKLOLIl|c<%YqmUSl>s9B?!>MgS_fjJly`aiN{UvA%^0@Qs7s8@$SrZDcJn z_Ix}2g1dR5T$OeU8sGX|pqSZj&!3t66p_J42AP3Tp}MJYW4kw1NPX1uSmK&;Hj|%r zI?=PLvJW7jmc+He3e(=)bk0CMAV5Uz4?*$s7)I>TEw(cdX+Yejoo;#S9t9DH5E-h+ zPK~yPk1I~)Zw9(534+(pW+wGAQJ(n1x186fx^V$J*Ugn|{Mtit>?{lDOtMH4KqZ(} zWTRm9+Xy;3tLeQH@%pP5kb(#?nKD?p4IWavK#zo4BMYc$(N4r0X{1E>hET=Gp^8cb z+V`YNJ0iirMJJa8%SoH62OzMgq{b%eQl()6oUFgeeX*-D+8!n9ESDIjid3`KShPR! zN<13_*=jiq(>w38xO{TsGu01=zs`R{Uhm<|GX4Wfg78bZ)1oDquV+DWs5A}!PvH<$ z6o4f0bDfzrd4UHILVf!tnT54blH#j>zf@<~oXYJ)Bh20C0FM8@N$Y9}!JwR*6no?v z+_Bnxnjf4y5Emisc+*tnFs(4x{$~Zbycdxy5*^YNQl(rKlx@xn5Y7O%CJ5y%|Kn4f zl=q6bbL2m$9hUbBdCL8vMaNt(X>wpa-~+nt;SsCy%X5?0HP6*6`Vb@T@TRVh?gsa; zl9Ok@KH@lqOdXJ^l)FpG{j!cutR&J)uuM7qqP%m{fa^~bQvlU%1RnZn+}^AUcf3NDtFC< z*4E;;uByi?wa<64M7fp=-qGdC7mkd&#l|R@+`B_btV%8F%s|we_PpA(39J?RMZ(wz z0Nj8awk+#(+r(;%e%D{2*t>z{l(JmydN2M5@{XXl;a{p;tgbxwDjC}x9AU>C^UxoZ zW=T`mSodpG=CzEp@=BLAQ;^GPL9DrR*jXl0o6SO{Fp&Bq`It8vDQp)Eq|*n?ujwf0 zQ`qC&oy=2+Qk4NT-7x#4Tr#H_9Y8AXQ1dZgRd#1_jpdoAS8KX13fuMt~?dO1t?=O32^jCL0Yu6ccAl!#=*_m*m8fdwOp-K~!dYL_D8| zA>Z!g%9HF_0U_-(@=EZ>Yc8+b&?u8gGfKJC^MVEAsH6e5_*C)=%T3%l?Bf+CH$8g{X8$;yt?ogF- zhga&kcKx|z!?ztBe*C;WQ)3h&m9J|+v!&|Skrw^&wUdfoYq+L>=4tFP?~FE3zQa7! ziO1#BcCVE0O#e(&k1t>+6xJD+BU>C_R1gx!YhH5z-~sP_A_c80$!70S!zj=syg z-#MG%>~ckRIIv;{=cF5+*OuTWze}%Q+b1lp*Hi%aWqix=?cEk@HZz?@ zozkDUTFT9@xQTWs@g#41dRi^pf3rHB4TdK#V%kQzMj&jnPe z2lZtomCEI0BN(26pxQs`6|dXY&>5c3Y48wG02C}HU)O!GuUg*+uoGSH1!ip7^1BJ; zN4HO0aL2LAw-?SwOO%7E4IB_hGgQWy6ms#XR+@P$eK*e>z z0_6*=!H~8wBQDH*Y3tZ4k92$tN2`?#socFCjYs>`2wFuz(E&`&7R2RfwCiPTK*1VY zYqafc;tw5n*jGcS2lG-BVqu|R2ILF?)Dz2+E0r|yn8GsU>Ad$_lH=E}K6k!4X5$!S zfJ4ggGmUOp39dP3if2Rf)qWgaSu6MOX31IkzIP*TDF!U&95fsnE~6e!>YCLjL6kxs zmv5o>4mF_g*OU7C`p>|@iyCdxUN+9kVyJSXSTU4cX+Nm(9$}oDkSIzVSi+7rI{jNX zeYXHZQI6R)xP_kzP0j%>&d{xt-%k5j%b#o9jrQsY?Aaa@7Xil#h@Q96>DHe(S4$D zz-EY|iKCGyr)?S#h{=;&P#tyxut|oYy&$T}J2W7k=}CNChkJbir*!Cf2zc~3j4wqm zbAk|JU67nBdY`-jzkS#s7g&Y{L0%)Zrqg|Wjr~olg)%$Q_OkL}1zD#A0#tYg9D_a| zjiqX7;&MD@P`8qJZp(c(Rx`VjDc0@1fI` zcK5}0ZZKlpUfHho*003uOs>(lSy@@1FXmBtnrl28S$=1kWwSZ>g3hp{Gxo|))eG_Z zw7jNV79%vcdxujfBLvla|J)Ov0Lf7t#>23bI;qm+DZsC^qY&zMYIRS%dW|E>#3(!& z{mubPqIfGzmdA9#eZ}=D4BJ%~`R#Lz8vZjBfGh||E%{j8Bo;x3+t*-UbCkY~|8(n7 z#n7;%mntr$tI;V^H=D)1erfdyhk(o}9OORe`por1gf20?T}N$m!f!s7FpR&{qgYFU{JrZ;oOSdOX_obp7!$`8Xm~N&lO{yX1;6g!Q?zAj$x#fT-7Z z?@37~_Qkb}XGeptEP?XioE+S@XN8dFOF<|YWj#M7KjsgHYd7b5FU{LOu(#&u{e;>j z$1e^lq3-#3{oPvI31f1|{P&65Clmu!znix~>JteLa|xu^006j$^xaHYXy|MBK0QP~ zPZ_>e{r$9ju`UerfR&k zxmD(}@ZQRBsKyxqHlt3-*F_(2sr{37{d4FQNdOc!tsRW`(-FqV@&rLwbiTMvj?lw{3|q zu8&$}?M=a+kHhlDcmuY5iGjh{lQ={^`-ybgmGi014p)R>?_=I$S3)1X zpS}w!?0A6^z3zai5X3x1UOSyBS)$67iep;Nlu((h(lB&8im}PD$glgIXT*y603}?z ze3k}AghT7@A4AXHbdIz8npkTsERzg^z`8Mz8~*bSGG_!Ah>tCL4~A7qV|N;kHx*xf zy*q5^#+hbTODTM{JWZdR3~Qw8$z@RnKOS=0FZvaal(tM zGKUE8YX}aC)~WfNRH7umV&QSYmfkChgdJ}n8ZuDQcou5cAl%GRM5d+3zIlo1^?~1D z15r-F216IyRXl3iR+X<3 zDS2vZR?HhLcd%gkwfIdp?JcL%##F&FB3$3WC>Efw*1A?U8wXNfQt{^b#-7=XC)# z7}^^ZET_HSAJTmew0BC3UyPqF7o= zUcpL}RsKnQl`aCy7~wZqu*MPIrTa5gHnE=|nJ+AZ0NOkKEj{(G0e$?D0$&yUOmqvx z;EH}Y*u}2ze)#?sk*aJ9agWm)OGLXPi%P^r=*DFLxflNjfPXb6@^KS`AZ-0=8rW!c zH$4oC6sy%L{tN&-(r8FB@e{yO&-geO+^sM|PEE~TmB=pM_3$_029ynbd4@icJb$sNT;;=gst1?Yqmm10F{xtr48)h@wB!RImqcs~TDXAifHgOFU%# zdCYo~`Rxkz$Ew*ZpEPf^rxY1@Q;+i1(l9_NO@9v0L(qX=jdeg{k0)HENMSkP^ zYN}H|PA%2stX@P$d8c ze*CEl-x8Cnk<-~PE}+BI{;bU=kXq<@A^aPXR_dPT|FF|hrqFQUN94CI*PC7hR zL5#f?bn>m@r>k6^$hH&{lAglcl$QQPycLBJTKLg&wLB?wZqR5iKh_ca8$h>*&-vrV zyVH9E@BLf8|3leZ2gTKWYr;5Sd z#+j2>elt_^)&1`M>r_|Q*=Ot8OP=*Cld>G+0j}IadY#rp;vh9w6r5&9l%4%a(`pIA ztiMPwL@IFs$rW%~+Q8p0t7dT^&|JB^2=(TY?ina_xm>#+P6w>5Mmm`LtTliT=;Y>v z$!z76P5u_ML3ZVdB5fQcm6^_@VJP2;VzBGY^m5GahwmBwWSiAv`7F8!!kVJx2HTKO ztLQeC z@;T91$t`4@H;$DcH}Wj#PW1BHyH3FEW{$erC8>zxU3*^k)DX{t-?le>=35t|gARUITRQWO8ny(w-J>j!?T*Fa6?X985dVh#nwRPwGBP2jP zaI+5=2K#ML%qgGKw2;>!-_jrUQrKg>e%}8`*hW+vQ@Qw^ z_Rhonx5wUC5}#dcZ=>er>N{WhaE`}VYv)FbiXSju(z zq%D7U%){xLce5qBEwPzumoXbrBRDFl++BlvTi@4;_rJ|+!Y#o^jHu*LcLaT+RQ`e8 zncAxNOcHKxx9B`+T|y!a4G5DB?C6$^8sD9DUpAYCBYh(1&3)eFT>l@8qZ z7lrqeQXrO)R%w_pyv~td7*>=lB9-#N7VOt0b0HouSax)B##!*!lRU#7Lv&WafG9!R zVV%jN@94R?N9eXtJSA`a#Pw!p4{Lt8>CMCc%pl|UjrHHL!+yGj_N!OzzR{0K3v0Ur z_`zdEQ@*AU)0_s_2{;v~C9_WMS}feAc_s6$O+be$fObJ}h@zcffs-UY!l=)5LfQ?K z3PVmYZBm{O6l@a!q^kIWuZ-aMO^N~hI?{#(-})$5*B1}Iyn~=m!gJHKb4_KFm%id_ zY=35*5`>QcYY3^4uaaUa98swcW{|-ebU%Z0;Jp1gEg%LIYSylRcTi|L=Mr3F9|l7Q z|Nj2!t95z*F#cd22i$cu0u65|el&6MS)nv9LG%l%0DlJR9f4ko`EP4WsA&{F@Xws) zq&|J>>yL)x3>*4cXe>ssMzmYBikevom2`}ymZ83ppERFX^{Rug71fvmR3eKWdYkxA zI-80-@x!nrE~FLZ3<|hw)YWbyT_I$Nw~pxWs8ql<$3_@Jg-gyb-nhdE|48cFBLdDj zvMEOps+`VzKtvH%W8c1yiK0Vwl$%CedSHI{ENL7(mZ005Bpr?T*C}BEB_0K;2!G9g zl2WIHB>NlqjMCb`M7{6wPxDLwCVr)l;^6{y@8#Z>eCC3`F9wKDcgf%VLtel!R^7@isu%VY^6p9aLap%XIYIAz~!~qe$hS zciy#~^zUc>Ow(WO|JM)tNYH1vM&@=mFZ2K23;(o99|x+0Yt$?9W%r{0AH4{yJT9D$ zP7(+;dM6n4v5ItayJ%k^Qoq{wO`5zwhKff6L1Sc5^tbZh`4ujU;L(gOfeKae^p4f!ddx^cX`XjJ&A4SayGDp_LH?dO#b6*`!I6$`;L$br2~=Aq%&f&HCd8XJA?!^h(j<*cpY+> zd<+0{r`}ocqI~7vvM&`;z08_`nBNooB5_?Ua=!mP#l*0lQc>i>p}4mm zXHnqd!?AQhd1~p$_Z!ApuJ>iUsv@6+yC$rzTYG0}oy3XXX2z8092^jF$HxuPDW&C? z++XF#kETpjn9@P~c+V86Oz8LFi$7Z_eto0f+mPD=G#?epUmzvkl&yJrE$t#yt5jt)X?Dd3#K>+$_metkPDjEA;n0 z`DepL;KDk5j3y$|4p%6cZ}g-|DAg|sE0KNW;wnmo{Fgugq`d?s^%8vJCl8VaN%d$W zV`DVyrDBEj=WkfI5P8_AKy|@vT14cqZK&oT8)#c(-B%U2EAo&HYfT=*iOQ1hl#VK-&qxbVI>1( zfM>Bz4ied0*Ka?xi{c4G!@^X9?^JA8w)F?PisMSuQo_gz<$A4IML#Y#an(X5c=&w} zMGR&K-9$Ta0@;6u1H=~D6&=rU?x%br9JZ4F58GBeMwW^j8^+8sr?~=olp*u&rdVl$ zKGC?ixN$IIg!2th0s!EW!BAnOU4V>mw)eUk5hLc|pR=O(3W2{mVY*fa!VuMCMK0hg zF^f9B0$_F2G7MG@etut5QGetg(mvQ?YtMUojajaK{;J1%shLLfi@@Qf~FmO|2?NJ~;nnGCe%ij2Ox*zhs@VKDt_wR8YQo1pi7&RzA4ZA<_UME?G$my)H$1 zy0^q?v_95WxJfxuVt+2dKVLshp=KFl8lOv;prJ>#nyqHm5K}pC&kx_}-IL2> zwSfHRw)-#EvC$pOI`8s+LLqrpfR$I=1M!{P`- zqUD+YasiMd1#!N_C&Lqu5%$hV))Itf@a{vfp7%JL)rhuZ{!-)q94r*x>3;0*G@%5s@(ky7s<|?|!}XxdjMDLVu{K?jO*Rt3XZRPXO|VsiD~; z`sBDB=qjItPPXY9=5Yr7e>5a~0=S5YmbM0!a1W}&Ba(sA^Ar`>M3C<(Jh7!k@j z|Iw;}X=viC!R4~T5B$-0m&b}%hjZ6%kUW8Z zZ^_P35z&v)bvCmfiM9*WfA)xkb7A>xKfTlN0%U&|*{Iyv;5N>5M! z`8T=yyV0bMPnb?J5~&`ii=lH4ixPD2 zZCY;avTTpP&`yB(NyktWc$Z2<8;7o$YyRkNL6V!|IZh)XCs#zqBF}zA#;S&?W6w@>9jwQIcdcp91jmSniw)1A?_)hR+XR1O#~Y&|K1SYGhd$HO(=|POM+i@n+6taOmqJ`YV^qw@S$22xV@!Mh2z(lf zz&9x#?59|2E)m_)&~>66D27v@YRNb**vTKu)<5Llm=80wazT-fyx@q_EBlNqD&lXk ztn7_*_D0FaT(KFg^b6LU8`<6-%m!OmFy#Z!;I1M6`+Oc>PDp-ke(@XbsgR7Mfq|;O zj&~kF1`OdcESns;ij!L1wUF_U6K`(o!%Ig}5Bd8nwt;SbWWeT4tW#bvmBaemNa)p0 z2xg?p?ES!6r7Tf)zV};*C6;(*jVWk?;XE{|}Zefl^moe}V;5?k) z%7-Hpb_((~m`pQj)A$aZwHcP%z9_|)YAhySXZ-)e`3CWmLPJ|RmP%* z!NHO1yJdx)9XulETJ!{&t1_C*2}YA?bqYU&{rb z22Oc@amaAS0>ZZKqWbiT4ZZLUitx0|z~6RgP-mW0r=HpKYXNuel8wNIm58Lw z-#T%pE<)!071L&6_iBCKz(vBS{3&C zi@iSBJx89mL$mxOb=C{1oLpS=ac#0n($Y1~&0!bJVq}0Cs0_`FMs0moQNa~RtGN;W zR?qlzP*UX;s~OtA3jIHu^uKzvi&OrjhDX9CP$D=RoZp(s)HJ=S&Fb2~C;w6QfR5JX z)QA@P(;~Vh`XsorQn^?&uUIux>^ws7Q^i35!O2R;8cWH+3QDM5casZ3#q#f!=BW#f zk^{gK$r#`*x0=OT}E1RH9X2gFr zz)*1!Nrfr$j8LW_s7HngqGKOosLe^S+d=U#Qn6ffZDSDh*)A`7BhOo4x3GUGYyKoa zgTi_yGjM22)x27YhgHZRR2K;ZIxRhj7;a&tp%)1E_`O($-*xvP`0h$W8`JZ`gu=-4 zD0{?GD9qz*(v(j9rA6A>Qul5z_*HxO~5fLq-}5P z-Aq%v&1pOQOo0{_a4AJyMD?Y2B@;``7Pz-IoGyeQ1Xc5dwt4Nb>5Zg!>lgZfS>=ja zjm~k?1pC><#DLm&8NZuN>6yHgTFR;0UWI$KZ`F5|UXH1gcQ!WK_A;(wH4?tDnM#Zs zoRgv;uZqicay=j7vT3jRRs^b>Fw8o*G&bS0(pk0I~5Ck4+4>18Ki(v5E+z)xu#<{(80&m z*4q+~_5Heo@~~`0Z&7c55FG?G3Yiy9|`$lvQU+)V)i1i zlo=B?Z3iC9D1^~oSvUfY^!l)FGW3G*Rvy=w>Qm*I7!65ah|8B&s53X9(d2aNB^(zp z=$8q52}S#_RwZWt#4R*@bjK%BO{YE9kqq>I;x3hBv(_uhKMGVzkH|lqEp8O+&UovV z3@Bl{&FM>Bf#TAjtTEj>%fq31n{odO3!B8gsBAr}CbHLQ$?{H1h8ycBNsJ^TVh`fh zs&SYUMTYmQJ<`!Caf~?leSK`JxQj1)eB!_*YD;dzNnB3creko2OZTY78a4b`%V%fo z#kfQpLuwPjkfNko$8oQVJwCHjTI1StZBDbZ#s*tM+?9!Zf!P?4Yl{8dq3+Ol3)6nh zpkEN+?xO_kYfK73DIm^OP|1GUA3yb3(#a?P@)taQ%|-$9AnoFsa6nDP?nxi6G=n8AvV#zgKL9X)3 zVx6Avikg>K-d;k&?cqDPfO74s))5wnx2Ur|wxbBlRwCP8EvANyDlE>xcwz+9b9>)O z1FwF32&;nL2qLhmIl;llAQ)$$|0c}6m)wQP$Fhc;J9mM=-s&yv^A*AKkjh&+O4Q}! z+ej;;XCcJ|8U;XzMNq-D|M-dDBh7TKAji)azUkgN=G(|Wp??_?q1TQn)3qMNm$ zI&1RI*N0uY!?E_vNYY#aF(WW-~6awa;5^Z7JN=-zO8fg^gq>npE0m=; z)mh=jX1S-k7Q_8Nt{6b2oQylEQ0kCraX&2>ZgJ0*l-H^{%c?p`E`SFSwvO7wfB){B zY%(VpU-rO4D6&JcEPk+et*R-7OOs(RKt8NLD14J`JljGo#b{9DKmFoEn?1GuYF>^` z>e2kMg%6Z07raU?y2lqWhSl*kC!Qr)c}I@G>S=RF&eDH2PH-P^c1#KE{Q|PVXD)gKa?Z+U9D6I?HHB^MZw!* zT@MbN9xTH8eO2ef2mDD&pqu239gnEp*K{~ZIRy=FR?{_{Y7FLEc(yt}P1IA1vhPB{ zZD|DUPY6QpVDW$)&C!(RkG}xNW<`}yk8quZyz;P8X_s-NmDnix6ltyp;1 z0kU7h1JsCcBpEABYz(x17ktglIpnSN7S^JA<=`s1;i){~R&8qhgC3-30?flW96UTW z+y@zK`bQA+$H1+&9*3O#C=%8FYdcFwecI*Z-Wt79Y=y923eE; zYg089W4n=YCL@NPCiC@5TbWTBW8U))M&F<$sb8S+ljMc=3!Y!q4Np;}*nJ;u2$*CN zDh%YKNTiCm^*fO>e1XmNps3<@RG2Fs9y8fb-Jvlw`{d_6^{a@>YJZ(%zH$e>Y^^aA zNpDtn=UsqFv-~Z|fO=1z;htU`)({G?2VERm`tp|ve3QT^K~SBxE}dz{K5x>gqmlCnPRC ztm(+>u|4T}puQO>#H)3v5h^3Ve}E$B|$lA2XFz~yAc{Fk$u zdL$+Tp4)!vE2*7+i!m4AnU?dbIdlHKdc_&1^#tfpWPv~#&63lN`jiSS91J6FPM=j1 zaN=B@3vx2I9nZ;@Qx@RxXZ+C_Y5Ntn6D@63<&L$fLhCqREu z=1CUoT9!0ETL@@;);a+c*N}qrmG`1?i`AfF!-%hgilJE%e*;hxZY%YT{GTs``y6z+Tr1SoL4!%0=3Y6 z^3%nD-{r1PMzQ&@EEXe_&yJ|K(FP2-~q%rk4q(ja%VuoAj*~}$y&LK?F zHM=ZBlbW7~0IQ|0pv#Mnh7^ZJsSU;eK;!+i4^Z9l?baqfA9$VXRtPASfNdR9hErT_ zxNJXKjTLn*RcgW;mTJ~T5Ys`)-o8iK8zWfMt}bn>c!)0QL{?#`TtxzI%C{f-PcL3V z8o1K~TupnjJ`crNcE{=nxZh8%lOU+)fxZu^GMZAE8ELG^E^6L&y_4|65bFs zOd~#n&F>sTj=Ys(F0=E6-fpv`a4HcZ+F0DS%1UH5t3e>?dgl3bpLL{Xm;eK|;}I&t zV5egT(*k8AX|GT39S2yzhU}k4H|w^zHMEp>F!2|z?RNBV_Xyw1(wWnT{~W3Hyg(Z< zVDtXL8*R*2*ECl#c;qEm!C2VpbZ{}N-k`_=CPqhu^j2WGkeV&qRw9z3 zjO-^_Sgu0%_*8wl&s*Jn0C-%@X--}f2;Cf(cu$&Qtt!L_x;CJ`d+X7K+w3uKCnw93 z9;dFnO!fL~G`Kv!1_h-yI_?uC^Y(T1-r1%ZQRRK0Zk=mw@}}m)GRP;&gnayr6zil<$!OxH|KeWXG-6 z>d5_WxCc4~A2*yh{Xqdy*ap`R2mz~d0#MzxBt}Qpmm?brdJKOz(C6p{9}r<$bUx! z6~om6Mdf=Sey7+;V=j6eVZgCN5s9g1blZ1u=~`hs2Iwry^(m<>F_0nTEvWbUjqVV5 z=i8hIW?Pcb&83Y7c{;`R{M;Jt1pKllC`PmNL(at@1>kq4Bp`r^C|2rtFkqOoZ#&>` z;K0Z#p>%xsC;V25??~G*y+&4QqE(@z#9?7FD+V36E?q9`i`Z2lu4*tJTtT~zIi{P} z;$<de@eqLOO^b zCk^^yjGqe$au8+UOuVHKlV{s_3Jm4!lAQcsZ$f^Hu4oomUWwJGfVcesT%Bm~Z z>c8Vxd~Xm20iv?>?FhXWRI3p%wSjvw*f5EY%v2kxhdzy%;z3+h=MmE4l(7ED{(f6m z@cR_w9ogtFjY?s*UZ*^FgHU~U+FQbkpT1tKKCQc?7hL6xk>;6g7iN5mbud8c3VkFK z&BE-(>9$wQ5X^X(ml^6Kh`4UM$C}6)iJ{6wc93-au8oPdH#6rUL|TMio_v;y8)hyB z*#3?@@3!Ac7*UM`qeJC%F|QiBwJ9CA@U*u^I4dNopSRC4u0mXC5C z%CVH$(MJKeB34d**^;{o3s0YUkvF7i0rz0csNI4ZiC2LN3I1}WMQzq6Y0_Papv zI+#9>e$LTzuj7oE+=eR!i~oEm=e#!IG-Dsz8wc{!}OJvgv1X;>5^9ooDt(re28fGpyVh_$rOWfA+GsoZLH^EF!i z9yzXWkx;wH4e?30*2k$kkJX z7gNk~QA?Q@n=wsNTsu+MA1E-Dy$v*!y-fE*WkS89)PW2u3nBxKAH0WdoRzp<)XLb5 z)1XSd9{okb{K;?OY@;B`Z3wc%Q+S)0%~ZfMwEe;WyY1sE(m0Wqv_c`SPGYAer)7miLnW=+m6^O)^(6;)t>dvHf zldoXshlhpG)3CIntY2YJv5BdP$&?NXzkU}Y-~II6HZ7rTK)P`pKV2l~(v1|^aenvm z^7v8G5fe=~#F+<$q$&ranQhaXSQpaI> zMCc0?E38s5F@s_*;_PF@P8-ce6(SaUNMUPaT;EwP9MueDW8SrxH2K6S|ESM_-2v5? zokxQqtZ9j(!(%-qooop;G|yc zAgXdMQ9C;|&F8vlCjqcinW;Z6Zo^2u0Qfd_RbDt+B8UafsMeLm$3A7IR95_rxtP&cjsPaK(R+5b9)t75 zdn24uSqNi@kI?Y_`#@Z*=aFZBjXz?dGkA*G;6>+)6Ix^Yj~ASzOMV#o<72^=*ndL0 zhdD~mYc@lP1G^nXRz-(=6Q^C_l?@+2KIGIx$k_@Hk8D^)`4V9}F$sKr(#isMEA8@a zF~h93Wky@58(5Br0X`>d&+2QDeckpRzc-3AB0vn1k%%J!ZopI!f>T|wt#qQF1rANOl+)y|YUk9|-V-7EH4<0ts1 zFuO!-2G~`Nms-XG&5?UhMTnooW0%0k)$BZ5?7H+E_b8LCS&6#-%2DJr&piL)hp&Sg z;x*u(MMPFJjm`>phA}PsEoBfZ5(gCoR18pdWoIPhRZ2i0)}jqZi$C4d8;=d#28-ok zo6Pl0N`g1p8fY6NcltvWl*+i+l{>G$btKxbf9HETci`YUVK#GmY55>NL!HuDQcgCJ zU_PQR;`3K^X3sxFMwUjO^l7n%n$1CcrnKl{gdos-@=Pi17NJp%w zllceIUfY|cP&b`JHkC)Sm_38eWbv*gXF|S>6EPGl z(?*Np5XSXZB0T%qxeEtl-`srpsi)+&bC8L6-ZKBW^EK#8l_7{WA`^&PL&#q%j4;<) z4Z9QE;RfM^NQo!1ha~@nt>(N4-E5rmmRa z{gPt-R1$tk_Cy_EGL zl!zE5WhvG(qt@s)hUar|xCTA~vs@@EES!Mdr^@wFsBj+BOyh}AjH5RjHhIUzCRb_) zSKIxhOoNovY{i~Cd6(%^5!x30D+^15FYmjMDuRq)ec*yKtT-!7htWLgt;E!kJYjom zl*XfWS@iPRTKuyd`8?n1I%4{(MT$$LAhd&W;OQ;a4-i8;g*3_M9|(uwMgtrJ-a@U` zVS*Zn#H4}`HODCaI4vkCt(KY*eTbj4nRismqe=$ltM)irSGX7KLaB)H9}=mv%ldtrTy|?+G2kQ>okd5w}hzF`fi;G_!$90MN9eB^x893=5zU;A`VQ5N#0QiW= zplGM8Vzw?Dr*xFW`BSJip$c4;26}3+xBjGtU4?1=VmxAShF7xB+reJOvqidiYU%}b zR?5^M##(0@i`QgXmymG1Je4i&crYw4hguF8C;`iNc3MCeGZhfBF7C=%1}R&B(k= zm~Sx9HPe6GxU-5KORwnBz0OY|@qpZQostQuqfFU2YY(1PvUeMdNo!A{VCi^jZM0dL z$!M&yp`PMyP^5u=a(cqHosH8ivnEss>zzPS+}%lN(o)Nw4$uLw*f?Hw2 zTlQ4pC)ZI2S3(`|Pmx=pu8b$!0W3C1Dmjb`a_-CPmn>9d_uBRw>`TfM?|=vx=0+QZ za8?8XQ}H)*MK9M6YMKbS?`8AAYq#1AB~<7*0Gu??7Qx%~^gV@B;{Cv95S(z!$?j2N zgFfl>x=w_0&1dXS?5&^z5G?hz5n@M6Ds%(Bquc))z|*fgaeS?a%C;VXP|8${_R-9@ zR49%AE*CUKyw4Ve*=Uq{dWmBCWj?&GE=*02}W-vZLT{ zFkf|?=kT5x`%yQS=S&nIh7X9k!k-=)AsErVS4c0&49A#S^#~**p`1y9$ z7!3Fd5!VQb=+bbvFADX{FZZsO;dQAzv)_d;zprQ)6qDqt&SBKBe)>AR#p@4lDSuV@ zsJ9bODtQ`2h0E&sgLl!!Z+jL}+&w@t`UZEXwy1HCq>d_iMKN{n&9h^(^b|tikFPMh znLBrDZ@F!NtRtFRzTA^u(m*q`y}R1b%5)$9X{>V2esu3Xw^T5!0F!ZO9?G8Ru_!7n z)+u@j;arF}r5-xU;8@03mX{rp-D>ayy>y`?=y+XO_q=GYNO#m6krDL5Gcm-9bM3og z58Eo;^&~qa8JyvBfN^&Ud3HR)YfO(zmqEGj$c^PHp%=V;i>P6%ko5+M-niMTL#Tc! z*tMV?!NwVc&&{_|P_aQc9{Th-kQ9OKwWnI!9$>z*ai}leZDLRpr0Z>pBc0!oi1&a;Zf9j$@nP7%uhr8cx=$tuFbuy{ z-0FyeE$dT%=>;!5pl@XYHex%Zc)+iy$~C&76$Ra9fWW*8=UB83TyCgLpt_L&SwO$n zUv$Az>ZKCH&2|e+zXJzYZA&{en<*$QEA3(W_b-i4_-+*m-)&>8p9b-TZL{Hu1U7SNbMmom)>2a1%7@wJ&d`(8nOOg7Rf(MWoG|JL-uDr+1)cOhvwKV*bz#KGcN;(F1%*T-PUP$MBL)x##y4bJ9y za;rcU<9gt9tsuco;}7Jib6ZKIhg9o5O`6ougZXG{`4M6F!Is`lk`&Ez2bG3Tamu)p1Auuz2-~B@?D~&yXp2pyTQ760w+{UkLM!(-@`Rgg%MbZ6?cHT#HHmEX#?@orT+_(>?V!0O`#RQDUl zn+jrEgUwOjHUA7?BksnfFMbCnF6Wu!b$6hz2CIboUaQ9sc3jlFaM?7x=GyIjCs-#t z1&n(=M!GW{EoF~K7(Re=FI!Pyv{m{>r)#w$z|l?fLbKp={PUneLe@+2J%yOXhj`m| z!eyJ7t@NQ`6Y*6gDgS;z7~s3t?)cTDS41wc+OV+>!Vj^|Fpj^(POC z7TNhy8K}!A4S@5ook<8e2!uWc!xirMwJ}t?U8h?mSrNJV zF0dA_=HVcQA(v?VbfizN_pABLE3I^SQ{3!vBGv-^owJ|#GLA0W+01&fm8k5_55vsd zc_e@hW%1(vlw6U^d^`(4(-l6RV>jafDH*|5Tk$NlrhBYx@=WauXQ>z@5-PKd8kSed zb-5W;H}%>dX#bu4|11u0_x=Oc{eK_Cu)?cXdK82uYsF;QdP9cXBWx2HhnH3d0CccS z?6FwA#lnTuI~NF_s^!TM=!1t7unII&kt8idQBv}R9mn+)l8iW-QijTR-N7S2e12P`}N@T6*jE`t?NV4I3^9%egY=-O!5Se z90S4f%)j7$b*NXry%2Bj?@Nx-_}w%6Fh~lwHtz~(e58<(5T^@;D)hYtXqOKfUtsc+#@y}e|CqO{uV zfXv}y5nh^LN+;@Cm^#3{go=r&X;3%Fz+k_F-|+M9>QF_*2xf)P^)srCeushk&B^AR zqSb$3^Z)PC|9@^UJB$(o3XaANznyLVB1?6V0Q|=&=5D{%OBO`}nS}=XW zA{CHOJ`4*F$BXtjU6Dt|BvYj-P*YVqIGfDD*XR(MEl~?6=w+}zHEIZoSdb;~SETRm z|C%9kk4dV$I?`ME1<_v7_d$o>cl#yNz0!`&sf7cRfD5f+x492mym&jn8$aFj3xcWT z`xD=VR+Auhz-?XEH5;!$aZe^)$VW7q`<*d3{n3`$c|!5}*2~))DT~5K9Oy-s-8*-m zpR+7VxFofy3TI`3vn3MqE9JNJ$BXJa1J+hMrj ze^TV2Oib44nP8q*E+cIe__N8PTNyql0mmAE$E)flVP~<%Q%J)W3ZrN|zB-yQnpZ1f zVWyxdMQcY<_j&C1cu{SY;bC^(u;s>&yu-DiUU|I)CZKW-2nx9M-MUvhmpp#w!eB0Z z#U2L=7@D5$Uw%cuJ{w(;ztb~$MJDL47Z)jss#hX+Rax-Ayzrz_oMR9(8LNO z0Q}*ydt*&Zeo(EO3*3!oxwQk1J=bG4CxUX^#+(X!CZKE&Z*vE+Vx{#EB#!cx;$lBO(bumDvh z5Qp=&_(jNV@r3LBTiAiVeV4&ObZWHSFtMtE(k&s(C)E?~g%3bGdXX3;aQX+duoX?N zn>OS~tebjM>c^H1X4XG%Njru>mKW`{!k*qbY;e9?*lOwnANP}U5v~XI8G^U$a1t(b z`MnC_xJD6$>7aJc(MNrZ{)VgOYd7h7nsM}wr-fC-&wH!?YYjqP^wQ4 zH1+qeLvlg=ySS~_rLf$xAIv8&y>7K5k6-r+QAih6;+yD0TBnh%G24{bM@ z%3>@i&+=i^yOn{VvK|x$DHSC=+iL5x&dl%e zM}|VxM6`{sGp>u*vv3hBNoB5X`U88g__aVvnz_T80=oE%*|OpE=F$h-sO@K@6PQ%gwF5B zABKdv-$T`v6YKmoN|5DnwruqFaeh7%Ez93Cy=SV{Ez(>)$6JF!z;2KUuKOH_(Osl3 zYbhg<82EBtybD5fIAv;PTrNIqv0SbFe&M~;<{=#J=Fn%n&e0)?GH5ZXF9E*nQ1yDEt)f!_P*n#4(37=3#Vquvc&5tp{%^YFbi(A= zUEkc=9NI2g;hZ>qdXFta4=b1KKf-^9S*-LbDJJw233m(szmiy2Ix|{~7=YZTU?}J` z`;KDK28vco5#k?ikysM4qT>oB3)TR6>FELpH}C5~VWyZ4lx3wHQ*aesxx{bax8-!0gU2SOx7I!O=k53@H&!v}uIm@22_D}`K8vjX?=%~N^+ zf!OW&e3`F6eDSnVqv3;{wy{f6uA(q;lXB&HG{w>1kf7VLSa0=x%WqVt@|Om0yFP?u z)CivHUT`BN*n*_Jf28z%GXO3*Y)RoXWtDb&TO6_m5cAAc78$b3f_hKxX)j#tSK8GW zTun%Nr~Ug-3~+lKIwW;cQyP;HyL(W<#5{#cg~L|MFg#*GA_gE46_@7~@mbF^a6qdJ zuO9EGV<#gtfPE&=EGFUlCRj+w`vE^X`n%-1Mb?90&gs_DCrWtJ(=uNDlJ*6Rk&0%$ zx%Be=xi9^7X~m}*U!v5Y=w!a|G4dAL=e3eV?KG7tN(Gp95bs!3ug8fEQ-nar93&Og zU_wK$ly$RPz+rXVXG-b%35dDGxde~udNjtmdfOs?TK&p0 zU~!@IHN~?-+<{1cQP$%+Q@NdHK z7;?4m91}{xK|zTDWooSwP~@5uo*0B@ig!-U#6Aw&{!vebgTJ?Uo|WX}+&Av0AWB8< zo2wcl!T_SHWY?G;Z+pgGFn+DK72Jm}AhG2zPT+pBu-6xS;(+Qx1ktS?{$G^6WmH_z z)~<_req03GVJ%P&o9?xu?hI9;5sA*T3*$ z*`77mTJL;bl$QzDHOD0-!Cg?YJ?wZ@b5#3En2i4`3bYz-?hy$udZLmJjEV(_;U8s0 z_Cv*zw*tDFOxC}jy@s+$fAE5~s}P)>jzhN#So|4A)Vm`-dt{nyx#bAx4ZE`KiUxb4 z(t>~oHfI+H)|>+cDt_0|0s)!t!Rkg+$*{`vQ72L+fV>PP--5ee8dRSo_l)Q->|QX9 zf1+42MVLiWhCXC~!?T8=140efQ-IV%4Fv+2rEI~a-A&FwC29f>Fs5d zPPX3=MX3*;FVa|Vzx1v@{VgM7?A|HS-7%o_(mZnVPM7vf?xg~%usJ(SG78b~G>)(T zMjLoLUtwHvmA4FqhhHpqu#KkI#qJ5Xet^EJJIVCOZ^53vUV>D*RgnJR`1%~SBasI)Gcks+3oPG*C;tG3t-e|*RoN52ctiLv_b-Ch{V8nY=u&0o1 z$srb*29?>=lY4+xc-Y>&YkT^xO}?Ia2cO zPE3@tA36lF_q2Dv*{co@KI`N^CohgRsHzQ!;!(`CFqnwG?K`6RA75`YIAklrvZl&@4<7U#bFSgu3E3o4nKTwMFwX@IMRU4Ivw-fr!jqt3aY; z3DEG7UT@h$7+gX=qWeDnX#%V?Uv-z31NNGKqltudAc6FX83iGhqiH;wqB;@iWv5lnHFR32;%Au|&l|{{W9o`N<1XAcO7=rD+koG)@diF6w zhfyy5{?DG#OK~K8^}6r=g+DVLCX6O)l+uqngub#*o*1;*xm(@K`KiTufAp%m$f(Q@ zqZ{*3>^IGoR#iJO;aW+|k}uJd(Ty(XJ?vOS94Nt8!?+L=aY*u_!YvkTAT!h@USdCU zB58+iyr;wb+H;2f11I)-r>=_M(#}=KNb8CH-vfZY`W%Y3YOH0}eTyJ@SwQafNKJN_ zeYS3plJN7VJj(*&{6Pm<6_q$!DzDZ!##X>rR%*y;Y8koXw7JEct40)Fmfsykh8c!y zk)zaJl92>8uG0 zM;2CdX;VZ~VE?=`Hpr1i^#$5{Q_A`5F}}NO%$|h$=8dz>wVO@N9qT#D8e(^^)phFi zA@SKgHlIgOfEzhRppw@^@c6;K^*vKwc3hqxZING~AAelbE+ym!`~LBWD&aGfrVrsx z!eS}IW4lQ#Do2mEj3p0I1gF`#OWp|k-G4z4B*W?_z>aLSf(Vc0FqR8#&IkILM zO}$=X%YHW47XR3O{JC(%ez3%*&aTMbg|{=_%ukN`X-xFt(7Wjl?H+rV33peD8iy|8 z!+po#^ltGNBCihQaucM0JHqH|aD!eVv!KKDPL9#@)e;i#S-_O%pHdrSj9r}Fy2D4H zta3XHZuccw-P8_LK;HPKwjX}vkMS3@KUknJMdh^`5qjz>k-{W9@{aZ7C1xS0zMaSNd(AaeAFZ+>3`%?!7Sk z!AF1}Vj>=A2U=`9cF#@jgEM%z_4tDH=F213Y1%bJBg_cn3b&wVqn>^%a-h@=fs^T;qE%bQp~^ zWuME6buAJMv4NgP8)z{_Pbu8Ux^>V|Ug#Ib1FT`qftCcXkRTFW+VAAN6}xVWfv^h` z`l`Ec^?$MeM#-)}jeXcH-Z5JRD|{U`tqL8v^HCf47l(mu0naBN8-AV26J~d5>o6%pd6`BW-k4t=^rbi2(kMBMK%l8*6>hbGpe)_F6A4 zlC1iA#}*a_uRsXm$NuEFxT?%Ib)li}2x+S~eJ=q`45#Um`?1NJ0Oy3**kV?ti_B`= zp3bU{2N~VD&+vqrRh3?$)vckQsSNS|$z3iRj|- zlqpxF9uj-jz;)9yekU3>pE)8D2>3Ytk)W`exsL&LB!Et`R!SFjTz}o`d;$-aZj)DD z?OkKZ$$H!7ev+Snou?dQp|BrV3T6bDTLNK31fU<2ds%*;D0(k3dW^lfEN3GktG(>z zvne%yREXp2s@cElLjl#s)a`IV3VRmbksCTDcXhXshkWpQ=@3awys&ixUUphX?$}{V$MKI)_%43zCu=roR#vg<&E>;XRw6Veuqix zH}ac}RU1^$0BpC}KxleHZ;&1Usmmn1SHlk(2Go9yfsDU`S*pb@j zmy<|zN}w2Mb!ML!D3RK@@d5eP!hJW?<5`P*2B{|z{gWXzR4{gsDBWY5!<3$h?gpK& zIEbIX0nv7?MgM{>ey*Dgp~qqL#g;PqVf{osX$=wSq3woTb;;E@0;z4q!r!gu?{BJ| zL%|>FZO8UQkH6A~WIgZoo$h5F4b%zcs^J!(_#VRDy`*Q@aq86TnS$;)so|d%>e@OC zMw`ywzDxWsCPZrw|3K3W+@e#n8#xB(X%kDxI|myB($P7bp6BkwNY>_q@9U$^5@GYxl->(`Ue85_Fiw80gIdxkE@hk)diJffW=qqZPwF28G8eXAzKy~8bR$ms^U?(2 zVyfgAqwby~jn!a()(t%!&*YrBJ+eo-eZPQ1NhGZDAuYo04n@N9#cuqaSe^|>eb`Q8P zVGsPG`=nE}`6OY8XwH=mzkXLWIL*D@+J`dE{1)U95m03oojjgm8q`A1*>70nnJ1W3 zXCAGmynO;qDD;8p4LtTKQEp+RnELe*dMn$1t~A#LPWd`VCU@)Bom&Pa73xORZd|>U zO<=r&>?cn&OER;_NA+hW1^z8Ba}g#Twi+)r&Ft`y>ozqs z%IA3mDn?Ir4#v~$;{Z!S!akjV;A%^**WXho4D)G52#{bP#M8eVv6W~$DIp!GEY7we zaN|fG<7tm`T1bUKtq20%AIs!{bRXxmcapXbwTAHYp z!VP3=sFHXzHc8LQibq~iod$mYjbO@ajMX)3_=^Eg^GXP_^BhP>p*AwG*5^5Of3>}H zqzSXNqUr0Z=gJI+Y0vCW;#tUa&WuhVz7V9qAliv=spIZ05C*_B(`M|>7r{23{7gxg9*t6rdl?}V-wk=Xmf=?$#5c9EDxSFA>}Df;-hSWxoaVeG|A2!JK1NOU=61!Q@Xtdw&yIEZ0!Uk54|1Gzwn~*pW-@Og@B{wcsBH z0`yQduVc~ZiDAAM?UX37bGW5?PhpXVzgY+VXBKB1soMXEI%|xpH3suytB-~C@WiLi z%wz|x;{v{UJH5RIvQny7LC1F=%%tgH^257a^N4uk0+mrm)st%{+uF_zN%_toesk8&gn{~=tZ)Xuf-XOZoue()=n3)z;MOHKA8}l4j zq*!5a#92!3?t<`R>gR+D%ezmvR2i2Z*UA;B-t-`KB&9;0)A+12^U4uyXPy?&)bafd z*!?cBFGQXJh|ynnILc!lj>#Jjy_bX5j1Dy$j2!bX=Myr-)@d9PIT))kZiipIlKnTS z_NAf&kev=u4|_tzyehsq1uUW3cjV7DEILpvfJK6Jqk$UvyZAGDi&f-J#tj;pd0BTpSV2;Bn**cX=8t@V&1C#??fD4Z1_>Svd$s+=nC52b_)nW% zFJVK{(XS$ExHmQXYMv6LzNqxgI z;TViRoW#2%k>u8S7I7FqVfMj8%BimBu270=aqfp*^$gqB#h!n4nR0lgmnHKg3%*;Lngs0WZzA=ffbp~)+JYzq0&?>%8_T<(Y*vx2uhRI#c^;4>H zejYDe>r=9z*-vEGIom!r`=>wh{ow zKC<_H_Zx~|OQ5esXgRDWqINe1b>oK*#yioRW9P|ox7;Dnp^B9*@-R$5KNJ0GCv)- zdOj*BM|`#-v{|R75#>o4`O&H_=|?76R~zZ=qG?%^X~eX5Koj9D9jvauLyp?RzFqk9 z+wd>o(n$FA@4>d{nO=`3JDxYBdvZ&58@{V}r~GoA>)!}62khGLV<7Igs=Gy5AC(B_ zO*!@{Soy2p>q`!}s6gP%g@I$O_O@wDa%D1E`Pg+O3jcN2$x^-3(5`DL zz8z=>tqKHbN?`%$ha=kriN*;7#1#SO;%SC_eTiWWfo~A^t-1)+d3rD-n zRw?$>Bv&FYEggNaH?ff?*oS0L@@!F48DN6GyLV13ElmwVR+21!;=(Uv+A8(nOMl#G zy12T1bCaSwbX+57iHtLOv^wlTe*3)VIP~VVtm1iV!yLJn9o$B%;Zm}xwR-g5tT7#mM9qUhBdq35jKfNh&HFU_Oe&!SNAymQ1R+nk$TfB>!iYifq=D}BOe-a0x}n#T z2pxX+eDr&Lw>Ds3ZTOPnDtw_w+}$zctbE)=PDuLM!kxmhip6`X(`d()4(cGi#f+H) zF30Nj&Y1|4%NJ}KJRupe48?r^9`~FuuxE%oV-~5^&Y9dTmH*J^w!u6)`9`c??c&@3 zWRO1optqYW65TS~uLuFMGsXUzcvc0o=miunn1^i@+o*l-ZryxOcV<%ZokH!%g>&>! zXiL;Isn!ig{M?c(s!+`1FFT|mtY*hAvwdt!=+_B&iErkQD#FmGQ?(^jpECtKe&DD-i8}tjzA#o^&c=~*oQV%@;rHHfs$FTI(c)N(Zql1(Y@G9j} z^FasPcuO5*oPhp{qqb-}K1M7bGUeLB$0uaq7eR$!&39&;*ThL_AGCb0FQND2S%k#~he{)pt z?#-87wupXCl)t{RP+}ElY!9R(MtBd@`Ei*0DiuBo?fD^J|7;JCVr;z~nhP?Au!ux& z%y)#AtgsSeW*DMWL25O?nG7*u^tVgQ+D1=i=YbW85uv4$< z4jrY6hV3t;|0@r~YX66VR>i4!;HKSp90A-C%B`c77(eZCvD7|Pbo^T4FN3bTvymO0 z%r}IG;f(Tq=Ij^eHqsU0+bh2(R%4;uY+u{BE}4_+>Nj+T8%~6{zMAFxrkL+ZcDbo% z(nqMWmA@c&?EckIitN}qUl7lub-^v8|GS*fg+Rty<@ol?faw^;l?Y$qd(`ooIf?h~ zJ@v21@RqC<>L?SCdzuOd?UJ5<9bd22*-(Fpe~SEqe^@=2?73oTG=A11!oJ@TJIB3m zIco3w^JVG{Ahg;Y=1mpE_DFpZ_U(r?6{*dbR#DoYoGI?%1CYrcXw@@Nqj~#rS7*NF z&-bE~Gu;@I`@s>>!s(Gmq0xgTo?&uZOdHD5&VpKMXCBXXb+Op`&ENozJCaQ1gr-Y$ zhbDKn`CiEs7W9=M69)IE3(%Jx4!x;-N5mOweUKe{fCkF5>1xsRx%FBZgSiEzhitbg z6wm{<{FG5W4eD`)+u;Dsi^uAORt(nMl;Ria-IYPpw3G@jiRS8bCIj2CXf*scgNit( zcn`d;Tb`^tO+WE3nRO}@b$s4!8s;CbiTR1YaI>_xE8_M9Yng@$Tze)Qlu+tu#ANYg zJnAGv2YeScgh%d~cfcmG#d+T`2mcda`oBt!Fh4FUpY*mylU9q ztynTDc$X`4mt5~I;?V;YWW0?h&}*@AlDe5xCiapIxJ$I z-rr~xs<$qjIaKu#^`{<#-@7wo9KbvbvYJY|#L`B*j%$zY<^2^7zLT_EA{c}lWKMfe zd5#)Ah(h%wqHUMY$#Df}yvFt~)JssjKVF|r4_AD7D!2RYL<@Qi2fvb4GtLOpxN&=| zd8nuqYO>eb>;A>2?r}C{`3z&o$nQg%`%Zp7@98L76>~IK)|)xQ`l%Mv<>Z25E4Za` z5~9?rms}&$B$HI&KgyTm7e8YCzG-|EvRgBlH0n|$YGrD?+jPWzCw7@bnU+3OH%GM# ztXE}$lxu-R^Cks<1=#&~xi;0(_&4|Ukod7G|C0w7FA4t>M~RzINT!6CwZu{40}?#7 zv0qc>?#rr=H>uKtu-m<%^ltIw(37_R7BRjLlQZk_{O3+naOhp;HF06q*s-b5=t$Lc zx}Q>qjbYQ-zxMp!=A2AjnyhB9vY&ley=^FHWODnO&(6uP6))g)4gfb5kkNa#_Ry*> z1MuVrAE(a$rF|1$kmyR9^-#^fooa;~Ly$c`0;NCy`H_4E$IopfY#@V<2Fn%H5ld1E-v2v2;{W5%E@k0XVSc(0 z?ULC4cTL{^gD?xw@`(@1W~UnoZod5gmqyA9A&`)yyh3Qr@09-f-vRmGXafK5|6iP% zg#`oM7!96n<^P|c50KzT2P`5czYh{mx*}paAp%DGzk~ii`9RGQAti449ku!@7o!fATDxRH%N+Fn?0AueEr69k`S7h{?!bM#1MzCc zy|Z;@BSB>fXlk|@wZo-G{ibPW)k?8n47rG*8P=SuO#gO_Wu}1ZVhzCf7wRXIH{Jwf zXCeNf1=8$q1O-EXu^JgZ)`qva{)z_c|(>)_GquUo*jp*!Oqxz~# zTDF&u;XVT#Z$Y1!k*XK2A7zGhzX&8{`d+xYLQLw~0&25u0hb9CyQV@w0LaY&DLlMg zkFVN~x)F73J3T<^2?eMhf`+Tr=45}WKiBb01GF75y7nb>nF`3;Ug^BAbNO?2*P^r& zCFcDL`K0X9*T}>kxGd>NqaG@u_y@;ELQ>rxOO97hwrlD>PNMRCW;5`1rDE0|Bg$(x zh))Mp$pFpL7^rdi(GR|vB0F^7_(>r8wI_s$+Zp6#=IZdy-q-hc&WYaIWHL>&b@jcnUz_0hq zdw8@0D)sGV#62IL`;AIF9snRnvw^b^*TYV(IZNVY^FJk(?tiFQ^27+Hah|qV$agWyr{?o7w@8wEwp}@Eq0M<^<;l1e^ouXh& zi}c%imS}Z-*Tn@BAhRdBovtdh-JFc`ziNXW=dE}{{eczy&MpAKqwhFW7CjE%=&3Vl zEqBIxc6#{5+lVP>#jaX;#1F0a>?`-)$E?gG{g!Qwf9s1Df!3Mz*rk!+j)89+GBD90Z(jO}T+w<#iZhcf zNb2dS#PQGNYf57AGL=8yUTU=&f1N18!#}&6oi8Prp91{0zg(b5|QfmLQGHLQZ2o8>C<^-G~zzDkv%m|#?R))mz8}f^L;!7o~7RbL{@tSU6 zk1f-T(r~5T81#Qi8v9*)$rtfYH}IO2RPmqg<)tcBTG`;uObtGM_=JL}}4Ww#nE*||Fr!;*yR^No&d~2}p ze8VW=N5P#kbm1){434wwzgN=#@_UH`ADk}d!}SBGJ?57a>*fwB97!&_e|=9ELZshx zrm_0b6=mj1BP#zNArVb--D{U9d#7x!*#paCNrUih*af*Ypo&P?+XtTRzw^*@U-v0i z*yrxsxoQdD!}&_ynKFHSCPMyqVl6*IW6q85%(mT;>=T*{Jy%DKw@-!b3IG=PJ*T(Di|6mHv_UTGSM#j_Wsv!5?h@h#BnClfv{Y&eKsN3V~;Zp^TCar55Xvp%) zO0wk;+!AXY%~s)9x>VdT&69E3F4(gcVl@}_6zsFShuq8ZD3%@2*=kR$6jPm(uL(sh zwzy_q-cx>pv|sL7{wixxsm@B}DOI^XnytSG;_?z>SrM+cEy1MfyMYV)W%_8i4xP-5 z$&c&YkKnOg7QYUjSd(}$UtEJWpKVI?H!pXHw5d+ghR%4`{9rJ=^YAVcjX2p1!%`gS zd9v9=!}Rrb-SOrqisT&l?59G#7kA!qaC6y^lvPNx7O8q>2N;PKOJ96uj(NmGkfeTi zCB78>lJx3Tf=!89MePrd^P}cYkni28fl#R^-y+q@%~;ZS{PVMAjupmFPn(Gb=LeoG z1jQbiH2jaH`G-~303+o@{GmDO4FT)6Bh?}eG2L}?uECLomCyWET)2ja)}?gnrTH*! zACuNe(=S)xeuv8&ewd&CaJ2=IV7+l{Stzq=NE9BB6 z+`UfC;s|odgcKqUqm9)R>Thno)$r;KRnT5=5!Nkn=4o2Xbyh5MMtUQn)pv{W!6iD^ zdBl({{;w7&ANRe*lDK@#Gxm3nW^V!W$3N!AZ)KmaKKh-W<5X&Agz**2yp-ss7?x?N zWlZrDgM6op`LHVWGs47FlYZ2`n^ZeoE#P+bY<@FSq8Xnk^GBVDmm(vHpubeBL{X$o zbkjx6R=**iKI-g@bwZwiS+$NHM|EZ>@sx?Bh{jy44`7Cu!LMPch{%e)@Tp-PCb1gj zCDcY8&e1SZN=nL?NC2mju_!z47umT(^TIycRaXuVvl@J?Az1!4V4k-jpTM5?n6|3` zO8@5Z>G8|{x!RRO_^bvyYr=CfPgG@%i37CJfI0ung!0y)%`Ibg?Go-e6U%2D^SJDF zC2=xs`e#;mBz~LKO38eY;eM^ErD7o})h^}qck!8ik7~U8p@G$(TR639?@6q-@6&+L zV}VnIL@NBu!8{Ut`|R5~i6izTOS~ULA*RbzZ<3!pzVS1sILz^;1m1TaDg4J$vw2%< z_n%>H^|W6~Gq6Rhv=U*mot)-?kBf(%F(&qlfVZS-=c>Wi`EOi&?>$yM7m`VdiH+|l zyV@GHqL{bOjug%(Naw2JX(!fiPdk=GvI z)*6Zoy&_g0JG+_stk~BcGm9fGskXcqb~fDPX@bx&euf0#4zwT{zo(dQDtgzawD+Cu zi;CRyJ7p!sg|1Adew9ouC-dLqo$7DBGsHztR{T{@(s-`z(YFTqDi)e=H@Oi%4nC7X z9^oXNBqgwSzNW;k^VH(TCt8f8R;_53lTq#oG=_>?iAOYU^8p| z&UrOUkj1Ezx6<@;fn_0>_rT(n$Ydwc=w|WT_q)&VFo}|t+^@~}xe^K9ZmBhY0)H}dyOt(6L+O&N8O!n0M(+--# z`}>sW#~qw#nELlr-bxjG`tfZLUCkl%L*-ftH{OtOl``jJ_iuz{^_d}pzxfqa;V;tT z&qowizwt|AN6x;BHn@#iHQv-pt8$qK^a=myNPbI0jpRfRUr&&%8Zwx&7DHNQ_wlCP zB;nz(Z8F5qR**8dJN64NdzU~dQxmXyXXKI5)X<|tv))RNb-n!vZP8%{sbS(Ofl#C) zU#gLYH{NV4hZWd%y0!^yFU9?>bXDgyy!)}l|JGUmAWL5@cxonp_YGUTtB$%D^&@IyARC@2rj95 z`d@rFWcHGKq@U5@ljytf3G05t`S9D-l8||k5i*&vv?7qzLp)%%K_+Z}e?Y374_-Bb zwaAb$P|j_}eGIkRt(R1wGqZ8 zY(U0#eRqC-=QRI5XpyuTQ)M~pd+P1#=5guN;@vJ=_GwIx+4bX_9UA%6cib$7(vBV__BW&UAUwX3VO z<{PmN3vXC?5Z+RSC~CsbsFVTiysb zD88bRtM~0L?#_ER(^4coGaoMH{!Q)&xLSnxK;!LTNrDnCxms}-1Y)nr)!XXaTBIf; z&K`EdjvovZw(3LI5k&Y2K}*5yLNusRXz>oes^v!4X?j~!R1F$REOft-oN*3Fz$9{T zNARs(-Ziaxi7gtH!KRnp@|5H>qK=cx&D^d_6Ay4z?t{APypyo4e&R#Cqn;3#S?KwA zi_@l1+x)g#mDp?Fs|`Lva_YOeWsTOSV^(u|$l(zvU3FjxhPvmnOQsi;*_>A_)TkBP z_CqrtpXr^YvEE*BNDd>F4Z_7%of`(BQe|p7!?5Lwl@*PH6oYDz*Tn9(S+xG%Xg%2z zYp*{&*1jVp*Dz0A!!K|92A#8Cl)Y~cNlJ9o&tUZb7epPAa0?K&-Y6e9 z^>F-!&0a#wxSl-I3}=hnXSuthqIznNPtOv}PT!%Eu~h15s@IG12W$D+__9P9h>>UE zd(5=Fp-4%z1Dqc9J@~WBTDhW}%ohD-bafagud{Q(@W*7>2RJy!0~F3La40SwhnoVg z`lOA~bqPMzHj_4`X7NXo-rbS(dJ|J_RJA`w3K7M_(n#+y3dubQ$Enc7o`4+W_Ovw zH!vF}JZm;cmXgw}B_lFrJZqrZ`3AyU9JygHS_^Pi)gB1aDU(Oll zmdjG~YkL(?cj`=8NjG9;MLby&V$3>iI@QtoF>tf!LG3N9Rq-jMvL-y#(oh8!t9(&E zv9~fj{*I<##`k2QDNBlGZCyf}7xqe?i*IGvYdON2ZBvyw)T57GQlGXS^%W+Ns|F$E zQTE*R#D6FX14od{3LN*cnz{6qzmim?*>*We)ztj9(PZQ;;c3zVy^lEb?P&oOp@qU* zFq(!5&+149^<)!=$C6$>lktT1N-JBM<|`BH$T>7yV)~c_^toe(_zFt9Dt|ctHM@*Q zi+=;cDd?u^s6d%9@)y$+;SL!IlZ1oB5;_Qc!CZIkv0qLu&upryYTCpu;9U)@o|uwH z6Ah4(?Oh_H?9)06+>L7OuES<}C!`lj6&$!DM5Z0P9*1mivpu(0jQA**I9xZE<02(s9g-(Xj@9vW5sb%S*@qZKB@7Qf4?lY2@0BHn^-uPH4bz*VDi!|r3K zr~54ya{jyL5~&YKrn2VA?`URk#-t2}k2+%8R8xN380F0~eC!OqI8NNlQhf@)d452S zx`j>iJRvf=1pOdrxaKm}SZB{?(-pBNmdf=kc^%Ry{Y9NM3l67I;;g;${*{M?))_qd z%;g}!YIb@d#z%=Y-j{ht#)*JIxc>bE(;3k$&u`y1$IZ&c9KSe;VOzxaHWO{+GiLnl zgJi)pt*<({6&HR_WS?@ z4sEfPs=Im_5jz1WI3$JlyE-WE8Jr(P4%^0OXdNYMOaJ>)0oTT$O1i)gE-C9q<1>`0elT5MM43py@lkELgMMv=8 z!K00P97P*gk>MNBpl>7+5aWs;joWHIyvZ9YvEGU}z0nyoQaOF-`WA?U0}dhLAJ!%= zBl=;Gwf_e>W_d;4HmUe#yXV*Wp?bf{eb%CE@`eW>4C&&NVh)@R% z#;{w%MYj?-VKbV5M~C|wwZQm)SBjChOx3hM3SEv-s~FpVgMxL?nDr8X`r*N$0zMnK z)&ZKMF^7T7KJv<&RK2=7w(!rZe^B%=*AHh(MuS_C8u(Z-n*olvmG|DWnU;&^F z5et439etsSzL!MJvU$A^GzYT>a56({ih1>UP$cotbBGULbf1w{l zbcKQ30=+I~V?>D|53SKlBVf!OfhEx}ik3B@Q^MmgAV+y?5Ea85*8H6qz9EW_^)S zfHGGZco_Dkq2;JfO7w|{*MwXblz)6#UD=Q4P+#S6xf&h(gcH&um66@N|NFYmOC{2L z|E4y^WQBFhae)xnv+v;sq{}WrFNk_IK(gh7R*IbY8^jhaQi51m%;2mrC$b;=2s&kt z71~+q8=U@|ya%Gs>lGSKXy!B{G!Pq$QG+i`7&XuLfPI#&`n?E&sd`Gsg1;ic>Q5(u- zMy;5v%?==(^1bn;AM(7e9zseWhL)%^NkIryzO2ML)nj!t-a+qh3pG2FS{3hJ|LgTD zaSbv>;%+1*sF{AfzpWl$4i2h+0Yq%0bkR=>rs$C**frq?7%DjLT~kpxa7=AF)rj8uxb4yzOoW+yw&E~kSz$J4##PyRPF1b~ z--OJVB&}M~)zuVUoX44ZI(gIZsud%}ER2|%D^f~HQuP?V6SD_9N5-r7iOiLdRb51e z)Tl+!NA0&u7)^wqJ?k6bJE6uGayIhOv)o0XIb@x4_vrPly8Xgc@a#-2EhhMu9{7c* z&+J!Omd{n=)El2({Di#R@#(s9GI6otLlFr6^G)W2_xus1?n0xbig7u}Or%MIhe&71E( z*d+VJiv-`)78SN%{oRVGksBf1fsX{AEY^Rd@pQisaX*^+K3lm~jM;gPBsy$eU{X<6 zo0CJ10;AI0o<>A<$i$M*na$P3Cxt!;EEO0dVSYx|Pn0J}yg9wnt@B);PO?m_R1MpV z#%WaFAaukY5+w9X?@>u#S8bRHSF@wg0z9n2dkh;Tkw7epLSE3yH^Y$sDDXTX@*}h2V^q2^9LY44! z^=?S4={c^f*Ou5z-sBoijd}NPI7awb2T_P=d89`pzMU=OA7T5nM3P;fTWlPgPf{#I zk)Dxr{HJqE#ql$q_suMRl3#-)xGf&uq@3X$+pTVsurx(p^np2l*ljMXYoSwW&R2-R zGMn!7`kcwzQH8Q*?!JHjP;yubB^6Vn0y9`<5e!Lx6Wn?*Jrfxm@cvpJ|DTB!4IFKQ zIV0iOM{lnmT%FAsmfTj{Q?`pFGp{-&wwE8A&SgS&BhT|Cr16Qri;;s94E=XVcxkZi zCc?Hy)8^dJ>4~}#`%=tiV7eKWGk!-tV+nC9W{bxy5d-nP+Ku+{pMN*B%0cT8-n^|p z(;=FK@PzD|Kk_66n(pp>9XZ<<;oRk(ntTyG85(;jd>y+u{TuiPkZM+j$mrB_!A_pmq5RG8HwzX~)^y5Y&x&a;5Z19p|R3TWs`VD2Ej! zoIHAeb!A&%Kne-)MbKp%ul*bu#)oPfquFMvG8f_;Tfd6Bw}$nfJo%t-4m($Q?6Afq z<>rB#TS!i%5!f8-t+>dp4nI5{@7zmLx8V)Qdxm+B@1-PxKkinxQlf|W`q@uo-`D$L zU0Uv#!B-xakw4#&Vn`7O*q`rNF5L+6JnOA!8jDNtX`St)WEmbh&}4r1jGyC`(kaq` zZL)N&2I%1QF-wRa0*7Y_bCzXNZT6uGVYI}O6fct-azWMNQ+<*lYi(X9%yjg0mo_t? z0&tjv3WT7K(6d4aW!|6Tl_aI->PMtE(p<0YTA_{{mHt&#Il>xRb4nJbTQPb5+VRMS zz857Lx>wNO$}V2%T+g&sc}E+~TqKFPeNXdzhHE~UYjyf^Hg~12L)M05{*E}i-biiw zP*DdRx(w8f-&5Srn@HY|u7XSjh0jO(A`<}1k)O2GRTNs(exD3IIMXs;^j3LyX?S?~ zpCgB3B{-)g<{Xh^QR~dheB1826ue4R24}L(@(ZFp{RZFGPg_O6wj@SaUDoaA*A0jn z{H(CFG$^C|bn>x0^0JF$i(`l{DbRO!@4K&%pcjq{Fr`PbGWQ-UBl(4|44F$T+OU7&_X$nlCl<^_Y1eteZRLNh zr|(5`*cJ@Hav3TgQyXTO6g=eLf>{EKX~}Z%7m%jj1jG4NQ4Zd04JjK!yFw$bcJEKF zWP(XGzD0ozn;-!Fs@r8^cBTIMlT9mKiR}0BKcr%s%-WmEooVPV=>vVbm^Qc;H(lLQ z8bSieWBuvcvP-4uICoKDy3o*}(9ny;F3TCcju{WEHL~yge{Ck4F^B{{seXIMXv!w+ z@!Ayj8gs;vyw9PxkZouH6FwSJOf|bIGHOyBXXmi`kT}L20i1z4eLKoKL^=CaEXzWk z*(Ld?e?uXm5i_YBjQ$6gi4Zq?U#pRRlSqR`CxTEaF$o$QOj(aF<^2(%KPw`)0q@j& zLcmN~oDbIHOt^|h99gd06sXtqrO&!8v0xuC$)F@1)+*1PWlNjhdZIMXJLfgra*fxs zci-BrfK;v`4o$yyZO;1ITR8eRb54fxR!6M$_t7T1PVKgzmRuzp5O9F@1Kk4lVbuS- zFT_zm#-A;RBnwFd4*blbntehIjhdpo!jY#R)$v@8KL<_Ea8@o@iyqt##)rIs?BO{K zGGKLb=ZCIGgW>HKUam)_VE1uZ*{O{@Iafm0)#mU1rJRy|&T;mDNjWoS0+)}|!c_xwS<|3QPIrTl-^+<(0yc^k`y+0|SGJ?<8E zTHC-}&2(OfLizL?W^KSj#!+Sa|ub+{bd8n3})yob%mpu68k@vJ|2uIsSTVBi6 zrQrU9!T)r80&uJ8yHM7_vvTjM`MH?lAP~s@P$J7gW~k4=St(jhZqaCZ!7N-K0Bk(_ zcar`;8uWj^R;f%actzSEQz=AA{M<+09G(kLxCzHz+A8(n6%PzRoO0xiD?i z&g8JJSYC@}M*Yy5o9MrCw`9jO{N@%r>GE`sl`7(nXS-PYCa*}|4_KbH0bMg?a|z&J zMFof26G4w#Vt_CN#;9bMS;j!nt~4yTjY#@iFrlS#YmJieeV}z;aL=@y!FILm=C{0^ z6-kE))UBs6PQq*CLb&Tpa#*tLao3cDX{{9e?~S)B17BS1buH%0MVmXl!MD!$*0c6* z^kv(QcnDrTPW*ki9`?skmLn(6D_thnqE>eZE%i1#dYYZZ`_9S+tCPvoYVh}Z(U${U zKtoU2;U@)G4LT}STPdru*;^#`KzF4(Nxw?;3WcHIV}Lp*$Y1AK-j|ZXrZ9$&AQ&os zIhtkzHx|Sj;Eayj5|gfAd>QHgWjSm5>twCw)$>b-MwXU)o9!NDnu+z;Bb}D6SysfW z`3`T63`;sVAx41BT=+|gOi*)LfbV$o$ss|j+)%C!Mp->l`}x|d)rCn$%|3-D!c7mr zk?WQON1VIM>x)1sEA-UY`d_qmL#;E^AhJYoZAXf%mSaWi+ zQ`sSz<9^vQ6}+OiyYhcdf!`LPP(O;*XxA;}QdGs$W(j-$^OB#I_(fy+?qZLT>_#1*b>m3t_SCOhh<{ij0-+_16*{0i_cptq zfrLV8-|}?7lKkHkb>um{^By_${dP{T-Oth@vje{NNEC-IuQWlpeJuQ|4%)Wu!8MkFXFG(AMTX zTMnPTV)yAVO&G1-dL|MuxgRvl{cv?COr^H;ODQpFfASpvFXV$1I=gO=D3I%N3VV)x z%RvW31TJ8=$G6uf3o=YbFX=)Nz%58RQ?Kr6{^_#8xDh0Md1d*_3$Snfl{MXg7@oi{ zK5mDjKDa3XXr=h{hUGVEb7=nVFjz? zu|2xzWm{T{>y9Lq4C0B!L7ykHW5Q{v>2z@Oqu)9tvWa!o!)TIc-sAq{rELi>x3!MG zE}!jETfptXT)){PbV>qR z)M*bN4MDAq$pjY%7a^xz@TkYlec^*bON}pAmsTUoeM~PbrdiZ98-SVBiwGy%ML5%0 z$a_})boU7y4*3bmzZG~r&t=4Ke~R`|v1EQQpWiRfA|FMitjICwm#hlAdM5mh)9E?U zhJEDL`p%Bj9=EmXf%~G?6y5#LVwDkOJ-$sB+?54^g{go#AR!8O&ZaJdWt2&$MyY-} z=3`0(C_ytN6k|g zaRl_e60c*tC+|rODuy{bA!IM=UNR6*%7fs=QSH=xA=Bsh#)~ zk@FAxS~O_jpaZVgvtBz8Ji-H6veAtr?;pm2&0(iQPc-ZK>Th%PHWll`_$Gu3UCS=? zyH-@Cv2a0@>qF2Qf(;pQF^F_OFMJGZ4DQx2G>*CPz}ZrrrZ+EUJb%&+E%W21;A36q zmi^sPti^rA3384GAr$N8Y(s`hVqsB0sV|K|y<#r&Ci&}fp(uX^d(<*Z2TPRciZKW1 z65hepHTN)sPXW$ms~jTNlm&>Cl*(VL7CVPiCnO*exq6gsZzN{Xyem^&lU@HoScbOD zd^uI5rWrXE0U{rbijFP>I`hUwHxle2w-miJNMcrK8_H~)q0Sw{N&_IbDDeGI#4)A^Nc0?c3`_pCrlT55d6L)YltKzKe&-hv)6VX>F0knwI;Uh z6pK}U+*`diS#pCQePuD66=N3&?eb6&dKO>78ry`jCpHMu`NhJfBmubANfNpJnDxgl)eJTy0ynm3NuEtFg`xMEEmKF5AW8-6+e?X}68;n5>i}DCI9- z{|1R&lC|`%3Z4LT15R7xMMczMJ-5-$QVSZqwkGL2F5N`SibwRx#gDmY%(jZ(E{rsYwv^vxaI@(tIunF zidx>QgyZ#=)8+ym&xnmV(1waoAznb4stVDU0Lh)7MYy9lwkub>^MO?$%ZP_6TibtC z6^{=c$>}a$V>~80TSXhmC%SgxTV^DVA{8Qq$idj}WXA*J6Myq)Mb485FeDfrif*60 zb!x#j&21Wq13KmX*50`(KZU}yH()l_xU{CBHCIQWPoxP#B5#7Rx;Q+dNgS9Rh(+G$ z!ux#w_A4w9p&#}9U|R2c8^0rA*zhuqTz*IhZGPVQ3ef;z$j9cG=*XUIWB+2|GV&p# zH6k3m(zu$;XC}EEMuwz>gx@1VS3D1Ho&6<<+hk-C8Rd{p+_vqSP|nbvv$r(>vOV?M zLeL+ZO5z~X?H{$hCulJ30uk?Y^*Dj0>dNewS2M{y;w7mD$3-GVQiB$vk6iq?o;buw zJu?$c#3N)9X_&UfE_6#uDdthb&rbo)Q;|p?gB-o^spG1p#Dhoh2+e~wwGvM3XU9V@ zIOv*+>-W2&(Ihg@&u(_$GpiG|A!5L&X7WSf{wO(9El9;F#HK3v;3U4#s%sS-9)Lx2 ziL+a<408KIVMUAgs6Jy5FuSP$p{d!b?#!w@mHyg@Xp;VwCSqb^mUa?QN^nxZFLkMP zVo#Dz!wC`oi4M$CVP97)MA;Bg1*g1^-U>#>gRtQ+>8n8xru6>;AcB7Qjc5ibP3L1C zmBq*eo6D4QaElGuL!Zv!>Qa9!WabPp5~xu>ph`ud;rdG0f-Vc5TMoVp!kLer*FNWo z85}{r8$2k867+LaBzuHx0V4&=u+3q8edMlRs zW1fKQv*dA&s5U0iWD6K~L(wBKadN6uWqo#bsh^UN44Q)X92CLyG9Htq)crnhXWuJX zR$Ze25ud>#o;XD(pic$clrr(YK5kXbL0Pr`PxOg^%~XUjl|-uV)$gbFvyy&v(d#?g z_PAYH22HQyW5wI;_)4>T=4HB@#9hRjx4_ZHi_%w$z0>}iUqg}pQ3ZVq#vD3GNaMxRU7TB9TqOj;U)Re0D)mx2v=b$!R%K zrM}l7?{r%-Z6YZ9F`fisXv~UIZ%Lce>TI|X3Ez+J+6k1xUS~Zc77NNLG}l^@_{7{r zxe(6d$3T8jqX_AZfjQEBWh3gYRB|pbz;uT2sv%&4*@#$j!x2 z*QQL>_9^rMa#)ql;Kw(`lY@yV5SCX)^M)7dQRw)|%}ikPXMDhdzYR9osnCgo*oOk= zD77^%iIIK_Mhg;Vy-s`G3nHwO(BP7ObLherONTDuNVkDCrwi9+yfsBk2J#z`LPFo# zGu(t0(~vIq=qpu&9^=h5 zTavfmL1fNvriDsG6_~CaUcPNd%soAmzf^~4x)BOj5in>4yl8)V@p&~SX#y<3N+{^u zEn5u`?n6t<5<7cG5)(9Skwk}D*(*m|64liND&XU#z(3Cr9#dvCIBNyc1T4I~ye4!g zIr8dGmjX}WFOJU#OB3jjz@doG?*K{62d9+AT|&bklJ3>#_PSdDDPCH*o0XE%yR01C zS7%%`BMjZ0C-*iK^Tc&_;u+}<_J^lqdE@&#YrdobAb08Vkvhy8!jmd$?ug-JXvKoi ziSoRKkWNDJlz26~c7>w)N5ShbT~-A4c5l1{3y~O&N2i3>!Yr5aR`SRo?y*T_rmt^; z-A^+a0Upm=EcdI$Sk<$C`F_FFf|E^W++G_zK^z6@L`*Do^ES6t57}DN8~ex z+fk?fXwi{iVLl1?5BQQ=-2XmoB_9v#^=sU`V*$U&R`+r2-*W-#8O`a~%ihw7ZlC|S z>iH<5Z&FND<@qHMhv2~(^NVm$h6N^~aSwbB*ma`*=;GBS3CHdXqm&5sPxO{#I+-rCwTdSj$z!aSe9nO*fF4njL4uUr9$>g7w%g8 zTn?9xYa@2}5y>e1nF$+6;1ew!_8*jNI)frnx5eEyNVMv$)%MjnnMYd4o2nc~*UOKr zo?(e?S!^C<P@)Z*2srT+(Dd1X{}oun~N1`i27AoAcO7%C`$*`AFn^SUz>&fkyCEe zUeYN>VfAF~fC5LlJWN5desTXs6%02yFS`N95`K?8y>DI_K+4(6e97R#G*-B!N3?o5#9i;ZD%J*ur%UfWgmoXvG_!-t5IYrf*%I_SNU0<~bU1PDtageiVJMV4q ztc`hGH+EHxM}OUR+qWgaiJfbrF z+SJ$<7`caVI~9WDAXkI~ocpt`qtrcBf-1q3NNS-oRrky41s@*j{N9PFZK?lSjQ0}0A z+vLdo8s?HTyZ5(l%fY=9v{RSE8M(++1IEK&*#)tKAeypr9?x|R`fdjt!}l`B*Mt0O zKN0yr76Xd!A6Lu_voLR?=d3v|fAIb$%+2`VoVMhC0j@v%UX8ZakTj+thlI^uaLgvT z5<;s0;Vdg(CtPp(t(>yASUTSW8R%^Qj#5k_T_qVDyB32}ZC$ZTkO$5TJ@6s&8O4n^ z0kTG7S?#uoo0Dc6WEzdJT>x_$n)PmFat#J~WufdPWSd zOB0|5aeZJ^{+<6>?e17^p)J(fbwf9B_!%!F#E1zKI(RoYEljpQV0rwaN6@2h*K=J= zSF!Ot&O3T_Dnh~Z>jzoROY1Ms5fZi~-LkM>;nv>4kROy<_*~KXq}`N9&;{`r^mu2` zQK}LR1g?3HsFTsPse9KqFfd<{-YfZVLVPYTH{c?Dl2mFnpE}C|J867t=8rWsd3$x( zdc$M9-fyA_U<|Y`NASa}Wf?`rr%{CLrNJ1MwoJ}3PbmXE-lbkGBwl0{;dwN4I}ewF zLtB(tQ_f|>+#S#(^jrVOtH*EjaW8(?2Ei}tIW%~BYykjZ3^9^Xy{v}^_S>nvyj1z* z@oz+rb-OP?H7MPC+ElUTYAT119eg`$+I`ryWh{b@4{|zN#{Lm&L|)0Xm>@oZj)}V= z5Dg=HLNxRil7J01tf1`oTO~Rj^|W5&tg=oR({`hKQI@;V>B?OQO?US!tc;I!|F-vb zIZ#-RGG^+Ou@qvac93_|Xyy6p?MMX2;X#Ay-Juow;6ZhgX5+ji6hZkg=avK+pnJSKTqo_ii3<=`!-8?5w{9C}@1hfYIbGG- z{o$VTAZoMLA8NQ$0i@;lY_DWyKJhob-zrc9&ohNz2CiYvDY6p7@2dT%rwoBQicPFb z+CEev^)l|&(q+`ZW2?k7!+7>@5fJ?ZL$2=E=p}A6P8?EYaK?#)8x+yttee-ppnA-^ z^0@XkNQ+iR;nk`@5I17mFRU;mWJt?btJX#uxI1xn3vHK z2YLORbtD*@fAaol?CdgV{#)57Z|QRP-5u=c)QVU(M}1~HtAP+}D*pP)WxmAAwc*9) zrN#iGSBK)O)r{U^K;fCfPEMTZFY`)X9e8i%Ho&axWT4b9bsW5jh3=;FfnSzYFx&QTdGjqC%^j-ANGb1S5kf}WP|Yb;xH}4R&kdJu|~k}TZl@}neoxJIRAn` zI@{wP5f&WE&f1+{5~mZsFKq#ST2GRd`*xL6pI*neG)r!GvRa_>YKldbX>fFZvKoQ2 zOke(>Qoj)+Fp(~vUfx}w#UsypFRAU7ncsPh$+xu>d*xDbXN$2qWYtooUx$oyR*s@d zionDC_s;*Xgk4NHQc)2?AIaAxRppL0@b<~&@4b7x<37=L?R>q~U%P@~`0u3?>D9jb z!Z)JV=YN$^tj-ZD`lJxUOGV9<#H80^LpiRYr?I}#Wz}D>`1R#gD560 zp`4GhQEb@WGP{qF*YeMx{qOhn<_f2`ha$1|9pd?-h(<#mu?H${MnsF$2uAxS$Dr$p zESZY2>X3K0EB3Y}DruMn@p19-bgQSv^ng4UfP)Gg)M?d|n5f=W?eP7&J^;#QseO#0 zOC2t>+8-&AS#}YEPS;2op&v@WOidT5C89ik;jq2?VXoSe`-j(SapRx0&>IRXm8kIO zt(i-zo%2+rIiKV2Eoyefal3OTCj?V-p8)l*O6rcvT?dIEpy3%#kIa*?LK_*e3=B%Z z=wQBx2}gnZ*TZiQFD1oYOO5CRVLn(O(z{XWR!(Nz7uWAt`Fu!n((LuMjjPhYi@$w5 ziVU^z)an_;J$b3^g?;FsUv&4^_Veb)?t%6M;a;wuRqu)YM-;8BNSw{4liJ$b0c1Z7 zQIsj7EYF>p;a4r$a^}W?RBsFUSQQ5~%NaQUPWs zJEBqVroA^f@<(`b`S()w$^sq&z6P7+?91s2&95kU$LeI=_Vxr;nNYwY(!f8G=mJ1V z;?9=SN~$eiwgh#HJCq3i?Ynq`sDxC+>8o0{Mk4ofaVP_$`Xhl}*R>Cj5PEtn$x2IX z`|!eJXEVh?xs z8aSTvY@VS$rWyp<%l&x?kT^-5kvI2MP&-BZSD;F{Vpxb!IzkymrS}fk6Qp znYK&d!+S8jPEibf&-C=P=S+S_YxR#7pDaCV4a+A`jtQX#`W5eHPbi1Ifha#=j3Id;aE9qjXm zOMNp;wa!CwTgFN9FzdhTQ}(`7sggSEC3cbGVr8t6Qnr;o4;enG_>x3ww#FLio=%~Coc&gcHkD9UOT~sS2glT#vBlvfk zmtuD*$;6nVCYR+!cM^&Noikio!t4qDm;)@a%40-BYbxnHk8U$b9BVE}-)nvqDo@|j z@tC<(=uNh>5x@3#E~-=8(eO$vR*smpT9V8gs}*kc%huM|q;631Ld!f+l>X8B2$Q*- z%M!o5@t*uS%t;Bd8K0-b?H%F6d{};F>wZ`&bN(sVQ>CVg_+hUV(L#aeVfX14>ZZK| z5YTnjwo5HT4O~b2&1cqk>J^%${07+FJalwY2hm17C;Ag63VN6D9X}9MGqC723q^yY zw?DtOY})j`B-CrS5xBW_Q?9hpMYe=*wcvGNOg~=s%&O2@j+iVp5Qeju+J`IAZ=@z& z!CJ|ML=*yD#ucqacN99TXn}PxuH7l#YD zD%9`}0_gmL?d>zG$xf;0tAlKjz$XDy3i;O`J_3>Lc`I>L+dV%-cWy6;IZS>kIBb!B zBbILzzL_u`QQrFFZY(Nl_9G5IK}cV4t^vedZ}IqS3gA5-|6y>;L_$s11#2z!hM-}A zCz-F9wLb$L%nGR^cS}u+qd&q;wrbNzOE$?e{I>cYOlBp-LG;- zw(lxm0JiCBT?WOrlpdGS$PJ=6)mGG_3;+UGQ&a|&O97*oIT>Pk39qNhL`xi0PRuX8dD#da27sEEPC}t2B zP$S^1$H)QV+g#5Z{^Bn;HHa>prK)NV<9!pRXNVA^%)A+?3#s$7?Gs6fi*(mxN3*fI z*j>?;8T3ORjxzmk{eg*NVgrS+va4$YNu@#~DwAYXdus1%bg&Y@rAiGjgIa?^CC*;G z&wU$+TI^ys6Ww`C=z$xUK;IWz3ZI};W8untGV|ZAhVKbyfI5E{X?GdccA z@9GLOVt-9*vL-vyU30-daM_Bo){RO+v*f+g_5` zmRmAe1s(&in2N&paq}S7^^nMs$;wKviw_MHFZ2fA59cY5ahs2A5_5QMIJ=+w5SS16 zeR6-=8aN+sM0UDe_?%X~dstTK5&y$%t+F~(e67;Hv;Dx+YO+VGTfl8`g)G`haS3`X z;&PENcIR|)$2yHEJ+($7SK_xmV;{c*a}@t9X0Nj!Z~$<9>wB6!BX+EuaKC&Ol>8w9V^MP_%x<<+5IBy`^&+w4KFAQRe5l(gd&0NT3E`)B)byvhSTsAjm!N!JDfz$1CXNL#-vFG2_lcX zWD>r}znVFTpZZBd=(feImAMPHT55^kSop|EVK6?~sp{>9Jz1g{C|zs#3C1e}DqXCu zCe-Y&<#Y8%KCesXc!c!H=LT{4S&>W3*KQ{P?(`dM%B!FL90fkqTnRGom=1Ez-wswU z1w}hZr4F>XRjm&IbxsY*5qM(+s`!K6m>}I%X9OscARV-30HdaGL7!Me#Gb_hHYuqI z#vuOXP>eoRs{2(u`g7AKXO#_V3{- zMi&1V79Ka6*04Q0rMU0kz<5pB;tcs*1G%J*`_jrv&JZzMK)eE(Tn4Masck=W_!i zGyEOcF1s+RU;or!tKaRR<3Lz&0)||@&)B%?|L{9sWSvKke=Uzhe*rNUDHtQM>!$RW zi_FG^r+cAujN7DX0=rb^F*!g=m;H*EP0-jwT&;EXuR+@BejbI_734-3PW~r#JDVbZ zJKR>BmNH~Gd?aZ9{qcSL!<3_NiVZ)izP>)_70l`c01fom2$CXYpe)DNtvWdMG6^ahxJ$(&orP$0)#k`XIS9iQA=_f zJ?D?0j4{>d7S8nRrul0^7DqJrF2+%Sx#X?-p31;#8Y_v&^6?8#CS9A#$ocHq6e982 z0Wd*Gg87MOI%%HczwZXLbeqbPu7pH}h(Y%~c?z;+{?*g`OSSww$@?6{9YFMp)HS9a z`wjTb#~0R%)dS*)G6uS)%wOJpauE3x7fzZl1SBs0lSd!;9=)eq61>Lb4-KTA?>x`sKl9BWvfi*~=3iw(U6;l_Nt3 zSs8Ba?*j-$L$-*yaa2vE+0B3Mk@yp}?U$b2CM+#24SF-;(oty0sK2+m_lz4k;|U zq55db@#g2eK4Q%R8OPIT+Iffp40orkignKOOd{7_Xm$ilnyCddFnt!F_f)nk#xalD z_)V&D<^TkBl?Y*tP-H$`bSrb=0g*g;@p^b>yh9V2?D&S2H8d;NqvT=Q zZE@3cP@m@PO!A0l`wT{v!clxjwpyys0(9R%ZJz<1m9M+#knotTA+T-kZ^)QMcWh14 z<1;(rHQ@d)RyoW?GcaQuJ|+IySaB)5i6jhZ zqB1?y;gR_VOa5E-!~Q+o-{SKh6#4)CgHDJr4zv9lHCY$a`KLVk|NVmh`MZ9yI0|Fy zY~YG?YS``mYo7pxkZVy)ZzA&!G4cr}@7<)&FrnEB=Ua@wPa` zzVW(M7D`nA*TMKdE(69Ccm+s+`@ja!yHk>N5&2Pr|KcV8a)>GsV{F$n@*y&#{0`lhM==$wGR5*HyN{ zQz>fL-3q%<*j?;$hDEj-lpbyjsBi!ICDU+Jv&xIuaIhc>sH+(n0t9;j|DoAL5@`~2 zh_6RJJqPF&n(8{>`uF+PT5>>6~px*4ayM2Zvvj!{TEuN#>X1bLiUJM?9zp zq;mD-4=F&>X2Bed{@-z!%>dM+wIjGrd5-*^9O>RS5Av#a}4K#b&YH zYQYox-`IUZ=U!5^)kSe1cRcZ*qXRfIF4#0(xN zNo)8F&3~m`^vCD}>i>+zRnzN}ctHI5F>DRldV50hpRLOke%f~7JKRWz_#H-`tW9WW zyYO-(Vf>;kgT(JAr=Oa04Lx(uNbtR)HV5@(0VdsHcd1gRT^d%DfK3N8nZMe3#D@{DyN4WV}SF)O^DFoWUbO8r-<^)jDnB3#ahs0=Q_GLeHDi zv9PHK)8nsz)*O-agHymOCQT_oldcr(@_uJ!736d>PF12^ z+EPBuV~-H#D^xg`!*2t|BUqrkmVfYI0)sblmlk=)h8ORU*)L8vhf1%lXX+w<+boCx zs8Z2@=z-Z%j`3ujJ;LePg<`*b`qzN12cD+(yV_~D$>g(*s zADD1TQV!iu4CHDC^!G(}*}|NDyX0z@?jB&pWxePa^~ZbZj>Dza{|5TKb8`TORlQHL zZtU^VyML_^5I-01rsH-FW;Uh(&A-bJ+g&0{%r#argFV?r`n>oF2tP*fSL$>~&$}LE z1HqA_KSU*VcIz#pSDWg+>f1GTxe02Bi;?#Ion3V#zKj zr=nUnV44!!RoH#ODtCeMkE6N6& zJ|dRbkO3XY#Yy8UL5A)$|5}@nvAg&E2?N%%fCV@MgPizmXORuh1Kt`L}Grs4=(c*lD(OA zyYE(x9s_hmc|U)?d);FwCSA>11E~EtB1~ILYi8JQ{$|Qy=5X80%6DsoMEyC}-PuAR@YE0{OfI%tI`;Rtnf2QR ze~)jlKXYm64>HPwVK?&kbXs{>?bqke+)q;BR7yVC&wJ1Q-MEa@;Q^#kMZ5=SCqHik zH$ajy|ErjN)6N5&czbmy{^Y(VTB2DV#GMbKLL}xEk%Vz)iC{_%m zB%J~SC;HghrHb9Rv*#nb^~8WJQ8|pF=}oGpFKWVSjLZzzSwyE>Y{Qa!1C4x^VFR55 zwsH28pP=+t_xfYif6lAgxU?Eo&z#NX;~)<_0#jZo^J0w7$360$ib7=ih7SAxQ85?Z zKCuX<&1>RaKFEVqzlHW?e``F*>zjDYM%3fVwyjL4o<`5~1aZD1;f;X4IhZd(F3g3> zRgpY>#NgW_P1uFhj3uA(gWbE=Q_t5y?K_jHNWT^>+QLrWN30T-YMxB4vC%Gb+bo$- zDzut#S}fkfQK35W%FKTr2m)3Go;28UdUe8^>$rbv$5-+b`z0%QRyvr0S7O?$Lx~-j zi2Jz0{0NuNlHMUYyo7T*60t|wfNM*PSxIF#{h{O!{|rU@9VR9wSKBM9k+w_=lcvrr z=0M=M|G=+J5(M$O!&LUXr_ys|^^?np`p&hV&2HW=ivp6Lw8`KrnjIqiDKa}6CE($G z>kJPx4A-M0`%hGw#*Llkkf6`bEOd&q^KoBPtJA!+|Lg6qUr=SZ%OnehWc)U7dYPz4 z1WX!2$BPwmORcv?XV+2_JJgk!E`jL8ACi)@B43U`Mz(hPAZLF9sf*RQQB>I2kD%w`()UjrS5g+_IU2G?&9031d12|;1O$~i zmck9*w_|+tBYAW_unb0XQjg27Zzu=0M3qUCkBLfttBBNtF3WH z9EeCKzw702H721-i_&m}Hddx%J2wcXjv*F_<3kfzZ=E%t3%6;?z^rexAg znvvy^zF4d>G`+0vQ7;7lDeR3*0o9??V&PWA@EE|6LN&?gOF+$4Z01f)kXGRI6&nN# zd06R3w5f^wgw|DYACPCGDff)xTXKcY0DH5mf@$W|^P@$-2M;A6(bWn`FQ+0nrkfWZ zjrKeDl&P7dxmv-r8cN{av~5hGsUCiq{c*6Z0Ds+^hR@sV5K~ zfhWQV6JG=k>gCl=muXrHQizN(`PGDx!CRL-t>-Ns7(Qvd9waYhf<^4=2Y6nJ=7&Zy zzodY#hWD81jo~$janX)|l+#>5*AXiCbSvT93sB<~x~&xr*;O%rH66Wuh%8b?ca)g+ z9HCN(J>{ycglYvG_LLY3%S62ZBBTG=2zKqjR$B~8xoM{P3TO0#iPeqkRi*=7igNmG-$vF zm3}1sB0Rd!RUi5@RT220JtT*4B|sCit&Wo zyEIG8UD(TBb{0^?NvBCtd&UiB;{Xx?;(^_4j5Etd z+&LjuwqZ6C4NAcNO|)??dZ0a_ zkSW#Sd}mWny{z5#__jL+o$NG2$%eZmtEK?kq>vKrd^LCV{C8)T`ck)0#A75@1un%P z7PY%Ur(WBOmFPC6oX*p*?zt}1q%sBesY<)vVZR4=`aoW^T#fJ#G&BJ;59~60x?YHYcW_jhd6=A(aTbks>r(GGz?j0U46zwcr^uV;!nCQ1ku$2X3dMJG`rXzeVKN$^j83!F7qrKH~AzjNK)+}Czr3$!gNf~oAaI~9}shjOVDl@ zM%+h<3T3Tw5$OGx3BJG+di;`ta-owjZM*+-@Aj<`wFuu*Q*06S5!(Lpw;)eGfBzXr zLd%zzJ3te;|L7Sdrur|TE3pD8vt6RnR}6!USPd#P44EsaN7nqPgqIX+PV^)i3I{#B zT}aR0|ML!Tk-FwDDJ~J0IvgwW)1{8c%I7p?uL0PaAtO9gajk-vKl2*&hoc`#){)(I zj<#L^W(;v524O_7vV<0DZ(Z_JOYuG40UBBNqjGf>-HB3d7lA16u0fwy-;J)j`XHI6Kd7IKpjYm{_W0Md3~ixnN< zVQyLa>L|VyLMz8hF&Cl<+f+VFzzn**Hi)Nnd*ggy=;3@VSM5Byq?c}1SK3B+ZCzZ- z*u&$neFIyelly!_Mk;!MkNAu!tzpLJ4>Zgk7ux|WASCmR)m1r-V2iiG9Rc=)(ulfD zr{51#>TP60Lqm6O-Lr1^hR2Mz`FT!oJ?*j}9=`fLqcOdP0sQ}gUQ+aU)DFt3Ba)Y|eZB8D zaqC^$tGzfk$EH~Z#P_-T>lWUQ3|^Jr5Nd9F-A;rR3Zw=-eA;&G<(C_h|NFQkeq~AK zw#?FXsz#d=uCFeaOLXh-ZA^O>hF?R$DOR_GT@r~jrPU7{n`6JIsi^W_Lk&9>A))*- z0^U;#95iP$bD8|2-eu@z$);>cl>Y!95m^7hUl@4em)ePpF;2e>Qu9L3i+BVx6*DML zMVE-3o1)fJw@FCPl5=5?DDj42gQ#NrSG>M}O=b5V9jbT9SiR%3&D_Im0~2Pb*0=k(o)`LBsVovp@3eez`3 zjus#+_r_ofxLkkJGr8N6!U|11H5tsru35gFS>%bB3=$gyNxAnoXUh+${rG>+djIMcnkeDH3#Ud*h;><6Lg<9o zXy{Lu1048nr>O+nqdy)vEH^ovfd&J=RodL70R6$I4l9*=4~R;O{R6MS<+U;%nYYSD zbDUl3Zy+nHK$bAb<8nu0Ofj|}sm*-NeXiPCCe5-xs*LK{n(dBKKC&h8k7wksaHJ)o z$#Vh4mdQb29tUg&{5y!MCPG+g{PSsVKh|iz^m7J~wbbL_-cB)Ed(lBf#yk>aKB;_O zsosLqZ2tF<(;5UljSC!iV1j~&RAs#v$#N+e85O1^4st_LFI7_}j~i+I)Ma)*m5fDM zMNRRKG~!>}N|!V8I~p1g`btEwZ-c9hYU#8Q=*Wc^{O5yekwRZmIe38Ue3S1^ zcq(m(szn;JuOK&F??HAtR%IyMR+~8F6~kwX@w(?zrOPoGWHS;BNzB=7*@ot)y-{V9 zb^!bvwqok!A9=0zlshbDy7=o*S&Bw3vj7O|IJ})|{0D3s*VzIl9yJFZH;aHa{>>3*F~o5$S;(5=lSCnS7Y zh1$#I2!3+Asy;g1U}*4CuhdiCm~lCV+@#E?ICGHImdKDy_VOC`wL?OS6xnw={Q^ zrg!0zhQ*_9OELywbIr{phLu;dPoQ^*5*mfu&qh<)V&Fito>zT%%ik-PV4Q%fcIV{H zpt&xow|jQX12C>s4As`oM>9vXy3OtvndbeakxN+k+RdFu~EzQ9}))^5seT=~|Tj!~@c=%p&rs`P``;0^@mS{|S zoC+c$=hpOdh(eNcHG5DE{XlG6U0h|)@5qps6jfGv`@d@|E{K z)Yg3`jPq)3+~G5Yojy=So^H310@)`)gBMU%2@tKLrW)I@ zM#t2Qv{Mva0H7)GR5s!RlFCHEz3o=NO1{2jFA^Ti5)zuS9hC&%|1O{Q_4k)nC|((_ zGhBSF)#<`X%HtVx|A0!1P*s^lK2X##IRkP8`o)(XiZ8PVhlr;Y z*IuS6l6Hu1(5tG>Dk{d#Mmknh9s#XZGdIwZJ{7nn=dLW_r{!rsigDd~v$)afegVx4 z&Yr>aUn(6sW-VK(?1L+gs01kQ$ZeU=vQ?IWCM41wf=htiU8Cfo{Y9IwcSnaxz7n{8 zdG=HZKMtX@tLYfinG2ZLGGw zwJQ4baBewullEc5wnXQ9DJ$B4vjD2oU(5R3^^8b|K0$w)!$8usMl+uG2KkCc>p<43 z3nid7pI760aj@A^w#gq~_#_$}MFInSlp`$?t#1HEHmRVjGB_G0kPEnL{b6lnCRz3V z36KlWV*sK<6_m>=w)~wtMbHi*!3pi|siHnctAp4f84A z8XK4+n)--Fxq%^a)ufjUX9?ExOD|s7=wlG)&%LSxmrh%+H&k@A@4H-qa>k35u>sUK zW|9Ac-O4C7An_)js?g+A?a1;bl{X`Ab+M?t1Re8)&*uV$^}PP%`3{Xf1j0R7DbL0) zvWhyJn_Di-Rc|&&8%Y9ar6rD@Y(sY0pZaQwb!ORM4)Sgui@^^i{l;kI7nZMQSu*Fl z%jaEBJgTMo!oP@x=f!$}&)i2}l4Gjka#38-t1`+ok>M;c*UElc-kGTM{$4s=F_A#@ zqyWGLs(SZ40LC8_N!X(e)_9cL0DVq?n`py~7ma;`J5+<@u$So{MS@k-_!M{-zY z8XmQjxpHbo`sCy{oYoprFp^eCjraM_CDQ5O#02l_#N6)m9S@P0Cu`e6MWEKC;;`<3 zf=Br$8C||A17F@vS#U8*pPzo&I-JojJDc3e>&+QtDs}*lP;#v_o|sH^RO-uCAZv^u zi?hi%7wtp>m_lgwVuMo|*V$h&m!}fj7wmpTJTj&Cr^C_DyC4@a!Ow ziLfc+BBkeh8#$r$VoHp3!YlwkPm)s1Ocuhqiw(<~om z7j4*4bf=1i2+)b=?b|o-@La;bf;7A@9~<{ZDug_a+Cl1w_~!IT^dUg69#@gd?1G17 z*KR$oZ|&=c4#?^qxvQ5hrlzKX9(=Z2?O)Ka9O~P+VQ} zE*cyX2n0xQ_uvqG@L&lVf&~xm?hFz%xJ!bD;1=8&EI3RE&Y%H;Gw2|L+|B!bzfsM}w0w`?!wkrJbBy0?#0 zTx>D6tf$!(`Zkog-WNc>1i!K@H}vnG|6QdrVle$h)GQ9<4XymUobB@gf5**NO~ifs zM}>Ofb-})c@+i{ zNB;al{mS;N;Z7aLFH7}{(|k7$1&8fPZ{Z!c%%9`qlVOM8QB6TXpzR0Cc$+3MR=i%Srw` z{FJ+CPDhF5P2M37^?WWnslt(gg{9?C*Apmjznvz|E!ObugjxSvAVxlu$YCjTubp7k z!tFtO`j`TwsUFv>%cGMj!{TAhVt6hReoSGsmhsGnjJJaCN(nVl{t=XhnRLl>a=OT- zde8j1cVb-bW&HP@4Q6H4Qr*t>8{t1XY+7ZZJ`!)Y-NaOoQZN;$+-oLl` zJft4794KeIYtA&N1j^lJn850Z6jq9jw0M&)2oB0-NXbs{04i7uY<2Nm{q@TwFEHy^ zXxIpnceb8xFMVJBVLsr)d;7K_v{#fwNqO51EN=xJCWjd8oV%K6e{Yc0x-Q8te ztgWIc(9~+aecNe(+<5vDpEktgOpc~gQa}s2m*l#|Ihz^a-QbJjv|KxF0BdpjS=R7Z zsK!O}4Yl3|GxtGO#lf?Jfii;~D_oePL(@t3Wl-oAYOPPZDy!yXqYXb|pp-IVu@|}W z`#F26o%oKIC3hyo`Ev?uBJ0?fS4ol{d@pjIAl=>-z27l_Y8e=`O9(+SE@Tag+R_3soEeac66_M}E=U^s` ze2>a=9ZMytu?#*Sn_Ri&`-?9Pi44Gr(rBMu1OH;n&4Nc~oY@3k^!d@pL~QRJRYe6@ zU2>ZhRt>W>ou&J-eXG88xG{D`s?25_XK7I{M^6CH2`?ml1$B9wUo3Z=EE?ix+zo&H zIK)hPo8^j~>59_T;_o^cd6pQ^4V($KtF0>o#R2R2U z(!8a9b?uHUj~$mc=^(Qb=2(g^IL%vUBAig~ipu4{;&zbxcI1`H>e2w4F@A~092To0E$e(W7&&S5ecV!F!RJ*1>y!#4F7r8hvuuH;@jC#-ffD=Ka>k$Ma<}F`5cD zCv9Z$ruK6hELS~tj)U<9=x{7=BVNk^zwRFzqM5QQHhFKmk<1bgjfC#t;p_)ve*|$} zGotKL$|4(Ehf@Ch7U{tmEaiU9$QkG!=XmZ#A*-=9H@VGH;)Z*$mDs>;gsOMlnI=vJ z@vh#xn6|z-ryx^vCeUs1jkWURXm`xl8qQy!j~wm`*zPqOz^lhiu{FX=ueT< zxNOauk3I#1KWR@Jnk17b;NG<9Lfe-(_a!>-gbF|r$0RsL?Ius*f8O{?ypua;43i=4 zS0N4Lt@A6mqu$*SOi=_V=Dfq<%f+4oSCxT_yGbPw{(6)UOK*QV$_$VAY>Tx!S&@@J zd+;HC?hcy(H&%+PW52Lk44k@+wyY^>EM$%A>Z-IEYlO~oW~Sysi)*nu%el6OAyKS+ zv50tb_E}?asR=?N#Zkz>qtw5`XkIbHq1i9q_hRQ!fZMQk%ipNrlOd9oOfw=D6@lE0 zFbVKqN)cqm^6wH;ad6f6t0=K8x05R&+1LmS(in|uTfa|)H@hR4iqkJs>r#pTW-c%a z8Gtj}?}yTBwR-zizKYDU?l?_>kNiWw5#S6zS4K6V4_0J(wElGow!RE=n(8cX@6s;? z!_wrtZ--$t!}DO@J+}#|PV$>%jj1u?`qz$a$F_Ry?k&C_4ifI$?%ps!ZD%X!A|r!7 z6KB*1<1mrp2A1E#8?Gyy7PCGQXM0xhxHdY?zVY>t)a&1Iiw@l)POcF3u;(6opZTHF zpi!apw%vNpGZ`xZBe{4iC4xFodRbjoWCJyc>FeSK0d9LLyLQS~xW8g}DKA8S)}Gr* z`JJd1gOL>PUJ4%T?BibCc@z9*V|~qN6ugF5qu-;|EMh3BS!n%K;XMVvr$Q`jlO$oZ(Q}D#yp@xuJozhSkIUl7o@0mF9desL$0ZJIu*FP4tDm*rTh7%scw=%<1OPe zs>2^QR;w76xC>PtZ_QqJUG7H8Z)S*s8;9Q}Y(#M~1QQY5GDO_y_$WMtcAu!S?(Seqi z@g;8c_nyrvsMkpp)R-pAF9}&WP`qu-%{M|B>xM&;P>je(=S=cKx*vmrYn~FfV=eC4MA1D`n;2i5tU`Hr#tmsBM(^LN$Sj}e(Y!f6 z*HzLOt6|I_5R1r*fi%Ag8s@Jch9ehz;#@ zyQ=27P#1rdj>59?ix;4y{foKEijir*mSXZ;UJB}=9GYeC4B=lo9Vth!7O9Tk&)%J} zIG+NyH6?dI!*Q;9i zy<{;e?AlcV{h+rw`I6FeD#`}e`{i`~8CX!Tw-}wdF@E-|kVp~)I+?bFh@Lpxdh8n6 zfI4-KUkX-Blms_<@>d|AcM~F%lTMNwFsCe`4{_1tn%`iKpl$QqUtscD@$A4UkHryJ z%*YGh#H}z-$D6~t)2?-bZBoY-;D_;%z}XWxwOz|cM69xlh+-{=-&RAajh3`V;iwS2 z*XdF{uNjo{WNF%w8ToT9(7UhB!}LNvHq!%lJ6Qy~_HC&o{VuPGrKwC}IP|f?x#fbDA2L2GsPh@jLE=4pPIvqx`WV@~ zN!B}hPl%0b?F%KW=1&kDv6g)LvaJkEEa_ufI(5nS2~y~UUC3{+OI($oZPY!yVw*&` zD&rS&$hNIZoa8B1!GX}LCCGgcS8uKM8*6il*9kRgt2G6o?XW!Ro(y$tGQkfW2?v|= zNKvHE#E%1J{9lJ&#(vs4{u@Ru@T#xogK7+TkK62PN2u*6Zee4ycLUx7AA4Anzy=X; z)%jIHiPsVwM9KLHGrxGV1G&icl+T3zMwK3k}kMZZVaCHDyKuJoNV1 z2_PAI4FXsEbYaQ6Rnzr;MB}AiAvH&2 zJ>On~BOVq`yGRPsN+583R_fp4`ss%n{<2>=2nj)2bY0z!R^)eDAyU3pbr^KyKF#o- zk#xK=os%QQXCy7-p%{5H>7MuGCmN=jx+U!>t&9_&1yM=in!He%&Qn6nN|VnG9C>+R z^d!^}(&nB7jcQfIBdY#5mEFwn(1c`cJyD(B=xm>b(p`+%VNzC?73PwNmobsrpM+d2 z<^iRQim$&48-5YAu|5Hp@8K309X*b~5++s$?DYxe2-FsPd*s#k zEP-CwFr4E_GDauWb)5z-gYxC$h#HK#MFm^$eSeP!jX>}vN_{69OtXpx zjlGWiz>0djPv4D1r|!XDJPkZK3Uo>ypl%)sg}lZ#ef`6Fv7~dz>Kz< zMD%4T$qF`gy6ijd{8Yx3{yn%twROJ0lr|sxr#XI+J*wl$qvRoiT4J1jZ&J{H?U2gT zZq4WN%A`;Hb~pG!`5O@3!oeFUc-lh@aDI!o6OWU85Mhv8PIay7g&dpB__bW!d7kK_R-^g} zR0om$comA)T93Wp)Nwz*IX`m{DW61zgJH*|QzGSKS+ilaLD;oWbEI5PPhuiM>77Kc z$2Z27`wJ?vLUIsFK2=_xXv-ydvL8#5%act046hsH*ErNNBa*Ql?R=Tw+&gsqwjt1f}Q?TM4l}V9LJT+7^^>Pxez4H+9T5Gs?CJFk`jSn-Z~9z;x-9eYZur z0OhnJp+wEV&8Td_SM}itLzhW=G!x?|m+E#zAg9R4ZvRbd&9V&r=XFuW`D1SU&~>|s zikd1L;X&3WP7C&SC5d-X;IPSUbze!+bPgoG%Hk}>*Sv>+ZrCYRQodKR_XjRMDI;zM zLFdd%_|Cu~e~tkDidWk}msK_eUM`94+=Q3Jjg0tk(gyK)N>30K=FmVcc#JiOpwnk2 z3!`Ob<&IVs?5sf%9XwD7Oo-csEn>}7(coQNHSbut z62a32u@Uy#?-h;a@uW|ZSBuRE`m8pQwzz>pT;$Y_EY;H#)GRQhz2Uo7d@Cqj`ozaP zU6!W;K073@>qfH2vs{uq@#BR_?xfl|tw%DRZal#)Gjm&a^pMJ-Dd)8sgxtRUbB*?j zxYyz~+hiq|@MTB%yRhHX;m=7wuHi)s6nqtQq=nm@rMz5#_x_l#P%^v4a~Q9~r+#;- zVuG5b*(vivWV5hXrV-D3_ahnVpjT5P3-e#)_PXr9r+<}L8k4QgTY5f3XOM=Wn{Z^fhXt>LVgh^-Q5#j`yJeTf{aWxf15`*%59-lBR^|$Es8CGoMN?=$%z=DRc^} zzknKEbg{Fc6vIFrsGTbgz$lEB=pi&9GD+`KALwbDHBOpRm!CDFjU&h5%tN=0`yo9N zg!O!;<);`nZiX&wne8i(8d>ByCW6yuEc&UgOrFN8fnHzhJgf z#S@C@@BP8NK@V@|w0cC;LEG&b&^~EnL-#v3Nq@fdgl1 zhI>Q25vfhFpnkU)_a&|LjJ;Iu+zQv4+6li^s_*&XK34R>*0fT9-(Rs1%6w-4R5#;3 z%vJE(VZO-nm1#4Yue_r@q?FYYef{q`gYkc?Guv-Js=_6C?CyGgcC#fGCImeR9wO>* z+sE@KAzqFjyz|3{& zffXmexR4(0=JDH0syP{2Rk|@)N5qEf*daQ0e`_#$eAK$1ZLUhl4n2~^>2Z3IlC{~q z8S#uT5U~;|ZY*sv2d~lBQ_nF=t3Ma}vDIWekn7;hWa_YTV;PIo%+WVu{Z-K|8CS5J zVe?e&HRBd|%&~1;O0-XwJO}-rDW2N%}Ms>dVE{plUPTVz0a~PB3p}3 zc>pDe%a5;b>b(tdi6^jn_6|Z^m0APFhcmhy_!n~=ZaU0$4kPFh+pfvRlxM$Vb3Ju3gx|h)n#E=PaNSlD;tGkQ zCgHthD4GX>6klhp*bK^baISMdc3Ga?Z)_uRvI=qykG&c5Hp(h9ZC|6aV)Dy`+#ffQ zw{X`zJx)gd_9>_%(B1BbzOhAMr*quorBz^+^Wqg1^5pG!CoQBsF<_Jp|5vWw&BORb zOn?~m;iU$3emSA}BxGGk`Z*yWjvh5M1C56ABC!x*#}zARv{^7?sHaic-e3B~lII#s z39ct%Sm*91b#5n{Ew#g5e6QW3O&J0N)+U`=nOR&TZbf7j9XP1Et%kl0Lapgc0-r7} z?1LQP6p1v@s@tW>s~ZuhS6etZx1-(U%otJI>i`=fJa0Pn0*pI=cxw(R;IuBf;l^xy znB5^Ue!8^#XVeTb;AkK2qQ&NyVGofeIF@Mc?)Jd+^0xeg8qpUDlGqG_rTPYQ3@5fBnYvK zK)p;vuda(Q*G%dEc@lW||9WA2p)!+8H4A+@XZbJg{`cnwnODVsfN&BY;UxTbSLpL# zZ$o-e-ea1@I@zZOFgCIs8fm}aW5W3h7j?0bmg%dMp7$(;fVhK`M3Y`6+XSC zW#`w#>q4nQpAzC}pegiRDfvb$=c6Z< zpZ_lc{>P&Jz31m!T6t~~n*FzFOZ*%}VMMJjimoxRF@6;PkGc*>EU^`KR@M5A%!3*KH`jy|E&oUbAhx134lQ}xsj96H)~7QpiUpRHu2kiQ)rrcP z!rEz^Sq#~#A}YfvDugMCX`WO212ts#H>Tq@zrI@lGj@Hw&fi3fEAvm56pQGya)iZwbUe(~5H`HN?#GE-5(A>zNi zS_45#GU)tfZfn2mF7U`KFn~>hPNO5jjPYRbnTY>og7OVG)ckw#me=F>L@8>Z&oXzq zPjvdwmj((8hx1A7 zUx+Zd0hUKQg%j#ZL$lw>mIC{bv4SJA`Rj>bbXHP}94aHhF1k!hJa^Q?cNE)BOmkU1 z*&085(0z@)sc{mdj)|~0@7U+SnygI?JYG3juG<)2h2 z&!j6`^kr^m8ewJ{QMr&XRfW~sbbPlrV41d^(NJ5PNlMV|**EU6PvkHLajQ-6 z5CCTcv$DX))3t8M7ka{eW&3k=VrmkHv?xNT{99Z}EB*JJ6q=2#R8K#HdE^TMF2BCA zNSmtCP`S=omKAM;9w-^BTlD5vVl>b});CgQS#&vE-)y`9Ub}<{J3>N27py4^RQQ09 zJP3mW5Qgi2gds~qGup{Uxd059SvQM5Oyh?)y07H_3^?MB{ePZ(^uIru$BNTea%ykT zYgw7I*y_+`JS+0U@9XT1Me?xHGl8wUnsTa-|(Ui_J*iQ)WiS!RAC+i_PB{wu@tezZyi2&Gqu+jr>@ZaB{iQ?flOxbmeXn{>|v+QV&3S(?@M;QOlOVVQdhQ!-Fs;CxBO~x?R}%HV6(AK z0hCOV_k2c(Lh7UQl!U>E>+u|jF5+;7+hLkvE}gi_;k?jc`6NN}X{MC!M%EZ7u)?qe zkHI`Myi%H*g}ad|rkZ~aX+Uy}(~$AHgFX3X#BuPM%G!dqF}zS;0+0qFc63(VgW;^o z0|6gU6iyjx!F7Z5xFM}#y;r7U$+34Ml2buxDD|?_)4E&8Fm#0$WPo@ZzHQ4kYDg0+Jw@l z%`W7&BHPqmEEk}+W$&w*V)g5Jm%DUNz4|vG=yW+3@FlYzQ0~;hE4@bNI9;Ew_4e$# zDc*-Mf!~$q8KD6w6U{z$=jcWmez}JcLe66g=Y<{-!myBDNBzsK(9(Se?A8i237>iOIp+`$9zz%sh!)Sh*c2MiTNiG6(|1d4nso3&0)2K*Y{RynkkL73yk(W1s zmhGMLBp*3tp_&qxyBzFJm9dSH&!$hIVR^(fe(*e4rm&ul8}V88rc56``UVUYt?F;%xQ1LpFbZ9wPS5vHwHD5u?9Th^X$N(@zer^$CR!oO!WQ@1o<8Fk1t*(HAkcAJw-b}98A6A{ zShdkEbBDZhZM9n#RsO{VP%?4jTE+LV`@FN+T2gEqpi7w)DUI`EYcBI=*FPK^Z5HB`^m9D6q^1JY@?*_$;LX~zylKo2O$k}}7NB6z2HxBGo z(-kggBGtxD41V~wE5h=djJxmKEIqDVHgq+q&V~T!n+I09koy5Fuz=N)UFDzvP%fKJ zSh+uo?hGt6xm`GxzNdq?jNehfakjr6z_Bwk_2gfCxD~EKSuQQbiZ$rF`1FoP80gF; zclhffv)1B`9~aiub?I;&(}<2ALi+D)m6jpx*UAeLO852m(iSO9#fbwdhtig7U>xP& zXX1}Ax5Tggdc0UJVq!k694zUsE+_-mxF(%fa_wDDy=Mp%NY+E_FR@_N=PF1h=fi*T z5YU_Yd%i~Id#B=)LUdibmRZ(sX)H{@V(os&FYV_lca5{N$E^KIE~0eQX4OjzMvPf| zl zWHkIQ?hPG?F<0(*hDo{GvKKX|JCX8i8uSymClx){7YCe`UxWnZ)mSB0Za4H>c6NjL zZe}IXXT{5Pgh~xsK5=M;?RK3ahohYW>~TY^5TZXx;qvzq-K0iso}%5PM?8_Q$q0Hn z8S9_S+Dz7#4jnq_yZn~h%3)F=pH8j`ykh`pyD8%=jj8u-hwn6(%Ni&~S?;9ud7GE3 zA3tJ^hmiCAChdT?-HP&0lKUXa36ZldM4NYFRTulPB5o!MA*aOCvm3mxO>eTlOq>6n z(&KZO@ps92vh20&U+8;gMLMJ28F;(|bsdP+8O(DENIItw`Wk#)aNi+!7_fn17TWLe z(eBKRGOtw_?qGnW83$%nvm#xY#*C+44T&)O6aAyvx~TV}anY zMBkhMZahKC8dxIZAse3pir`!&@lLR^%;6Az6-oVSRh0v`2qmFxZ1m_6UZc`xfISt`1bm?t%JulpPJ3K=rDpFFmLos1|m6Qe6 zq;(L_S9y}!05<;v;kQ8C{QPaeAb2L;5y30U^`UWCGxntHGVn6ztHo8B01y|;&3C$` z`=9+JNHv@-4W^ysk^1Wt9+z~#>>Jz3HQjv+_MI?zK+MGI&v=y-T# z3nu9@XIwlL*b{1TtsSw+!tGaWiEstJv^;!r()D)l0>V5pHfxX+(FXoTb<)bj@}vp0 zr~t&@U(iu^*2Qvuec%FZ;a;@`Nl?F8gH=Hc7|!hlh_sI5a4~jD4b_r#mdVV`dt^nc zdcQAEx}$QKF9VkO5w}k}H>=j8ZHL`-uH2}y7NK>&>tS_?K3VB>($tZW=Qlg1vy&uQ z`tFxE@h#jaTRDW61NU*lqzd`W2)}koK9T#udW>(5we%nltf8r;!3_SKTn8sJClr>R`G`7~`?7#VC9{-|O@CaPn$A=i zL)B9x_ICUt1i}JMp)PhGWrDuoG%U>-6maG3Zwmr{d_H`VmIZ3xTn_*Or(X=KPWN)z zW^3+o?An!MOzVHSbZy;3Cce4#kv`$$(mL-a{fc*IQ@=MCEjsXPk6o96kkt{@neRQ? z4|q=aPeW;=cp&X)vi!#);Q4f+nb>;lX}8tN*Ha!?OCvcaZyQP70gvC}8=l3^(;=Z+ zCA_aZyFHrQ=_@S5dL3$ssdh6XY-OOp+gtAMiQ~fkvRHDU6MdY=({;XH$t=dcv!kkj zEqDHIl_SdJGpkB1>{6?T@;w6lDygg5j{heRaJssmCpJ1&XOqX0TQ=Cli9JpwQ(}O% zvaMXYU12uvq&JdRKL6w{uUpvuU18Mb=|0+|hi&`q(J5WO_st5dI(5hPQ8-_JfTM~4 z>3dE53WZ#7Z;Hn}(1+F16N>?37^m)DO`^1;w8v6wVcm#jZ{KH8jg$n~w+!wlK}#88#&|a-pOkQ3upb$;+ZV3wXv<3@ z_ccPwbARgFrdGzvV85$8kS#@#pBc3MQQ-l!r{-~>TYvPaVfhD zujrvAbVbSG-$Lv#se3L=GKo&?w9zO~SgBj^T-bX*IY7^O{?nS;)Yt5iD?#r$mcyh&w zAmQa6pAv4V@@vVzh9o}j#eraroO<2#u`x#tZrHeO?DMM$$eX;DsKM)grKA%6kiKOpZwqrjhX{U+OX-k+{&%@bo-{K}dDbNzKvKcJ24`-<57yHf2O> z5|f{Y6_-h2Bg0OxaR!J#NFG>OKA_(mKuX*=vc@M=XkMs@DhP*}__vx)TDEL1jJz*~ zNdrwC0j6oWzKMe(EhBbUTp^~3@1uymB@?Eol&&ugojx>rsTm?x`J-8Fzpib2kUdYH zW1562)<;NZ9YNh&$Y zf8)a5{R0=u5Xvk|esbzGLzgRga8T!jeD?^qOMRTdpe9v5_V#T@u_YjN_?Z zlcx%QL3_(n_a=LoDZ-Sh`z`mrRbxJ@w6*2FSb>gCozvBD!(SpJZPM?uK&4yg&oiQ} zmW)(9?iH-VX6ZW(rsH^q{<4$XlE2aEBWL71^bG#yv_s?lordaqN1b8$)TOqYZ~ju7 z7A=V%J)T5I21T%b{M9;*Lg zcwJmQl#4)qU_O1=7>c6=<{EM+JgR0Mxr`?F-CH+(751lA{`_UXqEMhll+7=ooaE;2 z$oOo)kNVM!I;nzZDg2nfZ1}e*LfEfG7|&ePldAQahWr&$ux_rB_y3U#ra2n!*Mv3g zFQw;(O>}J%Fe)(P?vu>-8hEYVY~l6;@9@$pf@4BrB7Wa0r&=XQ!o2opf<2>b%1;dp zCg%g&Iv7mYf+dPQgl<9O-F+;9$=R`BSh_30aqkKQJZGCS5*rgruq z&F85hj#-Q2F2mxi%!^6Yn=RRUPROOzjs5x#x^?3CMitd%D(3yR{N6_0(f0?>r$O-sUix5@{l{^X7~^9!x$M>;dO8S z@kGsak*XFul*Qv2R*s(aJCx_s@+{`n>R>lRvqz01=5K}e5X6@+i?dE9?y5GmtH8Ms zXy~nhV=|CQjhE6+3lAFq24qtsYriKjt-im0@qPkjE87BdjHj#n(dUK~9$89Luqn@y zJ%7~4qDu~~7bcBlGO~0jq_~}sIHbN~vuh9zO{QB1$5p^wwJQuSy`0y(0|zBmJ-LRr zCWnKEnO-^p`Q;xgag~@6`qV?6qnYMKO#Ilh@KZ+?(Ly6Z=*6dRw>b*Nuk0w-F86(+ zRRlxr!-byz4n~`occ@_aj9$Pj!zO!mGc^Lw1YZsWaCvu~lB1na?^@tmjE*=3+Riiu z1wXHAt(h_R%bq`Us;#SWB68+k`{ z`>y&h+oMONF_C1zG|TU>0e|!L`J=EK^1Bfl=9m+BplVt%b!TtreZ$PCY(Qv=JoXra zLfmS3m0hk4@%}b@WoW>-juTs!nNuzp_L3W-R*}WMQ-q}c)@-x9E!b8QFT?G0w!0P@05f|j2u^*~#p?EZPA~Zig z*92x8;98;60Vw3EFUp?XyW@pKg*JR^XR6@DkY5&V$+j9|Yp3ykdzZ4s(Yt~WgKl{N zmb4DIRV$KYH8rgQE<)z%yHeBai2<`e+kneS=Y5%65gKr)mulr8@CW7FaTm|$5;^s7 z`Zq+(+lLF;le4BfRIi<7zBAhefR--~;JXP9o@CaBBe3E+@Hl*VybjSTrLQUb43@lE zW4{vNK+4iqK0$Qiw3hVzFTR;(Q%z5PZ$vDpt{>c)6D@F%t$d%A0yHCJf*v^rd#v9o zd2%5fjE8@W@DFM!kIIlC5pW^-fT^p5JNZSznSk*9ZgJ1`4bT9M$fiGub6V<(S0LGj z&WremRqjE3hP#wcirxA|8zX<4?@;Eq$wkR$Gq|@TXlU2QckIJ?XI%WRUl#i(#v@!0 zV0RKk2)7m$zzcy;c}1IRO1J&Fg}EfUFGfenc+q>VI(tK=azl?IGIr2^O6_)s{C!`+ zp~R~kFOh=ZH)nm9WDtu+P(50#2IY9_3hHg)ngd zWx_QYQ{n+-H^zFmE8yS_^j(n&P5aPM1cALGc6FF~2KK~ssca)j{i%U7aJrNBwe?`N zN2PRo7ZVE;SphpPiEb{=9vfEDzWqK7BNby41q4a>VL6p8WNbR))7V~OLK>uy!QZX< zIA8aN#0$WcjNi;$H0%kHrD=PM*+EW;a=}_hJtYeVe3Bj#K~d19#8S^2 zF%iB={(%_G-gC=fN<;52m7`~e-kKfVj<^$pc%duP@|`GKd@&4;w84`3{`1mqT;tp5f=%eed2F$y9g{*jY;VgSB!>l>Z@B?M?TVb5X zO2wewyHfrway`8c+vonN4dmiIqWDIIP>^f7RfRbp@lpK4l)%X1N-i4zy(dH%y3$0r za42ZQDsZ0Xe|3w!UWm)Ux&cap>7w+1802S{iKqsWfKdUlL~XP zTG-x)&RN^by*8j+3Q>zq*vtUmXF%CKd- zy$Sd=ZU8URJ~90$JSW*5%}A4llK+v}U(xnY@aE&k=Q+bk!~$vbi|LX>@7&iZ17T6p zr%krjKSjgDTXW1)s2o1Lnm&)7C4;}SQ+KGw$Lun@zf`!s*x@wBo%uCKE`K_NZsyti26OFPzK2?@5_)s@w$R>0V0XG3xTq&1*Z?0tKe|SlPxW z&&PiB^ZmBQ8xVJMR23GjJDro{Nz-4f`O7l{KM|PMyYzkf5LHnIJ_PC|ylJdbkriT2 zt-sHXbt#-ut289v%rZ9>OWxRrgZDm?@j1t7dGOmf<%yE+l)P= zd#tY=9}zR>X8QgXeQw>or=Nm#QI2((O4wSx{if}e0^XAZ4P6NJ&~@Ky_KFERwX}-v zch6}2fexV?{X?Dje8)>_5MUd76mYcLCPJrkF#97n`gXO`ZI;^aMK#Rxg%STTfXhk^ zKIBOvMwNcK4B+YD4Nec38Jc+Sf3O*j5oslc%C+|U*zwQr2*r-RSC=^}ga_VfQMc}4 zCZ9JQ2kx~(&^A6}Ir6VOW9zx=G**pWU;~!cK-PM;~Jq?6htWwg>Q1H*HXQ>$d74C3Y#ZA;bYJ z!GkObg`VLV{MY!SUz&IZ@;2@2@DJzdA`k~t4Q8(GbYSuHtDfjbpg_|bPQy-Dspq*M zQ-7%~A2A#>xWeDS1Cg6vkqf_38mc${QwyN6!vS=wku`Oh3#0bkV zLReGlM$-z}odl#EVTJ%S+PE+x-Nwr!zpbFWSWAJqe-&jWrVGbLw7y6qTDAjPKhf*X z^h_akwhN#}6{TX_jHiA6h?3GKZ>&6oO@i6wG7aJ1(J(pg&Ab)t!P9xR-ydrp+$ zf3Ti^rmP9Zh-nKhG8yDIfUO`toO7mUZA^Mk4I5A8cAMB%7j)p`r#uV5&zJ(*Js1k4 z(EPS|F|+HcOeBtJgglo41AEM=z2WObDb{D$jUe`2UUHAPDUnujN#H|cO@c6z3$;7N zidIMF+z-X)(YiS)8kzw1AAe>-R#jBISMO^iF;@fQd?2*`r{CH5pH#d|xeODy`$O-z z+=yXhz@G=!K543)@hOpP?~`G|7^a6@lcW(bKk8a1i#jDWY!b=n-?`Z}Bp%r3i2CNA z2AH7wE5HQTNwycNw1Ipwcf~OYaBMnKDyaj=f>k2+=3x#ifuP2a^Ei`gKU0xzQdB3d z8w}B_QKQ`T76&UA&|bG45Wrt504b5>5AQH_j)A0fXi?9De5Se|-u!gZAST}Txm&aL07EU^OFi+~>wC+;?lZXP!8r_n3#QhA6&(irmKB2_eyut?k zG?HAHDAk4j{)&j=Xb6;u-V-U+mNwQ?(s-vcy)1!jyl0X_kUhBBKz+QQKEf1%6U#mI zE5=5nYX3w~iFf-8FsqfP)ZK%u54%_@#l0w*t7nShsN%&x4)D)fLGkE{dCutcxAPv2 z(5nvTGx{xiz{z>ia^iZeb1xX^Vg@^}ivVe#FcyLxXw^f2)RP=tLHH2E+%+bNDg=J! zmBdL15<=B=Ou;m9a&MOWhE2;s+`}gKQKs#e1iPH8ex)GhiIS?wR2GM8ob$gx`fFDj z`2JV=7622?P6RMfmgp>o|G`-o2ine65*L@H0_{{)C&mYd7}Tr&dCg0l*RJ=Q|9Ncl z0ZRR!$AIi!O-&7Gf_fv~oh}jVtRW31b zDmt-nxIUCM%N%S%7&&D%_i)RnAwJ3xSF=Bj+?24tPT2(wOLwjx0|>FTP!q3l{p`}4 zZu5Nuqgss|L5qmz9^x{EH`b1RC5vb?A?S}~Y2W3VHhH!ju;aEV3eU!VhIRgfB1g`6 zlJtcQD6M)wi<~ax$W)|#iS2H`@}@P{|Ksb1=LgblSpXuqYhNI_GQ<{ z=xgFHW`yKE0%$Yx0c~E^-Pls&`ttxzu6A7LXGAMeW3vw6I$l z+qutDO!)?*W^ice0isrCA^0C47@$M2|L;)o10+5*fcz~a_s$RP*5P>{+XCtHJ_hyi z|IyXEdoA@9VkH1&;cRXulbRq{r+t>YjNN=&Af#d?-T^huvaJ#I%U}#TSh8Laf*l zBtY*hKG1TUn2ne*5*$2y^=GZ!RsKF$@AL&%oM)qb;4{Juz+bArg$@rVJ-AD@(+Ic2 z3wx~E8k1w!%qH*53cWTBptT~WLL4>T_Y8WG;@2O)y)S()O+Ho4c`#+YRAezJW61Kr z843qxcZW&cAY0}G>`%+%*8T3m!7(#1dlF57$GMgu-0) z(OSG_L^jM532^?0ZDa!Y3+1*zgI2A2+y|C0Y~|tZ=xIs zjH&^4YY&hdttbmH5F?K-Iv4cILzW^U_=`>Ij{38ZO0*wv&^2>{O+WXclLm2XpdEd) zwixy#K?_zFdPIVTYV_$J&K9iT-}a!qKpgzvGAV_ED@K(=gNm@6y;tOCa{gPg+Uq}mnlX6- zvHqL)dbo$+89HCBco4%kPW#uy&*nV6oroLKRDi@fz(NFSMARq<$=goOas9(u3?^W4 ze#ObTEx@5ezb#iQ^&cwZV7ipXVBKi*n=7MatbYlvinMd+Q8GDnh`R6~OZwm=dIAz6 zc8fo6!gUJu^#A>>A1puIE{qI?(f%LC-U6tOrE3=sgb-YlpuyeU-QC?`FKq)m-JfC^Yn(Y=clxqwPO?os7LqM>6d6rov5tqEOVx-`m5y zhxEEUOgwCMeXE+6WrH{rDh02J{CB+O1PUr(npWdt^Nv2YzqOb_Fag;yW)77OXiopE zQKaa+D(q+p18~r(pq*M?FTK3Hd}4PfvZR`6!YV)6KNCg)2id!?j@W+caEg`@MR0n0 ziiU&poq#INOw^%IgX~2`0J9xuvD|MDSn%igf&(>^2R^^Kr|IvL-H8?Az$M1ndkn3* zNx-s+gB3VMdgDD1lPgkX{)o8g)zkE>Y61=D43BMl7x43-hd0Htab}%F1CGE32K@6T z6iD8JC*2}^ku+d2;>@yqLpAvS4pATGT`tczMO{;G0^e8#wS;no&Z(!9wQk3g4v1>q zJZnu##q1zsQ{%xlAFnrhv}#f^G=wG>*JZlgrMl^3OeTYt%LuXU-oamEfPze1ZpV$! zxDv9g3j+fh#Pq#!sZmK3l9H01r7p4el{ZxUXG!iaQ4x?JsU;aaUFlCy0GYE-XvQJJo?37J0yk7XNACRQP zy{x&3JIWti=Jgu(FuULtpDEL|i73@?7Bd}9B6}FaHwf$FaO;aWWRkv0TPzKoDqs1v zA^gZ%tY&$0mosZ&HaJ6@n(G;!!en{PASpS&(TK$EO&(POec4pIpW@Z5bn8&~o4-mdC4T-)f;=iYIsoNY zK$3q?N&D^Oug;Khe`&hJUfGo7!#8K!{b9Sxxjvy0me+?A+EcF<9;Wn|c4Du>*^hV4 zF5^dan+8|VpPV=015YdPL){}-t(F16EK1@P--kwl6FrA%7svA*#m?G=+GB!?J@%;R zZa=1*JAH-@%uutj6gt@&zW3Cs*${KW6IX~s*YL{@4h|C)daq$kh%1h1L3017{{Op7 z{j(0F>3pMry1eMl65M-gsR6Xy9PdUr><~_F6{p%g)?B$I&%Lw-0oS8NLe*l`#HD68 z4L9=06Dii7DZ=aQcmtPSK0AxC^rb56U86I%onM0JpE$_KW@35U zPjIZ%%45df`cN#?+r*wt;WTKTC8VDU~%GUpT-u~xP4mv0% zCZ^ZO5NQbsWc)}Ebn-n<=_jGX>8!$K?YnA)n#FZxeW}^|S7ef#gQ~-~t>LLG@v^CG z65kR^Swx&nl6fq$Iomvx>k^NTbh%9)nrhP>+3l7|cs(yaU`7}Qi8S&f+{n4^X?2XU z7_4qG+3Xo7Gkb-^(P&Ntg@!OD53oa?R4fx; zwIs712d_=}=|SA{`|pL4rMvwdRL95Q*Q8?M4I1d~d1a_N!0xl_=*VudoE;B~-0IlF zR2{(}B(UN>beJbstVWl}-}gl4vOk@;Z`U$?pv_=cR(y)_)=VbDs7pLQKVRBgr{$i> zvw<`8=~|3{nORw>&i0g++38VjT0Y8H{zNp7qR`!x;odW2NULZFAH+m5zkb}}cm<_?9bVR!FFI_d1B`7Aq z%|F6_r45AI|EARdg%j}XK%{Kcj=#GT{?)LajjIa!=kPCkTh*uk>KV_z78aC|k?X^>3YZ&I(^PR z3xm-Asy{@a0j&eO<1|18@1>mv0Il8*BK%J~4d$|dTJpb;B!MN&Pv{9H_eUmCe2C?% zo#t(8#(yDA0e4h|*A-8KnLry&i>+gY<#3;T<}SrfSMYeFthrTK9e*_ zfDu64*0W;An6+i{_uFv-GGR~X+uji39{=msP|yU-&0!7<590~H(at-R=O&Sa%_U*v zlZ(j7#i%Rd$@u5dEoc-{9(Bi`B^+V1c+R9U-u}JkC(%rbO-*ZrnOAf~qNWwEDsX|Q zj=@7X@4khk8yDNx$EKdvC^d&)2Xy4 zf0-WvR8)?lka>Wmu}FrSB=np)iOJ?DeHy&KrJ1+I|J}(CwbsAqA36F**T;vYNPFG`@tn>XPezIohFN0h*>*l!PAt*FO4eawCwPGHOc2@ z4&k?s&1~8<7ia1e!K99O2t!B*Y2~%z5lj7)EQXF zba?Eb6`7y}(Ajf3UT4yLXV$$uTDH*rnqerADK?+2>BoqWGD`JVdbnngyI8wu8Oa8E z*>KwL2dlf#Hg5jTY5*Dq#A|yL6YRapZ_uj(c5la?y*HqcDe5_jN(+T7L6HCZ7lHub zls1@WsQ>a6QiYhF9JOXq7zqj4VB8nbbP~7jZ|9wC=s7W^N-5U)MxSpo87z8ASBLR@ z7&^96<}*dDQ#NxVd9v8#3-x*ZSldX$)kuoLUsZFwdj`q;dInWx=PB_0-YQ*54%L0k zHBEyCAM1liOxC$}ogO!<%x00Cyf5Qmt3V4Ms(IaRNecCf%UG;@KHZ=7nS+A(s-H+b z%C>VY7J<{?zZ)#dCKLl<_>h^%O{D+GbBAF7%(Jgb$R0EDywe3xJo~Ym{IA$v)*Mlg zK*c83A_Xgy^NDiEC<&kUPUdNeZmV~tN}=AQ4hijH!RasEHuv;6x>yoMM#hN}*|*6| z5Fv{XRkqViXPQ|`s^#%rO^fP-G2})PTv5fqFW{(EsL2CXaJP&khRI9m_&t)s;d4LOX?zwf2Bik|HO7xLyYt)Lh>A;# zI(9>O7mf!_7!K7PGJz8vCbE32cHM;*?5!d9|L(%8Zv$kt-6U={ssKPxN&Zo|hmuev{Dchg zi0F+C5@xX>`X=?A`ViK?X$^|c0!>`IB`YnmederigV?Tpg-!^~+P!ekN-mHE6BILw zTDNj-P|UX${*B)(h1n*?3?rPFC7S%~<5R>oiEpdQ&xlj0Vq`n`J zf!uc*KIufu_h81|zd2fJRrY<@#gW$w%EBx5SYJn%`D|GtJp8T|itLeDHQnz*_kq^9 z-){XeCNbStB!%7PU8}%9Z*C8%U3VN~2qKeQQRbFR-9jYJ1BxrGW1jfHVF%0A>b}<= zHH$9$7mjinlSFww)-Zn;t1aa2E*AJ~&M!wWZO_rQoURc$ATdf?mt*fE1*(1PXR> zQQv3^3%p$bxW>NmBp>wZ-w8IK&bF=CELhyT41o^2n>tw@wOF#xoTm2xoPs+f(EXc$ z*Kd&q10(tAr4!4)$`M$zK95oaB7m|QLpm0^)?{yy6!i?Rg3{z$j`b4VNs>_4-H7%v za?CiW>PNmvr;Q2vi;dia8P=|nDpH{#FP^bU0McN@*$)hlWqQ?mmj@%7VwU&%PyNz% zE%=N~+(ZEC!HZKYNb>4<1?wz9avkWD+WX!kNON=jK{RdsbaSluS;!ry1SpnVeRFdf zUq;Y8JMbd$UzdA$!y0^8F1_NT;&wZVX|`(?q-$HXrqrivlnC!o&8wPvh#>8~!+u=Os`!8y<~i*Kr-zZxE@oU5GB0T~l(K0mA4^ zb68_zmyYZjoct5SfcpL<91CU^5 z(m$XtzD~g$$dKWUZI2&%#?x8AQ zzW3OW$q}KWj8G}M2Jd~bPg7-*)7wWGRnwd@yLGDB?3~9T={pdf4Wj*bn4qFjMzO@U zZxz$W5tARcOuE1N^FqPv@p08y(X#8i1ur_?h39<~-Gt@w?IOYAesst5PaISa^c!@F z47q2KT7KG)p0BrS=WMfmAWLQgu&i(Hae7hs06=Gt{lKx7vi!9C4x;5XX&*hGJ7TPh zg$5lPsy~*M?traDAwLQq_42rb^SKc+tOs0dqlNxRPP(#jiN^5?odnH*%?rpi#u zFa~qg6F>)3eJ2N$5kT~r`LxMozEckF&BWhqj`ovK4mVY7l@i^Z%S%33@)O|0NK19`j;vwQTgqc$ZyO5?L~V;;cVR@cP;>Me5w4V`W!%sHr%Um zywIvna0~`-I!-qOc4oi5r*AR0?*ia_#=mxHeIp ztD}U49F{UoHRSU84*SQkfoe`7ou<$owht{_tTB*@Qx((y8kc$4fuAs25!Nw>-=ET< z0dk!G8voqJXp#Hi@a3n-JChjYvYf(7$2M%8$?0NjA2+OcD~V>;V@6c+z1dRQT1&O7 z{%=h#mIEQOso`OgYMQ?)e3z`Ten2lokkNI7{;mb1b+B_ zJGK@R1p&$4Z#H?2CEKe#2iPvC%op;4fA;jkl3`4jXwufVs2nf2@jp(Sa98T$fV}PN zI-4RczML#Du#uCKN>-C0hI@~xj0KBIC!~H+F&Ro2c%)J8?Qbqpr5Xwd4At4ZT(SHa zLUZ5>mJ~D9o~`gZTC82@@4vl>+hx?`cN+CB)ND<$p1BIN)bdUH%{oG9>mt02O%;2p zXAki6a(Ng;QjWzGo(Q-yow&++ah`HDegZthOmi6Ej8E^bJa#DsjKmpz#Qk=j(_2c(gz%CE-`A8!w;AN(U6+xb9@P1Nl@T! z**4BT2MwnFd3;Z*pNP*UkpQ9{Hg4(^s+a2h5cxnywO3&}wf5VPPh2&YMvp+p3i(O| zCVf0L$(`=wDNh!MbA+Ev63xuqRthy88whu&_hfVhdc76|B02~h#{i*+@qXX#*@=ld z`#_UmIbGDB&g+%yc~KM-pB!&i@(1_z!7Rlmie{5DGEaPT8LZC&E<62(^I52=49ndP zQ*WAN!pJdbw20nQCyZUq;7(PJdyl;Rn8n+;%fUG6Qa2xS=D>wFUwUX!emhdT&j#1a z3b1H^_V(E7dK{%}FY0Ck7M{20)o_7=eO$&oC0TM|0k7tOMSM-DMZKTDy2LX)CDT0) ze3LoN(ifc%H^BoKfwJAS@&#g#ehwABUz4`~@x@hR=Q1m_-EoW7n5s9?<5bgj* zHTWc!J|#OQW-w(ea>bWYy*1-z<)Mje)BVPF$G$LA469|e_3?h4$!i(tC>%P@1wfmt zbi;=juu1kZ4a#TgtFRj~W-hxh!-H<)D0K&Gu%In4Q_FA0dK`EC&o<)g`Qwhr-D2QJ z4?$os{a`frYgu%fV}2*!)qi7YASs zE(UoA3cjuySsnw$piKZRLv&)JJKqlaj$243%|=x%LQBXr5|!#Y-wZsrf3 zEPj^vGCWMd+x5pHuMnF|QNTA|{Fy}d3M=SukRUe+z?#sH%`3>dG)QM-nfBC%?Oo$m zJ;LX56hSH$@5Z7^t!#nHHoq^@Yhfr7x|!I_LGpaa6I+g_(o`Bfa!M&)!~<9{C~q?; znRC=5%ft*py6_pyrX#U^=^j0p^d0A;+GX)hcWH#r%ml3~7F(WkPbGsoGD1$psqq+_ zt>?qt;A`gcO`~uxBQt-tEjQUXX2%^fH2e%uK_?$@<`}SwTA*EBq%)Tv8ivWPd7H7U zx;I^vP<#Toz8CC05Ao~@064`2c_v*9w@`Qb`l|3 zAs70TJq+Isn!dys@uD5(L(LqPX-xbQ}Pv`TW)%fH$fxy~lg=TnA6D-zfGR?~JW3`8H>+hJYc~ zGEW^DjQR;~b`CMZ>q||bLK4f$EzXnp!yqpg1nqF%@aQ?u6gx~w1@il;({Yt&+yiLq zRg&RtNy#(CUg|%>_*&K7l>6k{ajoOp==twH$I&HnCO@xD!PKAO{$Q|tzpaKamB zd18Fz^i=V5-p^*f!-hqeaeG?1=pDLVv8XN?8r;TjF>T4JIHE^rzabr-6UI~|!Ho90 z;sA~dD=ja$)<=Sl-5tEvYOjXz-z};&t?fx~!nb~y15g~L3f!7!M#3R5#P$jGTkPH) z2}e*iMI$ekH+lP?e(+lFA>YW`e+0E8$4+Njmm}*b;Cr3Wd8{5i0bS+$JKtPKT}t)= za56weXdu(3VRVW;O_JM?(n@v`;qEzyOc4O)4IhX zqO87pvN|Sbz)-!6`jb3qDlBG?L>?E7PXpWdZMH=09R!x7%{l>ndod=|&P2*i z&w$G^8HhJu2V{1-^~Wowl2G1=2F1(mPK*Z?AI-OZGnhb|u@9BuR^8O}eYmexM~Exu z^At9@TGMgtrJB9Dx&0QIs>^Whdi-0mNWGLu*`;ez<${Q=yn3SY9TdVwlm))`ZSD7& z>eMg35FjpNEngdf&i3%Tpk`Am_UaFJ_4&Fh69?63 zFJ&4CeyXSQB?gcShKZHa5L$F6^^n3|PyT0Dk0X0?Q}3?n5Gi=S#alxKv?YQq;WUH0aI)hpjqxbtbx2Tl=<`--xfS# zgFBn!@Sww#)d9}{0CGHM2LhlBPI&c+wGLE#=JIH{9XB&-?SomYG{{TB+ItF}Y9eh` zHv%^80r~DbMsCOO<&Wp8{BU~P*g`wfm zY_o{J=w~!Y+Zp}OaQE-XxdbsrS*>Pm7>2<2oDNls?jxJvC9`77EUm-#z&* zI2R98fhA4WnNzUc{ZP|w%}}80Vp$j6$&YRyY(?g${l*%^)Z!X-5_5iCU2eho#Zd+E zokXb?3;M9>H+A`v^dbU)de_14(ME`zxQ48)yLy1B2v1cwAf5pSMNeJgWaE@v&;mIZ z+>GyMD7QaeTFn@xu2Rl<|G@9H4WTbu`EV-i2}wdo=nT{9mW#2g1Qi$YIWv3gQ9WaG zx0(#PXIZ#(ensz^C*QLWYMApdO2uo?k7uD?M=IVOc?6p1ooq0P9 z>Fi6OXfjS$ONTTDA!)R8Yj?S5(e^8OeN62WSUoPnnu4n|g+&g07qL&B-AS1nw-K{< zA304wF_~KN_3~8FK9gcHp!t)RwA(2;Lk9JU6nOcq+*3h0%P~Iw@m+l=-V8c@99}T@ zBOEjEE*)6Aj5dH;?nDngko7Fpt(j(sBMPgm9AN`^G2%R*4xd1r&Ix)x`@|_jgo&*h zNJLOfdv`#W_k^SxB&Md@@k9V(>rEjeJ*z*1y`Nl49vOY7M5-1yGUsccEDctC6YHq; zLRdC_mH}urg)tDzY1fb8VGhL=ew;dj=#zIcXUw5*#~Fp5Rh6pyxXoJJfg_EhL5i8! zNzd9?I`-O))=p8K=QR&H$#6&Cu(!?Bv=7E`Lm#(+YW=U19AB{&yrn4_Z<$XL(s_BV zvbqtc^LtH0&-|{V$t5^8yJw!B$@#%u$lTe?>|ZhyN8AJ%V2PReS_9( zZa>;DNlEPzAx4KW*Z+mZ@<8N0Dl8~)AD4GM-VbqWN45Y!1eJA|awCG=mSG>Y zc6lU8JW{o!_m#!&3PJhBPX+fsMDZJ>ZYM1+j+-W1=hx7;BD`(114CzoaG7#ypxn44 ztZjp29+Gn$2E49osH?YWMp__kP%FqCErsNTaFiITs(2K2?Qv;h{rJzizQ&TDiZKQ0 zPjak0TnHi$rw2w!FxAj$wuyO+WXmznR~M0$6W`N})d^o?f&ASl zG%i2#loaU|iNyTT@R3|Y4jY!ppP%!;;86&5!KqVd0^xtR}19@P)gj zSTZ^)0K(&udL;!m5{0x_JnymA*?{ie}&_0pvruuz@z`o+qjEDU~FO_&KW^si{^jo`5E-|fH#{Yi6EId5$3msTLh z3gYeYQ=1~X^XiD#=S|!MA^*w``zL!ruDGxvmL-Qs9w7{4Fvx7a;IMZKH#sOqzLvDB z6Qt5PoSpDus3axP2;H#R8Zzl@;)h=QwcFF%nGmg0`H!!YUvq|V<+~#H>wnukqDp8S zLpmC>I))w}M<<_7QC9F4`0yiwGIwgDj_<7xCer)t_98QXUntXQyTvM9tbp1-&{kCI z^Y{kIixs}QwOfgeww@yfw5iH5g<{a|N0hqCiJ;<4Yk-V3st`&@BfYT#A+^@{qizFp z7OWN7E!a4^cnx)Nb`&dXE-TKbn?S#AAR#2JPt<>1Lyu z3&nAOhdHjc&H?g*!6N;K1(`{16`CG4y_MII(~#F-2qvV5wH8gMA&f%R7n(AP`0D>wWwB$H>({v zmh3drBz`gDD?o96UF=F0+_KQ%o5_fJ92y=hR9G~B<;saP)Uli*KU(sW_$X9Uhy#7B zU0jLgoGY>ArPnpem~>yF>>H71E<-u+2!s&DT-axWbj|52xBw+h;fmi4gx9>UrLxZcLVyGMM@&rPN-_KvDVI;DS3~m4ic;pS z^riAapdj+&)Taj-5AxQ_Kg{ni6(rKsg1GtZ+O7)0_MQ)$W~w`V7AG^Y*Xr zc=Pxw>F8$$-1lz$&SV;mpn}^u4KH=zFNZ}7szT?8F$$IM)GU1vcvJbpTFbsdzoCj; z(-(V^G#k~;l2n=p;hOr#Sq?nCF#(}bc61n3MMRe zBT|=}2LmPO?|7el_Em3vukC8Qr#*97{5%cxk(3*C%p^}IHSmMxgR&ZrGcn>BA<`jE zh@wcIZAk@@{hcdOaJ8c2Q*`W9wxCUsPFWlrYn|trZC9n0H^Tm_?YQ&t=JIjz48L;d z>s~J589~!2gGhZDhOU%)F2ksBd#&HXD#!AN=v8lR&gZ@;Q>OD4RI0rT%kmpb@`}{y zq;bIfvT}dCJ$%l_YDT~Ob#rK9=m}dU!}kHb_4?K?zMei?4=Rz;W6VU6Zl~)Fa$H6a7KH3v4Zk|zE+7juY-M(;idop=**Y#i zNojX~9BTtd7$L25p_!I;quhhrSMobLgw92t@Kl2C*wkE@6D&r!6e+`50%tvBLB&?s zL(w+C_FNSrFju?(4$U#A)FPr^* z>ttle+hHgdKmt(#gj$o83Lux?i+?)ButH>8+dFP{Se+H~U&0T;yyoVDhd zx64|d?6|`RW-+E^^G?y!gzCqPkdf;yh~ojj`?Co6ubm4@F>UUH&6GURy0Qs!NW;>5 zp?Wxu1ikpyFuS5Vs0Y7e7>c7v{f-XWH*#&R^!91f!&mT=-$$21DIx!&d8`q5QtuFJ zei_F&_-)pkj>li$gV~}L2@^8&vmEVOIx2WRI%j{q+mOE#HSmr2ko|^dj5=UQ7KL9C z_I!QwW^r=V$WIZ8*~H{~b<>@V7qUZvCw@BVfrGrJ+Kuu;BC1Eo<)SrINHkJ!KTgZf z4$wpB99ggVI{gf!oV<|Nx2LE#TgD00=J|}VD?E4Hyf(g;hb=cvTb`Oeq2lyHa-gc& zsKk}LOB=0ULt}__gGHBFqcDj-j!8?sI#T*17<32>X{p^4gzG>{rkGsm*mL3^nI9!( zZE?<$>qjnjr`9F+R*G>hj+Y+ru&m2c8%yFexCc9Q%De$@rsD=`brT+$Qa&~jF{~Zt z&0#mX;DPLDg2nM6mR<9CjrekvO@1w-;oz{b*eV-s>V%gYS$)gNZ?7Pit>~Ilb@>lc z#fvo&`6MePy$)NK45d|@4bNIA0Fxv4>S0nrkF3OejU*H{jXR+4qqw56UFtNRQZ)3K>O{k^#EV#psO*@ zhp{H^e*5ZC=6TWUZ>q_K;qfJVCUlecyrlDXqvU@%&LdFzBe=w;@Ik%J5Z50Yi3g~m z94R2&SG&+L_jmS!D*6?=j@!}kuG??tf`FtuH5R~xO^M!P;+!~+z~E2r zzWK=X$klGTOOFZ)>wN9G0LS4AMkt5JD(yPaZ*bN;6z7H=(>KHNQ#6+Xk67ZG*&^G- z$0NS@lPD4x5a;qEz54Gq20_T;VePw3?qq4trkb}7$ojFeG2r+`V{97^H<&({dLB)? znGa_u7XV!GJbHy$-!+o`*_Kh=L0HprZpD7uM`bW^-8H-FeO?%rpFwxZ^s_L zFw={}7;C2y*I)gCIOva!YuP1=O5^drKuqEdsFV{SDS!QC0|AhaobAoeFm4LNdvEwW zcj+AgBejgykInDkV_v^_=GHO<&BJuFpBbco@4#BenPFYrPtW$-WN5Hu8U#QNgw-ay zM4o&O@s=6zKN{Zuqw604C50&Y@_a8C4(+_$+1DDg(>t(X6f%{IWftXxW;lKqteeU( zFD&(rd=O?s)#skCtYSDtx6R+53atcs!$CaHlfVqReIAeS;gUyHy!q#vXK^^rOAje8V%ajmw9!h9H4^h)BCT~DH_VYTGNw9*^b1&~Z z^SZb})Ck-{age`uTiAEnZMXnjxmO>n_7ACNY2MvSpEDwf-4k(95wdSSEIbhDn}$fP$y;)#; zR6pnX+PxVjNC)x9LcQHWygCzSWJh*xF$Lo|7l)6=9Baz;6-|@cxhN(UCZPNm;XKJ}XpXQRqy zqvHX>;v>zQ%`e~YLlf}BuR?el@2h9mZ9+4$QybVOfyAKK&C;crRFcABv&-i^OU!ZR znD;#gsR0>4>kMzyXZdV6LuNY-bILQYrma_^=kHwc=jRl}vaICp;EsrFJl}ji`W22p zDU7qLH-i%SHd+$bK0Y?T2^p%X&mlH4hvAq}CR7Y5;7nVk& zib%Bk4j=t7k1C#8Av&|mRoDD6dy@)3?&Okc{*X$83HDhuN9aJ`5%NTRx7av9*uAq2 z5X9%}_vL7!_gC3)A2PFKR{))OUphHti!99MDy{6kq9>@8uZXP85*4V07g=EZzoc@5SIdup(Dfdh{ zNAO4Dil=eaWP9HnRL29Vz0KoRScuYJ4Ageo#o|QQ=IiPO6J4cw!2&&{pRj?bnAabD zXS$w0=a1;OUjxc!6hvSy-tqFcrF!*WTn~8#5cg)Y^1|^5)Y;V+N`Qvjkv_aX0XAt` z)3D5AYmMuBdr}Xcu0QboZqjCK7Nt6KrIfOS9-SY05j`_)OBdlM2i%}VJK`*yDDlgs z@X&FJr9#O&&N<=x7Zze~c`jiN=iNRtD32{IrQwyRr?S~CN#fL5=fiD&D%9S-tg+vf zvRuv~d`}alkG1ZA1h7@A@1hD}Q~w7GAW%LOt);a0o-0l&ktXYM2>*FG3CNMY`mqMY zxhfZc%Wq~`HIZ#nOcAlr2tWr@a@L2D^Tg9AhT-{a7%Ex$HjupW;q*S|pw(;(p;Dy~ z|F}@EWOG!~2)}D-&f)T$duq=*AKKxEUFN}m+-B9PbtQH=oTAImZlb?PueZlaS$r-bFFu0dcK46{awl@76u6I4J7B=b)MGv^t%ZKaNAFyyQ|^fG3P*%T zcKXNB-+e>}H6w%kUXN+TysQ**bQmgJgIKfm{9yu_?~PdRMI9VY7s)&e9cD{BzIhC9 z3h_3%lJN8Evx;@|ruM<5r>EB(cNlK!YUMdgTAmxU8iV{VaOxFT(sd8j_O^`6MvJ)9 z`?T=Zx2{eJXW!kT9(!Nsg8%S29+0ROYkhf!b|Z>A-?NP6#P}qA)g= z@DS1Yz;hC>$zlDLPh9+qAwY9*4|H$f`9bERb^nr(*A>3eo1c754(nt#3zebWba+{W z-qzrA&Z4j^{)ikp=A*d~ssws*KiPCaS?_l9_Fw(Ik=n=3aa;4tYA11!|Gta7qOzD1s$LkwYc4iiE&-a|jP^ouy z0n;k@|H?!1Y{6(eCCQbXG_4k&re%%l3Cgcr&!3oleK&!H)YxcpM~B@_SSEG)GpX;6 zM~c6S-2q5Y70M3f6^#Uem-LXQzYk8#U+C2v!l&_hML;9($nLY-DECCBa)f`ypcdk7 zcFQ&-epFac2oa_;tS8oh#Scs$-L^O&z&T20DTK!^N)(`z_AYbkA($v}SunS^K&@ah|7h z^udaV%HP;*Cu}z7nJEgFXZ@8$bhjRYH*F>ha@r4J4~QOgY&2Dbz@kMWi&Ew40}2Xa z+x0tfWvKRxZ6il3wG%H@;)#)rx{k7l`})6;CkXaFu-V3>7|?R{mH62@iUJFsNe*bI zwLSQgbQ+Tb^}N5{^MdH8KOiZ?s?oGu{63Rv{8i@S1ZNPbJ1o#!7a7S;-InU`xtOGf z+SY2ms1^XMW&9e}(m5EE>bx-rdj#WN;HaQ>m@*wF&d?IT=zc+~;V*G(gw%?9V*iOz z9r@94QJ{YdHr8yy#`Z_{-0C;bk*)1u=GyLBC-k>JgZ&QYbChtbr8prc&|GUg>~8Gu z)wB}JBL|*y#zF+-X#Q^z(Kcp7kK6KiyrnSoXg4?o@p$ zfmbp*6Je!y2nPY5HKI&ZEGENFaM>+P&-+r+<-VDoPa(sc*^nnhhgv%??l-yL0dYkJ zd04IOcpSJ|5f8mKH?HME5vPrDxupn7XH#Md;|5x`R)>NgcuI4V)}#f^m;k&V zbOz|eajg758>c@k-8z|!Du_WwM93JwA77+EU}3v!Jm8Zw`9K!Vf#BEAa1v^uU}5nH z^lUl_9Hsv*^P`!gC$v3Bh2i1&B4u2QkZ%MgWyrs4hLaFQ~4?sUW^JC zQvicLRH2qCKF(3iudmXv(?kWPa^?UvSaZ)77u%D^D~D+9E@H`R-mA9emdm6Z zGD;iP*{FwuM=*SVVvKF&w4xCgO(*~&*n0l{;eIDN!e+NNF=Q@o2Zj~ln3}@9auFhd zD=N+xGWh;M1=A+(Rq{sX&g)+*LUZdFEtdR>q1;dHw+w?TQ=VJeXAuAGbFHw)nO&bxSf}4G%)C z3J_Sm${!Yw*dT<|XBhXBtK&p2n6X-TA;IIbL9eZgKt|tAwOt5hxg~{jChqYtVIBYD zAFo^(@y`qlv#|SqX`mjYbcmVTqHevpQf;;@R8Vj6N&e0v7JArqw^&$~)_3`q`7mQ+ z?dL3uSf*o-V~uk#_fTo{JJ-XMuVtCXm2jcsq3l!aG?Urc*{Vr9f{|BS7r^0_zP`Tp z-RFw{@dw6$#}TkCnpb(-{i1>n2y6ZKFK%-Id?e5<_4{@EF%9w4 z#(cc`cu3@e%&vfQe!jPB(%(+d-e0G8^hVf)hAa8N_j9BHJ73VJf0Vxz>KC8u=B6Rv z!!0?hMIn3W=m4K2D1rd zya#w3ICJ@hv4TX7>Ayc!BB%BM1A=APdAc6LWG$2#>ga9Jd%)l^?dE7w8!75N$hpIL z@_Ld@b>P>8zIh2?`bnX_7dr8H+#=LkE(Za1xzW-)#*a30Kb%C~k*BS0Faz$<`;TZt zQU)FZZco7;@3ToXj0)hI!Q?=(}s+PmtG6)kK#YNt!) zQF!?;$(y#grkJpUA!eGA->`-yU`#eg(|NcEp|GIq6yykc`iM!eLR~h_6vbQhKgRJ} zpmhxUGB7)K^O2MaFknEA86ZbCk3o&S!uzd884{0#XZ@W$!7g6@^2J40BCfDGCInwz zI!JlNsDclM=g(Mz69wJxYB8otu}sq1)X#t|2Yd-e^T>nXytdf-GFf`LYcv9;p=_^} zVZLNPzj=Nke-sI1ARMo`x(usV1^p(ro3CT8GQ=Zpxj(+LxOU4ZBr5HN+hR<`d%X&U z^_Jt21mLxv=F?@m2Ju)Mr~Ss1z4||IRDDq56t`(GaTXF~BMI}vyJ}B-XTkwt-Q{G4 zlGP*c4yPA<;qmvu6^Z`L0G-_K7t-Z})`ksqNRI%2w#PE9=4cc-WgI_;W3hGmHXLq< zYg-ncuN`ku`(omwXDu@-Reqde+=Y*IBq1*Ve&CiaSob-ejIx3?_Xr8t$H$GS1Q4q0 zyr%oCx!t>4kbCt>qRrp}yZwfF6#aprS7@e_ZnhU&|GH6Hup1^`c<~Arg*rI!Fu8rH zYMDjeFyT#JS>aU=U<$-auH?rbRQ=AEWvvC7qtcYBacGMT-OlBqwOeeE8^`rUNhf~;_ip2)xw1&+`+l#XyF=*$Y&|6tlTdtk;7LCM z42=x?^{t|!VjWk%*e#5CufwUsF15h_qz8Zi1k$iM+)&PeVzLFUX`X{g@ZCpK zl<0*3I)X~0lxhm+gWZBG59hTe_T3!}+mx76c#2DbdF7uP^G=<;Nk0M{d-N%6&)Krz z$p{u@|}%S5S8Brb}h_Gbe-Lj%yiLg&YvFXiIG(H0(%7%+G{hq z;A+$+^_4!#jX@JQDy6tO+j)^@H(T}Xe*8Yz>ng@JnL?()2H!vDk|tqpE$=vm|3!x4 zB|i%jP(t)-TdV%3v(5;)pReU&4(9`(WdpdzW+e+v-PU(L;X5;q|bH2LZ+;vOQ zx$a9X_axe3@B}IRJVEo&QH+pq`NXC)hWY9$p-L)Z+fX8H-rQW$yD_`p_k@;D_GXr| zZ9iz`$Bjq}oRbYMG3mr9j_1a^-@Q`_@;u+m4;!+YEveazN4$f@3~#@sv4IN7um5|Z z{_F+**Bnk2IfRITE1}~eRM>_0S>s9Jt_+7?Zw+OQ_bw4tk6{QtK7&N*MXHhmxjgQV z?ogy(Yu#^cG>cS9#g_06d(!*jEk5FSp&EG}Bt_TQ9ot$?HDJ|RTvgDPTh9T0j$V*Y z$aXpBn#TM@=ho8SZR5YIEimW+YoU%{n+9CS-=_q>)amqz`1$$4n@`cGsORR9e3s$M z*9{F`wut+8&`{8nuWbz0x(1yZbZ)-l?lt0#q;rPT>9z>aKdq#yWC{p2`SOC(&nnmd z1fMx?ofjJSp?vV{yDU8)cA{7M5|djXlU(%B8W=X;n!(fWRVY6QnODn z-qc!8CsfSPqqX?BNEu)NSgkXPL!rNawQw>QSY4&7#1CDZpXVU%&+J%(Eyy9QKR=-PrzWA)5{24Zpktp>TIE#Dc&>;8coSG?i#sDT+aWLe$6T%eyiVR;L7B)9q6fR%u z@i(G&fnHjQ3PHR4*Bb&HcH4P5z*&-IjKuwqLPkAjUXwJ9b9J7B0bpB(0dkfL8wu$s zzZ!P30^d&g1wn@pet=y|JYD(ji+=&%*w|PZYTI)7>yw`;`e;uFa)@{uP$MUJT|U`$ z;{^K@zPlRmP*?8ddQ5UJp2Fc<0@e$!J`t%)Bf#;9>6p@dQdrVrE$DphqSW%#EgpGx zx1Nv(6qICd8oiQ>&i-dy0mCE}W>12@tB(l~v8MQYhD53$+?RSwEwKIBE2~!cPzaC{{%pa(K%2pxqnae;ql>HmIZ zh&jFaR|oFEwjx@4I@>U2xnO=j-FL{VzFohq9M*oR5W_=I;ju6GyJPV9pDY#J;3<0k z?shb&&~sX9s!r9Z7FjBE$qV_H@Cc4FkSLqxyXSN~hoaSKiunZhRw)B7^4Aw-v)$Wp zoO6LCtva6P@b~XwYc=d*gU2@@DI@xTNn1f_eO^m5M(ra zM8i{0%Ssr*9uNHgD0}OuD!=aCTLkG8kwy^d?(UM1?rxB7*mR0WcS(15Hz*C8knV0a z-SOUh<2mndoH3qr&fg+qv(|mjJ=b-8=Cv*@N=nLBYQsl?S*;_@2KkNJ%1TRCuhG{) zFO|z5Nv3y)VAOELb1lC#&sGQsR_4D`P?U2lwpjVXSx)P7&Hs5w0$>iuxr7<`0qF~| z<9*ge4A~Ftx@h2>h7$C`e*jOCAUA?Oihd$sbQGMYIHaKWzlX6A6T@gN)OB}3C**Vr zy!ya9>`3zVD?SZUI?-$*yVbHR9NLE*Yo4pZjg#6GUN2OV(tKHgNW2o>O2qg56{9q& ztltOYsEg440J8biE|ES;<}=s| zB`+l^VyCUD-ai}y5%RiSNof&`JD-~0giG-M-C_}k!v8U1D7Xz(L(k6q$?qYn!=~5n zxv5q8s?K6O&>Q!W;!Pq|5jhRYivUsOTA2aS@lz=gXY<>nhFUNr z{-xHxG@qiGA1kCI5t;x_PZpa&H!HQlX8c9D&vDgL8^TiF?Elgym!1t=AvqZc^oQ(n ztKsV*ZQKSL0Urs{_U_j@sLJ){y_~bI&3q>DcTZ9s(+Oa4RE)&FuvZ6?@LOSl12P6{Yl9-xfqd?vFj)D;@ib!# z6&;zzSiRDPH|#@rP-GpaQ=H~QU&=FXb!6>CE+(wUc+Z7CX4uzkkD>(=}PRMX9-NULkAbh zj#zANZAs3Qi%rZ3gk@R+U*_r!95KRSRf&)YoqI}d@^8zYmt4;6B_%Y$!OLG#>kl(a z3=CLzPnRvoxu=gh??V28MSxHA-mmFAae^#D)9(+Ux#`w4e0&7Xp?RNXqyjR#p)Qsx z$DOtpK?yp~iV6(x8T86Ns)>q;)hzCtln{MX_vEpPOPPh?LV&C>sPnhqZx0#5*DJ*Q z;~d+uL(w*!&vEo9SE*;T1NI8_@cHP{J?Gs7%8N8S1zfUL7Rl*&{E=EEn-l*#5)~<= zPtUXBBga&U(qz4AWn?5;@EC!#kdWUyOiXDqsdqW2N~ZE9+g}Quir$Wvx5rWZQ-(Xm ziTSwu-u@1)LOoNX?ls6fuR-=WF6vVOWjody{A*NwBfBA3d{oh~~&+=*%8_IR^3j5`yzDG)%AvkEtlC17;%PzP3ZEoms7AR}% z|D`(o-+Yua0;p0#NCFX>K8}xVpw6IU=h;3=Pk%*51CHPMz83NIUqJT%`8P!^A$uUvX&(Dk6 z-17gq>cq~kCpzTPA_=enS=$u@h!L{cYa&Cq|4C%Q^-KUd=6c38o#F9q!WLgS7)MYZ z8;Y+_3(0@2dt2yWfHnx(i_s}b%W@a{wV2(eCN~+BYM{)Q$*O`tDS1p^j!Ug1`_G$j zFHOuaSo15;{73lgm#XOdemftm_nUeVrCtg5-B7N~{CBdguhLhz-ya zeg_;j6Y)_c@3fmgH8l;C*fKPf_Pf$tv6peXglcnr5*qr!UwbdiN_Il*{TM>IuUYdd zo@!H;TeAyrCaKuj6Bd`23eBU5g=z2aqqDPfsOag_J(1qq-JBhae#p&m9igD4cJlXM zk0pPk>|aoWC}wD!E*~$}Wc?o>Q~Ka001^zrArY150g> zYjS-AfBg87>g*1MtnY_CzGufBc?fN&aQQbyz`#ZosJls}pmdsf9zSZdtam(ba+A+W z(Tgz?6j|VJ!*E~=(+0s>2>Ny`o6PLYIbu^XR-)g!QM5sBP-u1Qr3T7i(f%l0Mn;5W zDU$s#Yt&oqXA#}%Lc1KwSP6a&jX38lpqkv>T^?Ow>vw-eq&_8ih2S?#zFN8>XF9{e zkPvBBvwoF&;f#UTHksATrD^#_CwL6lIHY#S`Rlg+7ICblzO1ca?BjTi!WMPHXHqRle^@&2|Fh+8>~&PN-`GpO?mWZ zXDdNG;i-)AC@%J_-h(p58f}P#JfM`(w3g3g((hO{TU)pQtM5XvZQj+jUo{_N)}B7) zN)J$*$yy5=D|pzDukt$|%AFo82+|NDbAI>{ZEmF-?>>Tq^~TByDY-o_8r>6KoJZj& z4S|;XKYKM!YuzT)3ay1A+Acvqb0PI$NnjKbs$t#Gd}zz)PregfDn;d#hZZ9t>`pCp zqy8x(b?1+R_1^?7iM50A8^XjKZyP&f<$(uqU}|cL3K~aZYc6j{ZP!QdIX6s}Wt*PZ zmO)Oqh0%3Qu~?z3n#@7w?DLmpjr31ElVY71Aa?IT;2I`tt*_O5u^C}$qVzVq4OBqJA9aO$T%qx}=5%@<&0 z;>}^jdcrLd)eazZYeHE&`V$4014%+OYd57p!=CQ`iqaxEF$qgp+bE$R*NA!iXSVXKY#NsrARo*Vn+gX|MUB zw~R;q{f^ohr_P&xNnv6_n4ff$33|&$C2zuPi}K>U1!}hGvVnFoZpF}lk}aJxHl>9r*TL9U)FL+(iuzSH)dilzfFC)eT<(y@~3m1U;vp&bfbU~UKsdwhCsHz2ja zzkIYX$3ekAh06-Br+8^-_Rs9_6VP1mY1}jTcAjk~SDa>zAAW=GMZk(suC~R>g^guN zE6+dIm6Hu~a(wwPFk4&F0+FL*+Xdww7<^||rOGp%Mmuk_vZvFwm)c}`KOD6;XE$ie$B z<}=HW7Oy-ucM;gM8r`P`0dB#wdou*pT1ycy%*+Qbn*KDHsTJVB3}1prR=^U!Ci9gP zlD(fK zR8gc?am?ztER49Ou}cUVLowk-?)8z&kU-$YZ_)RA_;X)H|8na5v9YuA9cv8g>1n@( zLoWwx6j^D|ca**pYSsZzAn5}{pS|39v-j8cw+9E8dz|(h;0d|i!v+&k>PK{gv+rPL zinQmLl`C9OsC9}1xy{JHxg73KhGs3yb@Be|>mznp^~g2*(Zi6VIl#|b>ZQBbB_*7C zl=~Z3b^%`+&5F2BF9rb}{Lg0Bz07iyfZLcY%RjcSw4=7^w8hOv)6IUn2NvhdQ@!-x z{nIc2S0oLP5(T=EE-&1*a!cUKRdcu&Rim<`#f`pB3X4yOg^wn?rD&dXLMGa*Z~hh` za)kMgex+ucca24jYQ;isT$XcFnSb^B-a0(momvXHdf1q7>O>;Ybh|T>Kb)nf=j*n+ ztKbV_IbN+Q_cn`}48jMXvu$wvu=$FuP%k)z-Cc00OrxEE(^FY2_vG|Su@?;7saELH z2Cy?0F)6BsTpuXpyz*Lfl?B>pu`AqQCLgor$m zhxN++ECOsa8sSe|RpD((*YZ0Kk`73?!^}oI$B$A(tULAGlB!K5lr-BeH*X{FNqk5c z?k^AXKUQWw2Rahx5UY)zs#^m$ zVF1q@Z{QNMiF9}-0ClFQQv2kjT&txSFB6TBJ*L^oN(5^tk(AQ5EFA*Q>79rYdW?_x zitm9Z^ws(BhAJ;EiPf5&@7!1*uU4Wh^MP;Vo66?%WrM&Tb1)GguTBB4s8H1pyH-8mPII=>>=9+k!#6Ed9%CcZW$7L zeR_SAJvXzbN_%^LUVF&FMT^y5GiluEN|2MgFAG*PaW9A3Y7QuzSpCDt%Z84)>RPkw z(c&_{^|!q%fbK-KF@7#5kTu`pEBrLDx%2`dp%@C*2Q7>NppA*B_vxIF)9RRl31f+3 z74Pl+)*_oWWCqhoIXx?V#&%iRniaWCj0@%*g?%*Q#*6n_fJn7hO2)}O!2|(z9pI^| z;~0k2%G59?@Hh&=%axOyDIysAb8WF_?cHp{XO^@QqF<*Q{S?(n3Djg{Xsy>npgxm& zKZjbVR7la?c%}5mhUp8Y0k9~UfJ>!;>Be@;*++8TJ%Wnvk%1{1suU@HwGU_X%VP0) zksCwf8x*miLbVdb=!i7VJaTG5dXl5{_V;1Ay-&%fyp&7QH3nGU^l{wBf_vIr4kpr{ z9+wL&Vlc`1Ix`V%wPg?k*6_X_KjaNE^4^5-@4NbWnQDZ1MevJ=M*Gbi7aYb%ws&=U z^Yy2;y=O8&CJ|(kM1e(=E>|)eSE5aL_M9o_#9vOFmm^E2l^!i`Y^?)P+mNrlffPa37<+!}Q~ zB0ig^>pj*}qGbTtHc?OL(YWFy+7=tD(dX;&=rT*WzuW{3ga%2qRTRtL0^GG=ImqS~%_X#Bp11Mm7fZII)$~o5N3X&jpgG=JBFfEEi!E{26>Lz*9_;_cBx%S1 z_!K?0OJppA_Sj4Y9=olyHTU%=RHibiYnpe0QwO|Bhf-qFarqAMF}%=rnND**l#8tt zZ<@~HOG?gD>-I#wgI-f*+F@8Moign&Iui3;_exl_C=6YB()tqZ@qPB{JAHpdq%c!B znVCF4Q|+Zztk94}H!Qf8{Ttz4)7djl>J88PeR_?pE$N7lzENO+@W$}yewQQH{vZTO zYpjaF(8Xt*H{50lGiO}v@(oT_vN?QH%x8Epl1=KXF$6&9IXS#YxKe+Jmbi<17;&T> ze!dds%Rq^w@PT}SkQyPRw^E+$ov54Om>yD#iEHoJ0mGb-WXB?HJKVfaeSR)wshG%w z@=-l4eJkL^_z{)33EbsZC!6f$TYAet$3Y65gJ1nDf11`1pqd~ zLpnpn*=i}fsF*a)2!81W%kevudu6?VrstJApC&-@C1|u-)t|RSy`45F@Y_U2 zZ2si$P&H*3n4DIYWfERN4B-xV3HBRcHjkGo{~KZpLS8F{hqRPK7!cixocH{~%?%cu z&xfe(mCdB9VFw1&r9+Wt2-O>;?EvFCQ~{Ooo z9xR4*mt&j2&^eA2lPRHY$6JF5szj=5Ycp}t@6#O_4TYx(`CYnUe^1eToqHAN5y5*G zFz<7I_^BD>@?(Cw@AS<9zKHjJ;CBKW0bS%3L&o*(1w@^ad5%uj))IA zP9@+lWPc3&(}9UPKa8~u{*LOug`KXS!;~H^yWQLk56kcsryj#zJO;g=?xip2Wq)Pq z-IGd$_$#MR_B1fha$qwpT4X)VySAzAe2eVup-%9$db(BYStgwhT!@Cx)xC%*Mj0;n9iqrld7nHF`$>73ERqK#%<8D~{xLj|`0J4pof4R#Jg zDxZ+gv7QiI_h`}MvJKJZhIfM;P!6K_#N1xL|HvU`Z;T}>z|_$5{pZ^6BgTX$p^Lm3 zT|D6-)v~PtB_6KgS@raO$JqFVg8s=K5iyHGU6px@H+oONWU^>yX^W5^cvcT9ZR61c zm*8k+2PQ2iK0*m|v~YBsH==q1l+vf5dPd|NsQafdc*;5M1?Z@b5d3%`Kf5&AMAUh_ zxM#gmhnB`=AIM&zBNbmflfR0n(m%u_fW;8MD7bx~_WgC}qK?m$KqVs} z94CW*uj3*pDZd|4DZMbHx`%|&L~$XE+d3v&cffFGsqj;lRU8QwFq?7Lq;R>O?zHMa z8?@AXu!oP8_*n}qs$}qigB^7RMRgSdDG~nM&bTR=hclx_*O`?T4Tv_mCnuk5H3N5h z@eI{JC_z|uuGcSFs*cwHeO?gqbWiq@T+p&}D}1XR7mp{;*jKYNE~A?&LqSQQFUXcJ zY~}VlCF1k=IFl4^X5x=cQen%XlSxA)NJ! ztmlA1prq9=yiT{-kj|q5h7v-B12cj^l<*mYHyO5<$<58hG%~@b9bmAETjHWXb3TI=a(^58ZQ#tSSs~9)@c~o z?@fjaX1o!3EUR>Rn5`jeZ;&AxOMU(nHC$aRNU^n1`tWUTd(190}6r-)0*)vWNSOvYJKV~%d0?p@sy zag{69^ip12g=0SLrVMlkHsoMsQH+gJH%A?eZ|_JJ#%Dw+T>*ul0lo6>v9m{y;vxOY z&IfzB6N@Hps>M3PT;Sr9bRb-B)ArS9f(APQ`RehyR0ivw1lGgxzNI`oa(M>(ksQ?F zS4$aweqP)Jl&d>|2#s)|iT#3I=8+@vuX zYrFUMcX*wQ%eu{Ws*im~U1OSK;FT<%%d@k581}|rj(uE*`&gQFZuQ&(-23mvPOEkc zQy&Dr=2N2ns)3rl(b1zN=l)SHp{pdmnP0|pc1VNfu;wR1;BhE+UNt4XU3jqjaF*^v@r@$=XS&_iE-}(@Q zxrHzkg3o(M->hUA7Y=7JGZ8=fxA8A~otxC9c}SX$rm@j)h_%rF8e11Z2(0r70P(GE zkW@J(J*&}d3vEI39k!oL%3Y~|Vbj=$SHCG+wRYd0a}+lht~<&Br#{Kqz4?QZ|lT-1U}u%M_H2S@7(mHK1BU5mXhIlnC9H z&m0>u-P3#L9%?pHCSDNg`m$xMO*-%aB4mTn%{?5=tNlsz5jehLpIu(6T>FQhsVkuq z4J#;&Yd`Dq&SkNA4MKQ%1C*#G5-Bo(eK7v;dL)JSvq0oc0N(==73doRA-o6LZQ$6L zy6Za~)(9zRX9_!sO{5~%9pcSBEn)>~#p3a~Hirqa4Xv$FyF?-g!oHTJ8q z;C_}&k`D(q_ozLz)96pT=MZ(S(b%+`r2v1zUO#XnebOIawybVwp&Yu?g zX85QlCtyH>iT}Ob&pCjk#{j-Bmj)+c38aYGYPaHZKV^Iyt~Xq*_`z`#cU?1db_^MO zXztDa1&=?o^>m>cZn0M5deT9Bh!@p%#iJl5+krhK;>*Ap@uK61JIdGFhulkjyGq5n z`Aux}&kNHP${iuz2ae?pv+%qL<5K)>$?;9gd z$PMm?JJLkZku2m(=&;!dm56RL_mIB(Qv$T7x zbOESWH>wd8)=g91!0Thp>6jbpSq_0@1m^c0#$1WH*_Pn&PBqY%Rp>;+{+Xd5O_gCe zw79G6ZUXZ8ahM7n9Ghs_488*1^~%RyAO5Uy0iW;@V@X^KP~oTOFVm`I`DnLMRWwDTm!qXAn$3vadH@x2Yv%c{EPW9U)~^GC4$?!*}awv46- zgF>BxIat2l9v3>NwI;@lsj8~p{LM%3p?Bc5@vs#rA;EJTi@D+2^ft&9hNN@k5f7*= zzgp}9d-&BTv1I^gC$BpQ8b0>talRm(n6 z{!mE<#%Yq|`wejrpr;NWrS-E0F^TxB+w@JwOtut(LM%3S_BQM^wq4#^v!{nvEzJbs zy-4CuGG*3F?t3^h_ShcJ>3I*wy2`11KKH6N;(!--=d@6uVM^@qHe1s3;rO7QR~7$+ zS%Sxw04Zg`{=7{yp7b8jvg}@FqglqexSw&Cz;7ofJLF2*d$0IQoFl&KFywIeIYs6m zyQ`lBv<3~c4vy3|4%(AM%^+|BcLnBABnUdQ|0&es7?=_0R4qjU|EiOy^22PDl!u10 zg3Q*YxLE5*Y|ZK~|&D#jRCS#%9V1#`)u8nP*@p?0@!T`xgqF*$Y{^^;C+j9Ns^lOvOjxhpV)e=JiVK2}UfB+LLr_m!XY46g08X!noB9e&*QZDO zOV|Ws@fR5zLCvC%~eWzBZRjuICg!a@3*W zy+*j^<3i@f@g?mR4QmUY>PWMWF)cRoEb+MFiDEgzD93icSEj@j^V*pGHO`8f_L!$Y zz7%CpYjEEZ{!2s;2xZXpE1pl;pe&e$^$HvCc`CYvJ1%{|sU;w=>$zhSMNb+b{1m_A z=xYai=d2xAw9izjhuE*VVth}pb8H3ieNLA=7b$zf7VfmqAyURE9m>REv6B3TE?Jztc4B6d_QEsr)3OIY3(ywerxO z3}Gu{{wpe?Db%+Kg~gj=h^3!mBWYAB9~X=PN_cW5<9ocXEjP7-)M)M`O;A9MaYpdiM z?j5~wAL>zFGfL{@2!6EFMJEyWC{bJ8Y{~2pg}_p?y^h(JbIckU{lm8RM1z*E;1u^9 zuv9+p{ZO$SpEr3PGbsn(PWFLD);AXf9GJQgEg25nkZ)s0=U?N&Jxb_mou$%%Qc`B=b|sE!Np}4uil`B&gvfw&R7L z(-tQt1(rzO7$8BS54egQhj%%t=~NxGfqyF{TOE`&0~SU?7iovhwSlTar?d-+lyUk5 z1({kLtN70+*T~$Hmgt#o^PWL50IZiyb4Qk6ZezUD<@ER=!l9R|O86nK8mrjB8?D*t zNv610{$NR!pWfjLHNhM~YE@*+POQ@lpPp3RxBlmQvCnU*?3K1A~m4flV}t89Q&r{*nV@vc}`#8sW#JMWaT2|r}!Vb?C&l8uk6MuKc?@| z%p0bY{|rq0*B}3PwFF1W?NW1CTF^rfIL!pai`sxS`kw^Hzd!lEGcdkwJ|JA}MA5-v z;xT`X|68E`uf?hM7vzt4{#lMbR&H~wp&-yC&Q2N?o8ciU`YDai)sJ4cS(vJfD-C1O zz+i@wpP!#pY;(_X#%5NS!)AGq6*kLYt@tq2GkkX!XXrq?`K<%^2_OWPYrw4={Z+9} z55K;i+Z6Z!6cur6-&O8ENdrd%DYSCHV7!=aN3>N2OS}2gC?!qJ;J*O{s-6Hhu7l8I zac{HC>9YRr@oDFsaQg-MSB?p3dfSX(cI30e#oN$#nD_@yIA7cDWjplgX&7rvUeBW} z9q0b3*Q}$v>Yw}Ge^Bc^Sx*c4NHw+OY2ON=nXzJo$ew$cw)KfWeN|oOHxblx#U5fc zs@~t&-T41yA}nv6n3-OFASnO-Gb#su)fEx+?HTZqhU1zGu8z$H-#98{CDUPs8KuU} ztCrL+>T+2|E;{yZNEIEcnk=x?*K#wkEE|#flvtVudyOs7YSvS0sFdDu_9Fk2oPguX z)tZLeN?0EFGin1K(11Mzs)rL0+^vBA^3O`ZFdmEfQ<2`|6VZzhw3U%qksf8o<3*C$ z<^_Gu+`#Jr;ks>-OS?o1&tO)fYSaati*UQ$Bq?QVlzYKZ#>H39%nAluN;-sf?mMUM z_J(hM>jP2`6#AzaRmOG&$hlQBNnHD<eF{Jiwa=>SAMxio^^bKeL=nVlcM>#%I^1@du+_c$|e4-RIvL{*;>M)g`n zd0Z(tULEj9sCcRUulNVm;G=o7LjmKhws{qVs1v(jMy2CE z;>k2kH<~~_GhT-A3LL($*TlCmE8bAH?GR?&WTREOS$fOs*&uPOk+!qj9=C6!62WIA z?Uh~y$y_n=8ph2?={V)Sic5{}Aj9#+rq_rs)ohN8wc1HzNNNRkfZ1ZIwQ9t&XzK2ls|Sc)9SXG60$cZWXA2=ES7_JO9Fdh}Jq^Ix(|LUHl)7q|vxS!4aWPcT0szI@h|r=p89K>olN3uM8F z0E4i~>FQ$suU{y9gtYI}@zi!8G4s2hZ)z=$#1WAxMWKQo;_Y#CanV`L=4Mrau!Q-x z>sLWRdP*~wr&sx}Z@ng3GZqsJM(#%9IzwX^j3c#dZ5h~q>0~ps6UtJ^re)z=J>42T zti%cUcyG0}8O`u(Had|RnCdiDo6Meb3*ovYR%l9ix4)q(v?KnK%%5nXTv8ghS+h-_ zJx!My(~yW=W(3Z6DNC6I;GYymllRtuoVa@$tK4Qm#_hJr`Mc6yU9Z9G;C-OReBlg` zS5f3SH{r>f5bnP)<$Q`NIPhccl(Erfxj8m`mV-+#**MZ{ZqS+4wd<{evvVH6+!x)Y ztc32>?9fCm0Wp`wt|oUsgKCYgb)C^{Y;1~K2PU`WJJOdZBlt!N&OZeJo|wX^q1Ycj z5e)y0wKWXoDq#(OxRWw1g?&#ar*KH)?b73RA4a|F`@@Y}?@%lRMV12lXJG7Rp96Yy z+noXM)lr!^oYof?m&_kFah;;u%+XF^LW>_-1ScMVaoH}HG)1z|mGcGZGF>RBr)%B{ zGMeex@}nW}(hdIaw$zN+;NW29kEJV(U#iMf+J3l`0~ea-n9$4=55NL6i&4IqN?X_M zMe_N+m;Y&(+p>&z!3k;twSqRE9IpqCYbI}Bm?J5N%50jzNu_436PCyO;J4lfNn}AK%Cv zGF@?Xg2=B4GcUe{X?4ZMI<4qKo<#O1DU4==p~bL_$u{?x z6SBkt5KDGkf8MhPM-UjzscV#B=VajX8WLNM*$N53optTE`C91=J78YFvQ_TqL_->}5vYlBpr{mkTJ*mfeEvRsg57Mad)DtQ{bfQX^G6p;|MGn8u_W9C! zoqZi%U5Zy1H`u(4ueqQpT`;gdQyz=#9;|Y$C*zw`-g3ObHGH5;Hi6nNt3cV_UZLDX zxroX)n%z6_pwI2PL8 zxPI97$WX*-fc}EdCx|pRiTJ2|;S=c)h{?jCL!5pJ@MlH*%;bU}1@6!mTn;5&D!4m5 zP4!*_L2^S3kg6rCjKLAB1y%f;wu&7|3P}4cGv6;*{MH$i4}~T3NJc!qu9gF5Y=NiU zSviaBDv}nd!yQJBslH55%U+JpL}w^QZ~7+z(!S7s&i^-!a;+ag?=%&MY=W(6uC^Ff zbM6*ZsN)-&gS6YwU{}A;?#QL_n;r0=J=*7%2?+R6r~zWjlZeFbry*P%PA@N{!l#-h zTc0pea)YpJ6F;H_{R!%M_kMQAh1VJ;$FZ5uCUG^p9a?p_CZ( zUU9p|oRa8re(Tm6EO`!x`XQ^BmVpVCZ5rupgz|Y*r!mRuK`kyL@pQhhpbF9m09=Z5KXr_YDIRMrsxbHQU}H3fL*B z&Jpwk_t${nQ9a-uW=UpZ3p^QSyVs5Ad0-EO-p)>DmBP=rDQ5zSES-+B6{j3LkGYKf zeS%Q18p!CGx!j!rQzQX*lR(=UM*^8ydCGqO1+Ut^kq7|%f{UMCW+1MtQ zT%LF>O1UclPOa8@hbWI0spEao6`SpZFCM>oVz2B?Dq_u=D-bMZep6^aNF1zT;%uo# zL5W=e1nUfDT!IZvE=szkq6&@qrLja|%jv?}1@d@R_Ab#UMl!I;9`_ssCU`1FBjs-Vc>kx~ga3h35)e{utZT0Qd> zodKG_r$BPlb-(J=RGRR$%AP!_giIi@rWhH4Q81EFAl~kgrBrko7_8!k^C?}nKW_=W z*6XOj@%+H!Ek*Uz#>=;NTaY2|nJHgW%mIs@Bp5jGZsmzJrMU{D_@e;LaG(V98HaA) zxVr`)!$#dmheF8?%m#0(w7U}|7a7k|gQO|~3l&g4lpqmYhg4yYj8+YRO4}||+T=UUVt2H*w9O^{PqHgIzqZn@YMGM*n%e!BN z(DNFfC?UYuS@JyH=Dtyq?`vp2a`L9C?7JW{*teS+_w01mK$Y)Z-0^*jEp>>UL74k4 z$!)Z^9V)EW8uT`{%DS$>nrTRb_fPs)D&fY~EFw#M_VLMQ4^{CZd zSLU6g)!07%5Ega8f`59iH*iPc<~k+^`f;xy->=6ZrZT=WlGdRt)-v`;M=1fsb$xdi zs4BZJ67uuAwo}4d6>wvqc;b`3u$Q?4h#-0MJ3vNKN&E8mmA-TJ4Fl>< zSiRaF9DE?CP54XLz;{p;fY?l#Nk_tosAZT*wN;omSFfe7ePJqD~n!$u@VKKv}e9V_*8KhA2 z`DCHkN(-~WQ1Gxq3cTv{nz@x{sF%h$TDmd-$ic2^C8q){RT;Rx7-V+X+(d5AH&WBf zl!L81L6^d=Ok@GTK1#aO@{L!!(+X<=KpKpx&B#*O@|yhtHVFk&iE& zt3H5p2t1y;JKv7#&i6W{(jbam{T?DI4J_*>8ss1JfF7ccztoZ;az+pQK3a1W}NNLdC6VU4>(sa@%`q%zPZO@5fjkf zKE-3>?HUd#<*zWjt$z0wPs*oSZ55zy)UzCinwK9uxzEC^XelE#PEX7w9-aSJoomH! zWKZwn>rhag-^lEDU>{4Bx1Q>5^l{9#_I-N+nUKYVL@d`WyKKhxHp$G7+x5~%ucbthlz{;#`Wr} z&n94bNAzVRTF@9$Prk#aYAeV<`q@W*&F=Ve*kW9%-vcC(l#37`=ffCdASa*NpZ+NO zx6XNaJ%a9CCfr^*po7tlu*mCnVMxQV8_rJ#gM?oKktidT&OQ9tJW?wusE$7@m&%uS z)<}9R;2hYHk*-Aty6#1GJgF`#GjQ8XqEB9c@mkvKm@|5KBCnH1Yq0#i^=SaO`t6|P z>OF*=S8J)?oCT%?1kg52@L?5r7^jgc@JxxC1gn&HM;kaQvhmjVAC*4@g;O6$m4 zYuJJPQ1gr_#fCzGXr40cnmPeV6(j$`vDKB=nOTOjB*SFBTm zTcS3xmmSLJby}Epg-VCQ^%h`mYRcsg4N7NhS4Ew2*c^6fhG&)CVW;HTTje-MMAXu4 z^S7NQo9iS1BM~!PDOH|0iJFI6fs&1Y$Ie5VaALk-es8(IEm)sOpP z6i$!Q$hK*f`Oj{byA*I22xB+qejx5Hyki6uG^{EkvylF;ny%~TxfVoDn$U8eSslZ~ zr>#~dH)I_xb%9?X)54i6ViHnKfj z!4LWwJIQAIx8OxCmI7E3o|gWjr>@jNHb7p?*+CrB50=)o;?DB=A*O|As=e!rGXCs+ z6NL9Ce3>srHY2G5y|OP5RcIGs#sHpTE}pbDQMM}cYzn)X%79?Bhv4Kf(W2)Y>-nrO z)^vkDy-??mAi~#4vfukl+l z#=bq=o{3u^R(_L~hk-YqteU#?ux-yyH$cb?NR#V3D0$Lpv6}=&N!2@(#C&~W!cY+W z1+_5>wMtT|$@{+911S$bn7)FcXD?>Y^A22*k$w<;Xe)bX(SF1ewd{TrL^<;MvAY)& zk;u;e`;L|j=H+a#;9d_=TilQY{9=rQV1~K2;g0}aN5Pg3T&>yn#%Nf2UNdq`-anS? zc@P_(J8`>yY8i1iH15)B+kE^f;3I?p$vBvnEOSMNn}5?Ajame1hYP@mI_Hg`vWq~< z7r|u~-GoM^n*CCXJ6zvlZme>ZdDg9@^JgC6QV(=_$ZT6@O{>EHTQAmeIaET6jV@d}VrSIM6DKsr%4}ee)=lIqwKDi}`%K-%+nj~9?k@#HaU{@XoOP;v>6{>`aHU7t za(_W9AXYEFTxXstackq2lITJ82-L}2j(EThKcwAZG?U(GgLYQdx^cQHYF>EO9mx9C z4dwf*UL0C)A0lR9D)plNPY*k~mij0t5la3ZhA^eBlliikCLPGO(D2*Ku;mPF)}ecg z>)AEspttvaEM{Qptz-H8ianGRQ%HdwOqiwiE+5`IC`N(Qu3RK~ysr>f8QlQBUVDPL zFV7>*7kuy1Pd&Qq`2aQE`HXrD_~bfCDK`fW_F^B_uWx|dyrLB-^GZvd>~pt;=e=f~ z4h4b%=q|d~tz321Q>=(O{({)GAC7);t--7NSdeW%ZYfb;jvvN7#NwbA-LDVJ zuj)7*gy3vfGFOIF(7)5YZeQo}$Hs`|>T0rim!4Cza9jsFo$T4=7&fwKSg{whgOeOw zzT;qV}U=M&lWyBh;<%_1UBx9rS5 zaxB`W8A(^Lmn|zFw;5di$^I&0@&$1wg=5wM34h-s9I0iL?SP$X#F{YjbUKzTb23l5 z0H;&Ec2!0|Jy*mLpZ>_nH?M`S_o&J;LL&`L^Vn_!nMR;dAb_W0cOGRptN%nH()OJa zD&9g0LN2(L{UF4;CHsJ3#`e#Y+}*;kWvuvRVV_!C{F^#eRK}IQtL2~tnflId^;eir zO65rJX$G@L#@d;y{G1zLgoyLuYk}C&Wk9enwRlXVJB9u@0$qOlw&TmN9=VRchxQof z1MKSP67t+l7~620^{md*%nDj&9q$LB99&1-QKLUl_~hy`UN@tb)QDz=0rumKLe^=Y zB=x#rjwgN4;|)>_CUC4yM%ds+9}Xsc7)?wTNlu1-zo&Dzesy56Y6&i5O4HINd}T>D zCSB?`7)PHK)9HS&zz;nqwtv-3@+Km*>@4qw7zLM zqQ{%7lZMMLc%+2$Fp?Z`(+?wEefW|(?e{c)p|^1l+kg$#5zy0`FVbQ#4%dvr@mpcu zI?N0^AJ0dH@_GEy^DM(h;`XRZrSLIzqJyka=2{#{zCLt%dczU-zUKd<>@DM>YTLe1 zKpKIeyF?{LknWO@4(SFdiJ`k&QU#>DYv}G)I;3L=$)SfvU@xxge%`(Jr}ut74a~rA zty$}gWBuz8v}0siZ`{VzepE<)C!aBjj_A-l^~&IvDdkQNHblsTDA2E!=cc6;3)3yz z(P9D`Nw4w}9f?B!4d)1D6CUmky z+RFS@`W)(*!l+LCbmVf?Iixafbbr?3|y>PW>Ps&a8S9?1~Y+zvGPQC&2D_;GNbrPCNSD}iyz zY3RJswgnOiUnP2RnaI7jIqqTfi;L2*dV@fP`Z6rL9c0yumDb$w<%nVgjDjm0L<>?! z-WYavImB>T*%ms`<=uB_S`?Bgzk~Y2G`QW9_rCA9-%!lKwC1SEQE4u3UkGHH^$8KD zh+3H1&e{xYuJ#Y(XPSisWeOWDdCS8aO{R4y*NrRXN6sHcoGH5ola1{*7kl?l&8NKd z_QZ<5*=$JBrf-q{BCp>Eb(p|s`|c9Y%pduX(2Pt&Z@&wRsTvt9j3k0P(dW6@8X*cS$M|pJYH4oO}RBeQ$ z4IUkL=H#9jxakPkskKW?AiOa)eijC|73oAHmd8%vcy=&~@W#ACC*~Y283Q>=+sb~E zQkvb_r3(p%7=jKo6vDthSW@p)(=q_(%Wfk2+Q#Cx1-Ula$FC%9lTP<1@42SKFy&vc zmv>*k3(kQ0@k}pXESb$w*Zz#+h&!(0ZC%~(mhN!q_W3M{Yv-5VnSVNGR}K4O-Rkf| z$>qLa$>veo;68uJ)Ue$*KIL`c6_33aYnKyS^mubm&;xx#g3soRWuit4+uVTk;4t?( zU6t3g8zv6U`ps*pwjJAOT%##YXIAA*kwKV`b>Xuf(nbeF+i@E!nr%bvP}=#MkAE^m zkKG?5MQ+xh-?mM1tljZ~`lF~hQwKcsoV#9V15#F+HKC=>Yaa#1h_2G5biGZ%#-{BL z36*5S*GR%~>#Kg@O}hgqt1#n6gTbXbgwxd_m;SVK8c*(yrnzpVU=J;wh(-?Its4d% zT0e#LxGRC=i+S^fP9^|cnWUSXL^5bF$Slip(S?8B;e_^Ytqx#A0})_0<(1O08Sizp ztr4E1QJpvLdrAUd&*a6LP3mR$fJr7U)%1qlxUatbxkng7y6AivN`mWXGluy-m&MIx z)68-1-RM476U4+^xT))SicaE7clxtYmk$@K`nCJs9u*!x#{I%)idJzEX}re~S+%NA zUSIN@DCApHOl=tWY!idw`TWuc9bC{H>`TrKeMGJm+g7E`+;42;Mbk?cDd#lSBlW>! zW@eEd6O`UGAs!@!dpcj{SM^davRRS;&bm}B75K|vJ{%ipg6e|rW2^}FtFDr-hAcbd z4r_(q0{z)5!4pvOrQ4VH(71rgy_C0CR@*{rcK64(R7;)0F{PQ_7d|$RV=e;g{GVU8 z8C`%WorB(b#-0LeFW2aSrEvJ(Th77cSk7q}-yAmwoYL;OEqSTG{D;6Dt0}KJzOPLo z!2xgg$U2zKBmbBpraQD~>7FUmWrubacgn6Km6GugEH5LJApad4DcW?Xhwn?Jq@ey# zt(y0_{H2s>Ws=~XeX=Ze9fe@qB|9>*_=nGXCbWe3VgI~PGoI;h|D6M|%U?c+V$;%2 zw?Q@1Q-NKti(23MJJRD(o(11Mauy4(Rv$Ro4r6oB(};)9Uo4G7jvZq>6ZR^otv)^_ zb0a0EzfAwf8!#=`gtF&({m|Q z4jtz1(a0GPpC$#{;Jo=O<#mp@O!@g3;kDiL@g^rQkg3y{%62$@TXssOH#!EoTU0GU zpbO5T8|{gEk-URcXPD}ysoPa!oR@c7)#H9G3w#$cT|gC9>DP(fOfy_7tb3zeHE z#Ba)av@B!Xha}0Hg4YL)ROuv>(`%Va3@-8=nM9zzwHW*SE{UZ6QQD+EBk+k<(${6ELsHQ5okr@ zg};KIH@V`S`@wBH`nYNy(Y?mw00+bM_Z?DbbN`WY3F?`!ZEZTiSOH$ zqr`Pvh&^IDi2XYcI#N&oEiL*r8dAsafneJHq{Xd=!0b9T5C|C=QJgl17FYwe<+0X6 zg1*&B{^!kg`ZXJ`*WPs@(48p>aWUXse0rpge9JBdeircR1?)R1UOX88>HnWT`|n<8 zztAGs+yz9$r+@!RGHi+Y?`i$#?EbGWUnB6)BYZIBUOcGXaQ!ci`p>y@An|ePvjz}2 zq74^e{BMpcHYL6`TcP)oxa&hv=axLm|LXhV(Pxh#DD^=ZvROX}_4HX#_)KuXYg1oOgPy_9ADh*;o#G`uVn^T%|O_$ZAPIw?=5-Lq4bTra6vW zcnhTU>Zr&lEH?UMI7>#~)AjMEb=dgBBsKV;TPtpWOT+I5C5ONSTm{^?;Mc!YEYouH zT*sdKk$Rp^nfE+ihso;?%#_Vpoz`eLGJ`R8qDBQ?b{tAQm<=jV-Nxv>AmX;0{VO}} z!RLL{rW>D9_g3zOOy32J*XpcpA*A}PI52|#(gecWECw^JH@EFVE|`x_1v_cP=R!T1 zdAws;v?}puw;^jWv&~btW|R(RGnoKQUt8yVC9+t1C03pzCK4Uj$+@;2r~0dC*mtDtGLby zE$BVzUV9@NJIRkqU6@b1A6!#lG-$Msa<+VFH?bK%?fAiD4b;-5kos!)F;bBy@zO_| zlygyQK)}d{Qz4D}Ek6CwOl)_rGz%&sa`UvMjCtXb9s8g;6j8F+db1@&68h?C;5^b- zc^#rknXMi_B(|_=w=FZ!a0AUZ8KYOJg}tYNE*-m9-!^wguS8=`h1!iL-~FE()cn{* z_jH5AJ7e<8<{*~<`ey^S=Hf|cBCfQ= z$Xw2^nZ6Bm!`TbXRbJf8>CsF#P_@#xUtJ=+D$8iS6fp7z#!a6eDMt6zzU!%06Na#B zs&rhMgrM7Dj)C`8Klzu8)~hLqaP>F|EHe5z2thfkzAJ*xX>#V#=a#!dw>?(m{>)zv zM9CCPtWm1+!(z-pme+nc|LfYv zj}=3~KqK5wE&XYob=GR?cRZtsC|mOw(}eJ9jauoURML@+~?0fng{$spfj3fO(~IF zA|@mSi6Ocx!BFWv?h7kE-gG;itIR~!dBSi6{9Yt994{({V*V+zh=ENCS+pfFt47yY z4sTeiOLSuV5s}DRZu5*9P|Q@y$tf!Nkk|6D$=EOSYS%GdN0uCN@-v+&&y;o~wKXD= zn5}qf{f@_OK`E8#`7jf@A{hSi>Qj*n6CVf{uJ1<)@0DREWHA&Y-c{>Rv_A?^+fl5T{GVQ-BDjdnbGJ_lIooa zn@zE(A;Q}33_>|m4Xr6)1ZV2YRW$|feW0=s=?U5z43xx}Wi-0Frv2XYlORr5j!Erh z5nb=#gzjaoDlCWj)0D)>%1PC!%8cA+`>>hMT~uDE^2*K+I2O01)+<0P(9d+y`YY6t zM=v08evN(lC<+MOBFPol;4}@ko3@>3swOC$!f5zvznsk@tkPDTIyZVDK9sdNGFW^-_tX{o|!%iXK|Zk#t<3|o;WMc5kETQw8j*bsBG z5!Y^L?0*H-TbzEhUl!xQg2G8y*j+w9qWgQ}y$TV>_f8KOqc?8bbG*4_8CMxUzAdR< zsxo{tl*rjCNAC~p-Ch28UCXhLKyxg$x&qrhw1mN6qwts)EHfoxN;`_(4+*nbbhTadhI%3qCK1ZEalj_Jta@|O2{f6oB=sHg_@Kvo-j z%ur`BEc|alp|yr!9;eS69#f9zSLQU zCGrm_o#v*;-sMon)z2(nepGOtvztH2sRUgsCkw=%oz9;Ua~YucBI9IdE!3D7#H2Np z`0!-RZ6$9ybyaq<*obTMrg7;NC6+yvS0qpPV5T4m@fR?k&8jIbcrZyw#`}Rd)9jBh z%PuDSl(+doCn#ixQAp-7H6_13NDLVX@s~EFuS~OB5+v({ZVPSJv%5C*xplnt{7Lg! zUDUl>tv{hc^F|zLlg=Lyg%HGDf${@yGB{;;UL71xKWcq`aPVc;XB5SH(DtxGZqgTW z`NLRY*s6+gHC3P}-+uc%ds}f|{hI{JUL&6W6gTm6h+*T0=JS-tJHA(TYVK%gOq#*bWLei!$_x6af`QkEllc#%E9ae1#?ETq~S3o{gVFPOd+9r~MwU$4wsm-ELv$N_BPMpfIU$78uPG@{A zJGXk_^D##R*y5SFyjlyFH9E1bfnZ}Et%XW9O%I!XcKyhEu`W`C-qjhIpIQNfT4jYX zZ7mXpAr5+!YnVdoo4TzBB5oTmITyp#Tn*>CTnmJ8CdNuQckryDqaGj|u7ety z+7c7Ki;qdrVufG$AIrSVf$U%;Z$|zyh_d4y4vlaj@+EWdDOx1Tb)gzAp}L$N^z!CQ z=G~m&#~7j#(1BxattO+^A}5>k)=voX1Zkdj)CZt~3heuWu#L$`D9&Gd9MHW_f3Zj* z(G2dOk=33fUKb`s{sQquyj11(j0J+3d|rNCD=>yWNFtAusl2F%Zxqvda0Kwup@XU) zd+}IXuMT$jS@-`|?DS_o!jxn7=RV#gf<62st9FLgJ~9bMmVP5o44`2%>L^c;HQ=If z7EK5-^nVkZ)(^B$B$Sl!4F9u!B}+ExUKh{4w|h4#cWy0qE@FHc0t8+@pr7cCin5- z6)NNDf<;_gCgOkM5XoJy{j@O>y|SB}u2AHc@mTdydb64I_`plm^47MJVr|D=;{89H zeYP-HKEV!Mtfz~|*);QM3zU;I?N-NYP#VfQ*KYyTUV)5rqS44F|ExO*^XPSTgm?f^ zr7(`u&PdK8YNIQVn^ah*ztd~0gF@FpW)EM+EDSFL8gCew`I!9eHsB+rMt2ibxx`X- z%^9;@K;tz--{uQM2_D1EX{1gHL<|%X2~MEZH~%gQ6b@e*@enEk@G(PtjEi4SrqdpFmxX6YWg5c`(YCJ+dk39qR_L)ZSjK98ks-c4>7 zWk#9JR?Z@ZW^hh&>>98{i(LK|2plg`tHy%$eHAuf4>n-Hcnn!C*_Jc?=K!OVc&-%w0$)8N8Tdz zND`Z{0W(f?MRFRgE)NAGtu|kxB(oR`f2Q#dateP=Q-R!~;^Vkgq17tz#yoICF4I5T zr!-jDALgMA=}}jIX!AtpdQMzX6;i(=SBEAc<^P;5HG@OaYiIz|7+NjwuZ<9V1T0_Z z6DB25SLpKui0TuiZOzW5zGW*=(C7ROk+2JP)*d7OEuBP?A%v3=lyNdXTv8V_Z{dSr zclHeN2^clw=AySn;@TL^!6Zn`NL;Y`_&woeJd=99%Ip>pfT>~4E-e-(eb3L2X&`kA zFntTlZ}!?EGp}7M%pHB5A9GywQRxpe`ZWd8dRv&qoo_Y5Wn*morlDZ?nAqf&hw+RRECGqr_I;drKY9G_tzQc}S7EuuNXJ1`nVbh69B7 zzMiY=g?lO&9i}^b1qUi7(MVD9nsU{!m9}?p&pbB{*Qyl-D-5I*62^RG2CCAxT*pLW ztuc;?ITwiY>h}NGT;7gTD2TLd(Q&oSvQPb^H-dM`@!%R`s|N>ZYYuuyZJxe%=jx{s z@y~#I8H})vf==iS(4*V$*emd+s*;dGqJ^^3|G=+6Ynfytqd_-t)fqXW1Y_0dd|*#1 z<=UoU*D|TVRK@_kcC%fIgi95KI{d*P zuuJugBpNsL%}LW0ZF_`P7DFFJ;%YYP{jP;Y?5wb<4;4$nMCm~21v%Z`rLaDn{ce& z#K}>!W4>nf97p=Y z4kF&|_`S7spOJcstSBZ%Q@K+0VMX$kNS{U9+RCm&(W&y9Zp-NPTkS) zA`zb*?#is4eX#LTJa@}Kr1T;PLG&kaJ6{?MC?XOQt8FNE<+8U#y$zkr>#>Gyv^l$p zn2CR0qq6^S=ty93*_#Lxxf3w;G06d=wNg2RhVukEP6jRQkTY12dFi`xjg7v84!1v5 z3Dn5O>mkiM)0JKr`AV-RY=kL_d_=F5Q(OSP zUh`#3NX^YdETJK&}>tCa6c>z7qFqAx2i>?wh7t3KvA`muqxLJy_+qPBp`ynl2yaqLfSA zkMHk@q})dwjc3<$JF7xqqE^WQX9$t9w|*cvp~frokq%d6!}tbyADM(?}89kU}uD%VtMHQPf)?h8qR9 zaPQ_(Lm6Sp>6XWfB!5g`YURBqr&wT7w*dKj0FKOltoc5oAc8f&tZqDB74=5J$Y=YD z>_X7B2dVSJ>}-Ei!ykGB_Hc2Idf}|ZBg~!g@X0FG&E1)u)@DP7T|Sc0fs_H?Tin9( zHj*&QZ~ASPwO2zb)RJA~J$hAl6uz;(PJJCa0`=>hnRr#arRsU)?a+Ic+ufB%b=ROH zF>6e38_uTPw}fjVn<70r19aUbBY!gQB2G2bJu3oso_n;i2P_zf zX^%?J^$=ex{fg^#Z%@xhevhm--T+R-x{*#KwDU71NF%=~~4Uc9=~%o%Wz?>1q)T_5w72Dh-eIAi*(3mdz1Tf-Ebae;|e_$u(@WcB` zlV+F%8joKei!p733lFTf_h8QoNK>UY8yE7zcsDPE7yk?uK^gj3l3&krNiNZJKS&$9 z-?=r(kb(=l^-oD`f?G+2N~;9@+b$Xnl2RC29J8mJ&!BCdENXhLy70)$>@3qoEk**i z?UB<>yYxN_{ZzZ}-T+?M#EVqMYge?k2D<$7^~FG^5x8KZaUaVo?5zG(r2z@6bzu6nHEp4e>1K4} z5R6U-fl(ZGAW5SqLmk<2eYrvghT9g+*D_YJEN*uP0#nB9!3w)%XG=FD?+(4snWoar z-nKa6r#V{By&4B9SdcJMN1-*1E#-EJgF;lJ@Kts5X+wt;8EB2jqCipKBnHmL(ndS) z4BBR6Y=H@<$^F@i%b1gYM|7C#efr`x%*#K|Qbm^hX6I#`QxaB@&B#)qB_8Ll3(yr@ zvyIH7Ac75;|2-}f@8?7?_%=Kc69j?o%tJjgQdX6C7Q!u50ysAl%8>R>rQjzDZl}dP z2185#{Q?I~5~Ss&5`fpYJ@G@^QNrZglq13r&q0bx3{Maj6z%31?df^P#^snBH&+pI zqqCzNG1qiLd9fryIJ#1-Fe zrRK{xB8x0h*~h$Zz2T)Bk)kT!Q@O#4qL1g#LNi3ikCdiWi`WdTXClq(JSKe-3wweS zi``aG&+ovUQ^7c7;?tsWoR_g7LoHP0;G@Xy#xBr8#hmWH&(GeqZ&=+ro5-ns;Z z4uHgH1Uv#PMw3ffuUmvp2}hH8QW|Vmi-&&3^U`pGZ_qPDTqC*FYbX0K%rF2nG0SIB zeVoXRuHJU<#pk(d&d{0@(w_BB&5CS*k}Q7r{fwpwX#|L5T6}pZTotz>ZK&wtu#1ui z>+R8*zuxT1k^8k)pOhJ!(D&)3@gxDk`diEhNQW%ZN$@YidT7eU((%aw7^@m3Dv8D2 zSob;#T-Av;f7&T4UKE(y(;m|Ka~Ip+KqcKrt1&Bam8u17?a;ye z=l;3YcQ-$g(%sSHKmvd@xWhiF}3c%V?9!41?nnSZjnn4-a1OYJyPmI+CLxqfWqAT09IFoK+Nd*a(exE(rIV z97o$_EygxuZJCF%cFB-Iof~almy6lKSPXjCQFQ_tzMlfv20L1AmtL`Qz`KJnV)1^C z87h;{R+x93EyLV)(bxPdoxaVE6~Vn*TLwZ{*m4hwBX?GZ5>meTq7#8#S-&k9!ffLkV#7019qonv)#=VIUJLmv9~@!N@2_KN%J#+5;=Lw%jM_gi_aQy>F(Y5j z&NyURa&F~iddtpp|-0CP2YM%iC$T-v-YB1v(|EojvG3zV^k4GP|1j5qog{&L zSOJqG005x^S@vWEKeX<&d^FEF9Hro8vTJckh#fy3O?$gRr7r-s2;^Y(mxn(-3E`b$ zrxAbb<$HGgPZmI*U<0^-O0c1%*w_1NVom(nJU9Hq=h4}&e))utrwcdBYpB?1DwC%D z3-}%{w?kEuTS~{DPS^OLLcrdnL$r%=e93id8Ng3>0}LStVFWV+Vq>P!@lw}vPTpurDW@Kr+Kc8 z@f$rLaakFQ3b-GYdtDsBD~p@RreCW^&D!a7=_NsoI;GWL$7mgY+i4~-(zr@mK)Lwy z#b0fBHtwmvBnDdRTRnV7i)tNZwdPLWJ02=C-jydH`oGsubUlKaM#E~J6nvzBg6{|N zk{7zIor$9@Io7b5HZqi@uQAg*F)IQ3)_{GmC<|9AsL>L7p1{AJB9!Uf3?M@P9_)zW ze<~ljZ!bdOKSSn!zd2{+vHGCUA1Mj{(39L>ik1fF; z7Jyas>E5gYB&)C_Rf*cmaID>HzQD`bkI?=>nd>Ocr;Wtrv;xgs_P>>h{O4ks@8j(yqe%vfD6sP4uPZy<;4#%M>=#`1ZTaFxj2~g;wuYbK%L=s;cNpa5XFlcbX zR!kQdwtUm-2`4sr-rTccUbeeYFbvTWiE)k@++>i{pt{ZQi~Z2wg*6-QZ`tH@!5nJP zBnuQqg+Si&eTIfSAPvtIU)!ItiOdWvG>BYZuKkbDAh(;g+CL(b4GB z*vg75}BxiSvGBnsu`c$herSf8VOJhT&L#50%%P=#L`FIQAjiBag!(ext6j>wddH zIk>ub(%VIM>ShFE_6Fh^2mxRQmZ;V>-)!^tP}Cu!!FFj+f&SaSwJk-XZ65gc@B60! zh7M8e!s9z^Ckyr+KrC=M0cpQW-+TFPoesgAdn6IP^Fbssl7vNJp~m8iswCQjfY72# zPLa~|uA_bGm3s^U$;xjopZ-#Amq?V~;%p1n)F`{f*s@pk+@Ps845n`3hpj}7%ohp(Qw>mkb= z1o}t4f2*lZ_3Q~h`h=YALh&AB8O7z#>*07>kUKK@Oqo<7%MUY_Nj-pFrBUx~;Z984 zT?LvByTz0y@Vc|z7R$VC@w-RG-k+1#;Idw{T^#HwRXWsV+J*t4M8%tVZyt%mjFvlp zKRS&i*8;|1B&)kI;oo2Hte0X9hRCxG(zX)>Epc>j<^*&H6xMrC;Ob=-F;<*dW^q~Q z1&6I}>N!X^B|4j4%>D>>Tpr;mGF2}FUQGvY$xj3FhTVf^II|HXeSt{b7wGhvz%6gF zKhc14@o-AgFFTaQz)QrU8$q@i?6cjwtRSF)v0s+=!Hh>HtO>g~bFb>WC$fOP`C!@AoQP9DE ztJ|U$BUATGQ_)V3yI{H5=|q8Q4Ejz3Lu*NRDt#&I3(|G8Gw0e(-!Jt{$G>~wkD7K6 zo%Whi3d@W<-?RqXqIU&yjQT1xXZcE03RFuC8zS?Hch|guJi@aO?gc1?flk)Bh^eOLo3s+4{0f?qQU0GlV%1wg@bY=<8=8Xw>(HRYntmz0VoTS@=w z4_xz`u`zI|1>{uzPB?E1Y0@%jNg)+Q z(S<$ho#oj3iZotLu1#d`NUQ%S44B=w(?ic4dIk2WU95$t3x;ri}=8_)00L+*M zmzrmh)AC5rA(^e}-Ro^_j1H-;%+5o8%Xi0#)k$pHv@{xwdXY|N`gX@u5Q9#oO;0wV zerF~t^;MzfkD_la#QmNG72$BZvVwoLh7V*1*W*57FCg3kBAOEG z{2~_~XkHKX?!rQQzq{$aUFtKb7k&h6rZ65g&%)*cYR04Wdu!^;KBij23~b9x3!p!* z_AmRAjSL_8$Zl00&l7qgEzt$UfMexsWB~%s)CsY+UFj zS$86U4ExH&ay+!;A>?bvRIs`A&l8T3h@S&JGljp49xL~IjXC}iDEx}*$3CK?txziw z!hZ1Onkm-Q!LaUjkqNl_UOrNVZ7O|k+|W1FBeBj#wKRb%imj@NiD%yrp4-2B=2)d6BD-YP~2jot1;{Z(9Q91Ut^G-qF14o4Mm%v zf4w`8tGd=}@(A@RdR*hzn;pq?j6TI$N9X}$B_FWro|Fv62S#lk9};m*@Q_G(y4--y zN52o&Z^w>fAeGGQy%3SN_@@*-FD8$8(SNzuW{U*@z#w{=#(VfwF6y`@< z4}U8$v54<}8nY|LhGT7yH2=LDOxJ0Xr5pMxV)vK)MROi)RXEBw0&Vnbg}Fhiu51%I zzAM&E5eJ75bI6KvayZ=|+hG?>ENAw<3w;a$)c$;uEKmJO?1{djtX)|&onO{~^g`82 z(>b#di)3h=;&wnmMzcx`PV5RaKY6W~mbdsNhL2}GG+7l_YnXx;oVSzA_C}{ejswA? z2*Pt;{H*5>kZ&zGyf!~?ToqR#9wI|_|a50XQu|ZP-|cOhze`nEi7Co zSBVLv9>mos>kB$}#YCHsrE)L=Bn|KOaQ!HFGzHqDs2>Gv3Ae_9hFo^l)W}p}Gj=yM zuTmZ?V~23l@5R44=04D*FL2G3@Y-p1Yco=4{!%u#EZLeJqmt%wG9gqP51KnjJI!tFR%6BNg zq&5)+%v(4)PeMm)gt*F5|9SDJb9q8@WwbUfrRJTIuR{1JI9#oIGz*31na|AFNLWs$gT3+e_G6 zHK&H55Cn%^8neCm>iEbHWC=pD91mUk0d5-INS;jh3w2z9p_b^z<#H#&NL4K*LrC)q zDeoB9Q5?c7$XFMRM}IK5E7kAL*L!z5!JngYmNx{e7nqv(;s`h&9W|Tm_GWj{zAC2S zu*k)CVY8Ix85=}VC(*S9R0x$^3)!KlTJeEx2ohPm#p=P*ClbXVJ@Q5q7*z;a)sN2~ zWmbBfh!t@1LmQ|<-twWHbOP?8MZ*v{l{wa{Rvh-AMmwmYi3y30w)2PPhQ3@Z%|1H1 z?(`0T!nPnLoqoJJxS3{2e~y1q*X2}E|p2PwV+tae$`}Jd zAkMf_9OxzbDLVM~F-|ldyV+XGGvbjO__nGiTMZhnRWp4+O7LUnbE@?N8E0XH7|-~J zG~fOG+Grs(tJDhScz~?WqC+T1p%bz+W(x`ct@^QplRKX|`+ga(-BBpF{&1e+vkHgY zP$cg1TVk2!&%riU=0DFdFe>xD>8hF0PMB)AJ?@@Rb zktm@dHuo{^;CfdoFCdnPJTAd2H{25MrwDKYOI?1A@ZIahhE3hAbmr3?5 zgJPS)X`ahK?ftPiapLrr8S8CK-+gBf`4|4m(ig0sM-b$50Hxw&E4V|!$nP%C3uWDh zoLG02MXdNQ;4V*aASOngP#eyC!{Yk5rbyn)_Ga+ZvmW#k<`BsJ$IIuON6-Jb0DOm& zKYmS}YMGjo#%8B0;C6r6m>__;JziAqPDd&&sFp7a zA?A1R{DOe$4RGIy6kO!j?rf{2yVUqv&E#{A z49-y@CmLSlwEy8SIBA$04fP6Q@7i$&ylP92pM#^1< zA9g|pwL&G)j9O5`?v{${MxU%RNXqLRobk&*)oN0g6oYQGI-*RY){!d+QDy)B!K#Cx zw<2>W!x+DJf#5gDb{prU?~7{j!@dk64+#nor8^NjeQ~}huPpP~$))qz%jjR}WF=yr z5d=u9+oHldVu)KWk7+>Je-pmP2{{~6P32>#YOW^Vtqr?lXO2Q zpq((_iP>mDV%PA}(Xll!2d&kLEVe3a_JxJ;*RT0cM6ffoFO&`wuXpJfvtY(YEZw~* zqIify#`BM3nT>pyoht*Ft47(^;Q_t`vmgtHLtp(vPWaC!hV;D@E!lIFTxt1e+OJ%o zfH1se(|`{ca|8}?hy%3CupjHSxcci+hQ53Ef6#SqHLJ+bGU7v+wx?^H{TXqMTU)-Xupq-11lYx3IWKeYB|y)3VKa7%u@A z9EggzkG#dxD}4aj?A@TaTvIvOQq`aPhHbo%3;Ycd#4pksyMKX* z3%TON7%|47>A1$4c&LX1p;snGYmhv^P<*e1wDm5`fduiW2X9UH8g`5}pCLf2nw)X33sq`Uhvi^0 z{X!&-YiwsMBoP2xw?Tj+Cih2-xYm< ze3Ft*gZPabQkLp+Y3eimO_MQ1f&-&y@)`VDE8 z2s|D}^yz)_i2}7R1&oEKjPdRFL{zL@wJKE3WKTCSU9GkZrA*(rfQIxK`CnCifr5Bc zYr16PEpHvEQI?L2`J%rpH9FE1>nDh>*t-xe zT89E&(+qHwr>?&!6l^V#UN$Q&Q z4R&ijV?}?7jI0S>RO;@gGu(S`ySOXw73OsaP2+D|IK=m|pg+43Ub-htOFg`q&(~o@ zyT=Se;MqZMf!w3vvUKOq)@Z)Hi8G+yFmP<4%uwdIyAcPUxwbWyWzg*DUo@SiZIT`h zJvpm~;!($H!-lOnlg}Ai&J?|#msxs*hm931SnHJynS7#m=zwjY71^oEzbvu+s zGZhE*B@T6imf3BS_a9B~p0zB#bPN)6LBCEd^sRDCE^6U;vwFEacsWpy!{ZNi^OTsj zeeZL1VE52VO&>ju6@s)VbC9tfqm`v$NTyP2lUcKQUGVs@KZPW5JfraRk%8ug zxYEY3;G9C%-c&4TYKc&VJ3hBT*N*BY!b94qJK^2n4;8@uZTPjMZ-C}*DOzhx@6F%- z;p_9XT8A%$0CiA_KHu56yN#7&X=u`R{Pw^vQZe+>$99S}A=q!G$4Cg)(ne)tk`yik z;WNNlrEi*5#se?=AO4lyDjiJJi`&dR*f00XAkPHXgm(V4nabK7;Tc=xmdQ{JbjHA;~XZJRlHs# z@w1CuqlSaY-iXeR+Z+6;o4x#g&(|yYGuK{tE%#Xyf=I>2-C-NH4J%3$XWpmJRKG67 zCLghk3R##P@FXAWUsWp}GfQqD2fmC!U@?tA;QJ7VaPve#%Z{LTRct4%K;gGv90-=0 zd+80K!&v`pxlc4~4(-WN+FNK+VV6{jyyPlLVtcnG&(q8YKI3?SFcL~sw!>Acl^NaNc8_h3I_p2?8{!BOek;%y$Vx)@ z%tkN%YYy(~H3(gnI2;_bRT+{W9&qkG1#`Z60?+?T3g}TB!O|D4uOCT!S>#Tc;jX4& z>bT>*`q2NzZ|oEGWaMO2BJS$23rqN4`fmXNjuEm0s)Lvh*0;a^=Vtu#B=KMThVcv$ z4}l@t@2lprC9;HaMh;+VoV=;Ra)dAeUS6fJ%t-YIPRp^$(KO!81YV|3kt`y>N}!B+ z@{{);PxRV9>{Ffj(iq<~y}`t3e9kS8<{aRHRhtiusve3@z#k;a0%#@ztDJwgwC|Wm z^Ll}qRQz7)_RtQho$**$8%0*TK{^FYhHNu({2@5p8g^;mcU~~{|4{apQBg&0*f1a| zsYpvp*U-|^ARW@3($YPofYQ?4-QBHp2uQ`|#R}$=^z6?cbD)TDuZU zd)Y6SQkxGY_Tg&gOAuNAsSVm8;_aa<*{dls%v0~74Cc_Shkm?Ftyg)n;kM10vKY#Y zv*xvybxji#^E@NaaJ)gsJI^mut%Elo{-N+|Gg_Z*W2$iEgN$a^LY{iFhn2*yQsD&A z$9Eox$WOxGJE6Q`Az0+%A2>1~@2%A4+)@GKJ~-Sp?mMBc!6Ua;FB|Adv?U*8WJjcW9<@#;}^)VDfFPo#QWQkEqT=BnCNP2p-W8d;1*J za8#O=4ixdyQ-?o!g^g$NNiOkLshJZn-@;XBy_Z`ll1cgDv^}I$z^GpEm6NWtxqgom zXDVMdDU&rML(}n>Ttblx5r^v8Uj{AI^i=!hX2lVLadl`|!`Gm|!1&*8x6P}2{g($H z{e8C1B!}a<1v0RN6a|{>m2XJ69HirTzwO{VQ)C0>;e!%)!S|n?*r*|ZwLuxEGIIG0Qyty^R5+lW#-{UF>y3*(ReQ4 z;8r~%{>0#LtaqUjZQ$~7i%I;aiU6s1l|QRF@@zmMv+^vr6QH%GM_bSt@Sa$K4(;-n z)*=#H!}0mCrgSO;(z>d(W}TmY%M@qipE?>FH+qiW(*TZIw^Egg+I_hPER5`DQUoX# zt=6X!{**fk=`@=mRcJTG4kxil0M*s9OP~>{u|vu@wFPairt0EjNP*>rioToTbqi%=Qv4Is*?qhCSl*KR zQK1;pSy_t*8G5Q*0=?8iow@9n_mC=Ncya=K3ttbBtePPp)*=9FuiEjP=VJ;75+0jQ zvcG=I6!yt2r9a@X9>sTB|C3{f-atF~aQgY6ur!PJi^!i6ZG{TTlVjv6ky4Gy50WqO zeoF#DXU|I}Vrch^U*W&ilqBc_{jROpO0_yMtZvQ6ZIlI5g}sH%$9`y!1$Fq{e@$e5 z3P0N!Qb^%2S9TfVh8q0sfZtX+e6jkq1P1}{`qLbPfxreowsq=xFGXfIAcA zak!F*q6qd)5Qq8b2R5Pe;Nc`XO+}rF*=~S&c~y^4XErSJU;k!qGAyoeybx$YtlTsp zP}~mpYV;zX3TVD*?W3o)CWH`t?0t1qM3Cv3PAi`wTcnuv&7;Q;=$4XmoWtH?eaZjp zCw$!Q)=n!Uff-I`+5>wqOONyRpp;QMZ11UTfBLZtBk!5&N+!t+{O=ecMtqQw0hy1+ zhKmXku?juQRJOS(DgyP zb0?FV+PjmncsOiRfxkvBnF?e=xY2sOG%#LFd!In_5;kxV+n>c1f5^+`?9KR_)EE#W z2nAFMLh(cao`QiKUFz8t0*D>~7IGD7uBXzo0;&`=BbdaaV68LuGE+IQRJ(%%XzvIS zPtrR-rqaOhEw-kyUTJ5)Bn8d4G`$c8aEdIzA6>G&25HG(x>Qnr&=ljF3j16H%b3IVh5YwaUzDonFk`-byUv1$_Rj$;KY#=9&BJ%i zFD8#09NU9Cc%`!xGx(xYPx&RD=_XW!Mc`E8cj@I%WPem3p^Rt?kkXl6UkTRNp!RUzm~jimN>0Jl*(6_oQn|@b% zo(bLW3okv18F;|~aO0eTYi>(e79h|I_`rE*W8-*XeVu|F45^;A-;?Du5D7HnAgv|7 z-KeX&$8HnJm!%g-XInV{Tv;*(8YaFS(BgMHe5c=uK=YNtH8X)>5Gm*_2Kvt)M%TGa zUzZF#{QcR|af~yU%q(icBU$!&t?XzcZGDiRFwKdbRuwHUM3Py4_feW2MwCcLBeu4b zCo9+@OyRa#=*bW4fFsjdxrvs=a>V{~#pZI&{wK2h<(A{tLYPiK`+sVDcuN2CEj3U_ z%?|hP`rU2O2ehm9a1W3pW&3i|Q%ov>C}$8tP`p&@c$3=$ZErdTrOU%2cRT_5avwVK z_MV_%Rwkc2>yS>ifhm$CZaDQo)O0sx-vYG^#V1Rfy_bg>RDek}S`s&S&&?l+SnNe( zCUMV$N?1ooz3~WFoyaz`mi$s9w6MB5oLRn8;>*`mKpoQ1Up*3M`L8B?&4Iu>Ps3F% zuYXS5{?2!$t?SGQjuTosazV_ zmh(YL43}`|0W02ibC9#w6negI7=ZY+n-uc8aJnTWyyQ> zuE#l;PWCOzH9%zy=MCa*V^af$T6?7)(QyTQ7(*E)80IzO6`Dp2efNhYiI^H-jUs zd^ZplUu3N=Gn`HP>X(TtiAu?EE0dsBr4b(RxMn&w|PGrq1^1ulFc$2g7DdDxll-w4={ zM0`vj6!MgQ4o9eJ)0wVJTg z>oT=NRMGyV<-yrR#l^B?kEnr~Mg5fsv_SrVw;PXjdE1=g>DAQ|*&f7p?=_zqC`7GD=Q`M~NA}6Wc4*_t*^=^H%*i z?v07+a-$XIjLl-A{dwWR;hwUR@Eqp&Ig4s821_RASH)_Q9cq*-<^ItH^(o)BrpYS> z8=-~on$C0ROq;4(Wxp5k&}TxbxK{H>zo6TpI9z^%o0noV7Vg6|kFM&{tW20LQ0S`+ z$Wq_A+Bg*d949VbX0zkQh-KeZ%NdtV=%C6@_C@$asq0bEq5U>}E}AJugzW+JC9Kvz zP(Zrb7GAAU=}A;QggeL8<8~-2_>A+}Ehm0q;dm=q)p_x#TUC1N3dslC_3YU7{36s7 zTy1t#V=3gKow-gIH$oPbArj8wc=s;2hFo5|&cZ_K)^+$-;%AJ)A`t+P@2-U0s$jXj z$Vi>$**Z`DmuB?5Y_6qW1H5DDcIRFH$iDT2O@@Hr>m$M4BVAG08u$9MEp7t+HoePM zFbj*E8?wm&e>~jz2!^ZBZ~C?dCMVHLU$P7}rhdwA>O&!UJ+Ru03?G<1xoMy zuk$VgpT!`_t?ygAb(yQ@ldrw6F}BqiAA5g={|VYOqExPM`sv8fDxy=PJ8jS&rQHUu z%MEboxN0wA{*OqOB8R%#Zor{RKS~6tYtWS5@7{zT zhmT@4>O2BWq`i;W&h&vDO)pk#7V1?b1m@40eADp9sV3@hyylA?FZRz@(xGuC??86y zRrtQ+cc8 z+<P28V01OhqUL^ad*gfRa{NqkL*)9odwou@(C%WE4| zEf!`)t#};ye&kmI*=%zGllrOFK`#Q~agy}IV$!0!Z%RKF$%iJb?r`OLGwnbU3ttPi z;68;Vw;cT?o~dKIv|{s6`+B2(MAyVG2LqY3T6ksMi!GeA5!s>pJUJIB9McN3gdvy5 z_g0(5Z{{M;{Z#M~cxWK|`VD5R=mjIr0>hKs!kUCSUFh@Q-#gE~eKUOvh@@>Aav0_R z`qknwk{+<>wHhgwpGzdcKU`cLhOq~`;AVcyVq*R5Y07DhY$utw*}WVhH?oVsl|9Mx zSz{NDDf113s>G|mt^0pGzew^KNy&tL{}Almj4J1Qel(WW)JzUFiFTj6Ufzz6=)!XI zTP`Da5&maYrO1Az)nO%ogcACG$1Q`lP}6Xl^N-Dh4d4>#XqYJs`plHv_WLtWkbc$; z7EElyR^nLLJI}Hk!{(*4P=MuI$LziI6>`XMt8Uuze=HMt@P^a%*gRP0>Kui?mNM^} zm*>xkGixlTz2B8Tx$$=?vhPj7fwAwDyuX= z2m_nM zUS6R8UjPdY8{Uw$zj^^i<4n1pL&Nf4Y)xkH843_O*YXhi=h=|=tE{7Q(fXJi6fh-naM#V=Ci?ei%C;(>{$&9G-T;JXXZsi2>|j0l|7nu*#S$7 zGndaF4}Pyhi3nmrAo7o}AWq`nmAXbp(5y1ZtJ@j(PW8Sy!)yJrp1Y%M{~8P7fX6;( zlTMMJB$#{fuT|Y%6*oyb^k@*EgQjPX+73Ve7teEe)$O&KPVG?RT?@n?-d}m=uh&Sg z&7a2AXId5f2L!z*gCHj@RQ`JMMGlG@M*$AoWz{QGA~-Dl`ykb0Zcr{~SBej7NWUnx})z!nN-$4H*n_s;{B zLR>b-RE+;c72S>@(g5a=|9xx3V=hdO zOZ6Ixqtm>uxVIr?(QOEMNieEZp_MG+d*uUc(q9%R#HChW&yG4?^hpbOF@!@!AG}l` z%Pp^|ga7d`F~K;4F_feaT!6~~W-}puYwc~2&Ta{h#G$a*>5M(ysZ(z^z{>A(P2XT@ z?0B5cmpfV)5Fq*;IrBYH=`K0p*=Dlhidh-+=+F_Fm6SGzC6!j5VhVexSYQy))^7Gq zI%7g^Ra%pM5%CNolR1f&6M&W~Ov;Q-?oAaG+=M5pj1*#0H}J6w%V!FQVv+IZc9J3( zk)ZoH137LCthquarHo^F%hDFFn%MD4D25s8`CC~YkfR8~1NN6ju-g=qo z@t?}3n;dXee$06ABa&M6y>ZOnUUET(uvMe96>m4yhh+NDMj%!07K({1jysRlXc5qrp_=i ztsmD_b>uHY5JX-wrtP$qv3i2bbAcWwcihu9~Ks7OKqwmy4 zxb40>_8))sZW3gH)4^nCC@VnpB65vxtRQp5;qv;c{jtU4d6_{-IDN&2xRXTN< z6(O~%!WL(=SD@XdSak{!&M&DvxkBF9#S=&We?4s+kRz4&9gjtVP58XoX$O<38PH6Z z-o6O0|5P?N{R#gaH8T(@rhiHYNRcn5fEIdk6T+gNPFoj0M68S(L^@mlSGN!M zBr_&ksn*Q4hd{U-9P$&Q z2~_ZB$+hy-OVr7t|5Lc0*Z_QjypjU(Yyn_ZlDx(8Hu$N6B680a%zys)a=6COcPTp4 zue46*NxWt#(^5riH4C|n4|g~RkRRty`YGnf`PhmTUA^+IA}ROTTtvUxAM=h0C87U1 z9S{^M0Q1Z4V{gFveqjv0SkuuwGzA(+SdQmeV9ynydbvv&FGF{7T&OPIU|KjRXmgn7 z11j?3$6j{4t4tUdlQ}lIP(Y?v`wLQGFZs(N@*XOMeTWsWG1;08;C$icR&svYmEotk%4eYmDa#z!umlSqBeN76`Rm9a_$HUlh zi*an;H(4!6XyXP^!dZw@Hy+;X&8`0_{MGmU6m$_?<8nAJYay(svADw~0?RKwUt$x^ z#q+E9Wc9oA892+VC@57oLw3+fc&G9d5F#Ahk{XL(0Y5Hz@B8z8>saN>%4(w*&>zmL zh&4Rm;WrFg9+`2ESsmwa5vB9D*RrUjV$cpv5=KS}MlGCIHO2&Z-cYD^=7i;4W{3M^ z(Y#7&AS!B@%&WZ)7{pudEaNG7LEiqDwrz;k*6k_J3gh$42`lR`7#uw7q@=$95thhe zXijBmyV^M|G3RgV`&c5akm+!X`D5JDEe4G^ml&eH8-7q$q)}eP$1i(WKD}gx7^71Ab16R`Y3lf*f3OO z7>p6bYyaToI_iOG+UDhIW3{_`qd+Ng6XNWK3ZCh?`*q z6a*p6rak)nu}u=ECbY}K?P1Y&D`Jm#i*4Hdeib@g4WA+xl)7Lqx$r0Jfs8V?7&*N^OJYDh$}0m@b{I&V?b97=JaT1$~08OPw1Wbf=g>l#BQA2rhRydH8m6V=m@{|_5_9I zYt`h^zS*pM@{;J|WlPXpeH`?-S&ZYcgP35nQ2+66R{iyzSn5_4R}5odB%Xmrz?DY; zxeL`wL*d|eg`8!}Q)QY$vy85kpikl_WB|Ja4I|b!1$%ur-d@SWUN(+jHaemugjkNJ z(mi>cDBg5wu5yP09aQYzU|$FRUXXwH0pT6a2qH*!_eXeZspS|iLSKbp*SxYghc%rw z$YU$TPz0R~(&o_CgW;7E@w7JfgiaxWIuINzVHy#9=(G8&DDqIuW3biLI_1(TjoZp> zRxkAJ*Y%r5k}CFo%ZqhX0bFdfbjrKfYz(%-5qz@&gnoTCCHiUX{T7)MD*M^yoP}-j z*A(!t!z2+v_z>>_g|mFt1ywb1)Vs^@&rPIx#;@g_7DhSEIv$vZyXW`!2FT;Vt?f!r zlwHZENMgd&c7MRFK2EjmWh$)8HAYt4pgzrxXnK#{p;uf&yz^TzV{H z@hif&u{gOTxGs){je9!d$sQqF@2YKs-ABVwHRFo%vzk;oLSjR|Ap*Cn$yYens7X({ zz8SvT{VyfX5#wniI>3!#CPyLa>ez}5!bST@08gKvQ!QmZSzgnAYENz&j48<%`9+iM zK>n;`+A@v%hHUnd;8n=s0>LWR@R6F5z3K3;LzmzHgLan|L7!E-e%yJF#YF|WGwO{6_r1E~s zqmDlVNp+UwWX)M;p;s9l_IZ7uxQAil?zfayuFfmWT}lp)4yhGdNksNpoGDh>cnybF zbpRNw?%nD8wcO2;)QWPH4E%pt0A(6EBFNE)mI;`>k&CdGZ0Nx(4MWj% z*S+2(Foc&LgYVODba3ygu|Q&?oWO;T`M`|h-(;tGiI^$uTv?SqR1yh(V5YrSRCkyj z(``z1)FdHQ@;SYXR@D>oA=#bcj7fvkP~6%{eCZ{85$D0$c_N5=JTLI*eOVFxw6c6bk$ie6BHGr zTPrj0KX=Es=ext5SA3F>>U*3*7|+Jcy6Vp7y^}aZnQYK2nQ7QXuyZR+K8@Ym-nf+a zKxYs&f}E2hm%BgWSX$~WRy<#|J4Eaw)jemgus9ltbuXXfOxXY}hL@V!bbUHiplfmG z?C&HHmiRZL5lON?q;<0^F4rf<&c1CEcceS>9JZNS_lV^6({^Plw4Y5YUau_KAAGfg zncA5zzvyyV>xRFaXH6{e>GD~YMgSi1g;&WOHl6Xz+Kf~WviVy?`$tCvbvEcxHw{cJ6M6HUMP0;eEl+6Bs&JR!;DE&Q0k=@%Sb0iinA#tU zxn$v1j=#F0gzA&pSDzB;C}x42d%f7RmtEkN(%gSsU zLMX@|A6DOuGFIgsOt-S3y~i01+he4NnYdfw4%_f3rxg(f-z{HJBQ~Y!Z3efIFKI=8 z{m-~&_%|7jmUMt0%5|kSr7jg-K^|wuOsm!tB7wuN_lUKGxakVBqPIGS3rFHn;euc`6ac2|Lcd|zeF4*?v`F}ZGZQ_o>0sOVL;*-%Ax%ZO@bvL zp=@?=T)*wS13Yap9(zf`W6?66>Uyx%-4mpJGXatz+cj z>+zf80ZQ6%!?ap~IkFc5Nl7c@ELp3MAgyI@0+ve?AsGr0G~31YPqPloKpT3Hd6jOL zN@1D&dEtup)o|gMZhPB}KAOi?z~1VoMA6K3apK_Ts6|ax#ZXV23+|H&lLnjQS zqS$L+FM;g_9uKEFAMViq3W3`A&D?EAo6)v2Et5hAWGH4)v&nTjom>k$*+mw^9?^Zn z40}BS8JOTjJpF4D6D;Q&hY}PF_<{v(W-F5-(|DE+x2H+xZibMd*En?RslZso$4wIs zH?fNLD}&9=>DW}r2ekySYBmK%=ma|YdVh`lg`LJJOC<87WboaK>URh_zc2WJy_xc8 zcX*A^LCYqEF!{lsf8cX7`lIW~aD3n&KCgk$;+(AIIwwTmc3#he=*@-5oObh@q4zLi zuql(s!ck_B-TvDSiR%o++~^OnQe!$I9XW7s!Y1cjW{a~?FFMz6!wju|PGM~ACD{FN zX7)=W_RaFSGGD0ZuMQ9g#|EKb%f|I%Itfms8T)7Iwx~)n=$GchlnbHap-;5tACrCn zi3#S5emAa7atpxz%>10z!moJCoberUS9kNtsiooG$OtRZ-GBtp;E{xg+oIin;ubT_ zC8kSOr^(h{NB9;dVm{J%q}pgpteG*A$9A51^D&*dDBVXa+m@cH-StNN+-LQPPP^VLg;~2*EXr<$D#mZO zDz7g7a$lExs8B%{@kk*5doc3lKH^yyur~`hjLmw%EBZBo!e6^`dxuPV9qX0-!P=lp%zov!-(6E_Nz#zx{S z`@y&Ch^>ep_Kpgv4(Ig=Wwk~^6p&bzKqdKIrg$Poi9JAneV7wt*}!Y_#dK5l#GPrc z+0lU_Ry@HzGA(0;L5HICOnK%J1CaI+OH{W#gf`GH91rbN>Uwh#E!3J}d1ZRSr*)U7 z!IEk*OyFR6{}(>u^0|gG^$6}Is72|bd^cFg$)BDoiIpJ7$`t*yBblf5s9~-KEJ6jq zwqZLR`}QJ1L#JPt+$U9%$EU=1{m)+Ah5KnzEWu2v#t;?kJHEmXx-3s^oR_FhRi1TH zfd`c9u&sn|9`0U`3GQZ30YOs^-AYq8qgb?TkIV+GdJvDQXnUIC`v=kJ23}-&sIVS7 z8pkA4I0YK;j!);h5MPQu!PbWWQRe3YqDy z54(d4Yy9MYY%7r5yd7&T%U+!xPLfN^x&+UZe$xNpx;!p{aHBvmb;i8RPV5?W;~RRe z@i_U1Z{wk_j7_gF&ZonPO&}xB@PXZtYo3ttHUEH7?UFZLvGc|2b@$hUBmpubbyTfz zUw)K&ooO&-Djbzb}k*iMC*rdMgl$_ra_8UW*doG2b8wb_$bWi!y7azf=Zd7jtD*F45O;zI8*Wsi z?S2FOZ?mb6#jwC>mDO1^$}t!0czK+(U}V8^xYPH}JeG0hDV1Nmvb8;weFJiSq84xt z!VxL_xz<^O)J;|oBd7iM*(EE=16GoN=*pf};DgTrZeDI(&^M0CN zE#m|U@)P9}W>DZHnIhy4PUO-4`tz^_Jwh4}V`fdgT#Nl&;AsBw4d3atX?_^vG43ZL z403Lbk8jvmr!7Vk>*~SHysKsG**m{hMx8k{7S^}$?-=(a(EgD2Q^Q(%>jHKdxw#d* z45r2itL%Q|EQi+Ss}KF~B4_*YgXZHP&L@oC7g#Sz1l5$I`m{j$31W0je2R;E&KR4# z?Y->OQ^f4HDfjOJw5a`euyG@fTN{$B7H#;okqq*w2OcRyU7T=fi`1D74Zwt4>Hv>n zpE{~R)+sz%2os}zYS@)RFWhjYUJRAaqZOGwgx_h}|fKrQkl`93IRIK-a!qMeyPuyT;7b_E8nB)*u1S{T?f!6pDr0B zd~$Bx_dVoCmSDGs*-{kg*}ZRc>f#H zwWDVH<*c~cLFBH}bqi8*leefh9us{}9LS%$*yAgbngCEYE7crhCf`w<6TV3|Xaph?ON8lfEQhq^ z_7kmv_ZYPshjofcp6K{Q(1Q_|hzNJKL5zHI6wGOM%O5FkeP+y8Xk@phFSMOU=bhGb z3-|*L{8K&$9Bc?3_={fE0tFLN zd5CnMe`gnfpx=4DcH(o90O8;b1-g< zreT_4am_D5Jr&LWUv`5uZ&T zKbaK0D$P&i%KsD@{1n&O?~N8jIBMUz*UDs85Mr5r4|Bo&QL@57N7Tn|@v^TQIqxkp z2zBT076@0% zKhrw=jx~M6Y485viM7f0y4|DaE#z0ig-`zXec6{^K@n-7N2>HW>j$Af({}W+$d4D{ zDm*Uy)aOQ!D%M{Y?jbxzzcHfOf+y;q=gVV}cR>r8Z9pTXE$Yi* zsb1SYyQP^nC6p7JIlCB=$|DByC#a3H?P2q`ZQQ!wbV@0-1nG0K#N6dY6EiJRjyt}J z!gpuho5t|SUB#LWY1xQS*BNuwhL2k=G+#XnX9veRDu2L)AN6=X&o|4e6_U_Au(V=l zMuV~`dXgh$;pE@p93UF`IClt%5ty z-?tb~p>toElG`3PEY8$O+lqqB5qWB~SzwAvAj;===VQv#_ZVF>9r-!@vGl!Mbd{Ri zbNU*4e!ZzvVsg$!lrzEdI#Z2acq!ARolDK!`o?LGtl-%5wEBn+n*tr6XQ`fi_MiT` za>t9aQfM#yr3*D$tI(^-U3P9;%4d|-}t)a=2&{njGEp6JTqj0YTBq3IT@CH z>e>}BrE5+qrrxU+F(l=IB9U4d;zB>Yvh(v!??D)x%h;th>b<&_L%qUi<{A34M-y^C zNVrK-7U8p#_mnb(>)Uy);kEbE|J|pJJYg9i^mBH{CH5K91s=PTL5F+jinhI3v@8F3ovPd<%tw&)|+AB_TJKhCXTo5=trTSXck9?3cH&vqAy##L!F+ z9?-bsfGfBbzV?@ZaQkDa+!z!ID%Rk-p<2!UrgBe{po?8yHo*~eahI?~$9Cj^aKmnJ zA?J7St~HVW6>BoD!0ok#w4i&3ozuvjareY6R+C}no@WhF5ZPR70XvnEkL!m#C;4>2 z88g49IJUopk%_LYouTUwb~}~&{W#G<$v(j~nRl7nFujh75YN8>qVC_Nma44)VVwoS z$q}!E$S|wyu<(Jl5QgUl^?d5TeGN8IwM3@A8@I?~7oKc;r zHB#?zj1f+zWMqd>N$kQyf500M5zZTB2lc!Z45-0+(~ik5hyN0l(2adiPSuL}7_eznwGJ`?wGAH>xFe}dOWYHRKr!~TeyHIS>@i$(j&?b># z$NigN(KZI@3F+D`Ba$H9;N)QU_iy%OPTIfH^_p>uwGLIW^6GiFplx24ddYp#gtB<7 zwZP`9%FAZFZ{)d!O?d==q>)2BlHY&n^a8rCJJ)y>Q0MOw5IapD*x4U6;^owS+Ovx@ zO#6aSceVZsJ6e~X3B(5Qt9X|OM6i2?j9siEWmvfk=i@WMc@fpOKumx zypP9d``^MEo6$IjbKnNdUea%#%CCEQ6k3-OaJ?a2^N(3B_(R>MCVfe-Oe3UP8X zodTc{Xozu^Ct$y&vwjcps3AbE>gO+xLy%2h~_jZfx}@TW%5E;s&q${8^yX zbv;gazSe^6Ko@*?P#kB0bK0vAW4sj?HxtJ(M40J_v9;bJ?HN$Ake$RuWkAQ|72@1gSJ!zy4F@Xxhc-Ipo z!Dmym0}sK*{R)uAB~wrobuQ{YoGI7-Vm6qxLN_F$JD=HuL-crNcJ*NPh-g3=3&Pet zAxsy>wc%SCWi`_)FWCFDP_EUzCW$u-;w~fqBhZ~4B2Zp*(vB2<$tRc8v}47E8;m9S zQIR#Sq|$=!Ddq#l>+ZYU)oToeUu|T)?A}|r7zz-WBy7d`Z?dEHn>35w`{Tt1=_F%q z)?eJUm{f!FG#+~KG45r5#0ha*o2r~A!b6`+QMa!1edn`_P3+-5*b3w7 z!~6N{FXnMf-iJK5O-uIe{StBA!7A_E^+KTr3v0Z>62i=7{=Y|fUcbEZ98jJQHLG%H zI$aS6>2<~FT%dK(Ht-}Gb};E)GyF@HfkDFTIq(^5*V}SZEhI@dYn)3?J7{7%uN+wl zVaJBFQ0(Ab_9eK|zc|C_{{j;lgSG^+Bbcs=6K4`nBr7DFwONhe*NyhIx{{3jomk zi=_=GD&XtY`P{sOe}QShvBICiak0n~94VBPbt;){e%?>(gQOGr#q>m7RBTW(|7>d* ztylPAvqxa7#ZQPT>2kNUPOD4v#Y~FRL8N9GY?>@?;_-9@yN|WkgQ7?I>w*@K!c+k3 z8*D;B(UbsC5NgLzykd4@2#J1$0iO(EeIU&&Ai&0-U=;+jfB8z8XOM9L&mTc*^M?}? zjgszFtxIbNfk4wa(LU1LFd@C;YJYJkV?-EFPuhEK8U@xrpduL%v)4#Z1H=Q0`=F4{ zqL9YLb<;WJ8luAH@wM;8+n4kJ5*gTY=1h!m%ZSNfY3Z?a52*!dua!Yj2!DZkgW1jq zbwMM()gc6ayAa(M-y@v4FbLz>K=ZvG2T96?cT1NxaY87Rf=9%?<}R~McS{YPCumoT zzbrnU#E31{m?mkbO$zP8(zAsp_BoNGKv7GTuTLm#r6Ug3oc1*JQwzvrr+pRGbOB;_Kr>9r~tx`Hg?iqAesK znCeZOQ1TSo_0S7>TJb^EcZ8e9C&06j^)|4GF(Mg+pIH(&f;0Z-ILHxUh>7F`E5bC{ zC8QJIKv2e@B~;cPbS>UV`SyCld_+O`{%7|G;?jRZRQa zLVdEz#%>keVRpR9zadraH#we4HFTb+_O2MUSv`LvM>Kvo7t(S^2 zD!Q~HvH%)vbfo1$yEurXg@uxtFQZZDxrKYn2ffQ3DG=>wB`(=^qMk5#f_9DNOG$`> zCNeEwqmB1T7zX!+jtUfh;>11bXJN~1diV?f7d;|WrhtoCxU;R6A^ELNktsK`N&u+; z1pMpa%X3-Lk%Tm28I zcRd2{ASf{)m3hVMTB5j-M9jk=ShKHty{Tkcbk;kZvS+sDSm1z8?d8XvU0&gfIBx?X zV~Y$^jSMR{Q+fyOmPtM@^QKkO5Zm1G$!H8%?1u{+e>-(<*j}bXONqvgbE{0(oL(f~%BwEwf>#y4%q*!)E^@tx#2~ z;#FCwIG=u*{YLO9hqIR#DoHv<-m$r+9?+E39wfEaMXRLF)+z4P`;BfSDGR0|DUY`2 zI?Zx6rfE1xlcdkUq{+-V_f>4CAJQ~(P<5a6r~H#HYjbs|?WA%j=mR$Kbx^r}T%|uJ z-ruZdN`LN61C6b>f)cM@4}~by!=;wIeshp!GRxF1h!a&MS=ZlMXiT2B+&HY^5$98! zsr2NzK`E4d$C;pz@caN=3dU>VDu5M zpKz((QCwfCD!l#r*4yGijxadq{!te?5El)x0rkd_z^e3@a4{X|=e+fuQ&eg<4SEkQ zvaCTwrNb2TOKR#RppSjhS>^q&{_LzybeY^{x)o(6?~&cjAQ4E&?5KabSaOXv@Bqkr zp{qosAg10DV>|0|b8qcvba`L9$APEw+%u`i zi*V?4MFgJ@>;MiPa#w$v@p=F?lu)_k(-6Tbp@T{J46IwI332$M3>^zxmZP3iA8L#g zqUF(NGPYY7Nix_*EL8CQ>JDVfV>6}YCvH=B7=*$B#!ZIXNvMjE1Zsiem^yeReP(gE zo8!4Fqa2YEY6Yy!JHV@;Ey7D35(U->VDalOOeH$RZvb zolpJf0Xy#Fg3NLLu-_N%+JZ~F2i0zd?KA+Mjx&M z1+_FS-B?-@&D}(*-S>w&bj%IN7mA{qy^Bb>?^oW=egx+HzhOXWG+zdFe;B)+0M<*3 z^IhTYf7|UG0do{&yF_UkK!n1Nr<9I8FM}%$7OEa8j!DI}NpqXZ zmv`TG)Gk4>)OL2Izuw7Ew4$zMfY#HGg#es3lf-Q-`@ZF-u`Ob5CV5qAurz@w|A?W; zoC9vv`Kiyk{6RfZUpG?oI6)Kk0|o()=^d1hqkfB{&6AG&O%y!jv^mx^f&%T#8O{sZ z_8{TDm@UyYM8@2;9JgD!eNP0t`^C<|-L-v=yzr_5GIzoYHUZD3C`3XEX~cR0v5rKIjgE<^=`~ zYFCQdS>k3Axs=Skvmdp6XZr9LXpNz^a;9lw{R3(9&B{S=9cIs)!90} z_u#&Ai2uv(^&kO)n#I_hdz31z*SG(}*jGnI*>&wxA}Js!-AFeg-JKGmAl=>F4hksU z-Hj3w0z-FqO4k6AL+1d$8-1Sl`~LW??{%$NEY_Mi_kGSjwd2~?_I>N@O)1C(jyL~B zPFeeL3VwtzM!>-(lZ`}z{0lvE4Fy#O*lELInMo44X)*G7UZx(WH5jVU!o!y%fq*J0*;bmWrXnAy}8skx~%;iP?F}MyyvQ0 z1YsB#v7!ekl4=>*$X!#zS!AtX1*Mc%i9lWUJ|G1l%<&oWii6tO4mYH{4|kBE!EOJg zUYiE@!ig0NuH{kAqw&qon(hYo&9LI`%g~7J$mJf{nZ%@g3ev4)9lOLp3;5l}mm^{E1>v-43I*tz7d^rjqL}=T(_O zr#s;JyWD+P|9e;oHlA&Dj|JGCUwB{5i_y-7{7{F*T#$is7aw=I(Rb^|S&tkw1T8+6 z2(ebIxY;*XHmw6Z56sq=)f?ce{bmP|)xO8jMkcmhvV&sEtqUS2ReNig6y@*Le!r%R zOb@VN8+M$8fjF8{~?wJV)TEy`R)baNJSpE4N@m#EzHlPn)*=Z+F(ndTN+* z8@u&M+%LeMjzE440OsEN_qns`HuEvK=LbCRkh(!@y3Kyiz7YfmtN8+u%PhfL
  • yJy^-cKz89y${!1sfWvd6y1EG@l8wt8U zYnqr<{S#XXErMpDJW?smS7TQ^f(sXa|Sex_h2xUKrDCey%U(wREg znjNQJZh?!%%&Rzcr}!Sz2!u+%k&-8;4KutWOgAf~_XjeW#9QIs&^lBtguW#&uvZ!Q>4ix2My~LP53{WPpD!-qFs_UhDE8p4iV52&U$?`_fs{ zaOyAK&)2&R;t^@avD`NuYrondtm!aI1Q6D8`Tg-AJl0qxyt*xKZ+d_-$-%8298X-PX40MZ8&ZdoP zBmk{3{?6X1A&LIU_MVNk+QD&Cwl6?1jC5l#ut z+0Nv1m@(C#X0|}Yqx<~UbQetoMD<0A=CV4*W-c*JXZVxIC$VR{dsN)k)8Z0B_7e_6 z@k8#PT$Lx$TN`HB+Pn$$I%PE`PiphZB7+WxUj2YN1f{tgGgLaX2@(pJ#ZB6ej*`Xf z&(|hfKXN0PB{-RvwH(Wdze4`F(T%x_xRqXH>bmP!ae?P|f&H_1cfB2DkEp-RshH>| z4noRrfZYQS{Qv$?c%ef25eas;hf}%cOIkt;N_DH=NAMjWT^-$&=X-`GjNc>(D0x(I zA^<0Wy+Rx5?D!Ms|T)qwItG0(O z3DLee1?#f1-&6=Y)P}N-MU9zTd88m7PprTvAD@JZz*{2)(XmgXXC2LC;$Hj}j!pC> z&Id$$H9G{V*O$&Z1#W}sTndL+mc3bbX8X(@;q{ig2`tA;^W~4+RxejTBG;Qzw4Xk7 z6SaJfj%1&(J0@}xxm($9UDe_$M8Bi7ovqXozCX>)nn6kH3};3%K+>yrsmLqH>(A~; zz5>FNq<;yNlSzlOX^|sjf=HJM>#f_nz1WJIv4#zml^FCm)k}XrS@XRIAwFIsg`a=@ zR{+1jAknYi;tR|5+RqW+*f? zw4d)regi`PYS=12QSR3Y@I#r1f7=Gn4Z#f400$Yw zy2lf9_Gh#HALzq>z5-Rh#Dq_m#xl5j`nOWZ8pV}W*;KKu<-V~4jvPq=kOg#|dX`7ZTl?f#o3aqYE7~Mlm9Z8WNKU$F z^~F*2L26D@6Zp1CGDe-5J%lE5VF@<_Kp(3dBIPvQhnOtm0nZ&+r2F43k3&UM=D<&f zx0kqVyZ69Xu9yi52%LU{y`R-&6%=CsMN03ty73Yt21mrK=Jv?MxA7ESON?f?zxCc8 z`{Ast#YM^%W;-`D{D9?q()7Uv)zp`=X2SRJPhQ zqj*(2PY>|10Dwd12f#R7J0!)mFok>G}t3ZR171n6&@8iYI&+v zZ(rO0tYgRkSMFaaW?f*;$iUF36U;oe^=4`LN{(l2f700d!{QZ_Ib^bY*c4IEbQ^8* z2U8kUh&+N#_1~^%Cp3gdNZ~%Kf8o(dXxY8=p zXg>W+hk#vg@ALEB+Y&wPqSWx~TvChI-|?7iF{hhV&STY)^HnrWAe+yD4pifQju#e= zYi4*mmy)=Q$7ec<9w;Sk_s@dT++LU1Pc_O17L9ITJ3|JL5JO#C;zvq7R!Az!LkPhGICPN#~vI&){maMi2QbWZ|q?45ZbVUsONdKQ7R;`Bodg>Jv#* z2w0T;+lA>g=9!KulVw`E60Je%vALN%1fjY5T+DTw4{T*-PdvKGBdj ziW14S7IU)o8561zA_akQj9Tr`7}W+gg3zK;b|z^Any5fg=JTW3Go^~3OUyjnouqBN z8$qf@Mgf!G)D?z>zBN@_dk4HJQDNXSokZFcg|+ob!SD#C4HKPBh=WNAw7Vy?i3B7BF5Y;3cw}XR^|f zF28>|)z9O9#7X$_lm~nG%jzb}k-x-PagId7EW1X7dL-72TCbkx+;O?PWZW4>leh0w^C z5=}u`K_6+cB5dcIC9K|-Gd{*5;{*%|Vdk6jef+e@1lxDWkFiLPh1^dPY|V6s7_BUX zO14({W#gGNq5#-6YQOyG4JeW7I~w;!JT694qAN6~dl?roanG;0w=qxwvd(sGsjH4~ zlkL)Kk3h12i`mU^TU*KPz)BF$_U=_HZVNH&Z3xY?Xu#&_qh&d9kPq2gY$F^Z zFIuo{k|mmn$%dZEydVW+UgaYlS4GOiQihHO+fLq{jb?WgWX@JNzsnLSf1e+-Hwo?g zwH|TW))o3JRxMv12|_$AQ}*@;pv+ZIoz7oG-WOKOmjk`a zLn%W^)h8V*s|EbOVwv8R7}U61`&->uRbx5be6Jz3Jejub&uzq0pS8HKovl&Y@R@dQ zy5(8I9p!0&q7)ZfNj}@?{+WKFE-TTqxK1BK*uJpIga|SwI+w z8XpGBb-&e%zM8+lCplezBW4;GKhe7mAib*$ERbU*uF1Z?TgI%}g`zGuEvT!hsVdg= z4N%Pw3d#b5*Z4hniP+Oc@ca0F+Up6j{jIT)yk7z@%R)aL&BE^o$GkB!gc)son5SF6 zB{$uhyl`yv4o-nx7c5lsOdip9%SpLlgjCGbP>+>4l)0CDxCWLVZq)X?tBpoYNd%WT zFFxnc0N!n*K*CWo7Nm1vfPp9M0iWl7-YmZ}QyRbR)bZvVi`hW=q5!nOooU z`2DCJ**0FV6VX+p(@4bF;nF8}q+XHUmtpFQ6=}ZsqJ(@e*do)8Pk4ddbc)@pbpOnX zJo0V*I_eI|gy}1j%qhN{`@lPPJ(mbq>lt<^D}lX4Z3u+tJiu-m5Y859)q5bdh58qJ zwCEmAYlCHU(9nCKCPU*xS4rdYSsPU`++V&OXE-g8&-sM%ZA|ktTbg&|u?0jDW)Xyu zCLfNkvm5MReq5kz2}_VuHm}1Sk*kiyH?wj+MH^eY1tjdLJjNbfpsji zy>IWqF{+nK4jNeN+}_cgOnQw;Pnv5(FLVgfpbfWPmc@E^Nc1p+osXBW@I$*PkSW|_Pd9yiWsdhLOi1Tt^4gggRXTw`yM&z?3!^h1`FT) zd^-f0-u0G$Mye=L!=hW!-J6GeX4U7zjh6Fz*LOdk)A<0P`^}zLB5yWZl(w8|tk5ap zG|uVo2aKow+qvM5pL4gcD#eyhDk|AD_RDd^o^AnNm3q#eJjsHt2^g22d5otQeSD0m zk(RS>;@28XHq#yzFsdAi9F3B8tN0ER?kf_HM&93yHH7UIYvg{7wmiXSjrCuc*SK3b2(;^LJGRZL>0#p3j5I@Qc1@e^$|!Ip z>>cyX!#}=bTM(&KcO&K{`+e zMFg+nu~O@-+ocbh^t2str@kyP@7_Jn5Th0Hf)1%QS();+ILjEr=@pMvQ*1RFGdehJ zyzD>gZ%P-q=)KxXW5svlQ<$san`RFL0e(HrCs2ea=WLpmWhMG84lLBD53+z~&M$Hd zNcu`0#=(`bZ#nA!ASiLKrF52fl;iAkesq&JRKem3n|$v&#cdMN@2l8-7SUAXYyy=< z2#f^$dW4e+r*N!3Er`>+_(*`d|8!cbFvBJ7sN3^?!hXn;U~sBTYiRJG%SK?3tQNI@ zu~y_xZMtO9pk295{nwU$PrqmS+`yVCQ+^ZCzMePKU)u9SVhq#dXejJeTf<4CGAU#m zf0-j_m5EhY|9!0m74BV%T@~c=%emINNu6!rQROWIT`zjMuZ7@SM-B`R!=J5TlC`D0 zuf*bMbNw>%A}VAgK)*8+lmA{e%fozp(*=WMg^TA8>brBz{lP|f4xZH@)*|r{`0WytCt31H-t=pwA1AN>+Afcfa%cE}=On40JWlso& zyXlV|QBTK`*gt+z((`oV-T1z(P6^HMUK2L1UTSb}IlVfT0jd2gK-^K?IFMN0cV|8J zzJcn!(@onZ&8}1?3X$wbr-PA|Y_ZY$rMvhr5sFx*aJ<1+lIQGq_{J`yTUr7NxZT`k zO9ayRC4^QhosO;X&GNLtrjd~|a-KXoz7@<Q@atp~Vv zdUCmwK3vbv+?tNXJS%W)yY?WUIvT!{TP7bAIAyqA!k}gh%ViHUT03C&ZPBZx^raHa z*)xb4V9Vtcls3fc&ZA5Vma9mDFtp&nQ8^DB%R&s-3UMairkuns>V8CZw1vKT+C& zUb`R;#%-IBtc!v8E02LIyzSX7$VHi^i}J3R6@H6 zyi#b0#`f7P|s~A38SJb}d*S3D5Ufy00`o$3qPvomA%ad267Uf*FXBnZ^ zUN`nF?nulp;38k1p2#L$;@LUP*3DbMKp}D@O-d?v(Q%Vei(5480`NfSR}qqVY|A_p{};Nnno-z{h-}cT$vUBVHKtDA7ex_vTxT%uxgL4PM*H- zrqEYldMp4xobbi*LM`4B1vR|S&@atK?}jiU?YJ5JC#U!oMxq$KDJO6PiQeI&T+HSV zbJzpeGF8dKO8%ZN{zL3Md`m77*H*@j{=`$|))=j2iZ%6GE&BdNPW?>ei|#BfPK&;) z!ng1QWf3;CK3r&9C2$mFNb(!9ncfx%HG zUr@h*1leh+#G-;z7S6%**=dny;aT5s-i80$9D9Bj}n3%d2`=g2+QTFk6{-zYalCL*DJP?{_L^4~xRi=^?;B6cW`jizz$@@0))>748E!GE-HoTy zbY{Nj1ezS*h^>{V6iw^;fUUFj=1IG9$2?~CPlFN}($%gj{01 z)QTz}ukBXVaGhNdZi#!0GkzNe6A$osh8sJHsIzMx!b$+=>A-v6eFnu45G zz|a$s#=|NwRp%A3A2;fbNuACnVNoZ#n z(Uz{;zsMMY0-iOQV9~k}R@%&rY-wE@#jm$Yxf@xLJ)L7@x+4ktZ)j&f`5o}ZoA>N^ z^XDA3vVSR(QVF7x$EE#cx0o+3bYb(^sW z=0!%d9ok8zL(z&fg@W0YGs>i2_lhtVe)K0uy$*i{)b-X&M zm_`omK(6Z@a=$kWr;aBwIOK~~(%ui}8CGM6O!~#PeJ7jYzIiX%g0YD5l5=~s`jR`m zz1FNRR-%}j=@G$$a|-7_+oZ%elwRrW7Rn-vn&1ve_CR1*Wgn?G7|LCZT4QG%6mb1}qeVE&)ac7fVeAl$r`Va{~?L?^E$9 zU)-M^l6Dj0aShDu^2?ub=Mg*&YenIDw4R;1qKZ^rjO#k2OH)#=U17wBTY$D~KX!&M zkd|`55dytR$8L6RA<7w1c~R%f_rVcg97Af7s+lPP>3g9F{qtH7O*h?*52>N_o|;eC?C__{K8vg zb=Dl58}XYrV&%oC_>8&?z2d3Ns~xy&4@s;K<1y_8$O7-;Z#3~qQIQoC2?7#hc(PyL z!BRop^dGplqDO-QiQeP+G9|cuzrkJwz_^?!eNde}_Z#Tm5)M14x0j1$M8hCA45nO$ z(Ih!z*Zn@Qwn(uGb(iesMG40b=PkPvJ4jC-G)-kguV~{+_u~!V>0bK?mxC>qVW?Q+ zHRLs6aeA~>1j3J1$Ku}&GEP5ICVkCe90B5YvvibXU8NyFX*$NZp;Jz_Gci5;s&S~1 zV10_>P1Y!1v6Ml|l#Fo&GhgUzIlB($EVU1kXE=C&`UqX3vIX$!?9p3DEzd!Ssiic(VS~K^9Hk)LtIDrEo z=i_zAW{(@MQ@=0%VM52lMx=oQLV}yRTXjAgR@5UDK;f<~j|n~_uwdBOrrK6-L?GaW zL;<~Yom9bE8ExVWi0fFS!9GJ`7r@9)?vTrMmF8-fnJU+w7-Y4dD*|cZJMIjEK}K0N z-2>es2Mt~HA31Jplu>puJFIs;Q6n1+>IA<#6spx>D&BeVLpeMj<>HyL_>B!(Z7IP^ z|NNl0CF(yb7W<@U9Th_h>iv$;)^)8OPu2MN7{LkpOp=c`DbQUl^AHhfcUTDDMm8uV z0E`2_=5!Dj_Oxn(cQ9@qL+i&rF&<^yT;J&T%3ULzOj5fOy%^9ZYG+>XPTUBPaf1X- z)b5bFjg-zHPb*1AONHR^FgL3NhVHJum74)|zfZZm6|ES3jY*@)^>hBMgVO~i9$_3F z{`R$5;kOH}@z+EV-u;)P& z)$WpZIhU{Cbhmbn^(5BVkG{pzc_=PK{>)E&GKdDV0c+ht8I_FZFzJVN3iZw-yzigS z;YL={AfQ^GTKj$_?Z=j>cH^!@nZ9d!XCK~|ERCEY;hJ3k;*V;38n&1&P*9GpCEcMT zQer(h-=~;V9jEIwA|-5pzkd-iswgyeoO0Z5*|wH{98C0_BKNzWP|R(oB-1Aa*VuS& zO~$p`k!J2-7%%vS0(bce@=cNR3G{J!<4Ll%i^04QzCfGr!1O+7eb%;)xpJV#7#P`@ z6a|m3poy|7sr!P@t!eVLM+rJcq;4uLj9V#j4LnKzlVjF0N!qNRfl{-p7BQ$_1lllb zLi+01UH9UG^XekkJsJ8LZ?ze^^(Ld%9<8I@0+&Vo#2@y{sE92lP#Lb&6FD8A6**Xo)xpsMlYYv2~l*sH*fh z`YBm5Ny1@(^Z#Eo>fG_!pw39H)MEGk*SNagxrJ*8=gSl?1zF92?@+x$ZT zAixeUnXbgUmBTb{Hh9d#+9`}s^PS=ki%()(>rFHc;VKLAx$=eh&;b?H=lI89zv1X^ z#)bdw_ zpXT3sr{{d#lOY&10=T}Qru3jlGX(Sn>lUqfaOpceKEs+z&vgDG%KT8q?oLiWYo`F!dbY6#N#em^QZfLHP8y4f40~x@~XzEQ`@1A}}{+Ia2;_YwZpW9dM`|CJQzH3occ^9h$81;SGRKlCU$cEOS*yP5 zBpMSDvme&agr3_b%2dz9pHYJ>sQwTEZ8q!zzV7{RK+tV)5=!hE;r9jzP^^WD(rgg+ zK6`horDS{*^Ue0{mqp^%5J(vLIlTl4t`Ki(Yt3-RI0&+X(y_42w*q z)tae{;OhvF*r3 z6lUChFiZ*ymv`&yU`10o#V#~6EY#G`IM44Nh!D+3d&xNLX1r5o4{A+CzjqS28Y~?d zTX54N8S*-u1Gd&G4oYogtwR3_WA^zG%ImZlJU~4Bp8)YdReEw?`TO}x2NY||)MAaW zP{YU$u_)UjH$Sn}>Vg;DJRvn0|3p*U_gjjbnJ>K|O;itAU(vV9p3k*1n8T5r6eyIa z!<{f!7RJ{-14H22kBn|RG!-rI=gifaYo}^Bcrv3QNL`+WukO6GAVG8^50)E0k`Xxa z;#{d>VY(6KMK#1A5=ZW7sn5IR1$akT|M8AgEcDy}fu@C}g}a!>wB!Dw@5KI1K!hy* zp9opwx&9;l_0jTQKsXOvnyucK!WEMSII{W^y@6ZPNd^x0Zi*@)*h3cFz&b(&LZDLt z0z7=LXT(FgKDlN4zUp$)wk38xky9p|Y-Hw)Z~&s56mzJZPUO(v5A?iB2Lg6NY@uP) z3jo69vm3(y>d(qEvd?^GH6zH6KT`NDN5~?Pr^`sY$rfWv00=jtbyHD4{wKH%gxT7zfq0&^ zo}WG>P7Daag^~Q*kDoo}GeA3U=Nn@5ZU#NVN9)nr&4Q5t3)l*3r_uehKOwda{D+PN zRP^??u{`vVChMDKJ7Uroq_yDI@OS#m8v`$TBtMQlsixBm$*)1SHm1MnWB911%*^TZ z5b=GGg&KLL`ZAsZ5JHEBq;dY*=%k{i`?r5*N7<)J6_dd1)Hto-*@ro;MQacL+X=P% z_&`Mg}hj4DzC7+ z8E!fo!d+>+RVGRpXp6}8^`Pzutyr~Ge)MaPy!OXe?4J;U5H$ndb3!oqA+8nnCqzA# z6X5#t=~MPZR{JlS^2^Vx zmhqVOF&BQZ^N6qFy$%y!#i2DaqXX5^04G5=2T_kjwxDeG>FN>Hgt#FNn))Us$b$~} z>Mb+F7vvR>d(~%>H@SUqVdvtP8GdtLVq#*b6yB92R9ClGJ#jNK%5mAA6te^0IZZ^Q z9q1d{Su5KP9Pk1&j+!Xdo%}GZht$~Hi4_Zs_$syS3q8T}JkK_3u5Tm67iE+Q^V&_L z9cdrr9`Jzq90-ReaAg|w^|8AyXkS<|Xvt4I4<2BcCJY>)rfU{dMmSOiC_lo0)H#wW z?}Fda`GUX!DuFUVAb^W~g<3D?pvPC}JyK0rwH3pL3Lyp^8B0r7+Ys206FTy$Y_=}_ zK%8H-+gqx7NBcVc1yCgGmpl0o`f180qWN{ z%V{bFTTMpC_A-3bqtP;n>jzdHyn;c0$rVuTje+59iDrh-2Zj;)kv(X?pL8V_+n$q1 z3^TL-1C^)=Ex_G!emU3-LO0+aP!ym@9-A`9Ag?3^40iv)xZt4xD2r-Yf)D6B#&W$x=>Fy-V0QlU<3thlRUwNuWE3e4k&X;-U;K9r z5AYjM5#5LH`(G?$o}WE4eFP{OfBCjbx~Qz9OhN>HZ)rjKzd~IB0}Kohm6vCf5XBD> z=#u{{KpKD#6n%m;K0a;$XNK~Ng!jlH4o9;#wPlvR;YG=lKaylcQOVdqAbJhl0Kkwx zqksTrcj-Km+NhAO{yKfI(m%Zeq^l3n>hmdo#r7`=TrvtISdhyaE|kWk9=qiM;z6$) z*D`C-B2z;C@a#@15Kv8RR~z>(R53>|6hkIK{P)AKg+W5z7gBEuvXM7pPYJVE1BqCL|P4W=;HpPYa!IQ;(npLDr}iQx8q|ES3=U z0Mbgp9{bYTCJ}71zG+)&)*&q?`{u-6+(fZ8&iv?<$w>QQW+?bk49u=JOXp2v??Uqw zC3aKdig8H8O<4ZSnNfzQUw>6Wyb&2WF%Xgl28sOw=~z93kS^h0`o_4IiBkYM>3UCC zm`4>rxVCY)+EM9$FD$3f^%9xr6ZMniM}MY;0AVvw4zop0Du$-mLRC&){^=*xFOq-^ zkfaj+j1`*^zQ0;Uj^nqW9@t~E1Vc9)I_m#GIzP!%h-_$mS?i);10aSZ@qr>0xsWE^{mtA4|Ei`oT#F( zvkyn?d>KY`h#p*ezyhJCM~Jh0B$YDrXT9{kMu;=Nl1ns={f$Kh5ILU*xTXxp;dE5o zBWL>X5pewJkMk3l2}0y#FaA7C5Q%;QZKTOa>Ce^)0K|Yk93=B+M$q8l z!_$b9AMK%uX>0vGy`L#U@k_X#<9*Zlzub&#U*Hnb6cj@XpI4RsIZFQfU+fN~r?^IS zjy6Ppixccbhc7XGjkV$dp0@!IS--{gU;c_J;>ya&;dsIIoU)j$|Ltf3KWHKH>JbgN z{q+%1f-4(#hFtl8fBUNsHXG6;TpBi;t*OA@9(fdIsCoJMAw(Z0jA}qOfa~mU#t$4J z=s-DYNp3j0QD8`Ryz{^5HUE1bu^W(9u2bnPPvIgKwFeXaY^wkIO5kPHZ*UP($Xbt_ zI%bvs@&@?%LM2ZYrwpE?p*E4_&p!LFR|kNbW7q~>Gig#2|F>6$Q7oL12wk_?#81Y* z>ltvEvmq2)nqjV7TD{a(_`9hE$V+9jWOINO^gOlu)n5uON)(1;fb#3IVatZ&?`9TA z@|TsBjq0o>upvnOYgF%%G+fPc-X{-h?s;N$;9BDiq%5$bCzf~^nyPG;g&Gd1kuNQV zF-a(q{L3-3NKqwVuRR0CjRrCR=*06ew?nj0*VW4)B+e^xPt z(ZhjnEBsxZ5&w08VCc_9Be6e=tqtYjs8o`Rriq5Bnjl+a0|_oPD6k)<(Wm-CZFp1V z^VHGwF{b}4F;Nu+0e5Kp(7Zb#wS$onCXnY(cP1OSHEOP>p|0v z=|F_8)5?nBmpsMP1_q@d<`0fnQ7Q#eJ9ZT$zXt#^Ksr{)7+MCu_tO*T(n$a);ZDJs ze@=0=hp!Av3MsR;#R!XuyPVL1S`!c(ATwtZXq!fjX#Y$IJqW^?pJ(J`$>($VJ+GiZ z`ct3TSsYOf@8TtKG=kP2wF(aPlW3M=@za=+dn$$rhuJP|Dr#~G3xukF2KE7xDmecm z!sQ=%WwoL}5(O1xC%Hdt-)kFOMB|5GgF1>CDsK+&P~y9d{yDKIULtJ74)m@{1c$-% zKBY{>DlF(b_?WGm3jX&8F-ZM!jD)Jm$ly9V9=b%CW3me7KLbxg1#}GU7Juzwe6uHi zCiCfMiU-u~8y2a=$_av?m+&%&!WBgb6e>R?!3cF|Z2u|~TBsz-U(y~s{22FnJRqO+ zV15E0v;a|6U3A_UPE6OOVZQoDj=LV1f1VpG>epLE8hpV_!mxLQ(m&Yz9QuyY5fnuuXrz|?zz)gUeL)Y4K(rSrM`G07hEzVe;az)K~W0GhJSZc5^?xBNbsxO?SY;V2&+N z-3Gu9|84aVLp^O|_6ftlX=aBZ3w+&?~8 zVP6fXY!Rec`h}Ht-|}f8W0B&(ax$pfx_#|l{UU1F=8<(^E43RFA{(v|V7=XGfaf%R zYoq9d5)6M5BPxjf8Q__~DD}g~5_wU>QI52L&!IhCk4Zda#$di*(ntW}v0b9FZ@w=U zPfgSdxRvSac%R10n+P_AQG8DPwpoqxpaB@nDFjK-)WPfM`0|F%4xc| zXW*1CbR&J_aj&=jlNhK!nR-!JTiMZ!nog~-ZpJf~6q?K6xEY60sQ>u8yR_5VCduAR zun-#r9TMeu1e8Em-diM2fxm1Z5AO9Q_s__mbl5OTK&bopaL!Oku0XZW^G7ZuHqL}a zdS^b$$f<{$sW!<=@oguWRT|HbUHwYUgba;bD=foP4>~=@skdpyUnlb4@0#8f5QgBZ z700~Tx9bbV@t6ynH4951Do{22%oyK7eyE=HS*HT*~6};&_^+7 zAFq}0i*;`ypogB4(w`)j#6{4NGrb7Gk9|T%PW@5lj={dDRKl zCeL-i%*sNRvwHjJY=5WxP_reqw%wmVQ*~ zI+x#{-ldR$iifWf_U5amSdGmhEs(IUYF{p?epja3Uf(Wz+Ljt@z#HHbE-(UEGhi}6Zq zLnJUwp9kj-P8r8DZTs0mDShQq%1q1N|9Hz7$41u`@{D`KcCls=!j`l8xD}xrtybvU z+j0&2(%VfE0-onAx~$yRZ@7ZAs}VzBj6kY~8RnwU-pm*+^FoR%abDQ51o41XGEkT% z{@Qi7H_~db>|J4k3-1|?<#-Y3rTv_yH0%|tgMAeFOu1fZ$WcnFkU6MW-I3Jg;fm>e z0xGSKm`rg^g#-;W$ll;;D@@66D6RJ=Fy!m0D?ubR(%3ZX<@@6qSXhXFlpfO=FAr>d zmiLn$KgHidxWhG5I2vyIU-eNw+cJlBeM$N?UM(ka^?l@>)2`ACK?lp}Ruo~Zi@0JJ z+D5+RdJv^9V)FCiM$fAuEF@p&`Pc}a%KrIC>SchZKCE%Bz}IJrG2W!&d~xaIZOMBU z=PfqQ<7YkAMJ36cwgsw(BgwTYR&TyB0{XQIA&1QQ*IMC&|Be4G97P?%-(ev&9iRwk=mdEu!gC zL**XL^&~UQjY-`1UnxCmZD#!0(Tx3-w=D)(=^xtQI*lnZVkblqTr`Lw!!cVN=KNU)odsuzJfg25R)Je>j&s7 z2G1yDN=Z63Q8>|zNqYE{me(XLnoDRWfN-$lhHe2)pSeAxOWmgwDgFsciz{`6J}xF?8-nEf?dL9*;K`7b+Yqd63$ub*R8!}e|BTR z{EfnGf4_B6JY+DRQjZovhaq#|0jBZasZpqflf20 zFYH%D1?)8D5H1K~zYL15$7kmneE{u=xB(RQ4S@ofE@nSs5R9sad|r)J@ZN^-D^^jX(m1o4d`^BWaOA|x@fdYrt5{Dy`G zW6*|orq-s@xq}-Rz2UpNr}V7%K_7q%KWOB#S8}tQ-H~x7b>aUGvpjKs18RA9{e*-T zC=WLg(Tk(eY@U#jJ|~-ct~-SyJmR8apn1k^&h{)goXD%TuSfsR0Ar$PmyW!dBUD!JdeR zO8tGo8FYF+?-m4t<&J1GebebKIy3^x9=Y_bZO)axuMJgsQ^Z*$0~94+b5DlNfeSHi z8iLF1>fY^LY|ufajKWTfBqme+Em{sZ<(nQo#v-SdxN*B6?Rf;K&7Z_pGB~@CeAC75 z6@d~3Rti?G1QiVl7u-xe_?(ze1TA7$KX94J+Fz~BXAV)2-kk4@PgkE2pinQG{^)-Osd|1N zOV)J%oOcs8x`Ru|?^IGUewqbsxebZ7btpm`6T9H!etLRvUB!2hjwQk~W~9<7qtR$G zX%G$X_gLdzp>d#<4B@1Yyr5Z|06JA*Wp>t$nwWWH;6PEOO-}2)2%|3G9 z$8tbHPLqLY((%pt*BTt(DPtO$blCOZOs#u8D7A+)Q3Bp(mYU^3VJR}<+v)Yx&lv8(chL8@X6NP`+%;y{)@ut3UMNZ|U*dHC%A=Hn>|coRJo&*s zm*aBv;|FZaK8olIHA$Sd?|F8Z+M{ysg_oW?HiRz${4w!)qGL)sfz4>9ZWsxu1H0QS zsw(m;!cBZZ_(my~aBN3#WXdmUXUa&Cy26oWOQw7%BkjN$NqO_npS;}be*Vy(-C>~= z@1U&e{mvwxG063O??VX-$ux{FYBOu2(A*Qx>z=-QFra`0D3o!XxX4BQpK9EB6NW0bGuH}znQX7Cz2q7x?D(GO=5fWMoL8(jyX*OnIV++rQuOb6 z-piE|X$*4S0kt<5)n`JAw^lyi9;6OIoa6gSwS8?!CUb}qseZf+P|xzeJs5cI^g7U~ z+1ddFC}@W$o$l$m$%wp%-z&Rmu$>Wr(kw@qOSu9>dd9y!A~2&U+F;S&)vP8*M3*JQTbi+j0So5yje1pFLt*k$W(MJt!^IG}v` z)#NB=9CojVJ1B#S$QicsIl$_hv$}Xid1j*Kq%@`$7Ck%07Je^3Ttk;V#F#*8#uu>D|iTW%ULMN;G&U znn{a_K_;ILF|XQ--Gn>8^xIJ_R{arUE)Zcs*E{@v>YMDh1n7Y8kFTk2%bp~2H=I`V zWTOpL;FnaEwqkG52);&7&;U5iWZ zu3NZ|R8~_q&1GTtk;tGdO*$iTAE0kw!^6;XI$D;SKynsq!(Kd{rBjq=CrA#CAfdBw zhBcnvGy&DHDila-qck9e_QLTOOFmlV?i^}pwoK5yGl1s>FvK`0;iw&y(%TIV)~2GL z2t3}(jNjczzW#yGeTkk784>h|;^>=nM49Dbn5-A$E<38zw(tZqg1RjFH8}zx8s9!Y2}O9C~qc1{;#Bmg2Fi-^^lSy1u(h z{c;@0LC{0J-o+;3V+lzULPpjXkd~mO4Qm31(V2@?d{hJO^Li*Ej2j)s_c+=RaoN@} z0x^2Xs5L5DO#|vE3ROP7q&|D;WfQp=pZQ==VizJA9Qu~nY4DZU!VY|csdTN6zJ>>& ztcU`BjN@8`t)W}Q}_@jOs@~?2O%1}VPc$7E)m0y=J{x*;Z@JmvXRfWsrcTw z^UH=;&P*ssa}P^Qmd2+~dk$Q+gL~qZiv*=C@BGV&)AOA5%S|V7XA7IJDw-T~zIwPP z)A!^PK&c2G@#~l{erk??04g4$0$Xrh1OHpA`H%nwh#qxPBd|JLHpGc;5sr;nlE8Hu z&ro(EFd!~!W8ey(#q@~-tU}(4KDGsRfREetX>$Bt{dJYg_IU42rcbHf@-3;1Br{p# z(=VxX?#lZ87F)_H;;ju>dTXD5Nr8+IaoTwOT^{x0O6q6-JcaUAnsXF)?>YX8`t-Gs z^o~;KN4hosjC|s&@VxOi#dS3-Md;)4*;(r+S2Fi>^KN@_?%F;ta$T30^Q1rs2!hR* z2j;P?6P7j=7tWE4no+DLBX<&E<0QqyS)7h!2+hWzz%N&%EWipygFUA<)`D zP>zURyQ~cA$E_%w|Hs%{M^zPdd&5XbN(f4KcS@%q-5?=S(%s!6-Q6MGNOwz@bayu# zy1#wU=ZcS_cn7U5sgy)=W!%TAZdG z-cg^n2+MAVcL*it#+?cu7Ra@qPm2|lS2LNy=4-nLc<$5wK5a^n@&tKOxWuE#_*|BF z#LQMCiu}U0Fp@rp9#?z4k449FM3` z$D&P>P4rWW3iT4FX;O%sbl|+la5h?G`R)?COqH9G*M4&$|5*IN`pdypw6>6vz~hfr z@Za;@fkut9e@}hj5b8f9?e-xA2BF6ob25k}a^Ps3>$oAS?yyWfDDn-8Ye!7JEKKUi z*lh{1SQL;XiQxg`{_)wFuSH*ZgLjv)^zb*&4#P2XuG{@(E9v07?}3@cCH7A0Rc^`~ zisnPnT-kH56Jy=V5hlo+K*LNQX9*(y13X`cZ)48zVR0lrvt*GfVev z1ap)y|FDtlk_nxKa^dJ0SHnQv61G0gIvdY9%-mO>Tj+YuDUt^#+Z!atpb=Qms+7HR zuDaag_&r4GS1@gq(xR8;OSAV7OvTBNK~tkZA)WMduwkfjhA=1c2PNk=TzZ{#MFYwr zA_DJoWZt;?;<>hdf9x1l>CT8Y9ky^e4WbQFKpV&^gWr)LK(|0$?K6gX6@*tA*l+@vk^OE{Gti0+% zdX8B3Qc7-qC`;-ZAV805I7A8l6^=q2j=X2gqFaDG;B`-0Vn>i~TfqCGt5=Cl#U|eICg#m?Q2i zgVL;-vbJT)*W+02B)I;-i{i2xwOy%kPbi)2kuxaxgEtff^Q;*YfokIeI9C9Nx-=1ALD~M;`3zEt3wR= z*UoT;58@RZ1I*Pcod>$3X-4x$)2)u*n22#$&e4c_y}7MeZ6iw3{x9@S8ZE6n{#IUhE2KQ30Oq4rZc6V)#Ld4Rqd8Zlh zU@1*6oIT)`e2p^N0z|lZHbv0VNKwKI0s}v}etHo0Im4V-AY$R5fU#T5GM|yg(Xwy_ zMe47U6I$bC88CbH=NCSXm63jQ8lOyxVzAm$|4B>zw6sAHI+%&|u_w4AIe9{Tyx^=$ zy<`G|g&}R+Yyt8%+^e?T20Oe~_-Wq|FXZTJ2bR^yG#A=h?2j{yU^l!#2blA%4+`HR zE5wPvlvoS7tj!g(0Ifb>2+^--oViI(L|yEbEA#;O#f<%)NmEWMZ5=|dPLKvzZq%B^ zg?!=F@s5ktheJ`A4ex7|M!mwsKL_#VT`a@;F9c>{)|EN5hQlz+b}zOi=LvM%>P*B!;2`q&^pNLxb@ffv{F{c_8=-;W8qno*9L$egW#TM0ZB0I%Q*1}r; za?*OMn4_ez!5G`bOnoZb)$AqC)u6ufWOlg+OW~(l#lauj!M#JrO};e{ADixRH#i}) z5}S&%XJ4Mj??{lK_U_GY`{PfO7-lT74qobGkU-iwX8L0J{zHg(*KJT8&*h{}+F(W(0i{q0uHh zV39DXY*3eeaI-xhx|!sr?@qA=W1gGqdcLtwTy}kC?Qqu~f%~>obLetJ**w4Y^T~dq zF0>NSWR-1oAeI!T(2t8>hptTN+|-qI=QedvwkL&+9;ZG#mQF$B<77xWcxBdo3y0E1 zCpJZFaOq#FxOMy%BpUM?Bz`ndX}ICoQLEQTABiykST1ni&N;++@ED!fE#`LrsET%u zJZCX|lX5SKQKoq^N;z+xqe5_gqCe`0ry#X6k{$!jYZW!=3E0VSS;VqR>+PUI$WL)f z_R@{L4g6w@DZ)%#*>G#L(l6jzdcEwb&hjIV1In1H?K!{qW37lP5eKoCz{3UZ#^hFw zm9a6m9Jax=-+G4>1OM0Za9;Txt=m=$%3;g1+>#lts5PN*6TIy`Hpobez{yVDP?Y^l zuU2k@mfPk0?P{?H@i~j~Q8_S(t0(Xw?Ufz4qvs(Piw_0eRYmlL$ng^O)};M7yY4%S znc?AGw&>+;N}TUeH&rgwpI=PScJ9d$uU;rQn+ZohboJsNEM&%bZP15a&dx`3D}R}e zU`RJ}RcA@D{$4LFqRrR>@2idGH5Pr5jgrA z?Cu$qyPO~MlW}?%{Z&bvi_I{YB#+$6M>KaG-iD5jn8gPhT6t0dWl*Nils}X&)rW(b z?e!lJo7f~C99Ao&a=dE7%koJowIV|5EjAj0162z@?U0muNp@kFff`Gt+Z6Si)i~2o zGCP5oL*azaV{{%FHO{K}L29&C586^25J(>S%Al#%RE2(k*wKa-BxQ$sW zelv#r(Nt#9)q;<>B+94ONbk-e?cD|Bv*b0U2X-1ac{}w*)BP^Dpv{-O9w%BQ3Uq)8KET>vp*_AhPk8ae2ntjt*Z52B_Xf2yLJ9f}P zis<}sz&`NoaS`W!ZGhOOSkyTBl3wE@3_;ypX^7C0m)3Phmt5Ol+8(w)^1k#dWq(Zm zpIZ984?Ds8i3QWb={1S=V|6Z@gIpeiPjs1&cA}6oP33v+H^*k*;pXKu9`$8ySbMZJ zV245w)eV#kU1~Pe0)~)fxxPhoHG>lKQDr$TJdcWYBq`czObp}*_s|QgnBIAx^G3eX zf#KScI}>9NATFf;Lv>O`-Up{JKh58ZE<|7qim!#P zPvyDw!&aXKT?kYkB*NBY=_yn=8K!^17(CE_UK8>=e(e3WN1ApTef5Zf$4@b5 zf4FTGWa08Yl3NU=MJIC)@92iP(Z&e;u+eJSE?S0!$$1&Dmc5&5RVa-JyF0@J_Ei@k z=swBy9Z=nZ_Suc?v4Qc46#R-7X!TlWF@FIrRdgqc@282N!ndv)Gg*6dugqP`sm8Rk zW#KR70R#mGiYyDVe`HdV1GI>L*EJ%Q09r^gL4F~ledObzj4#y|T zk6PLw)uB&leg`+`Axy>;OZb~BuCxgR_Wit+%p%%Yo+S&K53nv8D-PS{zXsSUt!}tg=f_Q`X4|td z2&9~2WbEl~Sb*`?4N)bFUFsQ4A%!5lR|v}#&)}yIYvTxr7K>@e|3t*jqsM{!%Dkf~ ztX+*zJG8Afyk3+u)Tl3AO<{}xNyW`pzF z5b%rCFDF>?3Tv^@u32fBfBD7q1_;lyu|@@-ZHH02xxDpGZcn9OiMEf7MrrOG3U_>! z5^#U4_-#M3Qr~QmBj=dc5jtqLWU}*GbqMWatJ8eU&SSJ-C#lX3bkCwNmGOe7$)n$5 zUuv7pCs+&M{B1rZ?0}7hK0kyZqKb#@W!?JTNb{f*l6WoIFHPLl*R714U+ zW{xnUX5bu1gYOi4fFMcBI~;HHk@L;8MIs-aVLz>U`6pc7JR@v|NEUlhO(WY+m;aBXMXTiXm)e0-md2Ohr#wJ&?X>L(3U7Yr=IgcwZD1}RvCR}(xYny~JOe(7ZdUnuG> zaQBf?DY4Qst%N4=Bnp=Q4pd3|iq^F>XZpR-qps1@tI z^+#gI$`Qlx1-cJm$(!3G9ouAw(!g@nDm$85g=^`<;?E}OG-g$9#}oQChdqa-WWjj) zFZ&rk(I|154SC=6%;dU0;>XizVCQy?hjN{7?N)3kR*H-rWQbqS)_UZZXf+AVuACxb zIsx8YI0#5xOa}3g(KZfH$@8Bm&>3dw)=*N_OzACKoY&L)@K+Dy?;bsyu8(7K2V35> z!y*$z1A490u<5mkS2V50XgUw2Qr;-@3V4s$|Mob^^}1cSQL%g1J{-U4G79#joSG@| zMhPgS8}RlTfFSj0&^q^TuF55v(xx_RB#0d0CQCnS(kyCzA@yeL2b#K41Lbw&OBWpb z&d@2h8jNrTLg&s6<_10}{`_iEsJLe-ftNbFbLnTHjRDvEpIQKe4ht>Zzvwk9>)X%q zqIiPPDdUaOjfswUu4CuUpJI5q?8NUb_BJ;Qm291oqAQfncvk7;p^@-dzPyDYk3QGc z-3~{CXR@EH&^DV6J%1?nBMeBQrSXGI3eUHTHrw4( z1(%Dp`>9hmHSX&hCDh?_KwIg*P(R-Uh8Iy;xh{8E?SsdWM8gSg8puw z%{I^Gy8*eQM)9)Lxh?*qly*2!1kI=tnH)n{hO6>OI#>Q<)Rf}=a#5x3#(OTOr{)v> z)W^!y?tQPpb`(Jk=MqO8qNFCX&dc9+joXF|Kd|4t0FXnUvM2Llh(C zB>LKdk$TZ7Nl{pdG(B^^H|c<_eO2WxoT^9c)w&5}Yuq1~(gm^A&Ao}9SsuACe|5I+ z_Qj$Tp=nH?30yO4>C~#F--McNP3476MK;N+lxgK71b(*NpELpbPW$C)E)tdR*1i%J zOR}rBSdk&{%naXyzSw=Nn^P&Zg(*~P4b>lxmar`*PVDxAY^>c z0=3{M-*agy)$wq zBM~S0Cy(9ZN&;SmGWSVX)Kp{&n>Es}OX;N5cPn=%BOX85)^2UxU|dS+>y~nFq7g11 z#nFf4dh@-e9LF%4yx||9r4_g-K?CfC%XB`cpXMtKQd;Jq*rBAhzSP#y6=5=s=^+q6 z7fS||c_kW9brC20BIvox3(H`eqxUane)`=XQ>lMTV|O^RLfQkVmd_E*RT?ml{k)%=yGlpc0>?*^OY>7X`+V=DuMiiDj&_=Q5rBlz zs5aP$NJE+pBHWp>6)Y1Z2uf`)QddYUlakMmr%~Ahb~#|Z;c&ichLh9nl62X5k4sOj zSIr_1$GKRU#tLtM+ws6+|62xT>!y?$zRZe`W(Om__uScDpye%lE9oxHC7iw zB~F%>Hwu6SrJ1;_WE)iErUQ2KX#IV)*Vxx?8BaYO$=tmf&x_7L@_|nZrMGwF>R>0? zAC8D%b~Z6MkFLmfu+NfZX7yrQ%5 zO$v-U<|>BYp)3`8-q2~-D}>Eaz3HsR$}HqH_K!9aa@6 zFHSXW`8il4Dkpf9Hk#2xtT&@6FM#JQq>o0fok%=od?MPZ*iIrR+{TIyYb`^fKsh$Y z^PE?~;gXx6Y_V`nmHf_bPKN<--*2QL=2Vg(x%mZ%r|DJTn?WD-Vj$;T#+8#9u*miN z<>#87Gcjlt`>2eLoXiIkDC;DMRisF>R&xnT3V+_4iqIgmGOD*KW~ ze+^_Z%p{X~|Bd&A-}PB0TN=X1Xs*VsyhN=*h5u=l{&p>4#89aymU`>qHt=FhUhOX( z_L2NFlcom^exFX(rFK>F$MBOcEV;0;>x$*xcwXGy<$fRFR#;ccBAsH&D$iW)wuepJ z_ulG(&yh=*&Kc{$CW?vPuZD?ZkE|)vHOfK*iVVe5Iq~e zXhBE{zfVC%FT~;x@l3U-@1Wxp#!rnOI5$a_nE0dEv5xeeo$EzczOo@R&CZ#X{4yD2 zXEi{nJz1`jV};U`$%t^EVMXt_r-bEov@?ZSAc&T6D6e2P)U zAH>b@6HmZX@qIBm&I@WFQwEo-U!lZRO54aOtg^?$rx_JE{{gH-BLK{48%_8CWLN|y zx93W&WwqGkd57bm)d5I@V4CfZh35faHgBY^+pNSo!|+QYn?s%nkKAr7?6cbj`9_~vFT^8kSY-V03iC+r74Ji6 zvY@U5KhEu_h{>ZJrO&JS3(fBCV!}0ED5N_e9Al9+PM zwUIKKavyph<+zFojy=SQpuZ{Y)H~cG7pj+5ZVn{z`TPmGUjSajKswo4qP~2k2!0A4 zvb?zV3-3v;FWlg%0*(yQI!{KqPAGeDP*wAC9w`S`M-$5TFD;biY$KpKHsdrxe5r90 zZ{_$ZeXAqc5DR9&U-h#DB8nkKPNig46mKmN4EF01k>^!gA zU*W<2exN^#*3>#UOzI%q(4{=s?J=%x(~a19LDY+IKC|&gXX-nBdpeezZ-V|Xeq`s` zicuBlOvhhf(t@?p$cizlZCm4Q;|qV$sa6@m5WL?ZYLv3h?O*dF$8ziAoVjAkZY-3`V{d0t~;#O#{pYFOn6P>`QoaFmTDhh zMf&>uCpPpAVc|}bYSVDtN6f_op!VBOURJDc552?al&4>e}8$p|wsYjCmK`97<2Y1Pd2{QERsJ+H3l4REn@cM@-|L0?9*-(V_2P#+TL zI|DSt_?xgum;H*9 zh~ORVf0^wj2suL>DXt`_ZG>t0H%SxBL;SxsAc9F?oPfZC8x3;!udnF<4KDxHjcXu) z_P{vkzt{l32Y`W~HIYLbE91$mm7v231&DV>(ju<#fYRXLXj!ZYg*`ll^QiIO8us65 z2O>g78bq3m#cfc^4uHAsg>G|=U&cvzT(*ae_cs6YH4DguDO{j^h42jDQAq>h*5n@R z*fH!Jr^WW7@%7q&hCPrLyr{fWPfvI`<5fd=X!6E6)>CINQuY;lAO*VUhScT($tdw* z8nvR})4>)wuyou21-*EfPQlL7n zeRrSPt~*F~>^Zyz@I;~UHILhIDo`wva^9mp>fflX2Lr^7&8PSm!U}-T*e_Irjwr+m z(nkW^KIj*0S_EnmY~>wa{n@y!g8FK)Lc@w$t2z=$@*gho9oGfQYS{vQ9I$DxRb%;= zO~?a}NwWSxq7cwPol&FqC^n`jwZ2JOVukiEWAb1l1bZ^KwIV;-7OaMcSVu2ukp=w=T`4>z^t>~Yo%FAA>nTy)_{y?lF{u&v- zw3m-$m#*^P8`EA&1U1&HvMueZqrmU zIRcU}f^xz_Q>-R>t;Ualjg+|Nq?AVK153TgSZ6f3U61(v*~@AL?D{wPD=+)-1O{C^ zcG++O9?Og9xhjLR&q3T`ga{0O54}kAst+3&H6mVu_p{I=seH77S+4g}d6kAsuhm-| zTLNkD-WOJ`f}v0Y;`_n*%M^W}w5j1etqks<_s?u-lMfbqvn_Qx9suYcwo$Lrk6z@m zn;m|MQh`^-*na2 z**)m)kiuou`W2y#jFH_%QZypPd|H)CHMvKdkoZ3*i~#upGKh&mJyN6nCmwf*c(IO0 z%Vzu;ZjC?f_s7IKI~l|mke4>sTx$ULo;zs9Ter0cf4bzo^&gMH&@*o|29;hz`tJ-Go~#2&P=zBO*uZVj&ryIi$DYbJGPZ$e&WXkTJCF)|>J+IL>Tq17$gy&~wk+5*h)jm{Y)cH;^ccLpyy@vCv&zs26d0TOx*n z`JTj`Uus)=e18IE=xbGf9>}dgC)q9(!;!)0 ztFkwl*lnv;Y}3MbIXO~=WkPtPK@_|`aHqrlTr`J%kFsk5PY-chg5{wD6(;lzlO5G7v@pMVTxYN5B{r-=evlZx|iE_=Y z*|QChvo8vXVyH>%Nwh)n=Hl)7I1r5wodIOt^u?b+gQre%+uc4ileviW;^iE80>2GYQ>et!4E z1<=ppgGqVr4b899KA<^RoN;A zwW2Y?@NzV@(rYd+e(6f1Q5p|er+0t+u4d_mD*W z!y9|aINkCOZqum|z$~WnpGg7QXE33QTw&>T7z3kr%{GN>m#_qpE2LeX*bpVwqt=tv zC*_HJ+PgbsEX6iIE3Q1mkpeBZ!|o7F(s8bZcXvV6$r8~bBG23^DVOr8!s_AQ8|Msm zJi$N&7)Noe{{*dnLQy5G8e0N}jG6bcP=@*v?C#?A8X>s0V#PQK(l?mkc($SwLaSzM z2Sf%~rTkvv#Yk(g)Z>4V5Gf?@$rGaCImebXeb(P_a{G%3UV$s@qsul~7l4$3<0QUl zKBj5e8a;FQn*9jzFV>e1y3R-#Z$|HQPW{SqelXh^jWW`^L@ zZ>cKZ>uQ@lbhB(>Q~i@$7Lg%G&f;2HHPUrkX1Gh7BF?BL1pQt(0P%Kh$V5B_am{a! zOBwh7#FoE--~ajfTM07ZLsRq!!ziGs-MOoQ3+en<+i3~}3sljoM6`BK^o{U;*Y-E= z{69WX<>0>t2SSAB*XY&!H-XmQ?>ta=FOK588v0vl<^S0I|NIJ2K{^>zxB&f!UCrx% zT~DAkG{XTM?GA%=QQXa?vKsE+VNZMu)GDE7KK|d`!s;>v*E4^=b-yp}zyZ%rf;woN zo=>4Ayw#dlsr)3}!gdl-fAFG3aFzeQ-99`i5JN3Ycj>%N5{i^kbRmwZG0OdKCDuYE z$PL6hrJL=Z#$nXm^b<-18pS|4BUI$9+flI+(MxlTkSFQjDH<}hJ&kI#b85Wh_J*Hz zL@I{OFIL4Rp8C@~gjaOQ-z;OnOf)zY9-5s9hUgF@CrR;nmS{2y2?>l?vOku-mw;9n zNC|BW5l)HneJaoml2R@$9tsvkBK<2D8E0a-=OX&`Gs68SUd+>h;lR%-%3RrF#ka|H zaiO!-?O302F-sX>#tkpi%z*$jnoH@mL@b>8;ioHnPAY)l09iy>>Mj6z{o6z8;GoT_ z4orrxXA)6I3Pf0G+9OsumB)n-_8?$D*9tP9I&2w{DS_A=<8GVZoh**OD<;>7myZ4* zwJkog4Pa-VA5uhghu;quTg1HXlgK{>Z^B=TLYb$xYm;iT~Gi4yw<5LFa&fJ4YpqCYs*N@M_duzaB7f$v|AiWEN9|wzpzWr$c9vGC6g8yDikj zRIfKqatBRCR2rRl`m7zcbq{9qmh=*Oi8X}beoJU~Li8V4nNSU z@~T){P1*oWHTy*E?_G}EIz(X|SL@G7p7ZyfG93dFDQYJ4)sX-Zz||UW&Rq*Q#R7rs zC)0^$%L7-?$N^Sa6Eh_3b$SAQKqr~oJlWf5PVwcUcr60D_<>aO@w*Azb@ShX1hhl} zK#CDVOx|#`T$`mHztv$8htuSo-H!fpCpj~Jf0SK*8S~RR8D=M=WXiMXK%O324@Aki zt5pSm?I2L2LVpi5RpId^9jIqVgbdeie6u$r@S;@#`o`up|Kspy(NOGiS!yxf^4-HO zx8wG>3Q-~o5Zz%XjLF&UHH$;f*F6FI9QTJ$W*hdnW^q9sLGh(uNCQ3x;ZKaauKh6! zZR6}+ePgG;INirmxsw64bs|KnzkksHbTqfvT<DNXK zhUd@>bKX20fy)B4RQVypK2(~TBGq^UT2-kdx8ec+FpbcBtFtUWgWbLgih<21?hiR? zQ6E{Xca2@{g2dK;L*4Wob2Ej{JAi&2@_4Wf4ESp{8Y6XF>(k37^J%R#zz2W4YFS0q@=EV>GRiqfeA8 z04TQ~O941pIGE@~ISBA%!;kR>IWfAO0%X>dsA|pjxyof~vgz)Jy4hd5A&`%JEhcE< z8A-EyWMk_|40v(HKOWs!ID*N#F_W7%7yQ{pZ3aLS<6s3u!rTd3W&>^=H5}iow{E40!a*)Qn|ip7F94*au6N z%yM)Z&DBazW@=98m*-OFswBY)Ws5%w6|+wX@cQIA-@JOxh6-^uQHln^9we!;rD@*|B;nlmSt}hmEVn8 zV%L8)Akx>^i|!t-@1<@didpo)>hsRb6k*XFK$6G5wH>HRMuxcjkt-BUG`O=lU>KznC3`=nFqmU^3~~Q zpfZ9N!Dpg9ywO8kBN8v&hd1;~Lf@4(qks3?^_^W>greO%RCGtv>HYAuO$-kJ^x@j5 zs6PR%=WCdX`G%kMP#v!#5Q_LGuS#2=EByo$W^WTt+*!;oW+jRuhUhmaWjH{H*zcDf zt4fPBW=nmPWx47cAM?z5!itq=_+=d68$O&mMs-eHFKZpBacAts!Hre~6 zSgtx2Wm3Y*14j)Fb1>IUu)Dn&>E zm1j?xQD?>Y-F7q>sD@#Ux?&Rh|3y>O3UNuT8Os0Ol?zw?8BP9a(xfw*0y+LIw(aeR z%kiSi-h|b!i(9wA+dAzD*X&iE@mSuoUElGa2ZXlUhwAEpho$+^R7t(c(pU7U>^f}& zND!3ih4FujK{c!GxrUv@;!&Hb*nQ*v0gq6?W}K?o19q>oNSJ4M20Q~gSVq-&{?skM5weS=E?svk0p+cPogaLvy{Z^fnr*Vvs&Ho8%E zSLu1DJ%jzRrV|~Pz1w_)fILSa$2w^!y-B_nU>}y-fHDKOW&JOSsdAY`G?==a2cUcH z{U7$?3(Wz)=kiJye)Ut~tpFl2y3D&|eoZnL4!vo2DERbNKs=tJO3LoxyoooV<-qAp z_$M2q^AD-Dc6H@5_Rk@%Me}IBl}IP6L}QUNehaVScTrK5K%#jGxDN8t0`La1s1u2O z8?O$FA0uEx$6dRNYCuH;beyOioH%1oC+wrmgC|}H`m3>fcA>)aeD4M{NfB)-!k64q z9(BpobNru{X@qs9{XXJDJ)ZA$=|9E6t(NgUM@M&uNIH0oKBIJ_k z4;!gWtu#<4@Mt)nx07K)wKb_wSBrDNB(G2=hQ8~P{Ox(i%#)R1m z)>0&$cvv9sgJ4gdyOQ;?qOCSo+#T_))?-~$l$TRj%Ad^hRdevv3A~u zvPX)jNYe0hxzvp}i7y8T0XU3#EUT7+$_~+l=LZdx>_~=!4R_Ax`EOgF#n;Jfz0z5t zwem^ta@Nr+w}N>WJl8W*k$B9UW^A-HZ%-$z!k`FJk0dc@&GCwVbp?7+U>Q-DAJkct zgARPSd*+J8PTIiDk7gE0XL;}bJVtXkjfiiku5>p5hzXSuzW!1GAd@qCF~qrhe`TX# zQ!46P?WUY_NDM%}jppx!@3(|q0(=@cx4s6P@3#h`hC}@pA4#Ew=31v5-;nqS&F+_> z4OIhrDYbT4ECxfY_VOR*1l4D{5FEcXZ@+I_sXO&Hr*I~D8$B`ego6H<09x`{kb`J^ zC-W*yQ0M(y@mc8O@q*|p0gqgTpX%UdQ*jQs>{7o(>)VUrzoVdWF#Q&)Kp@5Q#OpA1 zu0~^yN8a}M)L0o!9^5aQ#`E6ykN=zfq~WI#VzbfiQ8>uel>+f8Dous`9Y)5&7_GEH z$gwI1)6c9?je*lL`O6zrfV9sNye{yBqrVN~-|HZI1aS$lh5W@9QHnlma~pH(^E(d5 z6jU8Clv0M=(t#9$lN_-lfq}l`^YNkmq(~$IBk7Th$1ak=KMWJJF(bxq@kte34I3-* z@0(V*oO^$^vKlfnQEQkjX>@D37gaZZUl4US<&Jw%~b zZFvJ!LkU6rZ$j&K2Z?x~uj~dtVyU9tPUd;2He^!MjI9)AD=b4*k3wai`{KMhrgJ_9 zVh9$62C)bNR)kj~Be^m$^UZb#92RFs>1PAY*8!txvH1?+zw=BV9&Of`xLo;3J|r!L zLp3jJV7W5UL-BSGY6gt}$>bRg?=e>D`7v(mwfp){+8+A!O*<~|A73MJNWu)c)Ck?{ z73d{T7Uqb?1>ZfI;(Nqxfvqsjq~4x4y@bsMc+#l3w;l`Oc~+JN@3$-MfS1?9dFPF0 z>o}%Uos%&pYSO@4R!9z#(N(fHluKSca?If+eeJQrTgpM-KCwx4!=T5dP^!ub@u1|Z zRZnSwx(C{+Sq+Qr#ubO>r!?O6=MUp`S>A3<1ynfUI1&%_q4W{wxx_(AqTK|0`$%KY0j53 za0yOK&!OvTutAQ;rIaTXvvH}i=R<$o;2qS1p_SMmnMx}Twc$???fXdl1E{W*P1Jei z;NibpaIt$Db9SRuVOttR1opwq^&X&b(=%mpBgPJ++Y^Ls7CCv(Be=`*aiZg-KO+~1n;EdaY-XdEq-B)z9%ue3h0kwRfU~4V*DEa4A4NK z)ZeE#@p7OJi+8-X{CYrb9LY(oL2(>~il2tCk@Oz4(kLKcOi0i8TDW2H>&Kr+Nnh&| zrB}(g9mf0ZX0+?!2JeR?-x<}9l){r=)X~TlDbp~v*&o`}urxw<6<7VplghWV5$(ej zN$pZ^j~`dnU*1F9**v)Q-7I#2@&I^9y*jQ$@ z40a&9c}=aIKL-B+=OKRays3468hbA6QvRZiuPY8q37)WT9ChNp? zjS*YLW)j3MnC?2Ltf4z9j;}`2Lo%32tw}*8|LiqMftpyH3W_G%@ECp6(tNLye(e0< zvST&X?v`p*!%k$BvkY7I7zp)Qz452gsCx)5LxIw(Qd;ihnw@?%HZbztT&Y7SNXA&C zZfhgAyF>td9jYvP1QMxKBvr6T{np98cCRWO+A~`|g{@f6-qk3{6@I1u*7*ARQi%p? zk`QJl<9X$+%?fE&wQNSvstByMZ?Km*kq(^mo|pMnpV;5eb6dg) zVdQBDrN=>ZUK>s(WCL8U8b)7U%l%ccC2LiG+4>>jE%G$XE?J!k#YgDQOu8?XiowMX zj`0O|MwVPo!}Vj#mR)RwDRt&f>G5DIKwS_-9^YsBOkqTa6q?S#Xu+@O-a_S{4;&3M z;kQJwroU(sxL-*5L`QM`NCoGi~`G@vt1!0vKqBXbR~y1me!L`;?l z2^4RT$%S=g;^4^*%`U%K%{|~(;musfppI(Os!I4cU0O-?4t%=&Xr@@=&#RIdy;3({ zEkSy_H^UJ9c|uSRS#W5pS&9cASkLUy(o;V8>rdWhozg{#P8H`&zv9b^=6~K@5>Fwd zRa2M{v`Pr;y+XiP9swz@(L{5|7_QKiJ0asT*Y(iIh=ozvm5*NlY9 z^9iO5nsFSGWQ4+mNLe%lQQDK^F_#;n@{zsF@=eW8HJZ{t7hI(%;#{z(3{^kzZUae? zrIkT~@b2I0u8@!W)1>2%xERr=HQx?Brikr8ExR*GPam{>Q;(5+LPC?3dYVL-P^i8w zO`#jRwPP*_7(*Y0dwiFpCu(2bgnJV22S^qwuvddJ!8^iY;RQ%81Nt$71E2d)a>ByP zFciD5Zkct*?{;Lr`|!q2)VA-z7qU=VYEuC7`ux;(RVcUg)UeWg-N->pcMo?mwq9Rg z1;sknSo5*sbH_b2HEUGaL#JPe|0}%rfAW2O3HE&taY!9x#U{}_ngK=0DZ?LJqII44 z6&6x&aThrYZt6sBscmP61vqZBCK8+mn*Q0c$iujYGf%>$v5v2hgU7Oa z!v`N7On(bQo;68UP>(XHcy4wyQY>*Gwae|f50CX@M!JX_=xE?S)2Y`ET&O3v-6u<& zoy2Nuc_bDTWFaTwo`qO>u_sa3ZVeqTd`JoxT48w1wExOJ%^Ais!DlpXj>-7Gd}{#{ zo4V3y_}eCGPrP+cXH%;9o8K8;6Ds&6$@fE{M;j57 zF^>3N2?pE-kf{M$Gt`;#K#W;ZR~EWW`+AzE zWN`Q_oz5Oa9%DEwHL*78j`!@p+pbjY1l30a%@r=FU&5~@)2BF^tbUdKV++NMa;001y1 zVya?5{b`}DpYua0SEZ67g#<+|8wgVgPhT=>x*JDniE3|J_g?^>YPVN|ydEq(M4$(-uZw51^JlmufRN&0q(re8fR@>u~OWvca zjCqHsd2X?+nF{Lf!kb%7`-z%pg`xJkWAAo&*zI#fIf{MmTzj-rM?DCSPqgkuR8cGe zJE^!DbSenmY1@1vRnp#VW~8>Sq1@^|&i?L3*2S>!_XDdMr7Csur=77W@s*ZdM#%{T z6LObv5um|lQ5Hv@kwfjXGkcfVvl+?!@zf!O`1#i7)LxyY3FlzonYi$3L&z$Bjq)s3 zY3v_+Z>kbuk91rl6_e!uI5lI2vCa0B^2=YZ!grVjHvWV4ENFP|YtlN)N-8Y_$ z(RUqrf`wXu6in4)n4G>{q8}y7NcGTTOxou42hlmEd%4eWQ6R`tn;^6pSzg)Rz(`=& ze~|AF@lF^zbxlb1piS*P;M}=QJJFKfOa;7+qy1p>w1!;!5R7MrC`7pq^l53n7px{T z|0jqQp;?z=V8R178$lxWmNV2x+ibLU8*4LO^1s#wd{J+fBV@PLW=*i>D4zMED;q;w zLv{}?{OwLq{|WrW{2U;b`2J}FL+!8n`QLm(rU{bhdm&z=0L6*_3;){RU5YRS1WZ5- zVg5A_{1<=r7W!41RXSQb{JI2);=eg9;5*08FOEF--Zp6H|B+|^?|(!R!Tp$kNF_=q zwfg?gSoZ%s7#S6n_;&_$X!ylfFJ?R*r=uvqfhk@u>xsXx1&DUGXY1Xw{73k9t?G7| z1BDYt>xZ5wgj}X09$(6B04i|w;E>1qH2D&kMw$%M+`9c`Gz`w;kB+A*o*vh`rM_R! zW>i}AG5}m)wm_5U{0@-zYrDQ#D1Pv5BV>yZcPF^mj!v`U3!rJO@;n+`Msi!9MxHde z>Hf>X`~GNI{cEXS!Pe=@#<+`U6!1pT?sRp2!|%CVepGAIf#lKsItX2jTe}ogWN~S# z@eozjE0-Dz@O(?rly5&TEn)WF^IL3xz1~0Yj3~aI?SF2JtC!))EN_xFNUn0ev^l)` z^$U^3aFOiR>ddx@rYFN@$}7<3le3^8TKfqG2w!`A_kEnGp46T+B)1K z%T%Lf-qvfYM8KAjoj_ID;hMWnKu@ZCV||^^5p>;X=&lCaG7Lydq@TNsBQ_RwkH2@b zm>!Dk6k##;!kl2Keuss|11eW;OeP?bkWcI)v9HJg0eH zW{Nm*zgm9A7y3`Gp=ift+qU)qU|sum*_!Yono+w+zp<}>56~7viG13!X~E{sc+7^) z06P1fS^%h(y@x--TTlqN>3jpkcw@A@-o)DN0GWOh~t5s-s z4Xfd3Oq|*Dy+e=1g46#;*;|HH)ouO5bf@JKCB~qT$;Zd@6M?OBxD_N7hVh_U>}O0 zxY&c|=M3)V06>Yz)oLTYWy?n4F!UMc|7jc3gcS7y zukyd!Ka=l4#m2o}td@*}K*@iuFMgmBAM0(UY4)!Sws{>+4%7MU>4S+5Rc zsllHD7Hv&B1NS%khmdySwE?E-y+$4RyVY}#_5pQ>KRg*rkRL^msw;DlV#`00MjEnU zN6LZHcJAHLWFYcW9C{_&#`jP+z8KV)cyp+;)2P`>_m`#l*L*Nl$h@K!x}q8AFfTZF zWCwmqdA2fdqlWdCVozN^&K6E~-oDN-CtI#>p4f62ar{rCHd@3#O&m3*R7C%V81J1~ zJJ`m%T~i$UIW+Y37wte>f@PnN8MPg*_iFO7u=%gT?|u zHLcD&ZTCik5MpOk0Bdljj4eq?R&sgnFOO;Z-U5ubVz<_$oSKexa&E(xz1Z@q$ISV& zJtfXnaJHLV!^8=Ygw&do_o7@O^hU^OP|4_c_eJ|blo|{N^#oir71no%OroxSD=&6* zdyUJ5m)q|%*)M@zGp1K$fWwA!TCEXna63(e18#Y%s%1JCT)mB5afA_H%hf}PP zbcQ0pv)T}|MgG^zx_Y%?Z0p^{HrGS zca=xh<9bxdY(`VCYdnBVGnI7n`u$P1@KBPY@OG=*%j2Ziw*sZqDH&&Wy?TY`OdjQR zy<~Q0r`XnZ3yfAkrkYf1tD#sJLt?T1XA%%NndQ?`a{9K;{8pF!!JtTA!&`mOo;JHZ zx>z&iypEz*2JPz~Gx+5_DOK&+VoZ6uG``>edb`EjPri%JU7K)K*L@6Qgh$~`FwV_AsPHkyD;mEv5{?z~+H=Og zfdr}m97SHaFu@B#l6tymBF+hm(T)Vm^IEN||6!i!5`F<^aMg0FDuqcSYvNt{oFeh@cP8f&~lE9s||oy^Mji`pxrD*TydGgh4J~Iv`616R+&a5F6esdNjPF- z5A-ZOCLD9k{N*|63<8?x8?SG(p_xote~RE@(-}Vwz1UjK2F1MsXycRDg;C@}{xUT0 zUZy>{S8AmfL62E^t*Ho&uq=N;&5LJ)GLbk1Am_Q$7H>fESu_xO(sIdFjmd@dgeY(= z@IbyUV)%fi<}2N={9%uFnO_%xQGLYry%y?u1Db35QP+JP(E(NQc-{oOEP(~x_JKuW zt{#^q&xCW$wd0)QLP2#^<$EFv%D+{K?f*DSiX)$Q9km5JWafOry9u_@8U&P?ACe4x zC^I~~M&8Z10_WX+oY-g5wE-W}F-<_4JKwtHbQjMk#oS}B4PpT*wBtZVs1DRP^Mjr8i3gmF`>UwgBG;zdx?%T4rxcp%Ya%w!- z?LxI9k!A@?tHizGvGlw4eo@{_i#m5kW78tuCJ8nvU;pLz4VW)alR>(*7P19?bq^QZ z&w$RG3$x=!@X&n$JAS*s^Dwz^s*j(X>MAM?d1TTh^k@bNuuYx9SMj zkKlc?`s6prA45Q;8coP%+HbiSua!hn7CUZNT~NqBWt@uxRfF5x;qQH0X5SMy z3UFCTBEwp<#z;@?yE+*mLKfR6EESTzS-!2wJn^VtHpe^#m78w^hI)?ojE&=T6wof) zz_EKw4Xquz{@ih2$#%gVaD}KJP3%kc{>3L_vMcv%3Y{{2`#@(B?y$6H%)@xGZ<=fy zspbxM^POf^qqQE(Dn6A?$`AV%k;9&sMU{blzdg8|rnIKP-F~Mk7wn@c%?>3k7PCuN z?NuMCl><Mx0O;+=GHKj@ISC#?k_9|+QT6tS3v#mo7Sevg74dvE zFxjhECvX}HK(OejDutiG>NovGWg}kVIuQy*TypU z{d=Hz0Kg0W<#b}F0hk=oAEGe}dqN%K6f@ucV$pBzW*h-{{$_I@N)06e{s%U-9?yTh z0BG7CFn`3@-8!CMZcUZ=QJR1ItfL1&hls;{fS@7eBL3Dol=GfIozAkk*2_v71Cdjs zqwUs^=B`aO=!5>6B~rm(Rka9U^MpT`E&sYSr~@#3HHwIR%`lb~v?L-=EMjH1J^brb z@fAYlBi`kw*=*tJvy&NLpx$#25G~%O7UKpozEjS4k38Jf_2pM`A;owqO^e9# zKBxJ+VAy6aa98LNRE_Q&5FCth-fmsRwSbLUU#>Iz*8Qe;9p%olS!iYc%A$La(Re|= z<}k3x)B>8109I<7BUc=HU$%}Kzqt)LH{ zs^-H+WxyWW1*CC0t9t>vCu!c*g*K!c=ceUD?2Gw_)QfL#-Ap-L-MFbPReS3^)A{lM zBzrjUN-gro>Z;Si695xW;s)mu$kE8a?tK`b1$lOswq4#++(7sj2KgDmW-_%$vw_zl#Ge zcJ>g@%bZR~#TJoBwjgd2U(o5U4M2RnXi6g1)?gU9C85LKZ3hEj6t8?D#D@8VjX1m+ z9-WkCDRN|;2JObO-!dE+r5-X8eMX zg=^UFk}~7Q?gmf-%E-e!1zna*@`tHosc=I)VFEZATgd?4nc(ax=Zo+2-<8j?p^V2rC(%2vyjD1ve&&xJTs1 zU7DLuvf0mIWs8ZJZx!12-Vw|h5u^CJb`3C@-T38Kz4k#fUB}!tT{_i?Zfy{?Mcq3 zc4J41iJgRwnW}jsb^(}fn;0?7drrt%_QjH-$LKLc_@iOm+1p{RRjfrwlHx*y5`Vmy zd&f2)W*>ActLcl&rV>7TkPSi~fAYrW@C*QmcYh5V`T=J)%?*SI1&nHc% z2kppZZVhBS&!K41;s_rV_x7Jpasz6i?KLM?hx|+J8Oh_5heMA|$zk>Jek|L^69eJn zli>Prp@GIp!R8&7HEXYq-a7+@L7{=1oYH%gOD0{22%Htxx-hifXTa+%tar}!onIxV zi{uahXs~?1V&nZ>l)#`$rm9)lB~c7i=AwT~q|Qkkyfc)jng8yb1Z3b{N|T4bCKdH6 zjCzJpMslf`!yCc813>PIlZ|lkY}05Av}vqEKRM*ZiZ_;duLIJUlqpVRQ6H5lk=R(Olob7pEr*WpEwZJT z*U9m<7!Kb%aCT~+%G*Z}9m(suSvP*~yDCkPnW^A!^`-Gw6$Ihbe$eO7zr$GbaEFT?p}7-?lFgI;#z;he2OH*jmSQIHUGsVeI57=Hc=d(an@Y~;f=dlhteZjAaU z#+P4LM0PG~HW+`@)ErjZdjtd*j{MrTAYQ=1bZHB1cY+!4olg?yed_{Uga7?UaLdR! zK=)+&V#gl;;eXkRfQ#rGwxw(#QawQl0vokLG4CK&uw|wrdyI(p z1n!Ow;I^#HPPTz!4i2a-;@D-?jx7GYB=#ncMzdcH+yp5=lu7GKiSwjX=aY}8ycr_s zx%}!-qm&VVC39gl7ggt(8OUFa&IO-JLltd0*y_EJXLj`OGB;-o4vdH_J)MOUjM|I5 zMpI#}?|he_wCp3ZBb9r(whT&%(!B8t5%goWr)7drPL`=vP*o4fEejk$=xA`ihj|^u zcOa$Oyd^eMYqiKUcy6&2h?BnL;bNAk>TsRr`(Q%|8GT`V@}DkU_ZRTEH@w>}rQB|7Fy|(i*jMn z+Jmhp_Z8O6x2NzK??!9zvwi=_czX!NvcR%-&@y2|hUh1jbKKh*}$E`nl=mBX3!@d2a8e&bkB6Q_J zA^E+QILjpA1PK|ZN#Tn%pLs9hkRmcMYJe=uN*Oe{MC^C5GsFsr#(1S*UU$7ywLus8 zgGYzP1uuqCW@-`I*;tHZ@{=Wvx?#46GG)fQhc`A$Cj?no$j9|#wy3wG|BCIJkm~(r$>0xrhw;@x*8m~CqTz9MFW$DTH7mJ|Pwb@b#2MdV zWdhY>FhN+S;CC>uMrd9qRk~b$h4|gN;|0ZcnEJ-}nxm7VCnG=CnumgUFIXrG8V0SJ zLnUAa@Tyqv={25m9PCzY>P5`QsZ}h~KU+@hZGi((Ik(r*g#Ed@$#J3ucP$_jP1Zqn z43UlX5d6s7!1tDU0XJr9_m9i<^gA>dPF|n7xT_2wcT6A1Pa#3?XdEtVBfe5DdvJcu z?$()#qdpNA$p6n2U>A)3na(&6pUu<%pkw0Ta4)QjyolKMot3GsJUo;P@kFGM>C~TH z3+tC6kt?|X?JUpa_&&Jv#BuZc7om69#R3U?(=r>|hcIUflF>a;SmE%scZWs)U$jMwsr)y56WtO&84; z*P@Vo7Q}IlznW`uV+-CJ@FBbv<40so5OhpxU7USlVD#{ir7?hm}P4%A)I;fj~zIcI(P`fs>RR_K%owv?iPP(7PM&ff9R{j;$1|Vb+(- z`RWlD?|vxa^cCPk)u*Ww(!6xD7+sRkA<8axhH>Wixc{=;g2hp4S18(9eMxQPOF)1N zOQ2prlM@^Y2luu(D$@j4=CKzW@kxRu7_l$(YdAec0Xp>uUv=N_Ykuqx6T*rWO^Yi# zY%TjqsmRwEekzCQ8A@EKsjauQ{>bZH?LJ?=;MqM{pI-;$3;aMPYoPLWvzsao#Apx0 zG&&9ZiMJQ_nd5~=p+De`5%U{}axgkNXjC8f0H#5#t$C=KP@at#?E~_KG3YDcr1_39 zmBs>hBL;QNCx4i{ZOpSou=2}K9Ik=J7*`u)(Vg)%k zDg5{7q?{@=fd-D{ZgTZyd!bUl=nqQg$tn@}G%qWBrET#mQ5v|&`^{8UfN<6L(MUzM z*J;F?viENgY;-%YJfO2WeDi7N-2~CU!=sbG*mnWS%S!aBLI=7g1Foui#=oj`gxMf^L^7*ygL|#5l$D_Je623m4CMFEkMweLpv6^tT zzpBcTx~{!QFKqH42+6&QQO4em^kf^*R;yeS?S*t~1RmiS`0_f3T6U!EI0j!LI1# z$31@C;wzK-UR4SntlNG?|8(LGhjxUU#&y?Qr1kODLJ|Ld(*CvnW44#f_Ux&Tuj6wj zW2!LrI7z!hQ3*9cZ7asFH)IkY(xO(|9%r#NzgF$$EodIwCe03!L z?Gra-;dEqzkk!fw=E!dSD6V`Xj

    rxQ_6aZdbz6xluc`)Ay(?5Pm^+%TpaGm<}tB zY5#Nu@1xHRVWj*bDSsI=B+&d4{aP3I>q_;5x5bh6SW0%F)ZGG^B7Hx60D0I2zmYNW z^U-0R!tQ#Rp1h)B6!+qpjUqdtss}4s1A5}TGHMcFZ$s?gcoDr| z5jVM|Tj@XAzc4-@80VMGqG%a6iRKIOKB-_maqiiv5$Nv9OkUVXs2)Aj6zfNDBLkDp zU9zXowPJ;ix^+7< z%NMH3i1iVA{`QAutfO7%wNb+aHQT+&;Rc(h^v>sS|1TfIEhVuX12Z}-PWdfeFHjo{ z1r!sQ>z!|)jX) zU$8^MnYA?|Fi3n_zr@_6#6C~jD%4t|&bZHPO(s1j8`!t7$3OYl4W3P5MYjHN-hf-6 zm{${vX{$toZ93cD=o2?;zAE514q&n&*Uq};m#XyU4Cp^iQFFu6MId8`wSmvDeh|)% zM@7Wd^;kBy_a7hx3w*hZH6`KY*dm6kdU+EXbaeVjJ1vg6B@?at6!Qd3I6lWWBAKj^ zed<|nVrLcc;tZ%?rD=WBu6a7(C+#Wbqz~b@Y^Niqctys}VcT+Ww=@)12XYd5SlwA^ zY0Q01rE6jy$m@MeJlv-Mk0U_L<-xlZ;Zrf+`$a-y-S#x>0^YAKxVl1=4_4hI+v}lR zQmo#2F09J~m-(m*F|gEXQUT+d*d|i3$c>%62lq=$!$8Mnn=QJAs-vAe-)kTv-(Y30hqS{uiGsGMdvOk{<3_t$0nBF1~Au}ciD z>Km9o_T)29DM^i=U)WF<0P_ltkQ-qbzP4-s^^?l2)~5|bxnI-!SwiZZd**F`g?=~W z!)8ftk__%G5e`Rs$6zx+jMaeCL2+UariFQ_+Df%$>?~8f_MG>NBa4jdE)5Ks zbu@2n7}>7>1p+N>tX`gg1>}e1o&0YaC{TDCmQ#6n?TjH)kOr$7DOY%+xd#k7r0&t0Z3>Jm`xO;)|H zTSl6^T%|>+TtaU$a%uY2jq3}38$FHZ6et@3nDV$TcEb^A?ly+C zkcFPrjrR$q<9mnlwXld+(6ZTyUpo8pL_okyUndrH*ZLwTOj?ZXZpv+*(i0e=lq#NL z?X@ePQvX@+@6!Al;%wk4Rp?urn$JcrM%tRULJ!cBe}67H+Sx5V!-u6Z4Qj~UecAB$ zLsGsGz7Al2>&9c#7sv}1=no6~DcJAj;JQp)SGl|xV^-+tNy$BNe7c(owY#uv~RUIUxTT0P#EFf!-a&+^!LaqNa@2cpu*IB`}gxNNU%G$UkaIOss8=oOoA%+ zP7nC2fqZ?^54dUw_^`56;cvcud&Kx7TYXOd;DI_9s$`a8=Rp7~EAgM-|4)~9geaXC zXwES5**ia3>=l}!6Xa6?!f4$IpAgn*=@dqdPGTYt#A6spT zz$R0ISm}xIe|lCmgdd1-wi+WOokMwS`l$cM?`nv^9l{Cqd4+6aOK54~19f&k19g7o zqLV7VJplY9!|z|yTfc!P%?EnM;m-gc@jc9-_0Bg!UXY9%qfb6j*NvT#KL(P6X90I4 zo9otaSCzxTN%Ld^_HhzHTzZvMNt+<)+~KN#fr@Vdy}egP^Ufl>9o!#A20|e*3vI>*R4=R};z*(T5+v?`FSq2u z`VG~tr5d$Q`_|FrUfPu;RQf6S$Lx{^SsO_DOvrhi_7;Z>jGo+UvS9^nf8ufP70|D7 zf-hfgWEy{0X4LV0&N0fbRlwRpwiD@eBEec$HOk-rO5C|!8l8|Wm;dD^!~p?W)+KP^ z>D5;-V2$_d@Pm_^)Pd^Xc6~)D9_2i1NE;@9;cHY+HQ$7S^rd8}-rz=U_ENhmQoVUI zTld;nV>`|T@29P%0B2H0^N36L<#Ga> z!P6@DETSL)HpnZCh2o+QZ{*WQomNe*di16dcZX#Ny8U^&nZwo+^{OEMf&9Alc9_h4 zoyvLbe4fw$5@Gn;8wy_gm0(lc=^SHLg>=4h6h@b9dS%l0@KQ()=Kv)rA&u8o0&qSq zoGHog&_W4KuDJc|RH7E=1k(Dh{ah6RaQ_rb=gW&WWuyPBdd=n>uz0h(o$&tTSFMmp zef6MB>O@mJ)a15j+q;l$3tfBC0;zbsRWFNL7j^H|U89YCCGu&vQ2_xZq3B8BmjZr* zOELKChteUGa5KShWleJT=XcQ$Ow`m;d}_J5j!O@hsoXrDMki{WUnHAvvRb0bx?C0p z&$LPb>Ppt~$O*E|K#fDD+CkE}##%|oXdc|%#bf~H;Z)b>$ff{ob>+Bd&#+T5-iGXD zxdj9g5eJN4o1U_QQs0>de(W%pm_Yea3CEwr$lhHY&3wB&2@Dkq?SB^eaF2ga7e-Y2 zY)CVwfk?WIo){~pgMc4G8`dd0$}ZtX$bxZ7dRp(w@@_Qc4Sb!AFGx2|C0o#-KZ{|h zw&1-(U+5V~cd}dSxt~j`*fela*RcPwfE1n5tm_s#Aae$z*O z^P57}6PvJxqU-U&K*u0H}dr*Qxo_ zW_)s3eP(t8?@;To85nZ(8GD87^GoP}R*8S<=w}Xa*W*snr1-@OT_~SxGwW*`8J|yk zKLREPB@k$7RBgGP_X)zqR%1T8krfK7cr~lF4=s-@|T= z>8R%c+Ga3(#>_nR^R4>NN|m!%aKwC9v&Z(r`%$olW|AA_ zqh^LoP#-DMizM4b#78D?a)d+vvsO5J%2B%GyAYBhKw$UY!(L~LcS!%<{s=3yyF#y_ z+Z<^Z)zh!1nw0D+g*`BZzgoy2v`K#bR+7Ca5psrT8qNnByi>&Eod$EsYDz%V+`vKdZ@E zQLo!w>-zyna(3WVrY?GS^WyFzeVH*3T~OJ+`KHD-ARQTpFoemVnoaQSgtlOGlA`0o zE^(_Bq6w^=Re8)6(Yt@PAbpyIByHR%xplaSNgp}q6JJKIuM%jIrdLZ>)$4wTRgwD_ zprCKEPqid9vW$MqtOO{bk<_wi!_=>#Ro{^|Zf5V!2d$hto=ql2J?u5UY|46X#q(pc zhK}@=FMIs~SP_qL|Dw zE;YxZ{pHom$>uNS#qI+E%tt4boy{Tb(%smD{E2pYmP_P7el3Khg@Tkp$d5P0{ksrz zZ2qE-g%2yCvWmY#HLV}z02`R^4*+Lb18?nTPco2zoy&vAmjwOvZC#v|A@(ccN77=e2$ee2Zc~AhXSzsQ;61!f{~&O?p&Lfi_-b| zpp$>^A=NBZGvdmBQ#X)x@Q(ed$nVmMw+MQTH}vym;i($vG#M(ma74Pl`$jv8DNUV6 zG_nP^69RL+smRGoACO>NgLA?`E`}xow_>WV)5Ny!B`IWxU_15X6Rz{+u-TCh3 zv?C?f(xd3e0Cs043Boo`79pz!BNBU9_0%9I9-<**j_GlRpncY(=YVt1QRT*7Ts9+j zSpM}wCc864H@xyom23!ae*opgb0Vwmgo%KvqdxC1&f`Rc?Q!f=45<^l4{tZLSKgW% zPwwVgahY^+H?+APcSR_t9}EppzUr!TU2FrYG7;7WNhkUk!_Ox=iAx9$sHUl`bBaB; z1n*H4>6)QhEY-dcfi)J7K)-~U!BSvsioVwhSi@K9Y~XqdVapz4YTck5-C&`_CJ3qe zk}n8sec5}#KJ8fc4)N2&moNJ5E0`gHsp2Nai@|b~eTGW!fx!O<0~|4PjvB(wD7H8S zV-}hzq`$6%K5?g2(Mc49hO=X*9GAf3Y-kd2@u$k zPU*uiWy)S$5Jv7X(|s_a1q0JvtLBlPN>s+)RvY^y5Ac5z*0y?=&6ZP4s#PrWxYhdw zFHh!hYmxXNm-ccLuiQthDvrPoyElb0$4>3l=S_82Z3%c-oic->vu4aAZ4Er(T(c z2RaL!F{tx9ex1h2*11yP*hR~Y;Lla8I7Ghc5yHA$wecl>bdo=?bd*zfcfTUBi(BHY zTMQ)JvqLM@oQM8i2lw45N|c5*z7k#{eh7Yq#L%X!E%f>6atyo2c5htNJ%gxmq4)Ew zZGQ|&Z*z5jW}@sj$DBLY1sg${>r&^)Z-N(0)XFlQX+vS3b4>4J1TTk!UgX(DrjsVB zg=nJppn>FX;E++KtJG1wdj6>Jrzg>ivG}hYOlMPhM_N|DVR5maFQ(op)w{x))b5w# zV$nBy()?DghD|O+n|5AFrSKIYR%nJ&TgtM6nHAD*`sIPfxkAcREt@st#jr+&r@l(E z)po>$dQi~yoHo<^=%h@KOR)~bq!&|dIYu3Q^VuQf#NN3v@Usi!z=y?ywuTl35nZ*e zRa$yZD&4c@r<~5&y8;In1D&%%GDh@0y=2zX$}f)7h=kq}`y2K&1qQ2!Ci_7yL|7rn zgRIC?fYecY<_GJ;x)p}S4W2GggNJxQxkD*I2=itlD)_oxW=0%1AbvZ>fdK`-LgTxm zQbYV;rAm0J5*%wB!=I|FmYO}l?$E80$r+s(mlWmlv_H8;y&h*;s&Xoso)g9$%WBtr zb0|+RlAOpnyA%G$dmw_`n-T#nT(_$Kn{Z${V2cy5RV3_bt`w;gTkuXS%cy?Tqc$$F zy>er*KwVV*Q>X_>R9wO+cfsAhZ$CDg+j;`wmbwQ2O+(w8T1hfwci-wgFCu?Q3?HRN`E_0VCiJ4 znZl`dTcHPbtZf+)&hX>yz&TWO-rwmh^5`<#Ha{Wku~9#_f|I`;Bv)2fL+SGV+1@Q8 zVOq6J0}yuby^~>5oA~i3VEfT@dZ?n=Y(qC)C$v{&B5zz{(_ic4g~<|9a;zVp8Ruvnx1S{ueM+vvUr zj@W-7lhPEPBB|LMyE_dfB~*vttxH{*!RmE?p`5Rixh{4O^<%j8vVtKad?z=Ni1oUN z=_y6oRV9{jk3W*duBckS$NcE4^-apMNr`pPdZw<|ChlCq0X0ou^W(j};(Ovt+|&Fw z;<2$^I=F%05O~A>0og8FOb~P>@Z&(kgC&o6bkw9go#s6PEzC+kpNYQKtB%?jtW`jjNtY;Wlusg??0cw=69IK>H=Lt38l>NelxP1R2)GbxwH?QIlQvZGJ!6mqs#L!Bb3Z_*eVS$2& z@HFBpKAVNp&Y?YYG?g0-f}@|0jnA&F8@7&JGh(Cb*VDZ)kBza$NXBv`0jy^PH|7hb zf<5WXRJs;jq+}b)--r5i|!I(WyK~KmME8B5p?xWP66X!kZ zq4TC9^}Rf>`fFgo5ffGBE)?nTlZ;T+B3W9oBj%KPi!Ao+K||KFNV>;9%A#rhN*@{N z&)awpFys?Bt|?J;%*xSJQ5TdP)(UV3qK5$2AfhB)fviq?lusvTE#?mlKrzzCUUaYH z`e_Bbo8tVDnwzT4MK{&@?|0J4FV_XCV&2UCk1%kV{yrB7yTl7xmGLe&b^LtN*AQW< z3f7G!`~il}&~k%hB1^UqzKbh(Q@h)Q_!q13pYQRJaInbQsJ!55fYnErlc_mqLrHpu z%xM5Kw-w+#ObnPzEw2jI=p-awzPf=&S#hPOy^mT}t^9(SJWItm!t^&!F)Ru5h;dZx z-};)%e<()NWj^`}ub<$QP3_uYaa)(d&!{rFTs>f>^}V!f19~8mSp_>=9Q@&0Y5eGz zE5^4sUXRbU(3fq3x%fh*g;S9MwAS}NHS;s`?f2e%EP+rf=GBkKlf?`zdjnRL&Vd}6 zq7Gfch$!iZg{;c=PJY%OMY`}LY_|GFv>qR~fgd-O~!sh+jt=X=q z*fZIX8>Pd?VABxfn@zT$zJ8vcpMC{p#zIVVy|%sz`K= z$sQ)pw@fPR*#Da){DA}~kQYPISc<-1Flnei93}(f#PX)BABiQbz=h0GQ&B9t0=XqM z^qR9QU83B8H{Y1j9{?z)P{}+`UONs!_}sptzinjkMAaXWzEz@W%;<(|T3&`mBN7^t zZeqO6pb6prPDRTR_7Ltb_IgE9I=|6lq|hO^)xfxCtY0`l{8!(x5UKUx$B$9mrG}(D zw+z4kBHQdO&{q%8f!DSwE^UdTe%gXP#A)FpI!{BPDecr4k$#Y=UtdRj#wve^`JDf3 zH0{6p3)+fVT=}11I`~z|dCisS$>zjjuA@0jM%Og09X{pil04jp3?b-0YZev}lp5(8 z_oSg#sBuclQBdi2-)V^1&6aUqc{4NzndL!$7+^JaOQ%vb+H~$*T;U00PP3;Q>{%^B zuZ;ip7&cs`OZUQs)r8n)D6Vz&Qdlfc>t{Lsu<4ZRuJaQ9_}o*r;F@9KdS?a*hxiuv z!J2=$IgW{xzKa_Qb2=v;y`RsUaTDmD9cACU@S>BlK8i}w!1UzyGdwt*bNSJHjgY$d#hF8f?F#5c zVtCBn%$sm%!k*8o9Ak$QXBQ`T(9yY8aBgVwD2sb0Wx^ZD%u(wkZn;>oA#0}Y;^lqT zy0BVW_tFwQxezH_1epvx_T4|c)_kg9E^CRER zd2BY5`fgOLu~&H+*Fa%m`rz!PVxk~YSDrY2h|Cul3&)Z}9y`K1haJi<)xrA3+aaCD zB$In9bXtyoTuqF`7{9GlB4xtexHfgcVcIT|Rnm(Y#&t;%&uk>}+jK^@jWSOjN8rGO z-RfT6^PR!rvaLm}XF`q4BFL~hCaUK+&pEmgg)w+Wl%KEN+}J7W9oa05X*Zd}=yiFd zmnpaBa2JdkaR?S-8hT7MFTLiFZcx2xtD=LX>Kf9MDH!SlO}p>&9m9F zfWF)4eos?!R$GNvs7BiYDl=gA#hcfWZzvDOMcEP_mvxVi;P775;m2%sTb{2z)Jbac1e zt-^`E2_u;uZzk%O+0%d7|6-r@P|;pRw9%b9&%hgq6kgCJ9*fH8)kK`XR&-ZP<4ssK z6t?_0&$oylsq=MVaDHt;30d{BJRz7Sj!2^Ub7Xw3p}$tx8L8Qx`LJ2mo7YQ24cGi; zE*@KpQ2(2Du?;`MOwZe=spMoaO`zBQpwRM&KQvgxT{pwj^(}bNS;V!UE$8@^xN?-7 zanBrmvWT;C{KYkpQhm^PsW=U~{0)RV8~-3Sf`!parG%XulAzolG;r5)X$BpPYYvaz zo&28yoga!Q(S04A+drz4j_vkULtAxhO z5w*Jmx4$!AA7$MJmeNdwm~+>W{^UGj$p2-fI%mDOXVYb^8oGt4PW^aY*a&Re|S~ z@G9+215>0A>LYt1TX*xB(xaM=N%pB|w913?gh}qy^xofdhd)kKEPvP!-X>QLPb1WS zX8NFs9l4UiBZfz_VQT({Du4irZb9u0?{~memny*Ed&We_%Uggc=QDs|*Fj$of`7qJ zu>M#Fqr!=qLjcU5zj)g_4un@4WY}pnsFtu9rUWx#|9tvSI5s@3p`>Xq`wzhNZtm}( zQA9ysx4>-F{(E!a@7BLk{e|%mm6G{F7jlb#`~wUEZT*W6{x25^4S6bp*5CA33O4^& z7)=e~mc>VGNtQpjj^Tg%XaBER-^g*+Yql-n7ep<@1ZR|w{@2$2djkmtNr`L!{5;0Z z<5bNsN}>gF-M`6;f4l8}yHFjx0T3adr3P#MUt(_ewGJ>0ZEtP9{2!r2HH2UZ85x9= z#kOdTYyrS;N~w_hcy<3@{L#D77TUoP8%A=z029MgN#Cm6lbV&Vw zfvhexPr!CSic;a2$8{Lv>&JPlh-*p#8>i0|_Vdnpm}CtIqAjn)t;X~u0BjCLVX1cd zF^(s- z9vgoEWXl#n95)bVIldr-82Mc*ya%+6islSTxnLqz#mXPK;060^1VsT@yrm|4Y*x!G zE5?#CW@VqN|EyU#Tmz>r3d6abZZRT)RfFVJV)r}qf^el94X#}ar?f3j?YD*Paⅆ zc+1>1Vi!Xw=Fvq^;2ly0qR7Y^<;(_Ed#E6vInb-XFI26M z8BMDI!lb-_R!PiXF?C0L8UiOcp8ce1xbON=SWK=M+Loc*e<8+z+eL}PhoR)Pr6ur; z_|=#ndLH&xOfU%%?pY}`LhZm_K`m>gsawIiQwKaiI(`bsUVBM5Hb)vGuo=b{x&g;$ zYMbeY5ohFSrfdD}AF+u(d;Po<|Mce<^bjEFpiwlx_A2nY=Tv>B>1f$~v-iF88~M1* zcYvXN4oxfUS~rl$82jCljy{#;AKI>~|d{@&K<6(nsQlZ|2 zvl=sfxqfc@*b0=%%W@d4wbr69bD$g*z}jY-(FgIKrvy>c}N@T z(vV5ul)RoBPOU#+C1TBcRlaj z#)PVnymQV(wbl>)d0uYlcp~{R6!Dp*Vj&gM%s$+nz>RBtSAE|a*P^Tj=wN=y7V>!O zn;I@(NgLc}5T5Ro^A`xLggrYq$<*N^3(7i2|1X*sAoeVU@Lm>;FvnhjU{86vs`U#g z9rP~@hm$>UCw{NSZ8zWUmJ?(#{Ox;0d{!qK&q1-?7Sc~PBcW~!tH~cb?$fcB#iJg- z%rA>3x9BxB9` zpt1SZa@8N^d|Ju*^XpOI{b;HLb}Ekt{cO3Bav;2pu%k#A;%Q_yR>2X7RkrTdpwo-f zf5v-hO#j|%_P4@(A{*Ttu_wj6eZcY7a7VwRmcp=_zb!dG!&io5e2R#PjY9MZ$Zjc@{oYdf?>} zhxr~TFfl2h9XK?o5Wt$YMs}(EX5GZV3kTts)o5>{#_p0mqjT0NAEBnmz8M_Mb1hIT zk;Hsb<@@*VJkXmyu8CrAO}(YZ`T88s&^f;Tp*FvpX~em4|AM}!A9($;j0Iga@coPF zBj9%ngV`b{I&VMZ5zm%t8hNLW)#RjNZfF7v)Bc(tKEHh_twcO7AG`xp2HcqhX@1o3z+jWH2h00E2Ziz53OFo$GprwR(`F^|;i9rWP+ z;gVW`=~gt{pdF>|JmYpeq1gm-N^VZHHb#S!W4H_QR6$0Q=EENFGetb_^Eg5r^VZGz z^*$lub!!Li#p$J_o8-EP?b4`(k)3vn^wiwn?4^q(&R{09d~h}<=_8(`gtulBnT1%~ zS>F5ndoKPbAdCy+gxoPKN8Ln+e2zfsA5}J}pH$Zt=p`=w>4IWfLukI={)EiLJ@LSt z#Lg^I@@^xv`t|1>RuQ(;I-dTCY`K?F5oT(er;M}#1MjN+t7nGl|F5U>@Mo)m`gqJz zHKHiB6}3lIs2xhp;!&hxMa)MUtM=Zlt+h%CwOV`CC{+@Bm&dHFW>Q;i-rK(K=e>W! zz2}~DzQ5n^&|uBP6)Up{(i)i$!B7SK&t4@=dBR4G;o8Ew%V zgruu{IVp?xrMC511?YFNI@`7Bn$BW5>^!vc-O&&LEp_g7U5O~%4Y=cMn3Au(3g=@R zZNJ-N>XPEjBrFVEXfvEpF0!ktAY3d(|>@TxFb|0z%Bt$VzwR6RR#U(FwC=k7E>VL(eY*8!l?c2+sgqo z&n`SE@qSYajk9FZl2+~drI@=MrtI?p{+g->#z%kc=DVcmJzlVfN+QmAb!wJSS~K>! ztb+i@G(Av#TlAOYA;6l0`^R-6!ta#5#o!=LxyN^gau}9$vG)0&1c`z2n%|w+-!*PN zuQ|}4_ONuaBE56nC{gh-CipU3;$tyqa7GULTtd9Hf(h#U|LEm_pJ5~I=IYCXB-HTMg6oel^(aG5$I=g^kJM1-v0i{akgvtTT`iy#jY> zeN7%DlL8FtOyCf;4Vo~jzF867CvsjueB+MMmtspjq312kf#)mwsK5XNe(12~qsiW% zy)@V81)>^F@#ihCP*|=Pj+lj)D@Q5P9`<3LM))*Z15*7N0~rdVtL1pq$f%ug;f^Xx zd%4UqqbN2k_HEeY;;~$xI=) z0r(pEZFxEo_e96CdatjNoH~zZz`Lp`_H>MFo+4`P5yLBVb0P83S+4Tpf>yla~N-TMwHOsKWES+u{WNtc^Al)@X{LsMFsi$`hmGWo55jt^kQcEDu!* zWB);v>tIPk|6S@V3y(E%UjKoCbQu{&d-bP<5$d3W*hHoCOdzV11v2H|dQAD*zm6|) zNk^TIz^Z&6yeHa(cZj=&J%I|uc|x}5q@$W?+K{$B4TPg1mEYLpSeZz;w2KJL`YQ4aV`EAFpk=UJW+F0)Qzg{rM?!WO zCpOE|j+AO;*7H8Y8!m$z`jMu^|IJS3mDwG83hs;~9}1-3`6Y#%Fy}4(BaH%DO+Ym- zRiJ;TK0V^`KdZb(aX=|JEdLZRta~qFSxd<>S{9=J0Qmd|zNb}ZwwPN|kbnh5pc5b* zeUD-wUJ4U8@_!2|7^y`1dnvr>Of^!zcj=q6iAWJ%Xm*JBQYy6932Xle`0THhh}DAb z;=&@OZOCReBa2~FfNG|_^%NodYqseLq>2ZIWQi4{xMThOd6^@3pO*wx{9PF>Ay9kC zKiGR9;Sbs^L|ONqculu}hN|o%)miK2Z!$HSK{O}J7evrG)u$Atziw@g#TFeBnb=B3 zK;o=L-RqVTaT>AK)?&`N?OQiqz{ANjcMdnjnY2fqMYGz#h`9qq>km(pWvpUry}Xf3 zsBnJloBF5HzAk4s>P8Xoj^+~YO`UPYJl*F%`b%i)g@Ug0hYL|v5|E{(disf$k^eMU zZnb}5NyCmLmG(^Z0Wp^LrE^>V6~gtu0P+|;iCz-rfTJjB21vJIkebeP43$X_7i#;9 z6RxCD#5ooD#>>Aj<^v%l-i2b8e4D**g6XJ(3DGm(IW-cp#jhGXMW{<*zi#2>f`}YY12J)w%XYb zL$TQ?gopJM4Z=I$hFt0)2qAX&y9QJha%=`fp0|7~$P1=tB7&8gthq75tcIB;=yZ{& z?)M}=>u={+7}YS_^=C%L)EG0KbF_^QkO5YHZl8QH7O5@pBjPUrs;l54Hpo29Tz`8qzqm)nl$K$nxjg>aGuWI>pgC};XUTY%u#$jjJ(9X<-;n3R zPByIkTdsp9@nn07wsXvb;Hf&3vygSQw352})4KS@fp(d(^&=XNxX*kkx!ToX@1O6q z^)dGu3eZlV3|eX_*GocY{cyJ#N7YYD_A_Z>e7R~XI==Sfs?*qdZkklJmbiY;oco2} z-6Gv#Hn{NLXy*q`4gmu;J(zee^^pxt;!z0j-Y5NMn_wX8KKTqty^7x(uy7!Hlz#uz zhaEeuK=}=-1n}>~__S(`=$lv0J_xu;2zqbBL{|9so!6}sS9iA_cEgk!rPp6f@1&T!^CSBL%m zSp8TPkdvBgpbVUm5n1{%hp>Drt37IQ?G~Dlq>R1$z_X{5sq0{x=VqP%jwSfY0YFpy zRzBp;yp3%KvQu)_}bEa?x-< zaK9Zcxkb^CQ8X~MsF9sk<@vxo{%nR)H4*5mV~VaVyKbOWaNMXAH4lz!1Y^aYLfKC7 z-};~+1ur!M6{@O%v7C#*w6izswN~STDvv3nLFQZz!yJkeh?Y28@AxRy&qm4Ebjw5g z;WWM3?Qgp%_yej-;NsSey|M<&;AvY%r0z9sN9yDt+G(?cbw#Gue=d!+=05@Z`3cJ<;}M$lMS3eS1o_mMR>3%)u^JRCjH*ZhLE z>h`1UmJVs-%zGK7=l54?#P)Yot_!bZ(9&DmfE(E%hC^FD)8Grt3cWbId5-NUg(-S* zC6a1NbB+A>hu8T<)=A57j73GtpSi$^luKR*`rO+R{tcOPqPnkAJsNa-2?!naGi4N#QZdx;>J97OV=Js=Q8rmR7DBaOkx|`-uvk zr$PnW#6Uq=Q9pb_&zX?4@s95!S;ao1$Kxyv|JVq}nTM>xe$)W^HX`Zr3#z>+0*c>QVM9zz$X0=EM7m^;}Q_5j?1*-#*0{ zm=rXZ__&wLF)BR3NgUxl9X(|Mne7H*&ob`HY;ie=VU6!c;3uor?a=EE z-!~L^=VI1V#UX1BN&63t#@(b64DDYtoo+P#>>>^1Yd$Am`R7kbnF53K!~W^Y0<57A zv^kX4+OCQMs@*1Rj)YQI6_ghI1JiaXPG$sW&9!TV9R-%!!pX{tU2Q}9Ym-bk)}X+$ zOtAaz6!Oe%g69Uyy!A0{I00x|xyJ5=#EZpVNQbsF6Y}0Orq> zn6U<5K9TgfI_k=Z@l(Lgx$(OK%et`w3A#SLB@SDRf&U!RtI(GeKZ`_S3IG3?B*#lZN?wmaRHbuUOnW_qGlvLf$t?5f#T(pL4q=c(QcEWig_iYZ7 zPQDn%H0n9Zx50TsC`G~>_|rp~pNM098JeJbx3<&5@c);Pjeh)}4o|=3IRryI0ot z`!BBuE<#DYovf=4Vm#NgE)ChsZ))Yy$T)}s)n=|mbrWSIS5j|(H|y*b(R{{44UDE! zxS>`)7&iF-Ud-7a!KhZ-oO{aNxT%)fN-uV|1P0P&NYiS~!nE6<2T&3+$QK|UpcKYN ziw>q4Q9gZ-FAcimh0OUF>-Gn)WgW)oXZzZ z34Q``Gq%x3d&KfDaKeZa5EY^7y literal 0 HcmV?d00001 diff --git a/docs/argocd_root_app.png b/docs/argocd_root_app.png new file mode 100644 index 0000000000000000000000000000000000000000..1d4c79ca15dcf302acb1a44f05ffb2fbc4fdf16e GIT binary patch literal 67586 zcmZs@1yo$i(l(5{TW}Aq!DWCT0TP_x4#C}m3>pFiceen+-QC^Y-F0x+f6lr0=H$Hp z$6B*yPj_`ySMBbu>h7lqQjn8GMIuCkfPg@imJ(NjfPiX)fPj2NfCHEG@ZKVVf9RQu zi77~niIFQf*qWGI8ACwOML9+AI(bP3VDT zR7fQ9p5toz%GJ)^oNwRr8@y1pA&ywI_7_D(83mZNmeygdde^pty(U$G=Q&!>6?p3b z&{RU*MfaMA4wrFkmdvvsPIQtD1dFfwG8_IL^CntaIOXb2YRLt~o>SZyXjq%!gNMX% zf}>Kvov*so+}#2pOa9Tt9O}~poiHOjcyzr`wba6~*i)|fqlS_i>0xWr%URu&Ipj=Q33CuT(88pU+broRVUT-8`z z+C)|sf&u)D009|j4gmvxf&_mE!5;_+=;#0lcVLIRZ9k#^_YC>?7vXzl zF==V=x3ZCgv9XP#nXQxCX3+?k)vWnvbtiRM8Ga*MYZikqwuZ(muGV&cHGvRxIP$v!DF3G52S5K+%t}fAH;I!afKpvnfn3bi!I+$zg^h)cQV5BhoLtc1 ziwVDy_=o>C2mb<4nmIYy@w2kJxVW&maIn}qn6k3-@$s>;y<>g%ju}kB?C56WWZ=qd z<4E-%Cja9jZtQ5}U~cDRZfisS7heNITW2Q#CFNg&e*gWapT@4{|46cN{I6PI1zG>9 zVP$7wWBtt=+*I(dQho(V?AqlenerG~RiEsl= z5D+2|(&FzwyFwnP!E58H-Sl~1trq>DG(2U#S>@^#kI4@@2$JS*vEd#t;YoSUvvxmS zQOC-rxmg9imw1oynV+u3oDFiqgr%QXFIcer6Mc_hU}(oJ0* zzx%rn1_qR<2m%5m1v$)jkx#VxHYT%^O?1!L|CIhkf(tVY`~6Q+9}F{QnBk$JA+1k( zIT^n>{6$-m3gHisS{7q$ga7sTM~xX5O#kJQ#RCHg?w`!fc3`M?>*CfRCO!Y^!+$$e zO#>AV2?Ws=YsTDS@Fr&Qujw z`k!)tGya)_fpNa1RQl^m?7uqyr&tw=(#ukU8uowU|F(xF5~SmR)2tFp&i|q2hT47C zBVqmHOWU6&5&1+RVt`KRtu@#a>-8Ul{#Ti48K3Sl7@_G?N2WpJUmJsp0ZJZX4O}! zi%aCl=xC(U?9%vzgox?tt430q9smFK2f^1wT%_BUqfxyir@;HQuLz&(=7Hbkwgp{C zQuQl${7WrVN1tdYzYsS!>6+JV2m^3^9kck!u|%;?n&S7%`{{;NZk0X~uUhvR3KrHM z0X0Nqsn*)=Mw^o#4sN}FVoR#k4zq&Yet)(Exi^$FT&0MoeRdYklyc0l^KN2)F((F1 zdT;donVpN;Q3|)Mq^m1+Zx~TfI-s$Df39Q+hR20xxMAl*^Yy6YeBCKVqZ8t{gaq2* zVTKFc)A@!swDJLl-Ded0b7eeeWPAakp)#r2-1qnQn-2#uaS5&qWZoTm-+o(joD&gjVRPLC)ulpaEf@XqcCY$_-lGbO8TTFv-aD9=`@y6A=>n=%k0a>s zQM|erOAa2Fps_QWj)(azZ%IYPitvS9W&tT2NdwTs$Y&eaMwioXdFgHhe0&YAr+#DO zO)lufymt7X;!E3oUOZ>)K}&UFX8Y#H%QbI2o?O;mMf;q4+Fma4qjtHQ zA74y=gnVj$ZFylKJ_ac&DSadJ)-4ujWBO-=$tL)G0h%?=<*L*kcSW>OmX)MD+#e*o zrDJB6(Df41)=rHJblL9DbH6jWvqG)gl=wbi99Z~C$1X*%*A9D-w#7K`sPOvv}HG9v#ZbO3M#EJ!Tdbgzw%}#kTJR9IpT)#fwU?5}VUwoghm~Sw5oB#y~JL zFo^Q|Q`Jw@4(^`p286s(p5B#Bsa0G#7>s#5o{BKU2{PkGbEIQ!a)=j0mv?y^>gFvf ziX|w>AWFx}DY}p|jvyc);BmPFZ|7v!eq`IKS0V3({CiT$Ci_Smd5tiatgmmt1IM__ zzkU_xyN??h8FYdu9waf?JA$p7JfRzwk{s=U+0LVlxvSUypr($eN#Z+W$!)lcI(2r~ z_EP$=N>7L3U7}mI=~hx)%=TIu_}K%LI6)QL=e)IRJzwF7o-0RMhMoT~ZqxRRhvedl zm{Myc8R&CUe;uEKF`o%I@jgSOrlDbkBpzQSd&8pJt+w{`3wyk7?JIJ+*NrhQ)W#Ob ze;}rJGmz}ffm%ttM!q92EHH0x*Q?uK)^m?U+5#+iKB zYDCdS+OXDeS>bvnx^AR+p0GzlqaNQL#rUnWysV1nPGufFrd0SlmP=7*>s>1p;%F&Q2fx`+Xq}cqIVbObn98mjrgi3 zXlEb*enAuk-SB9>9`trTo@{%>J7FA!7v0gqpBgTM5AB&~4@=daal(utnw}d&#zjTh z+#6a`6>&@CN56fJ$Q1dw{upcDStIPVc#<>b@iC}C>wS1;#NN z_c|5}m!DhI+h{TLEIQgM^GNfK5`6A2Y>N*ynd81~`|e4b=XoaTsMk-k+D*rQI-_$K zmZr&2;SXw)lZ$vPB#m?%M>5&xD=XR`RwIT99ODIEP(`Lp1|fQU#)m&5fpEWJKDje78NKUf8pblO* zD#j9{IwEt#CZ%+hG~>w6({5x#ITF)5w`BS4;2GOpolB1TtK;k#>r|BG#&vMgcPd9P zL72jiGyO1uGctmFx-4s>Vxm+kfp&D0MYrl2yl$a?_Oz!w+sU`_h=r9xkp)qr8y3j~QI!Ejz)owYB9X;sfyz7!y?-NE3s zE`1L8X1Pjd{y)e1*8)@e4)PN;X8i}Qj?u2bO8(gP)eh3#Z2{(boIQbl1`@GjBt^|c zbYw=$SX{l)P5lq1RG=ck5*|&R?kKFm5qEmiLVkL#-05h;z?J4uytWl{%_V{yEdJN{A>njgUK1(VFCHO-iN>plfu+*x$qS~+pn zPRZ!`#*&6eUPO@a5ob%HbS+#~+$Ed5SZp2nB~8(S+uZM~+vzaHHZZ+9o!v@)bSFe3 zMdA)NHNqntFDpZGUUw{adF&;aDguz~Ev|TR<{X4O_OXf)Y2PCJS%agF^&{^|`Tn=kc^v7ExK(dp~H;o{9Ri8xVAC#U63d_yT%qVvGN7DJ6w3mlob$uh=!6 zx#V)%vlqL>MZ-;+Hee(eZgOF`)CwePeKn6_>F!vXf{u^v4felMzd$)%a8VdJ>p8Mo zW7Q5l_0kkIuVBcXBjIqsf4OPb*`KF?@H0DH&L#I!5xe$EA|Q0iuhYZ(9W4D7k|9nr z9&bd^zJ@~)Su$xq3k(l4D@d8=&szt|V&=;e(f?diStr<4_t0KO96+}sR4rlh5nY!wc>43dX#T}zN1Cj(YgXUx0;&TuN<`zSu0bqY>0v6$zkWF+Y_<7Bq_rF zy*yFjFG_IELE8!cL~er!_<3xW-y8CP#a82heKpH|m7supN#qKOWAQGcx|oB zf!Cgy*R^rH|BF;4S@ZE1;J2Fv?*(O`u7mGqY$FNWv=)fX=u&E$5(bOU*8R{o!|yL= zzFNxb;iOZbGyCHuD&N4%tDy^)>qFAWrWbGeGZPbUsHclSV!MB!!0KJhEfHXI=qHD9 zKK7v{Z6)(`zx9E2i+iT$=3ygm&$sBL%03&3XxqH@eib9RyyVI7srq2I=5gKJs%cy_ zXWQ%3+7jDOBJBQ)P~X(r*#E2bGaa|8>22HQ zY3|LkX_OshXppCpCp0n?hVJ|0XAte`%R0w!gBuM^I7tpseXfe@)jT4~{v5nH&eo6t zrqx?P%Ccw>zz$#dsZtzOX=}OV!Odq(@v!P!r=guDS1roNq??I1YYY=ZJG^Pht)Nx! zNVqr<%p5%p%; zRSMnaLiYU@!ih|bs?WhF_9XwBjq$^Q7se^;sZE0Sbx+iTG({eIE0q}rsH-Y#U zmF_*=RmJDYib!Rt9ADEY#l49lQ?=ywOcX>M+mz_|%-r*l-1k;Q+_q|d%ScE_^q0OQ z4i2SfTz*?#31zwwo;arh9{bMQ(lll4{*K~lVIR!y2n1c_{b>a^O#I$ zn~V@W3}Wb|j=@zz-&mnznQ4vrINS~91oe7*?jMy{7uE5d+VZ-WP8~UKIVb#+P=^OY z&NXT|ZW3I_AB^S8G@{;4c8WzN#B1+ypFvsov1Dp(+Tn|zA-qCVc~$T*@N(zKy4AkHNL#wTrDJ}kMIl?7 zDeNj&)K=r2j#i3qHn+$p_4?ag_*%+~$S2R1u)CQ{+fkg_+C#e8(L3WKk9cq($7&>X z3~pAIPnX4AzG!Mzk(cx%6YJPG`k%dh!#0Q<9F>t&=&ha>zi_0bsmbM;tcocA!QNa5 z$r{^LG?YM@wtzm|O*G_I=ssBKm&y;@^zBVwppBaGn0UIxM~mbWUTfdSbh%6n!vMs3 z#bmxeS9?xasB6+QRqe=-tHdcg(7boI3gWv`{isgN01#R~aLZ-JPAH0cO(1Q#srtEm3R{yTNM5SV+Uo#Hhyi zR_D^v7FtHt?+EI@rcd36#8p%m32rn7R zO+3!=WYq=Y)bnZw*X?@|PBiKtVlkj(lOYPmhSC=wsyVq1sXeR%+B=g(@%tlE0O##h@fTERr& zB8sK1pHa`P=$E3iMGsfv;wc8>NCujg;Dp-SQ)gr9Ouca?VO9i2e)wQ~6J-+L+F z0pP2eY9-L+TbwGL3v(JuQTf@ z;nWfJaS`7aCN#G&JmWJbYa|=J@} z^0u$xaw7JHIauju!%w|6A-YT)Y>GV5sH8am8_?54jw3DZ2m2>+gkW>RFX-qlPABtG zlldfyp-+pElI1`im0hzl_Bdhnx%p4XF*KoF1>E%)rCoP3}lYXvqw~3 z*fFe|ff9@uBnfuR$sQx41nZ}PK-&y|ZXw;7+RU2D0=yrqXhwJT_6GUzBjO>wm1PG` z$LAycqY^~6b;-d4LKNeL2-Lb#fkiQXilpoH#~vPD)JQH@7Pd+gS_Hfy>)Fx-u3z*q zp3+KF^NSr8a_en$(b4NWywoFO>LQA`1dEyU4UEIJ-IXfJDNb1>-jB09WO`R?f6SVj z92J!%@zC;t1`i;L-mHmllQHKJflUGrw+FKKUy(6Ao?wh^@V`pT0B_0KNt0_6{j>P? zbf&7FN8+^axG2 zAmu(j0X~DPe#{@_m|pB$xX+K*{xvnYA5chwJ>k&`+`sX|b*_y?IH0&l-rDHQV8`(R z#O;7+N-g1NX!|(RifQtXJp~Jb06`q(RXo{(T?dD<)M_!tC1PQXM31`}))Kk9LQ84G zjK@(F6b(xgqaTtx)=%Z1v4vS!Q5cmuMjF|VEd{Ux`OEv%WDpR1hhXjmI@ZD#L;YDp zCVTN(m&3Ih@3+D#TBh&snny+mV?poRs%CL;HpQ~Oo;O$tPOib)H{|S#>Z)5}btdn; zNyzkpl4I(JCQeuU-hWK#x{ypI5NOhtpLY$KL5?yVV>n+LZDYPsPz`W!H{KQRi~$rr z#LHJrR5+s{bq2-IIyY4ZvAT+AIb6Wr-Pia6jJ?i&5x4nr?t_l+FZ*-qr@%XY$}p2? zOYo);)bzE>_d7i>l!W-$KDxQ_X&_xkF0EH|VPypsyzS3w?(Eq=+?UFbxKjSMrdBMD zSQFN=DN)VJLh=3?rrLIloua(5HdpG{A=r8(R|v;X!dd)FkG_b|UTA?bU7Z7pTjJ8@ z7qfh<#it|@ze~+;6@5HZ97S)Pa&$e{TRHp^(!I(D&r8x|amnGhr@JFUkj0?KWvUCf zZ)(y4^e#WF2Poq&%{4c#5jlyHffAvr+H#1^Am(-1Z|YUQo|KTHx%*%rWEh!xGVI+J zOWnOcqzT9yz<@r>$V~HnjQg;G!c9%u7e*=}>gpZmzM`!&^9xU!+JS4!HM#PW;<_;> zDoNSx$&9j_QNwWlM%MtNN9gxvE;lRm&Yp!N!^w}*ksk`r;L=fTrhjaxx7a|#MY_0b z)>lxslEk$eZw##-q;D_i)EUGend@&X)qFC&fYRDxs%fjN*n8;o523E5rMDkhpy9Oq zw1L5HZ_i10T3+h8*cyK}z@AvbJl!$wYy>wfDh#JSk56grnnK~Ft%OgwYdlyoQ^B*t z_lt#MJQlJ1ERFvw8>*@!S7FkdsdwdGj~y00W$bZwp{EdfTcf5jp1tuKu7)?9m-i{k zDCGm)WpE7H<)KQ6aa1wg)-=pVA#2EPIdQXr@8<-@r}yFZ4fLqcg;BBO|gmc^67+8A0{?YASq68x$z^~+r$O2 z;oht`iIO)xPr1K-6>6+mSy@egAsc$t&iT&T_n=#!2xV$Yi)4VGc{naOMgp}MbK5{Q zYrB|r$Z=(P6rNU|Q*Vs6A|=&KT&Jz1^pUbTizFc2a&*ac8h~!YF`FAo78E&-KuxXV z6*(b18&3N1u7eto-5GMcz5mlNZ)hKKU}#E5R&r#AuOuxe$DiW09c))$)odJk9XS=xz-9htSc`!gbnVuKXS0l^bPeARweS9 zBo0CuFcF>16(7+~#%@{9{LE~D_p?9VHeYr-l4*WEuTzl&iQeV`{3ocVS_W+y1(&t$ zeNW77+T!BgwtV}Vdn#7?KKzOMdsvs9_7CL?38B)%+hn)h$t8>uK@MYXZEh}rGW#}t zAK*Y1k*I3d(V};*H?Y<6SY_Kil*`l0BeVjg=_^uq*i>Ip>)u35h1aFr^{bz-@vi#K z9Nfym^-^l~obU;26=Ps8D)x5Ij_Qyqd8Jp#TiyWxv3Tf^N5Q`T#w^gNy-HFSs` z_lr~J-P#wdHpo#`sX^xVHqtY~(bo-T#|O;6h{Afz2%gzVMA+L%uOzG0#z+rMPko_3 zij-=!MZJ(~Oo&kG9p~2Tafj!T033K{qb0e&cfx^pyjh}{+_p=lH>QdR&_I`kivx5A zCo2@2M4`CBM8BakZc=eIQI5%qd6T1=Z1|pixWW!w!ONA{pshG{Qb3mBX6SVax6yHa zRDZ6p#qq+pf|{T+hc_rUkQB3zXJprh-L`AGD{OnY=B`IwWzyq02JeiO z*Ma^${h+$x_5%*nnS#!-RI#R7>-Lz+<3$&kSC1Qmry3hZ?$h}^xD4wRXOSSQ5*?e; zQP-OZV8z+33McQy_r<5XZvwIUyN?qiGq7tfm7#_Wo_1T+=L`TMo(JMf1{u|jlCWEEnqB+Y1uKCLQWydFX4W@Z zbpmh94=>BSw1fA^+T?451DbEGkiE71P01Pp;e-+=v{Hhty^9xryvZosFMFx*kmUuX zeYLg829`N01TvOR_2FQT?qD5cM!1y&T<3_3)vE|LqV}d6Z;n(ChkS$pXWJoVMeu|# za`=t3s4H#9IBdAU5&-J6AW#VJXy4#F+3cO~k22Xlv;aNaql}Q(hD(ISeRsY(C2f=-J)!2&N{Lg-0ry42uc zCZ`#molCixx>E3Byaf940<1)No|B2+VqHOLGGncyDxyu_ba<zNOXFw2FB~ILpccwmZO6i};oEQ`N`JMI7N#>a zGSu84hHqu-Ee-xUUR}njno`O=~m`1CAK-pqExwXB2WQav^LDh_Fl3|#d6Lvq;U*<)iud{#QOAfWVm9XH*?8mD7Ncfu=_LT5vcr!rULSUkvlbOVc1uxzfy-<$Swi z7asaXu5_UuS~>rj>4PK+iO;R@YZML8JnzM*WH{s5jTA0#>ejF2kWtlEX|3->=UX8lmmai!PG;o?f2D|C-CN7dOZ|D}@^+++pj9Zn zaIBY_R3%xg0S#H>2H6MjgIx_#*#Q(11{>NIt4>?pgQ%Ad8b=^Ub&yQ+DkCYUD z+Q%jXBPUWs&UHHnih#!j%9@|C;zaw`Y-cM4RydKjbN=C57zk)_^|QH%^T~^f4JaW+ zje_p^iXVEt1@}hMOSvloZ;$VJL-F5;`E|NaQ%jCsG6}g+3<`tUN7RT0B2kXwD{@WpKqA+h)CVX%e@v z-x*9eyd7o$8a1SYn$R~lfV$GkVh2&x9K>{q+`oAU`~52uXKgK1eLr+da*E_bpdxP ze77I2U;8(Wi|Fet;*ppeELX#}hk3=^y`w{{_vIThWAn{c8B^9x!# zvi(Ci`$9p=Z!6J)kOVtMXOjB6tsjLHEGkXAXcaed?rZy;?Lv;%FMYqe|sKA7yXog{5#p2u6{j2 zY*1BW7cWh0&l8mUvu9sw@LU-&KL&O}esdHtmF7a?R-Yd0=d3Wj8OM2;HW~F#)_cM~ zli?dS>kEk+yq;ryuWIs6Uv{eOG}|(_3yjBzUKLHalf%F)s%#UByLWGqS=+)4RIxvx zcLbC)Vd7!tbZ@`5+`ipu6Dn<9f7OcdgIHWqU}>qd%?kLQ0sV9fi%%g_D(`Sb01 zr|1?7mK+n#9WfJLII(cf3A3!7gAT6!*nKc;S(S=Ve4T%K_~$;j7<@wxr!-fAKwe^G zz7j89n4-!N8dh${JuvR$tS{Y#TkQ5%=3|0?bHTitjh5%^Jy4VHm8?Mc}InyiT5~PFFIq8_kji3qA(rHK2&lX;d63LYKa=L@T-jTSPa9uBni-1N)E9DoO!*a;8QuqC(5o%+qA^ z)JS1c%xprfRcIrTSEVxXe>uM4vL;wL#w)YGEeY|YYdK|R5vG_ z*EAU^u&}Sb=-GGcI*({YT@Q*L^jg5egxB^egbjH3+QG7$g~y(~r>74Bw%yJWjV3ABhqk#zIg z7wRwbC@V1mCu>Cmx?IKaddE>;(3ybVHD?^Udpz7+1i%Wo>mz?iPvE3q55W$mvB8jm z^za$>#eBSi)F)u@;~vn75%IIvQO6@M=eMz$pw6F(2X5}j>$3E0d^iDQvG;+10^*2K zIWpShA3m_ft>@6V=&>?6uT6Ib-3_U15gg3fd}Uf3fD9(Y-@wM%&lhHL|y?=I|vy286jHO4zOLRG86Mx76C1 z-1!{+7*R*JeBZ@wg*HSy9mW8(nbsW~FBRQH5;8~k9lw;(2@{{`)8CMuK@>wvh99AA z2`37+%L~ILihVdVeB9?^>tm{P-LQii$+oc~Arxk@Md=bA00P%Yc(&SDh2NLeQ60F$ z)7d_|;5cf>fB>$cg0o$pnF_PB0dDIjGeNdd_cA1P0rH;=`Fk2w?A9|ILb`GHMt5R7YUOi%Zu+o2R( z1l|c69zvgVCxie^^ek^!a-;UIlu6~h%#6weU-dd~Sxpx8v$R@z_Lwfqoem{%8Z%!ko}@^s)0XgzlsC`smT zK&?2(r7%2h*)3;zVX3O$^E$SdFLCn^>RNFUR&ZoQt6e{E`;PsFpEHe?g1xx)F|B`0Cx-X znz(qlh^Cy;#&Vq`o648>)IBiXNK9qyFOr6UFVJ$Hx5grhfT(B`_PYML*`$;oV_=>? z*_|)k)+-dwd!{!)S^Zp}15fZtvPBFiyfd?*wx=1)F6L=`iuroJJD^5r_yj*W?o%Ex zUEt01I$?~?7HB-(!|CNAV~uisWX#GR0(JB2@&^#@yguHf@aS`fP~x|iiamfGQ>sIn zE&_<5osf)A~^C`5CL`JMQ|s61ahh6&w&tTWMhWL%IdprLLW@wFzb z>7Ti*=N{40WA<-g9^3CFI-+@Q2GZbta~WxpRvD|PqwN~a-$|i&3S9ePwm@2YvREeA zR=47DkJ`if>Gdx*JFjq=@ab2cCcRO2TyupJ3kU6Rrs#RJNdd0T z5}`{-(XtKjtd99iV zP?03vHws|qC6W|fm#$;_3i)^5oQ-VGsM{^=0DY2<7abv>_d$`|Vphg-0pSy@*!JA% zG@WdM`5^Ddt|QKM**0Be7M{4T_*1l1($roNA>i#`TxAi)_w-r` zV(tV%g;db{`RP*!i>Es?19jnmYrw~hmI~)-8ZXqDF7)=}q_(5l6pP`2c3avz$dSUy zBjC;GVcR5lv0|^G&eTq`2kchsv39pKW?2aIJzKbKx^;n_dg1$@Or;u;RrW~2cwo5F z6Yty*x*(Uv&eu!+xg2le5T)8d&y9Hlw5V}b%YE&qbp_2Jbla_X^9+yFmv?1#UyEe> z2{)CO*tvq%u1#acR~JZA?KdUX-?`8Q~<%+*SRO!=L zb>@DP*6}*J#13$#m^-VCTi|K#J+#yxJD$R68#NH(A_!?2>`WLsLe|sWbBODqw4LL5 z`|g@&y0Vznqk#?>&*vn9-P#;B$G7!^Ju7xd&4#B;4X#b^gNe0{FK&4(y^iuHkdO(S6U$D zEuZuJg>y}}5vnos#yIK=rsWGv zz|~P{E+hBXS#G>m4;jxeQ4M~sB`crAD4u;&>zP0e;08iAI~uo2-dN6QAy!1 zlN4yKex^RzpLr=n{`*4A$f~SpaV;QynA**YXc2)=7hxZ}{`N-0Vm;uccD8!hPc_k( z!V+YK!d}o^LgS^aimysz(s;c%Klhd9#)sgDxu}F?ffWXviU2-q!p1xAZzs1BOz@a# z97*3rM^A&ysB1`)P?$?EqnTp*Xej!#0zvR~o65N`d1~(}>nUk3YpZBGxkOoFq?`>n z^74VBPepFG%sGdvY65@yN`pvPNELrpy!_=c)Id)Qlb zP`muxO8s@kLqTQbh|>2ADSj{OOh%7WSPS@5F3RUyjy0+Qy>{H6dt;ALXU3MWqNmKN zJ#xQ`e1?~*twVw$K3tcmGJE*zUc7CA6Lg+lkCR#-i3q-t6MSZq=T%kPx>J`kQWd`+ zeROTgFGd?Nw@b-<3BKbJm#9sO6%p$xlP02yL;Q_7FG=>HR!why#qmk}UJw#x>L?bt#{sCzMZV%%Z-I6=rxY&RkuI zL+T$|fH|9CKo0fdpUe&{Kwa`L7+QK@t_rC}?OGUFQIw*Wcih#Y-O&q@y+Ix%oVMdr z{{w>zw(NAAkF=?)WA(c(BLIlKb51j8wPhforCG4p#@UIY6=kGrnOtjmVF53mhX1j= z1KQrT4^CZU)_XKOAsvY2&2le&4&3q2v){T_J{rU_F)?A|Dck=H26mrqC6ILt4i=7H z_e1{c4Tg!ei(g>Z`!+W+c_gGUx65PR)0OsCq$q2L%ggUsnkS)RrtGU9sce zL=k$})Y{&Av(KX#{~L7m*Cac9-%g#r7FmsTCy=-`8(#YXeE;;e;z`6s;Mbe@8Lc}9 z@4eylpcsR^I0)mhvb?~iPP-K!FW2jyJ-ZVR6!4{1`PtH#zI?`v^_*$n_0{17Zfz~M z`oomkN0HM7Z-k82M>yW@&4WHtB~jpcWgPx=8|$jawQ*`lc*lLBy2?lgRma&|MIqB! z8LYPU5gy$L7rj@W8xlb%VLh^^mrI^PW?x89tiH|MEP7sN=lSUCGxB?1$h$}72t&%6 zjXjArqse;qyRlDWM?CWXM>FYm>IG6>O&>rTlbPPhsm~0%KZ0Dri}UOE+S&TiAtZXv z{gL#5^q70+Z83YH^Sx_EiZ_`p^71I|1Hs3h-Cro#&>ZUNJUHd~+SVW1nx?U>$BWXk?&j z`}cJF@Y7_*>0b*A9X03hQixF z#|Ghq8w8Y}{jjLh({St|>_bS2!cXJ0sm2=dW`0KGkx2>FGF|b}0p4y6oqCJbv7|W> zkOqeYZ>+B?GBLk`WL=`IOZZ>5QVp#igoSCk<47;X$Fk-MEyvbwc{} z#{hj7koMwps-5cxM2Am@c(?86^*p1A!3i&yG+$2^$}rCMK#nGUVifc9MU5_(7WEDF z>(?$qqJ|g#gmpz}?rqVm+TD(+MR9a@Qe)=1GT+6wl) zshd_uqX5@09RT&`M^@L{EmkM4mgkMUyWN;u40khlUaQT8H?LuL-^Ev629X(6OS()? zTY~z`$BFXffyWC83~!>G|K%{?4eA0O98UfbhDkxziz=HS>U zKR=6P#=7ianMp*$>c~{Pyqrx&gX+rH=XaG>noyhfT+H$X zvcFw$zA$7=_tRm<3k>3Fj5!|zieYpcsuy_#o9Q=%pY{5rPql%$f?j^jonh9*WtRPg zOE&q%R{sh;%rszRqW9|FMZa@6zu@dErtC zO;pIBduy%)O@4-d2{b@=K>sH_6x{1-LU4Yw)<{nLzoV%icX7afr$^6Vm|a2I{*^}L z4E=E6gGTlaK_2k%(?;i??7hES>i6~;%5ggARc)PAXHW*G4JR52Aize4f8&`)w?|x4 z75*(KtG*teh1#d2q~v@mYwhzh%jhUC z8Fok-@Fv@PZPgV7x;;M(83Tt)ALDq;seZltb5cT5=!1jHH3#UFqq>3RMAX zTOmGQLG63tpQJ=M5s#CHhlf#ib`l};uXni%i3uZ*C|11=!E zo~>a!$HWVv^gJH1cgGqCEP1Rte!h5Oki3bwceL&81Ou$q6#8;*US9Bpzj7uomU92^ ze*wq^1Z4jD1!MFfyuE`;(w<+iQ(qSJv}Abk{rv7EPjC9A0~)@4YAR;28WYyvzdkXV zvU85SA-B$6!wDh$3rtbKv(VOk?wp8u6nI$C&|)M-2TcVP>~C7-5tGsg`+f#T=ORVq z`sRasrk{|JT^pO|ZBK0(Rp~c|mKt3^->R>iu8yXVS9R0p!HX{5GJp)6u$}d*x_b5J z^V1?Z6%}@1gzAr<&CP-ryi0`-V2C8$Xnoa%WuRoL;%0mgJJDb0xxm#H_rmMe79a%; zH(DtR%V7367~YzglGHUis%njlj4o-YRLQVZ1?0JVxFb(*9u`_%#ToO!@%NY57jt&2 zZ=wRnwS8i84D6+7k$PYyqSL=c7AwTeG2()^3FFm>P8Uw^`S>(OX>!QNdlX^prdqWp z`6w`mM#k`qxr_ln+Q(6}_nOZuPkVEpGN6D4!wt1CE!g(7WK! zyTQkR;4`|1n)mvnX%PV_DU7)?sUmx4eM$LMy!ID3scETf))&;w6spU)9(s^)ox_^V;AFd~XP%=Act(-{Mr> zEZ|(fwd7Z5Up_gM3#{^9d3hO2wxMB_*5g<4z$biGOQV1%3YZ0X;@To?39}#$tl0M7 zD_9yLqkZj1rdIa)^ifDv{m_K$PtdBeM2Dap1^oJ{<=XbN-^%SsD=(1ZWGx`DnsEOc zno*OX4ClbH_jUyW%pn~xk(8x@^R-R@;& zWt)rUanI7%*R$%W^1wa zE&Ck%JSn4`0AE_ze!T$_29ap$EbiS5nnU#$R__AUJkSF-9xP1L&JtgZGrs>tMl47; zj!YPGUEWmg?E#)i@P>Or?Ey?E@&>H#J56y_U0s)x(o3YcL;Ujj$qoXc-oOesU7wba zdf&$UPYhcx?*t3i$ydll-@~eFFeKbxhHwTI(Ubj{I2b`axug`KbI@cy)mkYEH}sca z@Dm0q;&)X?Z-aQCD&iWMZ!pkMMvFhX7d%EWb-4nB7!lZ6MU4CJ8yH=d~aa>rZ$+O>A0F;bhn=;n3X-$+ee z^mcMmfj-0cxsQ2qaj^*b`+Jd(hyd7HE>3@_dp|L$AF-*DLdDb!bQ5|3)5RQ7Q|fnq|x>uOAOEbQyzRoK}0DS7<@Rv9Ox1p6|+bQ?FYN%8U;8VL^&h-N^^R3 z9(=!*zANKY2B2xs_4G9-0Qs%Pt@s-EruyUix{TKQ&;1%WwtPxM)z%k+pUllkicE=v z)j}ad_#A9l7U*qiWl}ZjCXXd%N?zq^kf|`#1JemW3y0(X{gBWs|jc!TL z%#*PQ+7QaGfvMrJ@H*aGrE{ZKEl)j3jW?W8Z*yfWruD$5tn^MdUbC2{$AJ`V78`2B z=`K#-6Hl-XsDArTDEt%JvX4-ituhVI_p_`yi-^icFFLPEM!>pf@G3Xj6l`Ce#yxCZ zYo9(TU2Si?eXFfStFdB567z4>ahAp{g}sajXax}V+^G$ymGBDXkb=Ui1~D=YL3GsF z%i&o%P3^d0Q1B?t@8u4Fk|(^;O&Rjc@{x$s`)a4G88YH*FzO7rEmV^Zf!(ln$B=hD z2eIRw6s))JW^@ai)dvPSZnxAk8jKvE$I0l2@N_>ryj_G_E@IELs2bIy6Li~a~&C6PwR)O0qkr{S4UpXA^u@B zaFCjbKPWfs&y!j~2_84_VlW;&+8ih>1`FadF%|}rv<2Os$5Sv4pl&Z3!nKqdzfdqc zSS=n+*je#!GYXP}RAgJC@698KB++1(h`HHOCHj+K=oY@Vo&JGy$^PGlO??!msEfs5 z*(t#>JM7z@i?1JEuMT)$UhYDYS)3s}?hb=}J6|&u!8(DgZ8*ZN+!F+pmviv1 z={d~o#>B@A=7)u*lCfIVf;NLl>FlXqLfq;z)q3P2HgM3hLXhrek=0%v3A?fH zl~Zsgnw^4N1$(@FGq($ml_ujIHbViJ|sOs<&V(U&|-wEoCU?$!JP z(D$6GcS1%akg;8tih@-uCQc+=XIbDi=_4eqV>1{78sa$L{2b#A5yRjCl+(S}D zewgACc>}Vz9a$r4f?nXb-dDoDKmCUF2BgGfdWLVkC+OKfYh^JWgKc{nc>c_6ru~2; zTgsK$_6bpImVpj5B-G3MJJsrTT>zm(Qq*fzj(Tz`MjW+4hY!^%_4kRposg1?%!bOA zNi>!)%bib^qgMv{JWGtf!6LJ`u!#?XN{lT-6qu(`y>iG%hH11q$jgKqDmVFy%3Jd- zpV!@$?lDp+CS-a_g8pOG$Ayj83pw`j`*Q83uRPFzlSm37pWrm=^sPaOQHPgL)c#88 zJ*}0U{zM&KL~voliw)L9>%J$3CF_j_o+!3n?bZSz=%6qa)b+b(V15COs%OGrYS*bc z>~=p7du9-UMfSnKv8wFPnm-m1W~HBpOvRsmflVi_E*2t@aTbuC9Lg8rb~yyOcX+k8 ztFvHs{p=+x<;!0_6KHWR;qL!d4zX)e?YvIZ*pG{O{s|2FKBG;GiDmso?RG*^xBfJB}o0W}010 zRM2&$nM#w-o2q|U&0K3W`SneZgHUT`f-wU^L+^JN|0$oFxI+F^o~MQLkU1}rr(5k% z?bu!!JzYmEKGzr<^mY6aD05JNEG;vde~|>^qt^h=FCftD!4l8klfrL>yrd*>Xd>7rB4c!ggNs;=+LydgD#)BpF@`PvLc_JqNb7^lN0w1Il zfr|RK@+LdQ$oEJv(I6OtgmrXyrfa{<_F?wI0Q!1wEViL#T`@fD(8ox<++&7S(>7K?LAb5%$StX*ndMc?Fi0 z|J5`-#&0Id%KY=z=$$5JN89-7YFqf_-dXIPA5CaIJA^IuZubxEy|aB*h=KxryQ64i z%Ni*07tSR)G6ZXSh=g~Stwi2IX$L}~6!9(*i`!afZW>J4_|zMN_g-E36s)tN|m~F1$D^GEZ_=Y)+ER}?3=*~ft6t*a?Za=%kOBG ziVYsuX}LYsV(9b0$yxb?fVA#OMR86P=EE-2g)l?8WBBFz^4k zd~tT}|J3?BbD$08`}XL`?a7FnK{J<}>9UvX2HY4HXIm2MP!=3MWkvNIc9#$pHR>CQz&-fs&{6;c4>|!@E7x`}$!( z-fe6N>q)^j_%WKOFWjFsb>X_hKT1m)vGzP|n?oZjFM^W(7vD^uTsL?EAL zKImbvWL+1`BmEHE&hw}mIN{4oJp%gJ28+^4RTiW#-Uf~C7U3o;7HgO;46lM z#U~&DwfV@Y{i4+5DPRb)_|Rx!YFdoZ;8M}eMvT&KlZMDjuAUw;7ONL(d0eiN0Vg6jLBE^;n689E$%=7S zX+8VZ7BnYw4xl`xHJ6rlJsES>r01O$U94wLI{ZkXoF1-dcHUH__co02x3eaHV|m7Vhkz)!RC( z-r4;14zm)}s|gZM$y0~vaIoCdmR#!DG`S?=$>)K~2VVp#F_+ zWz-%W9UbXHpIQtI47!*TQBkX9bY3Y->>$s0ljHm_?Rpmh)tTuJ8gwfaq~MsX^b@kO zvR8c@Z=goab#7NdXnx|GDEHcG(|aF$c({I$Ok?u!!|xWlTl9cr&wJ2nBRNKFs?inc z64>$ijI!!my6cU+*TYidV*3R$#LG%g33gIyz|&ULM7SB@;&&TKF_pEr2(+lr8% zI_9QK7d1i)Jrfg>VE{BLrGWMx&99Y6#~Uy(i0@1UDPBoXn-%(Hev z;1mw245d|OL~mR-TPHU}S{r;gxFrAMbJr9MCC@qhZV+(%eWPy|9)v9L5DDQV9UZIR z6JUdkR(_i#?3*q;jSX@HUzOVO>GE+eu9)655Kw0ez3LD5THD$NMU2Ti%Q~Slu9JH$ zHARBDIJHJOfXl9%+1Hq^xx;+ghnbY!UYXDUjHxS`;o*^$beTKQGf`M#b+Q0WFr-Gm z-AGxYWnt)ISAp6>r5HQ~8BH}IF=#R#T<*3rhvl$&$#MyJ=eiNB)cL9c*OoV^n2YCr znKvO>x_Ad;xnXVZE4@%lu>N={$N&5g@Rccl>x9VX-GwP0e-RPfhTi)t$7bFP)@%9= zU>lSG4SO+2me+3j)XaOF5#8BminJzX&xd+>vn`}f#C_7=d%uyhZj;W95iEtZ8hI}S zOoejr#3z@e4|hM_BWMMj^i?({!MwV>WgPQuD&4pK+d}SFDBm^BE`pTJ-7WMrV4{YYJvL3)^Rw{KZM3p(f2@L;Tz$DN)wN9#sOTP4qQ1~I68?q@ z)tDq+nvs#go5bY)vdB#?a^p`yu8H8R97GGGr6~`^V)(gefM{K0liuzxV3g_zEDvgt}j20l2|*Z>+az;+LTMSw`bPkEI2VeqS`NI zeRtwA@-N@>qm=CWlxc_w3-ST!i71Xkb_d|Ue-Hn`C7~CJu`oZ@;j(|UvR%R$ItY3c zr}heBuK%hPe{ghtH9LynP{Xd}dK;057aIpBWwFi~FBqW%k=|cSDNEC_6U8D|9iLgL zz0AbNN;$2V{;}uX*PkM;rx@la=O<^8-Y`fhB_72;!J!D7P8w)gilcV>Ki%9qNRb9@ ze!W=Mfzz?gxOa@W{~-covAcD+GwOBs^CeND!4o>f7a+`S>;5c;2HX?(=g;UsqQ70T zv9&#HMGNc{0&-D!+z~Ch2_b#pM?nq7%5;aa@3_aY_sZND?;2P`4SZQ48V;Xv{osKl=<*1RQ*y}kFf15 zNC|DQm6@!_(vY}YEvq!y)u7+1o}pqf8JFY^^#p2F557HHY!8vFqS6n4&L}eBa5q{3 zb*m$M7TXIjHQ5=G!;VUs`59LPHX)u-u&_^xPV4mf^T~n|PV}4iEkzV`$hRF+9rWNI zNy)anXPyVWOhbHLv4qF(uU65JO1i&PcHV@c27p32#DqkZCpN!6}+YYe9MOBYRlx3++3RYJ z$)V6MhraMGk_cJwb(`WR78 zleJp!TA4KL*u(&dY7ECxy(^sjP7i4D| zLLpn1V5)*0r1S&oA`;6uc!1%1(nbwyriH?_EKq^e0HJYI+(N|jP^P`ufW4zzE*9_! zAtBCIG&={%XUQ2Ia@lPJIIsKzRI*`N07d&cl2&y?K@LmkaeeV{dP2l(0g%7!VkQI< zVr6A-Q)4mdid+R8QSzHgXaol`T@=%Mx#Pjd4WM^;AtqcTO_0sE!CLmXK38~e{(x0@ z_`-A>#8j!Jh*8M&J}noFo!kMzcBjLtkjcwE&4;L-b=g8LwOr*S&g*YghVUqv(U+@I zlI@u77cx-c6dCx0^Qh2&(UkBgsnLVJ#prR~t>pieON3#mA@;9L49=ncYDj;Pp5MIa zw|!4g$LRl{vGRa#fdX|=C-m+AlRF^>Wi<9U|AmMkBn3iz9KSOsb9$eTivEKgfB&KH zWklc?Ei6bc%E`{I8tAx=Z&$If&bqQc8?fb=C*%81wE)PNP`HOM^}IeNG37@WNc^6e zo&RZF#_}SMK7q}GahH0h`?*ZY<-ZjvAfy<&QRr!#Zx$)P*_C7Db-C2|V0}ipma=&B zJe(wCGzwU2vE_5Q7#uY8iryj7QiLRQ+@}|Odf}sgvMoscm?khN6%}+C^{ZavmAD- zhg%ODA%@xZ7lfdK2PAq1s|E+EHM{@o`biA_WNrJgwzjq(RO1m8(C!_HQK{FCIH+Xs zh}%2>`ROMzGYL3mHI-=3We;2Q=J>s!8V4xhLd?sl$Y3Zhqn5ea^$mc{S|p@>h>3v# zD)o>(V`5+k0-0(Ucpu@sbbLWbto#_wQ)`}F6{MQ*pF92w8;V{*p#2I#jmq?FfZ*oOBP!jmSSPXX|jKln=D`{)mYQY7}~a zz)49-=D5M3aN8icIG-}5Qc>0pJ;-!ZCa^a)79l|ytbW^6$B+DK?5#A%K2P! z$DmemTYVL(U*FxYBEC*FDy@5;y|-t)Xf<9I=WtU9V*b?$xZ}^|p*2O_$bX^5=9u(BfBP){+>Z1&l9KY!KY3RE_ zopE!0tA2-xiIL;|?1Ig2i9Bz+D!$EEuQFi~E~`VmCh*2trqhW6!e4WmJZ_s{ZmMtd zc8f*)QCMoK;t!f~rLTN-%+Nqw_N~%R1+ISMymg{>!jXJ3Sv^CMV50+ zI4x%1edN%%ueH1df=mK5oW)=aL#D_=OdqpKjT9l;ETYdZuWyBE-E)VqH+wr08U(z^2&X{_d{-$!+f5`Yd$A@AOviwlXM%qRzKeyI3s-LjkxciZtFr z8JZXtSzyd4CfMIhwr0e`B#i3IKklzD(ewG2wEkY62`H9z>K61KNQ7{j+*;);p%SB- zMOh$kyvSf~N={6C(yq#q5G0%kMFL2xuRajGKOXP}OqzJ@LA7NLYuVt^#KtG5r-6>9 zl5`jPp!`!+$L+_~@+=~EeSz6S@AZiu&KOXQ>eKDODW1LXdpJERDJf^fim$L?c8pkf zc+nFCe7U}cxQ|NCSDcvBa56H7lG}B1KZ*PLG(HrCQ>y4GD}VD*Pi&MDm+Vf1Rz}Ek zsm-1ksgN#C@Y;enwYnr;=*JZI#ub4`Z65$2A|~^?=YFO_>cAEJT7|kq|6?$UuwZ1A z>3_EUQ7Bwn57VfaU806{f!3WCkTr5NmpjK>R{IH9>GpDqocKEos6GQQ&x>}hQxlNS z4ZC|GkfyNxetPsfOTVQ_xHzOv=-GyVV@)3=# zvi+=R4(9NjUwYQTxfrmht*#cKAPu8xSKNxt5{k;;E{0{cS1sAyFBxG}ee4#KLPkzz z!4Y!VhGI0hRwL8|e0Y8brHs2uj9$xBI8dsQ*K+-pijsnMSR&|J$)uf7&_h0QP&LA& z|1<$B39lL^g-f}5!Q%vw5&n(Mis5`WKlB2lWg>IsS5?~Q4`*0tPv3ofnuG~AfG~); z)F?&*g?ba~Mu>Z6FCgmxu_*s<#Kf?$5PuGBMX>y6OOOHXBll(CYKjnDefB3&0PPa@{`Sw<)PtYwY16h=p~H4*w`nra?`~P4;4xGvJ(l) zD<7;%;5k^Laf&%|@_yC)4ZW=7N&$k3p zS6_7+;TFV1#}|%vGT&p;!s=e%CNgP7p(3z@ve-VK8+|@~ijRc_B}r@2N@G#0#x~6# z7)o?CV>IVq0q0W+F?%6!aN*f2$T739w&s*+!e@)6qpG2KgG#Yd^GIBxndG9(=K*kZ zacT-CDnXx6xnQy8;rMvDabf9B3gOYEfi^v?uvp8hgW{RaL-XoX>GR~2|IX>3sr|tM z!6WL!^8qJLaojZJs*cq#VEDL-B;Z%R3Z$%xr-3c)&b+4pl5(tnEVe=n2CZgMc2A(= zql~@%a^O8InIUd&U8i7&#jIpg?ZHGI+L7-w;9LT4^8Cx3{7R;GrqSfZ;TBWzFYBQ9 zl%6IE&Rx4n&O_%Je7XEq-*pN%MN>aTfpr_0P+p%_@9~xyp0DM3hTAE6JpU?ed&Xw@ z%50ly9R?YKco2SRM}tX1^v?|cIk+jp?w)M3>`qy+nSV@UH~uhhvzW{NGg>SwRd1PY zM{=uIsgu)vJU!^N-|V8SI)&DPsTvcdr^&}$FzuMF+=X7l2DY^AHVRlj~2Nkn6~{?*-kZ2;LQ^z>Y0F#cM=}mtb-dmM~GOeYKi`wpv!5;Th7Z z#Tod>5pKGaV~LGlTjiO^5*W?3 zusO+{rs@6j75b@7zEi5nT4y$1tOi{y!CS87hRv)0yC|8MA;>+YC*;i-KUAnTRa(kk z;Yu7CyMhua9C(e{VD5%G@3Vp2JiOEUWkya2G(cDg98!nFLmBx;D@uu1?x1MF=cupq z#+QQLng!&FrB|y$iTTbn3kkm7UojBZnnLANxR^&R=sI#H4@mvD&Ew1UC$p_?X@s_ijzt2P}L zViMP{{TV5y!ULIt+cA%O%z4xEoYZ{mRZIdRMaZe-n%!=-MQzwQ7~vdL--Aeft=ZOu z#@k?9oSXFZ^G0fJfSxAF@#ndsU=3?7l8ClkeZEGq@nel@n>mjU1dD^4EZ+0hBQ|df z=i(G95*i=oA?o*T{#}0%{=s^;Xdm9DiFgfTWi7)gH|51krm!KbAmq4Shi9?@nCyT zkshu>Z>#HJ#F(9rboz+UiV<|kcYjD&P+ok{Ou+xepVLp0_(%m+5(&@pFiA4;rlhTy^%yoP@}!2?>DCYXBU6#{_BSqU!k3w=#3oY zpzhGnaLt>2ATA^4N6Op|B8Nz6TKTi_)~AvAFaZm2IS#S=lWY zb@ZbcW;!+JDpG4K`Y9=V?0%Uq(r@$MS_VENRb5tMfTXU1CpfZVSUg`wwH9({z?t7s z?W;1JhGLdL^a52>cQbs_(2nQSufaf;UQ01^HCdi@;RFIo|Cj5-B7u&Atn+SS9?w;FPi9+%b1FYfDtl+${X1m=XS?LamsZ_%P9QyUvaQa z<~Nv(8umFLC-NcdHdtrtIbwHBHF}+DO_Zm{?}^4^mm;+C+tUfwGKw)-CRRi_i>nJ( zrbjHxfw;N*qxLl*)2N(v!UzH$sMX5kzY$mXTx2tavUyX;?SSSMs}lkP4Jwlg!uhkt zU>!|XOk)ju2Qd?&=Oec~{$Rj90-FNvJR}++B~MJuR5-T|8qIEGkcS+p2(5J;kK=bD zD;+Fk8T`!{{1~#@R84PR9><9jJk@sRREVGJ38`Hb+2zKx?}kU0n@#cda?F^@A^Q)_ zpFQ_@_cVZ}Q(vwpg5zErkb@BDAt@t$UPVHQSJ_Q%>Jie)?Cm!SgFNum{f?!k1;GG? zwIP7~d#e%cMRjXQgS!yfmFLi}9RudpYClX|h18#ncdVZ#=SrGr7??e6@`mI_2Qptc z@EUsZl3Jg}@D$hd=#;P1-j_@~et)%G%|<*uM9^qtB%ppA!<`fj0vRgf2>W$@E2L?$c-jltG)KDgd{et8@Se&qL<4x>YL4dd)_tvhd zr_+LuJ9Z7h_-^SuUn%SKp)`{fgVR%;f0%-HrXTx z@@P+1mVLqAp4j$jdLiRVcD%uEPEM-#65LijmV52{*76Z|`|{U`KdKxw&@l-WQ!oh< z4wBTq*9JFBEku2u(TnM62%l|Lj<7{jn{}l*_-EN-hIg^uynH6$k!qkhn54O*Iut41{oZFabG?HtB+t67+N^esPI@o^Vym7TD8O^m`D>#d0c$SND}?gr+msvGiEIv6%IhIk@OfY=E1-(C_>)zjxjg04iH#fz zft|#_CW)gE$oKFNWQ6U!1)&YaD~~REl1N(%NMva+1ZQr=EL-E-wQag#=X4g zqUG+dbCH(ivwbMIoA>5MmVOo7SU0c`ov`GzQ*YLgF*~XQMLuCavCWVN2jPz3H~Z*x zv;5vn!2cT}xWLt(3D5%c(^~!$$M$GeWEfM@(aJ9L_NY_ER{V8>P?A}H#BdoQq_nKSZm^3-qmu=vcWO@ z-+9p%gb?*%cq1JC@Xn7v1neYuhSeo=CAP6$GzArXpI*gQ@RP5HoqLnCwZ1CjU+`*# z0}K0DAMs3F`x|wm!8))bviu9dyV7B$>tLL6X}tfL_kSpaHai6BS$D2|w7lYfu{sDz zg3i}YsbxEo`=8$mBUgQb3Viu^y%y8_zu%LL0Y3GTilKNU|34q-ibX*lucX)@NJ(2l z4h?IL+_dIW53bA~JaQbL=W#~I!BLQ5Mjcx!(0RV*|M~&s&YxPZfV=Yw zgn5z+zEPGHv9SRq4_Fvw?;c})i6IybWi;&0ArwctH{$+4Id0c(CN>SV*|1h$qDC6O zkYn3agk)lis2u6or>c)I7aeC8^J);TP2=5A?8@A5u zw8S}Ao99<+aD5}uC?iTzQ?U4i1%%jikt*)q`@PYUT?E^YoEke=)XHF|IOY0zxUvafeJC9WgpLBfb5m_%Op}2sqZP)5dkuvYZwe9Ayc=iIwAOezRM#pAJoyn! zZ#ofxoqm0N`Rey!oii}e>B<}T`BE|t`_A)J9q>S`Go5Q6Z zGV=I^lb=4cHzBMZ5}bsQGBdz~P?LN=aKXyM*&7aR;6v|5={(6kfM6`W5!P0hQw*LI zX~qJWQ^>P-pyO~cD;ZNtR5+<(#W*>HMUi*#5|rWmRX-hD5(T7njG%AQiB3i6scAV2nFdwh1F(HBvUEG z?zSMgsVX0WucWB_Rz1|W$LlyCZk26#Q4x;+R_nty5Q$sjU^3nKwuwSo9fr+k!aQ%k zUcA3hr{XQ)mD0!JSGxN#=9r)$By|7yiRgP$Z+`**gR$SBY4Nb&BpJ=XUWDzc@7hNW z*qcJ_8WRQ?i1V(|gNS3YhbIIMB9rjLOvS^JP{o zCg0$_43qm5+;$-Ex=gg{s2eT->ZVsG52)g=JQO!CLP6IN=@R*fg z(Y^`!Y=erKs7Ga1AQZjiZ6|~t&g8;zDx58 zeXX-xh822Y#rjP-g3jkEov=!H4zCuXoZ zA&`Eo?#o_s+^ubSQv5|v6vV`-hH_2ttReU!-vnyllggBL^DG(_cY!y3F{bMyd2R<2 z0()1xVevHjv9OX8u~{D5bOi+{{`q$CH#mzf~4$iXUSF03_1sE^h|SKCvyZyc)L|Gxkx; zy|`7Ri;Rc2&7V5%xf_x&SoeB3E7wUI*HJ`YN?e=Fm+PbXp_Y?&iw!$p)F_I5ajiNFJ$D%!wg+R-B!MmpIn5G}hcN zx3?cB4s*n5P)WhOJe*O9HvrKe_hFK|=(Oe_<2gJ2+%~bdj9<$d#k=8`p|CH_Uok{Q z?Y0eU=;bHU&fz*ahNL#SMT6ewngo3rxv99z8w4#_%>%PzAJF|C-1q!(yzxzut8$Gh z6YUL;#@rreA$)HR$S;SKM6lAJdSCIi7_~`ZIt|qBilFv$d_Iv_>QChTf)5HeyWG31 zn>V6Hhu;K`>mNe0yh=kW(&QxA?WdJvdfZ}3(do6jm^pmzGMKV3-eWarKqbS0c{0^2 z*cQkpb{)L=V9^~pKYgB_NCi1(@sWbDgoD-GM^T_l46BvtBt_X?f?igX=M*ykaf*$P z)tzcRC5L5eXI#&QOJ-EJ!v^QNP;xU;Ymhef<;r5i7{K(VwwrQtN;ydmYKP1l-i-}o zPa}wtvpp*zjx>~yy&Zt!dKGD-HBln*>?I%wr@0`(s*y~Qi8tprWGpqPuZgX#s_R2P z*Q>y}|H;+S*oG6!VnX+^u4*}c04jw7!K>0U&h_PNfvO0GsG=7Sb}6D$?D0mIX07qH zwYvt|bVY>lU|UgVDqGKe|$qIakeTcnW)dL;zfT>zSr=%jG&g6N$p+^Z)LU< zd|==~@M@6}gU(V-S!y7aW7r1iuwyx{5t!#{sFk&s^oN>F!h>6Xfe3?xU_ys@Ier8A zD-L+tx62Y1>Ew$ETJ4TQ2Xf>dBEiY-}(77J&cFmg% zrAfv*lI+Fj485MtZlDm^-Z%QD6MdflJZJN;>GR-m{576IO;!f3?u+2@;9SxTk9Cm_ z)z}V8)SV1rkHLy%Z2V>E{b4fp?Y*0pjHGtURI3!$o&>*3kM5D0keh{Sa1&m%`}7J{ zB;G6s`VBB$bX~^N({r!;)Ah)*F5q^sf9>b9z>sp%1~Im82*@Ao=^}9chrk-W{v~DD zAVK(RSFGP{2U7%(5V`mD9MVYb$kRs#515_FLL#ct8`%q=lqW8uV?itH8KTx!SJ12H z6x1@E&bp;rCsahV-^Q066SdW;4WSzv8Z4K}Vl*ZGN@M(JAdv6*!F^YZZ>}H8&myR+ zyM35f#p&(>&mBuHKnt%$cj?)?*WE>tP|&N0bKhZQVFSqI^L%7o{}_6gm0G=!Bm?$l zIdGPod^Gxid&1?;hG8i8nr_6c^)A}iai}!QCnIE1$VV!nN^7!-$s1_lIFZw+&Dv#J zYZ1l8gF(zAjb{6|W7M{m( zVgXsoIjrtuqHR)uuS|ej(LC0P+lE4;!j;1Vvfq61(u9n-CIsvGb@Uhxd_AT@Ct;-G zcsAJ$$FsaCwj<;7?L(5IZe-Qb$>oGAfLswC+!KD~+*&I-WNgJlGX;lkuFHl5nNsNqJF zFYus_KXTwKQt0HFnQ-Hww)igjNzpOODzZ22QL}PIUWKjfa%2&u_;z;KRt?o^#~s^u zem`LfxZR;Ju4O&BFjHGMpT8z`mTW-uMd1%Ca3UejG;|`N0p^E&EAaH0NpI5o%~ZaN z3-fnky6b*Y^%t%X5f;tZOVy=7!}vFI@K@LV2vwFzjh)p%2U3&&;V%DD7*2@T^~IWt z=&}C~g^2*sAr7+S>4g7gOn!gFjEs19;s5vapC15#c?2$3b4s$A`5c=k(i>$glk~J?tknQjeg?^ z7dA0Lb-CaR1QC?&UZoC=gP&fLsb+}3DM)UX+sG5()wff{RQ~zQ)l16i3q!Y}M;3Hk z$4?H$;n5>E9&aa2hJApo8zFD%oVk*|?+sv$CbYBIonE^Q;)K&yi`->2N&qLSxbuR| zoJl;of`EKRJGe;Z-Jifk4+%bUnV7CQ&00tXqLX9MU=}&JY$iXh`=IsP3n!MuT3vQH z^$=UaUCPt)WO_Y**US?N$_pFo;?-Hs!;Wg%3X{6aorPm|Z@#w2TiAmX*v(TL5E@Fr za_>)#q&8TEgM0=s%eMvA#^DCQujkBm#HdUiXD!i=Oqf~g!iqiJOM-GsAMTvC$%Tx& ztoOgx{g<977l*kEfNW%m{U2b(pl=K5?W39wvA&(P0`fkaacM*!fWQsJ@q9&2q34AG zeKIqpfNam9F{FkjD`m6?KasLaBgaYOrlvUh6W};?o5yo6RpD^8gTn?E$uu43XUG-O zRGXC;$3{l{mu9`C>BEIqyxd%qE+oc`qBDxl(WbXezY%^+T+AqVpVy+@WB*KjzLMMF zQOyb|L664%8|l$HcegCTfFrd|D9?tUlRDoB&j~c%POy4SyGVGX*>AW)sK510a_fcl0N7TX zC`<*st?2&It!-68$fvJb3n{zwdNrtKvPtUVu1{q`ze;tf)GKyH^CLntCbZ9yy@1e) zZW)?=qw5ohYCk6>E6+2O3u5swfxzermx*+b4BmiN_R4p@3?XbWD=1>mI}1GUD0p~M zW^}(=B-HBcXCUZNLYuv%P*$$;g12dhxuP=o3n-hNF2F6VB9I6K{J_N@c=pPv-2D`` zc(U@r_C^!>FG+yRw)7k311 zkLfDjECb+vTF7n`+WMaEw{p9VLPfCjjZOv62UEQS;7OI(1M^vj2UBt9dk+hgJMC{? z3G%bW+aE}~L4-#QNr-C5sUqdI!}IowROIJ`x!xZx`u}ET5+lD^`Y=4d4xMJAKxoi5 zlk$^7{1(oB{77rU!YhTv^b~^!Auof@C|g}x|3ZbWi-ePS^Sfw?Q50G7lyTlpj%zfQ z89Zw#*9(C_P@0Bl-NzPUd-!H3v%B-vAhJC(Bw=6MP>5=B$#bPzpN<>7R_ZKuZVL;! zh`-7gt(xEKl)Sy;BJ>6e*!3xT*l{zV6!yb=NgF+U)FC=Mdo6W^#*Y|W=(UXjApL(g zGLlj?3wIK9@>HRj8Yo}1IcP7Wa`v0ig0xwG88SRi`$$Jdt-`w0G$E9g0fghOsDab@ z0?6d96NWJUrPFaA;dMj6`pWWRHoto)Szew-C&AbI=nEz}Wsn$G-ol&-a}PXjLctbn zVv5SR)O5!F2@>*pdIO>M>`mFmuZ6Sb@pAGj#%mEuA_`;{rDGo-u5#FQrAHm(;N`RahAU zqlFHPvaD#FnJZI+;_USNdc!Q?=rj=zFvG0S>-<{nvJ?*9<&mGIJJmbM4~(|QHS-;e zy>8D>j?)DO^eVEXq%D(Ub(m9po_iP^A7iQx1G+$?ARY4m=Qr5DEZy}Rz&*;ztJ@~}PG^)BFO>fMHXMot@O-sYi= zFRr83-o-a2wl`NQ;pg2pmi6v)=(Q61KLu9P;IZTjb-W5bzAi21&uigL=4sqeepWn{ zI9IqawSR?UBEsj{57Lc^3GCiP@3p-5J`|2d-@6w1s|a>oqYNFKv+Tx9rZ^=EsihcK zNNjGdXP9TS+VASqBN*5oI+N=3H^dJE|TyrzloEI~+ZzS2+VW;H?s|06+^d<@Ff+|y=WwG|LM z>E__pVg5Mna~8_gUIR{k>?bfVVK5nS5%mE#YhZyh`<#Uz#5;M-$q{d^B|r4{e??_} zE+|&>(v)Oew2kaitwRhez2JIVTiYXyBDut2dzznKi!Y`qFYRQ6hObw|y^i_*#Fy-_ zm)0*xGI?oO5%nz|?G+T|G)!AIGEF0Te{dNx)U;$5{ce&#M+!*?KSD(k4pQ}1Je3E! zxX72D4`-^#Y+8dmnqT)Dj_Ymav_ zL`Lw6+NUS%d{-xeT`IL)oC?NyqT=0Y(|dT|6)((L-RV@==!6e10h}Cjo~h?umAalyt~mtPcC;U~GM4LqReS4BH40#qOl$vF);9PG zMbEk1ds*@B@=KO~k@9pnCwW1e0$B?N*qiuXv}9D=u4P4>r-fHsptPvtiC^2`7kS2()MwV zmRdXw^yV+(*V688^tV~XdODq&!sVC=9Z(w^=ymGyBx&JlMWrRQN$dS`KJizy{w_c) z5nxL_4QDY!#YU5e48)7-Smo!0k$B(;bg5>ghNC2tDxib}@=4{^Z<>}QKrZMywD4;+ z#z;aQfMoVHHb(%>0^oMjvbdw4f=Y*;M{sZO)9ZutuK0fk59No20zTs>%K#R}9edZ3 zD5KzYr{+YRibR3W<6Ydre&^m?V@Cbpt#j9#Un_s4YPr!+Pg*FS(&~Iv)}zltV*X{g z^)2OPz4J{NWpJKBw@ASwsMz+tLVwmgN0mZ0TPJRSmhlua70jY$`w(nvt8o@8`C*`M zRWrXyO3O8Y{2K)kz1-%QNd{kC9s;Be7MVr z(-+8fl?j98@V(JIy%bj3J{4){P+2L@sYxWPp`F&ZE)fZCxkLxeIFnfS%}bQM;W_i+ zjFF@~2malHLZ{biQWO2yP__{J?GHL_9SLt`Iln%3f5bW-BJpbS zZ5NPtgI#y8!~vCIb&fr4o5WYk*}v{q?~6mp15)DgJUi~3_dxpaf5?*`);sX;6PFF2 zR>b1zjDxhEV}30RC0`=G(PN^p9HAN7H7wi+zrP+-4H5zJ$J1|in<6lWXt&GbjeM-P zl)Ngd!XjN4jsOzXS72Y%LZot4Y>beUm#SAqPk5f6xJ}~&Ww7{KE4iz5YaPzV6 z1fD_!uEPa%-8;`QI4f=5O6Dap8H2DkYt>r05krmtM!9ViX1StTi{_DIj7(~_^L_zQ zlohJ4?VsK0(yvE72H+FPnTzq+Ot4KXXRz_)h_vJIP*F=bdxB6|%=nJZ0u;|yRGyZ* zcfl!BdUJZ4j@SvvT?yyFlZir=@wkYsvzj6T zRkZZ9o35x=f%wYzzz{gg~XNc;@d4e>0H;fxfb}lR25{u9u^*Mw2Z%S#Oqf1&AI;Y z7yIz)ifuiF|qtl*hN z1WEo{$fN}8LCoc{l;G2|(WGCAt|Y10s+_$~ zmcx2Orw9O#Im0XW3wbmzz`LFsk-nD9<;vwPDO&?bxySokV#F3>vI9S6Ba%O#ZR-D5 zvoo3McFg_BF$?+P)3h=_X-~|(>D;wWd=P@7GDK*}t8tia=p_dseNjCC0V8R; z{#BAWsw)>M=RKgLW>qfd`wxG-IFxmi1Qmb zIofx`XzA|r9d0wm=fX3}SVv`^$};7g`Y;6Qk8&O4m#CGYOp5^pk-}zjTaWjb|48-$ zRK#SeNeyLZtTyV1A=HBe?$>ay;AeC04i*Ek=x@TuMQgA+BB~$1fZ4&Z!IK{+C%}K8 zXMxOnvGOgJKx`e{-ynD|MZhXd%kD;`oKsEIf~>z^I~Sab zPy{Hp6smG>&RNKN(JYi!5l&>fg2^7vF0Usnj7(@}XZb>e%KBwS{@YHaKa+x6`lH1K z%tkcE(~X%tQ2hM8+)`ac+hg3SKDyUsjV83MaHgAG2Dr?X_D_qycRyWyPDM>?9=qzO zvynWcL3HjhWk$qfm-#VMuwoBGl4v$=nBX=&Bja~Q?i=e;&K(ZV4Ul_Hp=mhTwZ&q^ zU+sPqy87W>pqNSe#J!v4@ma1Q(7O(3A?DtfBsNS_m!QM5igYN?n=yuj6!kTwtnML| zoXx9Gv>0BB)uC5P2hotNhKDdWupRefeeZi1Hv6aZ73xrY`xa_N>-Kq@43UopKKVBq z!QwS{1XK4vl{F$9ci7k7Af90jF5{2?{^&3HvtMiYR}BEuH2G?5^B1fCAA0G5;78Ns ze?F3GO!fbM@$IW5pt48}5PkjMz5MG>LdgM5$oaZX0{j09Q6`IYFi0}VTrmWu2!A!o zrKTj;GbDe%Y7Ld!)=Y)G1EN>;%iKlwd`L8Q5KM$WJmHS#{4XRrivE1=^op=^6o&+@ zr15S*T=K;yO7)nP9WD}ffjEcMs8pO^%oVr=TWy;p!O075)oqlruN>I!RqN-)#l%Vi z(B~|zzl+Tg0SF_%DXvcQf5is_!gI%?m5St&tCPSZfM0<@55kEOr$F{e=O*{@qYt6Lwn?#9oXK**=sHZ2#N`OfL;=ud z+_y5cY}m~=(@!W(S8z^5`W?l`>hUF&wEE;|1QJ#=5 zpx->%9D})>FDM0`u3P9(9hIF?fVdRD(^Evse~`u3t1fBv8;-wKYOR(CfqJ@^M(>NL z_r35>FgNy%*X6oeMiCJoQ$Aw=PzBjY7QFgdHhPIDde)zywzZ z035{~hj1P*h4r@ki;vYV(=aqVx^r6!2Q_-PL%l z2nBo(G+Lisi%-0g^fb0X0ak&3-Y9Oruar##NbxR4m6Jk}gw7`Ke1{Z+`6&Ggb?aEm zwNX`E9P-HF_Ss0${zs2ZSh67Fu*8ag@H>U%Y=4`LTfH?LB$*8Q15`l@_-X zgLylM5x`oDn3t`3z_^&Sp=!BMy$x~0<*qz?N9T^*B=I8Gt>MH@S}2^j=nBZTL6gAu zA)lxb@Tyi(1QmMh)AR%gkbXd{Xy?PQ;toyJFQJ&ic<# z|B;A-LT3+VTB7+50eG|LQvQcEo<7T|G-6O-hV@cZ7L<}`TH%Z3{eAx=779j`SNmSTm=!S zDMbC2*rA28eWx|%_h*L8_H&~7`T$`TuRK!&lv$&rZAFEcB_^VUj>JbsUtq_x7~!Jb zL!XV)9dF_S-MVB#os)hzJ$OFpHoXD*txf}Ia!@Qzh6BLtp${c6W|%@8|EYR`B^C+3 z8vDpEq&wOmCt5_L2Ea*zBO|+>kGA2mo5F36riILUUH^9Lc$+SG5LUf4l;gZttzK)N zNuK{u=)H2B(dfyJNc`|B%X}=mI7+LbtIA->vOBdzxhyg=qlKK|;yZag!F0J;8IQ}_ z>Mo+%jMS!N^9$P=iwFEEP*0)5Y_rFUkxcH0{9n6KEedUz{gEAQT5|USJDludN##op z+#Mup74QbXf3JmdCB4^p-CvX$buHP=_tT?Gfrz#GUZmReS4t$)F%hZ?|Ls-R-N^$% ziE24Z{MleY$j0v`X#_%UZxmmqdt{}2P=p#yd(nQa`5a1E($D^?9Gc+pKtpH?Nsdce zB5bYT2_0$g2QiQ7ubV((S>mFgx>*|%=821Nil^=Yrs9pO5 zjR||m=@Xj&RH-iZeRp3#Lfx-JJhKN|K({5#|4WMi(@7lIE$J6)w9LqVZJjO2v0;z$>%0eLLlUMUl|BX6zZ z@;ydL{b=LYG5q*mu@TW*4y*_OK2^U`8@kqnGrwP|mgow57$!03{@^pB5RVF~aY7O` zZ^FP_0tP5{9c;2FQ;{Dkw{p22%@)tgc|O?O*R0zKFTILLUG0lFC{$`_Cx_(kC(tU5 ze5ilSRotQz@wPKuN>#B8=W*FTU{1T#4rCkt(fc5IYi!<56tHCg#w31q2mX) zWBZJ7R{9VyhqQ%PKMG0i5GYvy0nlfBs-sJobc?&}aC(FxpR}?l1c?nP0Z=f^<@j@6 zUYpZSx#zFZ<;lI8?Ht>T)bB&3H04??-^U?uh<2YR!rkvYY$Ci9&f9Sw2WWhm3=XY9 zeB#)Grx)WQR@2!Q!hdSDq(r-1>iZ)I*g zsQF=uxlcG)Vrb(^VgJpNnuLuNFnjb3oNJBsV&Bjv9_sRNn%E}L<%r9n z!S3`FjRFF{;})~r=rnnU_}#bP4_T>0u1HXDKVl6hGOKB#+u z{NY}F*vwOq(sY>bM}GbYk`gn({TWW9dZHPtv}3b@3r&6^vfLaFCBO*DD4O$wz01liXUsB2dlah*Ie{I zy9c*o%e~{PG$O!oo$Je_8t=^L=pPal?PPY!6Lv#}W%R?($Js!2qp`2AmaY1#@j<-V zk|Z!5Fab&3ZKA!gB!;wIcm(GQO{aUa4E`U!*Ma6fsdyDLSoNAZz*ambs$W%rgP-6a zduB)9;v>lm6m#xpZwID%tsg0#Mu$v>FtysfMNvpt#2$nozz|7EqLIQUGM2L`297J@ zH^j3D*X8`Eep9h^(Q@Bs2K=U}PaU7V`w1>y~GAsSXF^brUBn~tzgtG>>6Coyht%6F`3dkN7>#Y zAIv=ZD;egPk_jsw6+2)LWSqgo{1>mv#DSBWHG5~es#^4V*2(ih^TVF?OkDeP$Us5H zJ{cXIqQ%#U79%7P0dj<1$@z{x)|Xe0tOl!s^%d<`iEH?JBp;2r(xu}A8fF||x9Q)) zN(gWYs&7#VN~(ucii=RKQ3=?;s01Nr8)eJqN+^(1a9GMLm-ai^Eame{vZrp0_nyx- zxZ!9o(WO}1C4Pp&OpFW6Urd*0cny|ly*lEp?fsqfik6wo%JH(^=#FXLSRGNTp~7E$ zC9~0iKY$o6)ZifUa<-Ogj()zZrRA2-x6`^d=a*BQt`9PQ^Hn#k9Z9X_5(W!P!&&$B z#OjvIYyVf*Gj4Xbe3<8)OY&0xTkcn)z$>=2va(c^sKIrzzDgvhLr`R6lu#`T|F_~2p>Eb;p+e!!l{brvXOZWu{4Y2Z_in2x7`kq58HVro5#-%uc0xz z$IGEKcaJXITB@$0%sCe;&i9{7!6*=2V%}Tqt%!}8G3#<{%iR(nfid)3Kgq2Jii%mu zlSV;gj##U8vF1K28Kj6M2IK$wp#c@Zp7=R+WkO@}-jaS(t+iCg7)jVQf*-Po@RZC- z-KeSlQE8?|v(GrA-#E62(mfcu#qMDHRU= zr&9eJZMH$=OW5XYV4ODGmQFj4wYFiYUh@&J>n(O0~mC{>}~dg9 zx3|&qGos7p%F^6<(eX@8*oknZRB-n-O>6NX*Fxk^lg;C}I`sXDI;$dz*8 zXJy3)9D3a&Z3uhk6ycs0UqfHEophkXmNbJyBNcoosrb91pSSJZbiTer zBNmFJ=0_a(PY5t;HDzMV zZoVksdH3IH!$1VM7Bp{(VsBDV{S<@|-acOvn|sWpG&YExFZRH^IGVx+v@!BVzcQb2 z=4-8sgO36@1>^&!%gq(6SB^q3bqgGhyQpMv*-(5dG=qTFxN@}uI~0W#Z(Hrj?POWW z2bEw^?XtnM`yF{K5OPUL_m7YxFfqBuI&Am{#fA%%(CncdSH{xleeX43@A&|lly}d7 z_O9j7U}b>;cR?Qt3A|P4PMhyqRcHc= zDOu9L>xbV?6rgtiKk-N7ZN7y|?~IZ>094%gV%7jE_J&a44=T1HWnYcb1DMvk;L?8t)vq%rqmj7ViAU8c9( z9KF%aIxK8cs7!GXdkq~O=Cnr_5U|+8Kg5+BXKI;oxMU*d_GuLxz}Rql50B#2ZnK|E z(jmKYCNBEoRwsa7A!eP8%jK4Vr`85Zi-5oz=s@h5jsv7Ye_?LIvP|NA82|1I)qeyW zHK-XMe?}ly%4A8zbe%%aR3^+SYyjcXDZr*T3NOpo57{#94?qZsLh(j!dVb}R_eBzh zqpkR$3i@8-{jnj;`&#(-LX-0S||Kt|Nj+L>Mo(!FYgz{-BU-T9L~;Dm#v} zIR6*!nbr(b@AtH|(HGvviWF9#+EWGEJ8pW$6#q;MJ_W#SY;M!Jpntr90Q8{swsOR_ zkWF?X?Zg1CKxUZHD#61P=Q-5SaT8$~F$w1osJrbi8XGjF0Ou-TI=vZc8P;41F9p=x z$vifR(12y%(`SHEHE`fW=MRsccM14CV)nlwF%7@JXGSo9KDE8R!g%6wBWv{;`F*29 z6){`oCHP0>RNEko;Q1b}%XitAEr-_lz|-@fRI?WqjiYB{fdtfhNswi(4tZ0oFb6RE z=4CbLctjBZlqXbWc(s3Cg}EH>!SbA>l?{pF`$hjuZA>K~Sg2fk3;M4y_!>wNJ-TEx zKmlgkss0GS8XF|0MG?v)4~Tqp>ZDrmus!Lq3JJnnm9gp$oU`-YL}PV|(gMOj=wL_`tc%AFYi z>PJRocV!h3>H`Ac10!xv0R?{DIZrLn7YsF#o-(}s2kTRaCLbTTKGq;*@OcgALEvXC z>LpLS(i#y~Isy;py>cFUtp?%a51C3TaI;$D*6$;B&+9??SJR0ax@?ZDQ(5 z?lOQPqmhFuyVi3gNZ4q8Xg& zL2~d+`2Vk_1a2N_1=;D(L1xUA3eN_AEe8berY-&aGpCbfojva)MqNY2l@^0vnv0Pk zLMNa>T4T1QvyD($*~nyp2ZW2=Ic&(Udpt1cRw4AaigrX>#{}qxxJ;A1KRAZj;BYnu zw(9Tp!EG5uB|l=W!H)T7Iy)*@o&=j-RsKxZ_x6TYxlB`3rOAE_ncb2y?qXE%i8GbM zD&5vs6T@6 z5J&9>Dp&mC`MEiRz44cob#nTRL*cO&}4tPgO&na-W|`BNJ<3W1Quw!r=AQ36z~*iNqX6pCW$m%WV~Uo-D!3ns0P21-jvDPWY+ z5=(5l?Zs}bLZ7@2eByIpk#@Tf(zmmic{f8qvlN^o@gYBa#({#8p_l6M&0n(#2MMrV7|N*Prh}vx9@oLLHHe@xWu~maNGixPru!WaJ?(kW>b2wnAtSf zh3CpUy?c6i1;p6*01jpS^ZKPZe0?Bb6A^-)VxU!_)LtZk!-+3w!3s=~GcgCxJ))pi zdpL#;z^(;Hdq(&XPKSTq6em3(L>i8eNLM->Fdbg~1GyP?ZC3O*k45>ny) zgg}zBCGO7R1YkEa9`Ug^Z$U7V$ShQ_{p~i_?W3=VO{DzoE}FoNYM!)d^}`&e;QH^e zI9Pn749dgq^V4%X-!8xIewG_SJtfyxO6%ozgmWB5C8y2F-=0Q4xu>AJYRmb~cCkUEge>Ox7Pk_yj;A!6Yv^!+^QL`Q&h;dh>&38MIPBc3s zgn2=ijis>iXYhbN0-g)DhQX;5wRR2 zWa{Vj;mgS~oflc0-D_hLI&}uE9%Y=}6uT?p3JEYH*lPQ%@9Fz;9kfm>OmQQ4$tb#b z=CENQHHF+t#$}%kZ1(;zP*Pp_RF(d7D;Yvg!l@D!j(9c)CP1_);Jw4%sLD_=bcxHF zwd1icHkzq?T!W^;V9MH+@_g@){(;V$vYd zg(a7{gAtFBdIuQE3Qmn>=+zuqK5`4AVFI!l1!KT7Ou^;-6It--idS~rEtC;5^fno7 z!S(2c9$0L~+k3RgihkN!S^P@%`~{qEf8!z8MN7Hq>^ho6bNl`*i5%)3b@|TuGSyRJ z?WAs-aRmtM_axPn69;`T*x4{@ z+@EeVB;POhH-a9%-)`W^7}JF7`d5p)AJh{SXB6yYVI6G=kC3kVa=&FK#-sry-yCkZ z*(dP@|3m&1sZ9d@%nq1>Fm@5ID>R0)<^-$RD`pUjY{Y_Hv;@+Lnocx5?C0HXDojOnU;zOUQnVPZ(-i^!S zIJVUphlhBIa@CpD9Y$})ZiF;Q)HqDgc4Kq%(@dAeBhl%`1$Q!R8$l$RpqS&{YG?Tc z;66sBYf=?i=pR$&LqgM4!i~^LpRYNC&>RE$n%>u2lO-2cB8}R+>1|;^26b;*oFJLp zKIXTfqtF=LWk1kGUr%b=pq{T_#_{!pVp z8a4OoNnhKnx)fj(jJ%KT*h}*uRO+K_11u0OA^hX_pH9Kdkp7^}%W1wz?>`AHKw~3)Wu7I;OMY#@;8|-sro2|fKTz~$cV>h?6PL~UE&oR@qsv=HiN_J$KWQ!> zoR-0eIZsJM{0^_fr$p#@Kiv_*I=`2_9_)ZUm-tJ+;I|n0wa!BXjn+UPDS-R%=iUSi z3IQ8#&a%K(q6B}F-BN z#iIoL!85-{trzca%BI9bH=b*9SOqwmNYkTjA)QPNvEh6{6FBU&nrY_d7_+hixf@h4 zVk%ofKWR=}pYpXSd?+XRGA0fr6a@lNB{_ikbBLlaqQ4sdYMR|r9w+im?}s!#C&7mk zC1GD}xf9yS`ILj?dpjCQmwr5ql*Ou@gqfb6lZa~pq0w@pDwiDrXNqqleJ+hm3pu%) z9LoY`37n0Z2~z4evlN5#s*$}ma=Dr{y~B?}4?OJHoo!CnOdhF9SKXKD;y-nrugWd$ z)aDy3LaMP$OqTkNaAz9|rZWa9yu{`Xw4(ZMHcN{1nM=Qu4a3cr{95(`4H$gBn}M<` z`^G|(9r2zZUk}b>9Daw7-V{K@(f$OEh&B^Umz3$qI)vx`4ofvlnSmHTI*TUGg*BT! zk!`))c~xp>JJ-r$ZNX+;hCemwk0uy^RVkZdAn<6w2lVnqecmWOp=Y?_)EmyE3LlZW7>I63w1Q@)a9RY6zhNV=Z20tzS&$ zzZcUAj&>!Z`^z^QBawH?IX7EOnIVb7$`IKlsnK5!h85x+nBH81+(qC$^og)K^@~Jh zHMq&+XC#j0QxFxf&~d3(0YegHg_~NQjhP&C+$Ye{(Pwn2*&%eiW?6!deyWyBB!Ae_)HQjSjkc(b&OCeIM$Cq`O%@YJ_cg_5snJ16 zZNm&<`bfe^p_5QNFcJ=7&2>Ui966#d6+C zg*H|PrAR_d18FV8{e`P{QaOkAAq1L40<2d&#N|xbKM#}Xb<8?#>6nl!C6_bWlJ_iq$*3Ng9&+{vI^UP>5iwD2YzOrhXq>s=ivJTyz zqFSLa{-g$4se$0E2PIOa3`E=m#yQNvw0Zy`6Y;$-iN11dhnibbhU+=hWJ1H9T7$er zk}ob>{QRrLsr=_QHVsXuoubJQ0W;$BK=9Gnb4$@ayAP1iRD?Q%~)TONV0 zxPI87#o96iyzn^DY_Klw(?4+A3(Zs1!hsq2*-gRmLsQjnC0_?z7o0swVh41&De~S#q7H5a;w#`h5v0g1w z$6mYPNK&VASp0C#1N)r&dxnh{i*6|6TAq(b9;Rg64qMj=m1DK_nUN2t;o%8iS=O-J zZ2j^8n0Qq|?LLff=+;nv;V6&E71Yr=Y%SNqbmKibWP+NR+P5N{AZsYx_w0K6?cU?u zkUVR~4G_SU_5fq-5Z_GT2&!rgm~jCwGfb5!3A+qxnt{}T!~?zJ6UMmXiv0WRQ=_AL zQyFN=1uz&}dSn~{gu6LfD!3E$YWeABb?6C_8uP6{<>dMcXkv{HwVeZJ@!tsfCtLA< z^ZG?suy$$hrJ@CA-YkfQ*VxcsT*Y^2>4K(7+RcK3dbe1fh8nR0xi-j1BASK zZHr$^KPz>q$ZitwHC_LKhY1Jr`DLmtSjMWt9igY`+gGtNQTV9b+jnkaM+4C{8)f!F z(J1cFRcgaC>O0>A;fg2T5?=E}aEKJLv2K85rp(~_1aH0ci!iE+Y!e>chjj`W)214~5tZszd@#Vodq$KrQ-4N)p(3D>KG~BzfjJbz7WoULsRN z67)y;?NCe=e~!nvEhql~68y{C<(9b`U?;wlY3OC(9S`llC1(~yFKfIZnLDi})DblCg(llU{IgRTjR_Eu`6ag*Y9G4C%^awXxz6Rc&B(%ACaTj`2L zcb`&%sqo9u{GYflG7D5yfsR`SUGG@U%E%I>EPSw_YDT*yaa!T;OPVo?nO4sq0?K`{ z7{aS!L>%v>MZhyMiK+Ma^L#qk4tg2?L@(DDQ<2LN0fS;62XVGL1O#7E>KMqinckkf zmKM~_N45VO=-Ye=ulPv97Dw+Xf?w;+U8weRvo?30P3-GQ%O8w0SL7>&f<0H`qPgha zmp4Hj!vo(Hf_eU4mZ1Q}X~-5uY_~Zm&fD7iB3-Th^;EHm2QlF-iHO$8GTC! zF^%!Q8w!NK*l1szk6vg`9DYQ`X?8yR+IV;E zaC#cUQ0oa&u~IJijzN`X&O}`f62;?H{K+Ms1G^<=gpyz?>h-gR!T(b3`}iJ8~)~DcGCYpN1~?C&eB-U9ir| zG}?IZIjv8B>c>R>@9lCYM%>jd9yNm1_B$7u1fg)zfaqd0?ssspD!$s zcly26F2}whVrWysr_b7a-lo>#DTykmGmXm$i~`?BZ{uJvcX+*)$oev^HA3`e>Br}@ zz;u%0^;LNkmEfPOA!S-E@5*%`&%L~OC#g;f8b&RE43{jI;cY}D)IjV(`ev8C0Hp$- z&VXjM>vHvGL+n%zTM>c>DU3b-ire@4HW3qMBiULzvVUB`r!8(Oz6=cmsHSpg&#I~h zn~%JljrpMHouk9!K`qyY4nK!+c9>p~!o=R&t~7<^=2C_=>8PrzRyNRwzBZB0QN0{8 zpDttq+=l68HyXE^HFbsk5%2{xyD6AqG;KpEe~Cst5S||_o~jB!581VsK=@}Y4xzBb z6K&oJx36vODkaKdm&Gc8YLFim>vjc`WQOUzp**oh-Fg6$jstu$5!MdV8M01!5@hn5 z1iV5r2#K)quFm*#`r5u2Pp1Ym0l~KZ<$(TBVUUgvu#KOgVlipsW+I@i8GMG*Y%nk9 z>q*Hxbt+8;2TxNhD@gPOB!$@NQbVjbPrB;)5g0=c@mE09>tg^;7j<$Yt=oA}VbO_;SI+OE=WvxWuB>nK#&risFIhO@Lx{sZX zNAuysO`DwJ%U{W_SZ|E%n1`Qz-1y=o1DXS79N9m0`q(3KCVsKkdEAJLPA*g#)rhDv zm@EY^hDA`Nh&`XG)bE6|%ptCzs?d;7=X%GW<4->`{&=jupLzqbp4%$oo7(t(EuNxC zHJ)Kj-VZ;aAKB2-R7ECaDn3oKR?%-lrNsk>Gi zPQecZ+&@py)=|NJ5ksL$Agw_2`Ez&27%JQVa(dy09)Awg1F6!$?Ap znlE#5<2jEzzlDabhLz^QuuKzh&+~rjpXJv7bDnO0ohRA}e@6;;3;3=6hS#T_>-CkO z@x{ZwDWhGNv&R=fe_-5j5`GLFo_fw*j~ma^I^%V3?)Slh`4&QI<#gZpD!~Z1SI2p^ z-G@|5HKR%%AaOL`5Wb;|kheIBsdGj&t-m9k`(E66{yx}G{&-hci!2@mmgiYvp$Li- zufhVxR+(BTdGFs;_9vOZdvv-JwPw~bFqg5Hv76>=+k1Xf6jSFbqBZ+#YueO-XV5$+ zI3jty!rihuT|#zvA<1Mb9~-8VBJBt`y+0`s!^IQ&adzxQ@Ta4r$21D3rO)1*3llz> z*Xxhz&C|ZYL^pOE$oGi0TWRvIBjBJ#68SdL;JK&!RvUSo>vbHEkzT zM~s~5n3&(!JhS-TMZzwol~^~a>J=ArIXgcgYqP0rx)cxZ81Xl$i%P9$IQMW4{FFdg z{}#S9a+r3_vy{TW{!`^bK2_Ua(()R#(VN*0%VZ;$_g}JE0eMbbm&2-?4Otg`*clGY zl&{fF_{O$O>q(5(hY7~qDHe}??${Or{Vwo1e`D$$Y1&e>&$r%~Tk_u?G@YeW5S?z< zmnN5eA#*+%fA##m)x3$(G+4c%YuF!h)N_}2gv09+XT81@aZWY&rP?9AhpK+VR-@iny573Pjq($Q zB^FueZrgdr^$$E$^RIZg`mZwMR_sp}z4o6XbIg`KM!4~4Tu4l|kk-cYTl=@LRc;|j zoOSp^0S@TqmizaPwAXZWB_BZ6T&ZciZBWC97^nA7IRerPiYhBVY=EGP&h^k zR3qM%GBEH}iJp^XZG=RrDEa5WggT2K>B@1lKwerRL$Zmb<3n9X$YBla9J!*TUH9xJ z2p*w$3-f_peB-?Cqw3n^jYg9p3x=4G-y?M85bSin48V0XF2uxpxApk(2;;mjQh>y7tKaS-46s$*MmW8dwT{@ zCuyC73LU>9C2Gq~oYJ1blc#%D;wHPC$N}QfA=MH#2Gz_n{q&y#K@prfVS10-zIE?4 zb5w2Ku0bEgd24sIx!k_LV$N`{K+Nr`H!c06~_%*<^eeAq|toglWsKI zh>!>l`wS9_d3Q=4X~MA7PnX_)_p4%cLD0QVSrvCU)>SMM(D!#4QZ-Ucj)ZjzJ-==C z_T#C@oX8-s{jkNGm;HM|u)&sQ@R&g+;yeIe;w!w^^XDzWE1^TLFVTDhmDXahiIc_9 zo@?~aB!2Y0`Z!64ejT)=$LPk133&uV$jCUP#GUORH#`|Dvm_z=Wja;3f{#-tWC!#spz}Hj$3F3y=6Syxxj3BW&~*^JSF#^%>vsY;Anx=GMNNNT_Sg$>E%@NLVS={&2^5V9nWW zidX5^0ad(?A+6RWkc1C+%|K@+)0b z&%*=!*GOy0mIfNjZKI^C@cYd8ZrH{sljLm z#OL{RH0$+&sO5@iY;2vcu4$;)1o?DTr-4*QE`$FGU5+ERwRn3?vZ z2~kh7qU|B;EOAQw{452mrQ{c^7eXF7)tSs;^MD0D68O=pBV-VARxKbV9@(D@!=<`i zmseISt{BAda9ecD5D-lCkXk_BoZsI;9VsFw%G5TG33oPp79y z8CXJvzJ)hcJa7;3egOG=>8bj_GB145JM2q&%dCe%=|W;9S08aOt)0V&ijnoO!>h$K z<9sQ^h2La%XC0;K*LO=Az_c)=RA+uoql zjd+HySuRx(lAeR+*T-Qc$mFAPlM@%ewAtIVZ|;s%Q&TRlZxa(`!4P-lUIEf~VPL-V z6`rVj7G`=r%5~z3b_NEOv8`S@$H$*`;0XaE^rWVn>X>kx`~vEfAuTV%ziu5=Wr}@B zcPuOWk?A%p^nyhO$>U}`I+afY7)JM#f$C3)mdxjMKl?`3~X||w&-T{qkN2EEY92@sRh%%V2HTKbH ztt2D!8)8B)ToLp>5u<|fPy}voy*l7iLMjv6;%^rdrHL?~~n|q5L zbOt`zLp1*ifnaPOSvt;fA30z@UC2|X$WWu3Ng9xK4#jylT)Bh=Cs8Wg-G3|Dp$rx^ z9cxYTR7~GI5KwJ0rQO=ZHcu=V-PUZjX{L^&i&)Hyk^#0Q;NycSpQ(3XmT;{@c$t=JjK*6>oK9f)Hm@ z%{Au45EC}0FufwJ>`vA!X;t6%%AXxpvPKx|%kpS9*r>Jo>Bg3m5(Zl3zRYVj+rM#y zB1flw)VN7-a#0?`Mdg@g77sDDU24ZXm?`U6(N%DLYvKSqyC$B|usqoJ)bUkk_4&a! z_Ow%l5bXiIP%!~NbR#G8Q@j+sADOAc75qVcK6iWQEY5I`hghJtvKUS?DMbQm7syXt zTI6F71BOzmX^Dh$8oxwHsig*!R5iDjmm;sD*Y^zGr^;}|>k-&0FQ|nTXpY|veRDiD z&#&5N62HmT-((`*;9R~d9DZ0{POjNyBsqxz-lGl)iq&e%hrUQ7Jhev_5{nm`?%%;3 zg3oF&)Z>=S#xfhjz>>%u zJMh3loINCYXmp_ud|hT!;3eR6Aaw23)`nAc+gPnAQSDON=V>BYuQ8EtsaGRXMm5Bf zE~!2cz>4mFCG}#9G7%>wJ(-I|wIYbIOO9U4f5O~dsxL}ztvxYAtQ z&&@!`>l4}~(^^C!l6@w}!bF2cL<8p(1@srqHI0rzW`ADo4^X5FW~{#8lYu8=b^NS2 z!Rxe1%52;^u_Fm7R$@H0>yH@y^FHDAtuzJnwVs9!R0yMgj-_vmSvma1u5-fFplhDF zQ{{`mE`41|?>=(9?Pt;3toCZxrjvuCZ=3FD>4SDcqID0{9nKa3c4B;f2*<5>kei7x zS1o>hF29#AUw!JEP{NlqIp|e+wLh=I?`SV(jW|HV~j$1y*2T`Z3&;ePN5^ zJH3zUuH0yHfK8x%t~!!4I*Zba>}(cYo>pa{yt$t|mZO+b;-P#NREVBZ_Oz*X;&6*t zDRN>=$eNisE`etyw^6gr!>yBLj2cGY+|{~2~LuBwMD=ZjX;D>d;Tai z(-N-`R!JH^!`>yzXrO$5b0skRQ|hYY@*>Wq~rd^Nj}H)1P8~Bsc}UK!FY}iy_(g3CE2rBII-cjU+^(h?{VLZ%YOCk z(D3ltRE0o5%Sro-5GCrh#U{eKVU-)837pW5o*?Xmkx3bRG3fj?}`am{fsS6C~x7|Qulvi(z ziMZj~(&jRi(m%h@c-t&4tA;o?v@Rvt&(E8v7VPjS;EDNeqmLnSLF+0Uc$vl!amxo< zIuBA2SHM0NwB+|b4kySbT2u1T5PHyEo4mW1D-hvFZ`pedQt~@l#6b*sH7wIc!jB(N zGxRFPrQGbf_d;LVS&7~Qfv)G7NTT&MrJMQ!O~2;NR#A-VTp1FfF%#HySB4IGVwr!fTm$64Ph3D{y`){&E4J#PjlW6n z4_bIj8NWBtX6gi-PzW*#OL`Dw!c0;kUlF`KjZM*Q3ta)1@xfQ5EbPos}BT7kl* z{Q!S?nr-p6%1TCTjouXjz1@sS1=sG0(&lCmAZXL@X#X$AnfD(fT_d{KmTZPp$hFPA+_l-?f5qOODCFJJy> zEL3lhFN~}z_JwYJ4f#`o|NRkr{k(;iAVj`gI1ddrP{IJ{3BZ?gP+-1y~xeNR$WQ!9vq7Sz|j(bd)6pQ0`Q!pQ{Q+}>XEN2@oy zKK7YyIC%zKT7Ld366@tqU}CTj*2(U}wU3IZ7g!-NC~wwxm5uN6!p-HEUHc8wv`)+a zSK3>KRkeNb!cr2_NJw|LbR*p&-6bFmDj?n6t#l*ZU7J=Kq`N`7JKn`Pe`TM0@3)s{ zf7s7rueoZj@ryC$oVcq!5x~letP4&qG5NnuQ=PC{oz7dUVeK=6;Yt-pmiqf+c(&>7 z@78)+nb!&WF&wLXv>IJ!-{y;0(a_EIHPoLCI5FRJHc0)sFiB7<^mTPdQV#Mw-%+S&z|Ii85!R3rY)zCi&WRpU>A5fQIdL+UQofosuY zhnLH&ijQWn4<*3!8ZS&(jZ@R@L7KDyHDB-{9*xMSB>G+~%p(RJ3S6xjaXGPSwPy1l z!ONg8)1T~U$4wz39!ls~75~2OV`|MEg1u2tG+=>Lo{heZ9jdCT>SxqCM|Nn{5PU?$ ze?RInLXd?HGKLnY5PURzI2f?Rt~nw%v6oiLSg=NkDZ4?{FPh5@OA?OI$tAD4mc@VF z>{GG++%;Nnz8iE*Qx;s7AmesIAapw=v0P{pGaV3c!?Lquv32Cs?=Y6}p8KF^fz~%_ z9DQ9NV%l)qqXxgV`EV^!r00=xz70$dyv<8$Y3v7IE*Us9#kXytyYR(*>rhcPEy~|+ zwcFgLOf)ipltAqq@rY0cT0MM4!G!ejefj6(6(tJVZ#4^fc7JU_^gDrpaB=@FoyAOC zZS5Sn=PDV}<%JFOPLOT1n(u+agZlK&d(NL1ofwOw0t*TPch78E?+^s;)?VMF<=Hj%-O#KB7Szp23rlRD8E~8y%>T^UY7v`be5FGNi666Z+c__-TIa^Raq;x@w8@-?wVpC{ z^7@+2XsEZ3UY5}@iwc|n}}xdYO7A0u*!1SzcA?R z04KJZUsACDQECFWWwj6Q@JENj=lX?NJgu3XVnVHhGjdm}V$-r?Sn8zUfDqV3-l7M5 zBvn0iCV=`5qjPDfBX~G+PW`IjtRmC5c%+qbAjZ^1@8Tr=;`BJz=*0=;$ziX%S8{jg zEj2alI9JBS3pp|q`a7yO;=-_5rQy1!=C(M|cjyGoNxMXVEK7k;rf>2!;)V%4R-2tI z#Pt9A7^~6+p?3{kwiG((SCgUf+s=vC-ecFCs8mxI=PUJ9&@hEo1pLsWL?`x&l1NwD zXdni4cuooF;lxt(>qlB00FIgQTH9(M&{7HMu;8qZ>nhd>!>=vUC7~#+D+wmXw^lbh zg{lv4)5P3yB_<`JvIyLutDdPo5RpIZ;pt^XKuX9$lIx>D-pwr3i8wLpB)Xgr&tw@D zbsWL{_~e*P{_xB}9^rGgZaEWh2Ylo_W9FVGG4D`$MaiY>ndgd>zDcK7V=yPs68Dri zrxF#<%VZ#R4I7$|dfx8=P4L^S>~Xip&(bCIHHEveI;k(dA|kiI$MeXiU@fU%2RCKq zvbv1>Z1x|t`rBnK;HQl9Ut8cA3H;sk_ytEEQk%+z>vyFU*k&&h3ZWz3?7C^3r_d7# zvPDn7B#^(hOaecgB)o0=Y~+fG(f>~FpHu9R^2Av1j}e%q{2Ks(BfW#{Js+;OxQ?vp z?f#1{k6-k0;HZygSr7**q`*!B*uxp}^cI=_Q;p<5*ZuF+Gs{A6T@g?oTE>H)#uhyD zG4E;rG%5HAGo$IRIerCIYJj!sl|coPWAsnu)*=T#H%iEttsT&*1JQpXEf8-J`5}TA zGfW4uo)E(m+CRuMLvXL5p@Ec{8V+-Py`io#0Jf>Q30NOCEHd&f4&w8wn(A$rI84kB zHZ}pZwVX|Eu5Wc!b4wjd$%vWXCcHpE5SEi;pHaRl=#~CqFSGacVs>bVD7E13xZ<4~ zHy1gyb%}L~F$`~{7Xk+CecG(R8S+S>YiVc}j89hYcA9LpDH zXOg#Pc}-1-#lyyc`Q;56538n!9e08YVOUrgnRO^W>Yqmr+9K00 zyV&SPk|wuuzK|o(VPE;|=qLfq{30zKe8wWkH2J0P^W6OrmfL-GPbyHS>?#(r2m%rR zx=RHX790TWl(uB=q;h(oxw_riUqueTowM6?yPi|$pfJW|)TQ^tcRaYBGi<3p!NRF@ zU~Cg2pcelEHy4eU`!wPEBGM0P zrOomvyc~?W8kpv%^|#85lUM})xE0c&b0$xAQFXX+~YC!a&F1a@!4647X?Qcr1mYHo#aaQE%MIh z61`4NPOEJJt^U{7Jg!GqdDKc#8|AA#T1D@6(HIy8B)@WLEGkH)n6T*iC8lGe;w z%!tKH*7+JBSd9QMp@x`}MYeF*kE+J_gPqNc^>ig8pQ<)r6^u z{GYf0#92{DsN-8?!&YGYw|hx#-eFuf1H4|n{4(9ke{l6*!VKbuKb-}}QFIuWf(JzL z*ASD1n`aRw`1rg}_Y82nD49tuQOK?!>@2~d;haop*5^*W0)XMaT>t`>3fSl)NbC^t zr+Wg21i3?ha492_nfNCc_6PfxpsC#nZXwED{*Ok7O8*~?4we4@YxKxMFun*(3Q2<6 z0Jux~e>9qG(w+na`_XqwXgH|)ts|A-*XN}u26|S9_yyD^+=hdP2H}%bP!cgVGZCD* zVyLXFdKy)Eaz3{9e(uv|iexh1V}Re<+5%lA<2RiRHeR}J$`@|ygSQ8PjtVJV?11>y z7j7}RNt=z0?NlQ{)Yb^#8tAPwN-;+rd8ytBd3pJ|*x1;eJzdLM3ho%seIU|yrwke< z?>gt_2`Wtog-j;kl_$rxW|d-M-sWsFm&`|jM*1O)hJy!pwObr#QP&>- z3;t~ZZOXp#{w1s)jcIP4n5%H~tr zs*s0O1%g6OVZZ|K%X@o%=^&wGE_(z&u`V7DS4;ZLtgN@C*G?Cafvlt0L=P#%me; z)U$WxL5xv)jTh1?Tt?SyO4ax7?w7q#7i2O)6&o8S485V8rm{PFpGVW_(J4$3pW`pH9+$Q%MvcNl6L|h&b7-7YgLDL#z-2^7GBBcpl;U$&&+h6@ zh`oiXnaxLi{b!_9vYu(%jd=mED~3i(8R0a&9dlDtQ$}vxc1S&n(C%(-dG5$SN_>hzi%NM4A|)Or$JaCv_$7RmFxW4~$w4fE;mPSc;o-q!lN7Yt zDmqpLl`d|%cpbJY8`}l=ODTl5XqTj}H0}Y*!exKCpqDV&QdyEVzLdfN>jyL=03%0= z+`~B)Bt|0=TlYRLHGh@UvEyqJfYYspr@~cg#4ZiPQ$wcq!exEO}+Pus)ksstq zC_OB5d}u4-6a2RCD_AVl&ql4wJjb^b{oNCuA$8?C5W_-i)Ebug9f4+CfXj2ln#1SJarkIj!z>*0%P2E zLS6zDWhhap6{geG)r=|$nt_CzvAG{?x%rPT>(q1t%#W7b|K2%C(&5SiybVQFwaKMC z!3RI7XEL#888-(3mmU;Bhj)7(4+?29^A4^p7zP;`br*xYw#CP*Cg0(8>dm9I^z!R- zUb$Qlc>+sDo=zz7d?wc^)*lIMv~HD?`>5ra(s;22Z?jk6QC(d z@mC8JZ1wnROXEQ&SGDWBwK2Q_7ytnx-}vl0jNtbx4JceC^N$GOrpD?z0Sthi)0a;w>Y8)dnQp=g;ff_j$?WOJgp!0a~ zx`z5JfEp3(6%R@z@&8B8sX{tTFU$Gb$m=rV;^JQ8ht}5CE+9;69^oZ3!;bV3jTv4{ zohT45Uwa4(eMUHL{ql<9w-%6=e)ZqGB?)3&nOrg{lfIPBB8{YIWjB-UMUcH6tyNTW zeV(^?hoVDndXpAq+tEsmc1z9&)e-e@71FF)t-p^C>Jr<`FD@2y7lxpvD|BXfjnAp0 zqf>4;{SKec^}YHHr}6Ro&{r7#$HyPv1B;(z?RK>JSUo*#kUpxd`cP@34f9H=8)TP+ zVCPZm9T8jX`p!}9n99;>bpBE~Rh8D<8blR(XPs ze-1i3J8Sp*UH-P|i7gwj$db&rU^6os%}YuOO2diJ-yIX!9Dp(KuX+W9KhPOd7u&Yf zFHfs&*}oM0*q6b&YxcWY?7oDlHJD3Ivb}CLb>4d;uS+Hk-q};AU-n|f z(D1FAUTG=OQ2(3%L6e?Z07hPgVisvvbS|abNa}`n2sRoTA+SU!O2^{RLBa7mJxf_O zbCs*mB6`eqYv0mRrraEPAlh+e-~QbF)i5Z9fq_c!D#*WE9Dk@^RL6?I8xjhr6!LoJ z;n687mRZq&tB67Q{i9!jb`2voCN~5I#)_hjPF9a?c^NA9i;&Z1uluMLQC7Bj;@MbM6_5m zAK)iZ1({{;1w`wZm<4(#s5q#ek&*gEnk|Uz?A0HvtW+|2#Pn*prrfm=-&&1LI$qc8 zj457|=%N1?1<*_R(5Q! zvul`=m=zr9sV-Dq8vFf=D}Ub@SaIlP|GJ_^LbMRh^88!n%mXdj6cXp+R1UY%6_C~z zc5%T=ROvi@XGXKpp9qgOOZpBfF0jA#89;MpuUn=V=jP^6(eXo#UVY1j`;-top41`n z>n!KhuTfMSoOH5|6*Vlqvh3;CrdwIf_jNRj<(d*AYE*RQaJ)`Gxxo}pC^RH8oqORR zk$8WB#8M_@kA0Pf?gTwC50c6z^2HTC>zKloq09He(qF33MqaraboZJH3JPp`?o9Wy z6B8W9!iT;A0TKy2-4!3%n3xomn3$N1Bj6@AR_&NvdNr{me-dh==3F8!OiH$oT2WUbDm%rQPPoU zKEAnX3K8}H03gt(eN#AQV$^tlOE6?;=fFLPtxdQgx1lOe7@{B&(}r=seK7PR+>J zmWxvJ!J}L>8G}l$Q`ETf((r=tY^HGUC8_m|^k_oa@LK|ibEQ-x7pkCO3EGJAB0drc zjUz8Gs1OHAm{JZWH1)zT9&SmZIqqNT3ZvE49Wb!P0vRGZoR}H}Bh%8JZv#V_HD#Y5 zr-g`mNhs92@?0kOYV##+UO3Pi&|*w>I+t0WNzd8$?0ql#zFYq4C-%}o058X!cTJ6h zaWPkpCF&kmT16-bTyjSM@69pYDb z5>B@2XVW(DOQQ6*mIsYQ@w1Un&=ZCf9NKKyE$?J0bBcA#WhH#v?p(`l9=r(sO4kl_ zbxWkS8C};Ciy@Wxe#l@k438M<5fSYfF)BJBxln1{7$2N#0Iu^N>V=2};lrP8QhQHza9CS@4po`O0#fC-gR$g^S|cHgj?_J*Sn%J}d%%bcxjrvMqo_O3 zVPlCan_l_t=@`O4J*-DJ7ay|3jj4>C3#n4|zyB}?NJ;b2TOa-Z2xqdiXI%$BJi75o zo-QvEWDJ<0x72^H7x)s23v9~TQ2dns7sr4vCFkm!2{_|(B~CnG0*sZt&H!Q8A$qE-VI ztQ8P7nSKJ`6kTqtt!HLrWGJ`-sSszn_ZD3?>TkK;t>lui=_6`u^VF_=FJkD)ZvvLM zSv#F3d<}jfk)BbHcl6oqfLO%p6W1OJf2XOON(=~qZGR(jbVQX#%Jcv!pT-{m^Jx<_ zISISk6Rx!G1@H*KyGYM!=e^h+KMRD5{b4C)Ofec^3K|&4JW;9lZhs>x_tle?0{Kw{ zXJv^{AKLgB%5rYmjGH5Q&{tMv?=Y9?jB}9l zExOoc&YCF_cIa9@!w1DswT&CwLY+Zu%LoDT+NM#Sz!+aVtvbJw?5|Bmc zWwyW~6NqOAPc69b419fUVNyB|tnl&~gNa zjPOQMk{OV&za`C7?BlF^)RzuNKI2OX=4RD+c|pv$N4Z^xmKa?u+;v+@%1ev>!z8NC z-kIg=a*Je6jCN;`Nst(4BYAiL>w2|uMZq}+T&Ix-v4tGwToNFT7RCHg4M0MPt7tEy zE)%tU6`c*+MnUh=*&acVh>81x{fL&4i7mxZ@aX8M`|B_>1EH8u@H5>;bC|(Y!9|Kf zQQt7BPIEac26Eg+{RQpag<2v&yr1n&)Tp{yT3dSs1T>meXnscr?GfzFsL`v!ONE80 z#T#L8(~!P{E*8zA2>QW68B`~Is*yPo4R8sVa-Kj=jhSe24&OF;`?Isdy8bualj+^V z3kw2M0~PL15f6tI64}CqT5+XzuVM>aj3YnWE4&E}g1+li z8@^gGGO{O(iBvMLswj}wLhRfoj)yhUU5U?+7%O$n2L$4Z%`o^*wKdi&VyMLSghqK>x1w;^bER+4Fu2(LAsXJw1=qadC~tE6*r9Ksx}B|v^} z&#oc~v7n#;H|=09%;}S<(e!8+&kSvp&e5{;nhNw*v*SX>A|6RHe}WIdJxSykA}B38 z$t-1ANn|NIR3^uSe9dEmSKKRBKVg!jW>Jj>so$NRFojN+@&qFp&Tp&K-GqlpFX^(* zC+tAo_^aH<0bJ_rH2)5Z-T^ zl9qDyY;z1Sz;5_;Y7S3Kn7B)&$$Xypz>fNGKu|j$T)XH(q+$*OD|2Q1F1bE#xd35+ z@pLR#J3;c}MH~S+x<#;N()erjyqeIiM1kEIV2*3cei1OeCBL*Cxt-2wrUjKi8*=nJ zs@RsZZin5iaC_{$dqC~B3OM5FjKh$P)DLOGtix)K*6Mb(?VbFA)Do1O!?^otR;-^x zU>H332GDPM-{4I42F;>}xa|6|BpHessPn0}6Ar6$2xOd$)fzY#-4rC;80I(GpSmPa zeog%Nz83*mcLJlC*v#6Df8l;6uC=bBqW4=vf6J3LeFm=%{CZa8(PStF95Z9AnH37~ z#?Wz8A$=N+r^=AS!k&y+fT9Sg0Z^^ubVvY>oN$!-3Gg;#Ur_@7sqp*Xb7J;}tK-?G z#3l=52>=QP{wP%PC1U80TqJ6CpGQTOCH1iOFIrc|Q>PT0%qnh&%{6tXeY5iRy&d3806 z#UMky;R^{sPfqVNUgwgLFsi2UZl^U|KewE(A?Tt=z^@M3#eV7(h=HCR!HeC;nuX6N zrb5=+)P;{@X}%`Dccpm6B;tFp;`idMx-v9cDP4(1!9lTh%}c-o=W^NS%cVJ@LsLA7 zbzJs#PfALN=@5QYDMm&}_EyMVT9>4)D7mm?7@$d-+#BZEM-gwIyuaPsg zD&20ZZI}t~?CiYY7IZtO?^H!2zzoXWFj=Exd4)lMgyM)o5uH?}RwpCo3KoyZnX{pH zD`qsRU?Ql3XXWxd`bvkk)8(XTxWjKZbCpn zpoVlz1z31cXSBei8I-N0U z)vp&}IRpCaYEJtXG$ux1QdJQj?*QOeRSQd(AQzr zcK?-|*=?8~*(votA>;6G3zV=R^?TBY7qRb2wUmfrrb=`wrHfg#0YeS>LU|yt zwZ)>R22D^L`ESh)h#TNXo3*?Gwv;Wj3N^_lXA5)|$kWz{kjy4io4*$o?7!yW!&SEpecELS z4tkf=JtQ~;4M33bnl4gTiiZ!9r&rc&ym=^cq1ldS5EWctzx6ypO~15HHSJ3MdzQfQ5F?-Qw?aYxSOMRCP{cD;Z_`P2 zmj-Nq&@&*I2h@V21`CL)=RZ4gV3dViVlF4eB!u|;`wQ?c?d=uUzNt~Ym4AWs=6fC$ za$I~?K|RcZwY4PNp-*We#EncZC}-wcbPy5z_zCb+Vz;<+h^p{-uk5`z^r2dBTgoWatW~p3LFmf-AC@#!QB>EHYzLLY;rMIVoJkrfihli_ zD(cYMd_Wx^@uCMqi7Kh2FKF$iFaMKH$+y-h>VxI1-;Z%R+}LD!B58t*k_Umi-U;=E zcyKT+JnDNm1*nI3wKr!}0$Lnao7jS?+-}ENze0 z#~wG;f2v9!9AW}xUw-L-yN>i6Sdl;*!qL8{q`0fD9L+7=hq7A#(I$6P?{G5Hp0lvf^|WGoS3{GSB#Of z&4ARWkl?|BYi}R#s^MmJ_3W%w?Hdv!P}>oL3guC}!`t%S@;niLQmBNGe$qZO=@pShLA+}vDU z54%d{xtwUVHZd{;wSUUFwQbeiIz@i3niBCJBW2ZwGHv|(46^(^JDlCSwhui}va>Yr zZW(EYI|vrv#s}o~t<;JUc;r$x8mlNzpO(EDtFScWJyeW`l_C}D5%;wa)>PDhM_EV0 z^0K4}DsKAE)abjYLO334pQ7ASjoTwLAujH@6jWjYpPZ67sB!%3r76*#4?Dm>?@7t3~(WAqp z>F<|@?S3Gjx={paJLkjO0Axs=%Oe#%ujI!-pBiMH@{7iI)dg*-Zm_X945R0;LB0Nd z{@b_i?kS2|zz_@N37DYWU`MqH?s$hzN<|G7zsD+k?=D~oNa$?s1ZSDJtp&%W$l&8t zRpS?mxij$}SE@0~Y4W=)-=yU@z@8+YN@OmU&_dGx-&S(%WoN!1BV#>VI*V8+yu{BO z^unDoiq@jN#>P6t7h|0qg4H#p>Le{5OE}?iTG&UAMhDhMc)Ris=H}g-y@qdfl(MMG zRFsts2m4y#?BKGp%0l3XyG1Aq2NxgT&uctJrG=A5>SOXn2h^yqAh)NZu zsi#LY=Wrr7)wE7V5gZa$D<=^&6D-%(RCJUps=%P}yrGujDhF9Zu3a%YM*un7rqRI? zPtxw+F&+OZt46dfms6*%QgjOf)1c(c6>tDdsVb{d zm#ec>{IF$vIxaA6K)7c|$qDa@BU%^mWf44??yfoJ!Rk^N|lf&xj?YdFCw^#SYB zYed*{#ql$rX?XOLERz&MOW#Z&f&vqv!14=##8W)^&ixi_D?kb^UxBBxaoB;HEs24w z4WTE@@q|VsaL`A^e;HbihQLqCc*$-xflT8g+d67UStf)iG8ObDM4 z0)M){%5Y-i55_kX@^7`+6>{!t?!jjXbJx}p0BghttZvR+|G)ChHLzf^a3Z&Ke#_5g zLL05+9V2BsW~;|2iUBZ)<-y-#(s1O~Nhk5sr%xS7lO-KZ+?8i%fPf+u;HZ&FZCL<> zpy7D1F&qNIE?uG0d>X40D}cX&`BF+tD~1uJ;_~dS4u6Z3^?fmHw{iyg=5=fjG?_vo zh=(U(@Dg0qn@rtVNcr%U(5c9%4}&Gc!h=OcFL6SNcvVdqO1}%qSrt2#3k=Q19juJz zyklc$zxXz(!J3^T+f`NdGBzQB?0>~k@gm^T>&*qq~2(F zz(W>Rk_V=zS}o_Ys{vkW%OtE)=J@$ompU8KU`TkdS=;UL<;bgjiL0OZuIkpa1g}4#ju6 z!a_8514sl$td;i;H6^^XAHA&`8ydtV-Z5ceZS$PT!QNotVTAP;Vf^5;LuX)MFz=_K zG(nzFaR_+oZr5VzHj6^71=R{9HbaAH^Hjd|E8isCmU5jN9ZDK$a4nk_)hsp#H}#{@ zTaRD2L>%Ks5OE@N*VpG70!U_Iu(P}zZZTQ(jAyWLE&;xX)h?A`D?S2q&HHL({PB+SImNJ!e$t6g82o12f!%mka|(_A;?8^H3y zzjZ4qJvgk}_Mn1?N|~cFgK3c0Y4u zfGH=&Gu5E;MfDh_F?nWq$7s>_q^taw`lmmAxf=J#)DROAt8;d7>QIBH&?QBG_6%xU zN>o^R*(%N5AJD)ddxQdsC8a}O#AhRw&!%xxKw@SahhnFgDNB%08y=lg9cO8)O51tY|oG zf_r?atk-xl_Hszl($cZ%g_ z@}&xAU<~(ZXc2@xKRY)}PbF(nNyp4>l#zDS%hl%|MA&@x5h#RLLz;7~gBYw3`#}$C z0rgVPo9_JXtYAnTx@w5&guYTa8pB7bpjt{*IQxovy1BX8&JJ)IoWw@}NeD|BOp#PI zdXkk-HENGTdzuRj`N3{6F{DG#ZB0!+yn*>Rl!fD$nA?f$7YQ>YOi~p|v%g;eLvC$+ zN)uiL_s+tKT}o^5Uuj!mwg8Ei8D4rrP0GF$en(x2v>YS}`SX`8;e7JvVSnB?{ZlmN zsa4Z2LGQc8d(^8xCBZG;iwXd36yzTaGnsx-X0_WNa9sG6s&{2L{CxqelL0c02=MDZ zDgt_w!{a+niz1h8Uxn;f%TVwNEOO>s=rM$b6g(+J549<93;}YM+K|$>GaZ-Abt&)n zo-P(p^1xV{EW?-=E}n)2;M)fk^gmmET)rI&A_Iq+GAIaf_q}IPD&YA(E?8h=$|`qx z0(c7!#OahxEKhIBBMxp614CYTQeR6*JrQv7bhx*-$3|eL0y(oeuqixIV!#Vr$^to0 z_vin?DkO-Kb7pfwc35e!=7eUU1Oa{gv?a?Q^Z=`-FsvTp?d&To7cF_C@BRM) DQIk*= literal 0 HcmV?d00001 diff --git a/k8s/eks/bash/functions.sh b/k8s/eks/bash/functions.sh index 4ac87812..cca69b3a 100644 --- a/k8s/eks/bash/functions.sh +++ b/k8s/eks/bash/functions.sh @@ -1,7 +1,7 @@ #!/bin/bash # error log prep check -prep_log_dir(){ +prep_log_dir(){ mkdir -p $real_path/log/$start-log/ mkdir -p $real_path/.cluster/ } @@ -319,7 +319,7 @@ aws_get_vpc_id(){ vpc_id=$(aws ec2 describe-vpcs --region $REGION --filters Name=tag:Name,Values=eksctl-$CLUSTER_NAME-cluster/VPC |jq -r ".Vpcs[] | .VpcId") } -aws_get_subnet_id(){ +aws_get_subnet_id(){ aws_get_vpc_id concat_availability_zone upper_az=$(echo $AVAILABILITY_ZONE | tr '[:lower:]' '[:upper:]' | tr -d \-) @@ -334,7 +334,7 @@ aws_get_sg_id(){ aws_create_efs_mount_point(){ aws efs create-mount-target --file-system-id $efs_fs_id --subnet-id $subnet_id --security-group $efs_sg_id --region $REGION efs_dns=$efs_fs_id.efs.$REGION.amazonaws.com - + } create_cm_efs(){ @@ -365,10 +365,10 @@ aws_create_ebs(){ echo -e "EBS created with this volume ID: $ebs_volume\n" else echo "EBS already exists, skipping to the next step." - fi + fi } -make_persistent_volume(){ +make_persistent_volume(){ export TG_EBS_DATADIR_VOLUME_ID=$ebs_volume EBS_PV=$(mktemp) @@ -378,11 +378,11 @@ make_persistent_volume(){ helm_redis_add_repo(){ helm repo add bitnami https://charts.bitnami.com/bitnami -} +} helm_infra_install_redis(){ helm install testground-infra-redis --set auth.enabled=false --set master.nodeSelector='testground.node.role.infra: "true"' bitnami/redis -} +} helm_infra_install_influx_db(){ # We are using v2.6.1 of the helm chart, which has been evicted from the regular index.yaml. @@ -402,7 +402,7 @@ metadata: data: .env.toml: | ["aws"] - region = "$REGION" + region = "$REGION" [runners."cluster:k8s"] run_timeout_min = 15 @@ -506,26 +506,26 @@ log(){ echo "========================" echo "Log file generated with name $start-$CLUSTER_NAME.tar.gz" echo -e "\n" - rm -rf $real_path/log/$start-log/ + rm -rf $real_path/log/$start-log/ } ##### Functions below are used by the 'testground_uninstall.sh' script ####### -remove_efs_mp_timer(){ +remove_efs_mp_timer(){ efs_mp_state=available # setting the start value for the loop to consider sleep 15 - while [[ $efs_mp_state == available ]];do + while [[ $efs_mp_state == available ]];do efs_mp_state=$(aws efs describe-mount-targets --file-system-id $efs --region $region | jq -r ".MountTargets[] | .LifeCycleState") sleep 1 - done + done } -remove_efs_fs_timer(){ +remove_efs_fs_timer(){ efs_fs_state=available # setting the start value for the loop to consider - while [[ $efs_fs_state == available ]];do + while [[ $efs_fs_state == available ]];do efs_fs_state=$(aws efs describe-file-systems --region $region --file-system-id $efs | jq -r ".FileSystems[] | .LifeCycleState") sleep 1 - done + done } obtain_efs_id(){ @@ -621,7 +621,7 @@ cleanup(){ else echo -e "Looks like the EBS you have specified ($ebs) does not exist in the selected region ($region).\nIt is possible that it has already been deleted.\n" fi - + if [ "$efs_deleted" == "true" ] && [ "$ebs_deleted" == "true" ] && [ "$cluster_deleted" == "true" ] then rm -f $real_path/.cluster/$cluster_name-$region.cs @@ -629,4 +629,4 @@ cleanup(){ else echo -e "Uninstall script completed, but did not remove the '.cluster/$cluster_name-$region.cs' file due to other resources not being deleted.\nPlease check the '.cluster/$cluster_name-$region.cs' file and try again.\n" fi -} \ No newline at end of file +} diff --git a/k8s/eks/yaml/otel/configmap-collector.yml b/k8s/eks/yaml/otel/configmap-collector.yml new file mode 100644 index 00000000..49cac93f --- /dev/null +++ b/k8s/eks/yaml/otel/configmap-collector.yml @@ -0,0 +1,37 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: otel-collector-conf + labels: + app: opentelemetry + component: otel-collector-conf +data: + otel-collector-config: | + receivers: + otlp: + protocols: + grpc: + http: + exporters: + influxdb: + endpoint: http://influxdb:8086 + timeout: 500ms + org: default + bucket: testground + metrics_schema: telegraf-prometheus-v1 + + sending_queue: + enabled: true + num_consumers: 3 + queue_size: 10 + + retry_on_failure: + enabled: true + initial_interval: 1s + max_interval: 3s + max_elapsed_time: 10s + service: + pipelines: + metrics: + receivers: [otlp] + exporters: [influxdb] diff --git a/k8s/eks/yaml/otel/deployment.yml b/k8s/eks/yaml/otel/deployment.yml new file mode 100644 index 00000000..13733651 --- /dev/null +++ b/k8s/eks/yaml/otel/deployment.yml @@ -0,0 +1,47 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: otel-collector + labels: + app: opentelemetry + component: otel-collector +spec: + selector: + matchLabels: + app: opentelemetry + component: otel-collector + minReadySeconds: 5 + progressDeadlineSeconds: 120 + replicas: 3 + template: + metadata: + labels: + app: opentelemetry + component: otel-collector + spec: + containers: + - command: + - "/otelcol-contrib" + - "--config=/conf/otel-collector-config.yaml" + image: otel/opentelemetry-collector-contrib:0.70.0 + name: otel-collector + resources: + limits: + cpu: 2 + memory: 4Gi + requests: + cpu: 200m + memory: 400Mi + ports: + - containerPort: 4317 # Default endpoint for OpenTelemetry receiver. + - containerPort: 4318 + volumeMounts: + - name: otel-collector-config-vol + mountPath: /conf + volumes: + - configMap: + name: otel-collector-conf + items: + - key: otel-collector-config + path: otel-collector-config.yaml + name: otel-collector-config-vol diff --git a/k8s/eks/yaml/otel/service.yml b/k8s/eks/yaml/otel/service.yml new file mode 100644 index 00000000..5f40a4c4 --- /dev/null +++ b/k8s/eks/yaml/otel/service.yml @@ -0,0 +1,26 @@ +apiVersion: v1 +kind: Service +metadata: + name: otel-collector + labels: + app: opentelemetry + component: otel-collector +spec: + selector: + component: otel-collector + type: LoadBalancer + ports: + - name: otlp-grpc # Default endpoint for OpenTelemetry gRPC receiver. + port: 4317 + protocol: TCP + targetPort: 4317 + - name: otlp-http # Default endpoint for OpenTelemetry HTTP receiver. + port: 4318 + protocol: TCP + targetPort: 4318 + - name: metrics # Default endpoint for querying metrics. + port: 8888 + - name: otlp-exporter + port: 8889 + protocol: TCP + targetPort: 8889 diff --git a/k8s/eks/yaml/telegraf-operator/values.yaml b/k8s/eks/yaml/telegraf-operator/values.yaml new file mode 100644 index 00000000..cc386bab --- /dev/null +++ b/k8s/eks/yaml/telegraf-operator/values.yaml @@ -0,0 +1,47 @@ +replicaCount: 2 +image: + repository: quay.io/influxdb/telegraf-operator + pullPolicy: IfNotPresent + sidecarImage: "docker.io/library/telegraf:1.22" + +classes: + data: + default: | + [[outputs.influxdb]] + urls = ["http://influxdb:8086"] + database = "testground" + + +certManager: + enable: false + +imagePullSecrets: [] +nameOverride: "" +fullnameOverride: "" +serviceAccount: + # Annotations to add to the service account + annotations: {} +podSecurityContext: {} + # fsGroup: 2000 +securityContext: {} + # capabilities: + # drop: + # - ALL + # readOnlyRootFilesystem: true + # runAsNonRoot: true + # runAsUser: 1000 +resources: + limits: + cpu: 400m + memory: 256Mi + requests: + cpu: 50m + memory: 64Mi +nodeSelector: {} +tolerations: [] +affinity: {} +requireAnnotationsForSecret: false +# allow hot reload ; disabled by default to support versions of telegraf +# that do not support hot-reload and --watch-config flag +hotReload: true + diff --git a/k8s/kops/01_install_k8s.sh b/k8s/kops/01_install_k8s.sh deleted file mode 100755 index 0e65ec55..00000000 --- a/k8s/kops/01_install_k8s.sh +++ /dev/null @@ -1,109 +0,0 @@ -#!/bin/bash - -set -o errexit -set -o pipefail -set -e - -err_report() { - echo "Error on line $1" -} - -trap 'err_report $LINENO' ERR - -START_TIME=`date +%s` - -echo "Creating cluster for Testground..." -echo - -CLUSTER_SPEC_TEMPLATE=$1 - -my_dir="$(dirname "$0")" -source "$my_dir/install-playbook/validation.sh" - -echo "Required arguments" -echo "------------------" -echo "Deployment name (DEPLOYMENT_NAME): $DEPLOYMENT_NAME" -echo "Cluster name (CLUSTER_NAME): $CLUSTER_NAME" -echo "Kops state store (KOPS_STATE_STORE): $KOPS_STATE_STORE" -echo "AWS availability zone A (ZONE_A): $ZONE_A" -echo "AWS availability zone B (ZONE_B): $ZONE_B" -echo "AWS region (AWS_REGION): $AWS_REGION" -echo "AWS worker node type (WORKER_NODE_TYPE): $WORKER_NODE_TYPE" -echo "AWS master node type (MASTER_NODE_TYPE): $MASTER_NODE_TYPE" -echo "Worker nodes (WORKER_NODES): $WORKER_NODES" -echo "Public key (PUBKEY): $PUBKEY" -echo - -CLUSTER_SPEC=$(mktemp) -envsubst <$CLUSTER_SPEC_TEMPLATE >$CLUSTER_SPEC - -# Verify with the user before continuing. -echo -echo "The cluster will be built based on the params above." -echo -n "Do they look right to you? [y/n]: " -read response - -if [ "$response" != "y" ] -then - echo "Canceling ." - exit 2 -fi - -# The remainder of this script creates the cluster using the generated template - -kops create -f $CLUSTER_SPEC -kops create secret --name $CLUSTER_NAME sshpublickey admin -i $PUBKEY -kops update cluster $CLUSTER_NAME --yes - -# Wait for worker nodes and master to be ready -kops validate cluster --wait 20m - -echo "Cluster nodes are Ready" -echo - -echo "Install default container limits" -echo - -kubectl apply -f ./limit-range/limit-range.yaml - -echo "Install Weave, CNI-Genie, Sidecar Daemonset..." -echo - -kubectl apply -f ./kops-weave/weave.yml \ - -f ./kops-weave/genie-plugin.yaml \ - -f ./kops-weave/dummy.yml \ - -f ./sidecar.yaml - -echo "Installing Prometheus" -pushd prometheus-operator -helm install prometheus-operator stable/prometheus-operator -f values.yaml -popd - -echo "Installing InfluxDB" -pushd influxdb -helm install influxdb bitnami/influxdb -f ./values.yaml -popd - - -echo "Installing Redis and Grafana dashboards" -pushd testground-infra -helm dep build -helm install testground-infra . -popd - -echo "Install Weave service monitor..." -echo - -kubectl apply -f ./kops-weave/weave-metrics-service.yml \ - -f ./kops-weave/weave-service-monitor.yml - -echo "Wait for Sidecar to be Ready..." -echo -RUNNING_SIDECARS=0 -while [ "$RUNNING_SIDECARS" -ne "$WORKER_NODES" ]; do RUNNING_SIDECARS=$(kubectl get pods | grep testground-sidecar | grep Running | wc -l || true); echo "Got $RUNNING_SIDECARS running sidecar pods"; sleep 5; done; - -echo "Testground cluster is installed" -echo - -END_TIME=`date +%s` -echo "Execution time was `expr $END_TIME - $START_TIME` seconds" diff --git a/k8s/kops/02_efs.sh b/k8s/kops/02_efs.sh deleted file mode 100755 index 4f781181..00000000 --- a/k8s/kops/02_efs.sh +++ /dev/null @@ -1,86 +0,0 @@ -#!/bin/bash - -set -o errexit -set -o pipefail -set -e - -err_report() { - echo "Error on line $1" -} - -trap 'err_report $LINENO' ERR - -START_TIME=`date +%s` - -CLUSTER_SPEC_TEMPLATE=$1 - -my_dir="$(dirname "$0")" -source "$my_dir/install-playbook/validation.sh" - -echo "Installing EFS..." - -vpcId=`aws ec2 describe-vpcs --region=$AWS_REGION --filters Name=tag:Name,Values=$CLUSTER_NAME --output text | awk '/VPCS/ { print $8 }'` - -if [[ -z ${vpcId} ]]; then - echo "Couldn't detect AWS VPC created by `kops`" - exit 1 -fi - -echo "Detected VPC: $vpcId" - -securityGroupId=`aws ec2 describe-security-groups --region=$AWS_REGION --output text | awk '/nodes.'$CLUSTER_NAME'/ && /SECURITYGROUPS/ { print $6 };'` - -if [[ -z ${securityGroupId} ]]; then - echo "Couldn't detect AWS Security Group created by `kops`" - exit 1 -fi - -echo "Detected Security Group ID: $securityGroupId" - -subnetIdZoneA=`aws ec2 describe-subnets --region=$AWS_REGION --output text | awk '/'$vpcId'/ { print $15 }' | sort | head -1` -subnetIdZoneB=`aws ec2 describe-subnets --region=$AWS_REGION --output text | awk '/'$vpcId'/ { print $15 }' | sort | tail -1` - -echo "Detected Subnet: $subnetIdZoneA" -echo "Detected Subnet: $subnetIdZoneB" - -pushd efs-terraform - -# extract s3 bucket from kops state store -S3_BUCKET="${KOPS_STATE_STORE:5:100}" - -# create EFS file system -terraform init -backend-config=bucket=$S3_BUCKET \ - -backend-config=key=${DEPLOYMENT_NAME}-efs \ - -backend-config=region=$AWS_REGION - -terraform apply -var aws_region=$AWS_REGION -var fs_subnet_id_zone_a=$subnetIdZoneA -var fs_subnet_id_zone_b=$subnetIdZoneB -var fs_sg_id=$securityGroupId -auto-approve - -export EFS_DNSNAME=`terraform output -raw dns_name` - -fsId=`terraform output -raw filesystem_id` - -popd - -echo "Install EFS Kubernetes provisioner..." - -kubectl create configmap efs-provisioner \ ---from-literal=file.system.id=$fsId \ ---from-literal=aws.region=$AWS_REGION \ ---from-literal=provisioner.name=testground.io/aws-efs - -EFS_MANIFEST_SPEC=$(mktemp) -envsubst <./efs/manifest.yaml.spec >$EFS_MANIFEST_SPEC - -kubectl apply -f ./efs/rbac.yaml \ - -f $EFS_MANIFEST_SPEC - -echo "Wait for EFS provisioner to be Running..." -echo -RUNNING_EFS=0 -while [ "$RUNNING_EFS" -ne 1 ]; do RUNNING_EFS=$(kubectl get pods | grep efs-provisioner | grep Running | wc -l || true); echo "Got $RUNNING_EFS running efs-provisioner pods"; sleep 5; done; - -echo "EFS provisioner is ready" -echo - -END_TIME=`date +%s` -echo "Execution time was `expr $END_TIME - $START_TIME` seconds" diff --git a/k8s/kops/03_ebs.sh b/k8s/kops/03_ebs.sh deleted file mode 100755 index b46397c7..00000000 --- a/k8s/kops/03_ebs.sh +++ /dev/null @@ -1,52 +0,0 @@ -#!/bin/bash - -set -o errexit -set -o pipefail -set -e - -err_report() { - echo "Error on line $1" -} - -trap 'err_report $LINENO' ERR - -START_TIME=`date +%s` - -CLUSTER_SPEC_TEMPLATE=$1 - -my_dir="$(dirname "$0")" -source "$my_dir/install-playbook/validation.sh" - -echo "Installing EBS..." - -# extract s3 bucket from kops state store -S3_BUCKET="${KOPS_STATE_STORE:5:100}" - -# create EBS volume for testground datadir -pushd ebs-terraform - -terraform init -backend-config=bucket=$S3_BUCKET \ - -backend-config=key=${DEPLOYMENT_NAME}-ebs \ - -backend-config=region=$AWS_REGION - -terraform apply -var aws_region=$AWS_REGION -var aws_availability_zone=${AWS_REGION}a -auto-approve - -export TG_EBS_DATADIR_VOLUME_ID="aws://`terraform output -raw availability_zone`/`terraform output -raw volume_id`" - -popd - -echo "Got volume id for Testground datadir: $TG_EBS_DATADIR_VOLUME_ID" - -EBS_PV=$(mktemp) -envsubst <./ebs/pv.yml.spec >$EBS_PV - -kubectl apply -f ./ebs/storageclass.yml \ - -f $EBS_PV \ - -f ./ebs/pvc.yml - - -echo "EBS volume for Testground daemon is ready" -echo - -END_TIME=`date +%s` -echo "Execution time was `expr $END_TIME - $START_TIME` seconds" diff --git a/k8s/kops/04_testground_daemon.sh b/k8s/kops/04_testground_daemon.sh deleted file mode 100755 index 9012f188..00000000 --- a/k8s/kops/04_testground_daemon.sh +++ /dev/null @@ -1,32 +0,0 @@ -#!/bin/bash - -set -o errexit -set -o pipefail -set -e - -err_report() { - echo "Error on line $1" -} - -trap 'err_report $LINENO' ERR - -START_TIME=`date +%s` - -CLUSTER_SPEC_TEMPLATE=$1 - -my_dir="$(dirname "$0")" -source "$my_dir/install-playbook/validation.sh" - -echo "Installing Testground daemon..." -echo -kubectl apply -f ./testground-daemon/config-map-env-toml.yml -kubectl apply -f ./testground-daemon/service-account.yml -kubectl apply -f ./testground-daemon/role-binding.yml -kubectl apply -f ./testground-daemon/deployment.yml -f ./testground-daemon/service.yml - -echo "Testground daemon is ready" -echo - -END_TIME=`date +%s` -echo "Execution time was `expr $END_TIME - $START_TIME` seconds" - diff --git a/k8s/kops/README.md b/k8s/kops/README.md index 251cf2bc..120148ca 100644 --- a/k8s/kops/README.md +++ b/k8s/kops/README.md @@ -1,3 +1,154 @@ -## Dependency versions +# Description -Newest additions to the code (for booting up efs and ebs) have been tested with terraform v1.1.9 +This page provides a detailed overview of TestGround, its services, infrastructure, architecture, and more. TestGround is an open-source testing framework that helps blockchain developers test their systems at scale. It offers a range of features and tools to facilitate the testing process, including network emulation, performance monitoring, and automated testing. + +--- + +# Why TestGround? + +TestGround was created to solve the challenges associated with testing blockchain systems at scale. The main goal of the framework is to provide developers with a tool that can generate realistic feedback on blockchain performance. TestGround accomplishes this by creating an environment that simulates real-world conditions, allowing developers to test their systems under different scenarios and loads. + +TestGround leverages Kubernetes as the underlying platform for testing. Kubernetes is an open-source container orchestration system that automates the deployment, scaling, and management of containerized applications. By using Kubernetes, TestGround can take advantage of its robust features, including high availability, scalability, and fault tolerance. Additionally, Kubernetes makes it easy to manage TestGround clusters and provides a flexible and efficient way to run tests in parallel (soon). + +--- + +# Kubernetes Cluster + +## Cluster Creation + +Creating a new Kubernetes cluster with TestGround is a straightforward process. To get started, you'll need to import the desired variables, such as the number of nodes, the type of node, and the network configuration. Once you have your variables in place, you can execute the create-cluster script. The create-cluster script is a shell script that automates the process of creating a new Kubernetes cluster. It performs all the necessary steps, including setting up the cluster infrastructure, configuring the network, and deploying the necessary components. + +In addition to the create-cluster script, TestGround provides other scripts to manage your cluster, such as scale-up and scale-down scripts to increase or decrease the number of nodes in the cluster. TestGround also includes a teardown script to remove the cluster when you're done testing. + +The variables are: + +```bash +#!/bin/bash +export CLUSTER_NAME= +export DEPLOYMENT_NAME= +export WORKER_NODE_TYPE= +export MASTER_NODE_TYPE= +export MAX_WORKER_NODES= +export MIN_WORKER_NODES= +export TEAM= +export PROJECT= +export AWS_REGION= +export KOPS_STATE_STORE= +export ZONE_A= +export ZONE_B= +export PUBKEY= +export AWS_ACCESS_KEY_ID=$(aws configure get aws_access_key_id) +export AWS_SECRET_ACCESS_KEY=$(aws configure get aws_secret_access_key) +``` + +```bash +./install_k8s.sh cluster.yaml +``` + +Wait some minutes until you will see that the script has finished and the cluster is up & running. + +--- + +## Cluster Provisioning + +After executing the **install_k8s.sh** script, you'll notice that some pods are being created automatically. This is because TestGround is provisioning the cluster with all the required tools to run tests at scale. TestGround uses ArgoCD, an open-source continuous delivery tool, to manage the deployment and configuration of the required components. + +ArgoCD provides a declarative way to manage applications and infrastructure as code. It helps to automate the process of deploying and managing the necessary components in the cluster, ensuring that the TestGround environment is up and running quickly and efficiently. By using ArgoCD, TestGround can manage the deployment and configuration of the required components automatically, ensuring that developers can focus on testing their blockchain systems rather than managing infrastructure. + +If you're interested, you can access the ArgoCD dashboard to monitor what's happening in the background. The dashboard provides a comprehensive view of the state of the TestGround environment, including the deployment status of the required components and any errors or issues that may arise. Additionally, the dashboard provides access to various configuration settings, enabling developers to fine-tune the TestGround environment to meet their specific needs. + +In summary, TestGround uses ArgoCD to automate the process of deploying and managing the required components in the cluster. The ArgoCD dashboard provides a comprehensive view of the state of the TestGround environment, enabling developers to monitor the deployment status of the required components and fine-tune the environment to meet their specific needs. + +```bash +kubectl -n argocd port-forward svc/argocd-server 8080:443 +``` + +The default user is `admin` and you can find the ArgoCD password using the command: + +```bash +kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d +``` + +In the dashboard, you will find something like: + +![alt text](../../docs/argocd_dashboard.png "argocd_dashboard") + +All these boxes are the ArgoCD apps, meaning that we are provisioning the cluster using these apps and the ArgoCD’s purpose is to manage them and be sure that all of them are working fine and synced. + +*The whole process takes around 12-15’, so please, be patient.* + +--- + +**note**: + +If *you access the ArgoCD dashboard, and you find the root application in Unknown status, please wait, this s because the services are being created in the background, they will be ready in the next 5-6’.* + +![alt text](../../docs/argocd_root_app.png "argocd_root_app") + +--- + +# Kubeconfig + + +Once you have created the cluster, you might need to export the [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/). + +You can do it by executing the following command: + +```bash +# First of all, you will need to source the variables in order to use them later here. +kops export kubecfg --admin --kubeconfig ~/.kube/cluster_kubeconfig --state=$KOPS_STATE_STORE --name=$CLUSTER_NAME +``` + +--- + +# Scripts + +List of the scripts and their purpose: + +- **install_k8s.sh:** + + ## Description + + This is a Bash script that is used to spin up a new Kubernetes cluster using Kops. Also, the tool creates an ArgoCD app to provision the cluster. The script requires the following variables to work: + + - **`CLUSTER_NAME`**: the name of the cluster. + - **`DEPLOYMENT_NAME`**: the name of the deployment. + - **`WORKER_NODE_TYPE`**: the type of worker node. + - **`MASTER_NODE_TYPE`**: the type of the master node. + - **`MAX_WORKER_NODES`**: max number of worker nodes. + - **`MIN_WORKER_NODES`**: min number of worker nodes. + - **`TEAM`**: the name of the team responsible for the deployment. + - **`PROJECT`**: the name of the project. + - **`AWS_REGION`**: the region where the cluster is to be created. + - **`KOPS_STATE_STORE`**: the name of the S3 bucket where the Kops state store is located. + - **`ZONE_A`**: the availability zone A for the cluster. + - **`ZONE_B`**: the availability zone B for the cluster. + - **`PUBKEY`**: the path to the public key that will be used to access the cluster. + - **`AWS_ACCESS_KEY_ID`**: the AWS access key ID to access the AWS account where the cluster will be created. This variable is set using the **`aws configure get aws_access_key_id`** command. + - **`AWS_SECRET_ACCESS_KEY`**: the AWS secret access key to access the AWS account where the cluster will be created. This variable is set using the **`aws configure get aws_secret_access_key`** command. + + The script creates an ArgoCD app, installs Weave, CNI-Genie, and Sidecar Daemonset, and creates EFS resources & storage classes. The script also installs a default container limit and sets up an ArgoCD root app. + + ### **Usage** + + To use this script, you need to set the required variables in the script and execute it using the command**`./01_install_k8s cluster.yaml`**. + + ### **Dependencies** + + This script has the following dependencies: + + - **`awscli`** + - **`envsubst`** + - **`kubectl`** + - **`kops`** + - **`terraform`** + - **`helm`** +- **delete_kops.sh**: delete all the resources, Terraform, and the Kubernetes cluster. + +--- + +# Conclusion + +Overall, TestGround is a powerful testing framework that offers a range of features and tools to facilitate blockchain testing. By using Kubernetes as the underlying platform, TestGround can take advantage of its robust features, making it easy to manage clusters and run tests in parallel (soon). If you're a blockchain developer looking to test your system at scale, TestGround is definitely worth checking out. + +--- diff --git a/k8s/kops/argocd/values.yaml b/k8s/kops/argocd/values.yaml new file mode 100644 index 00000000..f8c10a32 --- /dev/null +++ b/k8s/kops/argocd/values.yaml @@ -0,0 +1,27 @@ +# https://github.com/argoproj/argo-helm/blob/main/charts/argo-cd/values.yaml +redis-ha: + enabled: true + +controller: + replicas: 1 + +server: + autoscaling: + enabled: true + minReplicas: 2 + +repoServer: + autoscaling: + enabled: true + minReplicas: 2 + +applicationSet: + replicaCount: 2 + +## Argo Configs +configs: + # General Argo CD configuration + ## Ref: https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/argocd-cm.yaml + cm: + kustomize.buildOptions: "--enable-helm" + helm.enabled: "true" diff --git a/k8s/kops/cluster.yaml b/k8s/kops/cluster.yaml index 3e316d55..a393edb3 100644 --- a/k8s/kops/cluster.yaml +++ b/k8s/kops/cluster.yaml @@ -1,3 +1,4 @@ +--- apiVersion: kops.k8s.io/v1alpha2 kind: Cluster metadata: @@ -16,13 +17,39 @@ spec: type: Public kubeDNS: provider: CoreDNS - additionalPolicies: + # https://kops.sigs.k8s.io/addons/#metrics-server + certManager: + enabled: true + metricsServer: + enabled: true + insecure: false + clusterAutoscaler: + enabled: true + additionalPolicies: # https://kops.sigs.k8s.io/iam_roles/#adding-additional-policies node: | [ { "Effect": "Allow", "Action": [ - "ecr:*" + "ecr:*", + "ec2:*", + "autoscaling:*", + "elasticfilesystem:*", + "elasticloadbalancing:*" + ], + "Resource": "*" + } + ] + master: | + [ + { + "Effect": "Allow", + "Action": [ + "ecr:*", + "ec2:*", + "autoscaling:*", + "elasticfilesystem:*", + "elasticloadbalancing:*" ], "Resource": "*" } @@ -38,14 +65,22 @@ spec: etcdClusters: - cpuRequest: 200m etcdMembers: - - instanceGroup: master-${ZONE} + - instanceGroup: master-${ZONE}0 name: a + - instanceGroup: master-${ZONE}1 + name: b + - instanceGroup: master-${ZONE}2 + name: c memoryRequest: 100Mi name: main - cpuRequest: 100m etcdMembers: - - instanceGroup: master-${ZONE} + - instanceGroup: master-${ZONE}0 name: a + - instanceGroup: master-${ZONE}1 + name: b + - instanceGroup: master-${ZONE}2 + name: c memoryRequest: 100Mi name: events hooks: @@ -61,7 +96,7 @@ spec: legacy: false kubelet: anonymousAuth: false - maxPods: 200 + maxPods: 200 # 234 is the max number of pods for c5.4xlarge allowedUnsafeSysctls: - net.netfilter.nf_conntrack_max - net.core.somaxconn @@ -87,22 +122,27 @@ spec: kubeAPIBurst: 40 kubernetesApiAccess: - 0.0.0.0/0 - kubernetesVersion: 1.18.10 + # the latest version we can use atm is this one, due to the Docker CRI + kubernetesVersion: 1.23.4 # https://sysdig.com/blog/kubernetes-1-23-whats-new/ needed -> persistentVolumeClaimRetentionPolicy + # kubernetesVersion: 1.22.17 masterInternalName: api.internal.${CLUSTER_NAME} masterPublicName: api.${CLUSTER_NAME} - networkCIDR: 172.20.0.0/16 + networkCIDR: 192.168.0.0/16 networking: - flannel: - backend: vxlan - nonMasqueradeCIDR: 100.64.0.0/10 + #amazonvpc: {} # https://kops.sigs.k8s.io/networking/aws-vpc/ + amazonvpc: + env: + - name: AWS_VPC_K8S_CNI_LOGLEVEL + value: debug + #nonMasqueradeCIDR: 100.64.0.0/10 sshAccess: - 0.0.0.0/0 subnets: - - cidr: 172.20.32.0/19 + - cidr: 192.168.0.0/17 # hosts: 32,768 name: ${ZONE_A} type: Public zone: ${ZONE_A} - - cidr: 172.20.64.0/19 + - cidr: 192.168.128.0/17 # hosts: 32,768 name: ${ZONE_B} type: Public zone: ${ZONE_B} @@ -113,14 +153,13 @@ spec: nodes: public --- - apiVersion: kops.k8s.io/v1alpha2 kind: InstanceGroup metadata: creationTimestamp: null labels: kops.k8s.io/cluster: ${CLUSTER_NAME} - name: master-${ZONE} + name: master-${ZONE}0 spec: additionalUserData: - name: myscript.sh @@ -128,40 +167,51 @@ spec: content: | #!/bin/sh cat <> /etc/sysctl.d/999-testground.conf + fs.file-max = 3178504 net.core.somaxconn = 131072 net.netfilter.nf_conntrack_max = 1048576 - net.ipv4.tcp_max_syn_backlog = 131072 net.core.netdev_max_backlog = 524288 + net.core.rmem_max = 16777216 + net.core.wmem_max = 16777216 + net.ipv4.tcp_rmem = 16384 131072 16777216 + net.ipv4.tcp_wmem = 16384 131072 16777216 + net.ipv4.tcp_mem = 262144 524288 1572864 + net.ipv4.tcp_max_syn_backlog = 131072 net.ipv4.ip_local_port_range = 10000 65535 - net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1 - net.core.rmem_max = 4194304 - net.core.wmem_max = 4194304 - net.ipv4.tcp_mem = 262144 524288 1572864 - net.ipv4.tcp_rmem = 16384 131072 4194304 - net.ipv4.tcp_wmem = 16384 131072 4194304 + net.ipv4.ip_forward = 1 + net.ipv4.conf.all.rp_filter = 0 net.ipv4.neigh.default.gc_thresh2 = 4096 net.ipv4.neigh.default.gc_thresh3 = 32768 EOT - image: 909427826938/testground_2020-10-30 + cat < /etc/security/limits.d/999-limits.conf + * soft nproc 131072 + * hard nproc 262144 + * soft nofile 131072 + * hard nofile 262144 + LIMITS + image: testground_2023-05-04 machineType: ${MASTER_NODE_TYPE} - maxSize: 1 + maxSize: 3 minSize: 1 + cloudLabels: + k8s.io/cluster-autoscaler/${CLUSTER_NAME}: "owned" + k8s.io/cluster-autoscaler/enabled: "" + k8s.io/cluster-autoscaler/node-template/label: "" nodeLabels: - kops.k8s.io/instancegroup: master-${ZONE_A} + kops.k8s.io/instancegroup: master-${ZONE_A}0 role: Master subnets: - ${ZONE_A} --- - apiVersion: kops.k8s.io/v1alpha2 kind: InstanceGroup metadata: creationTimestamp: null labels: kops.k8s.io/cluster: ${CLUSTER_NAME} - name: nodes + name: master-${ZONE}1 spec: additionalUserData: - name: myscript.sh @@ -169,44 +219,51 @@ spec: content: | #!/bin/sh cat <> /etc/sysctl.d/999-testground.conf + fs.file-max = 3178504 net.core.somaxconn = 131072 net.netfilter.nf_conntrack_max = 1048576 - net.ipv4.tcp_max_syn_backlog = 131072 net.core.netdev_max_backlog = 524288 + net.core.rmem_max = 16777216 + net.core.wmem_max = 16777216 + net.ipv4.tcp_rmem = 16384 131072 16777216 + net.ipv4.tcp_wmem = 16384 131072 16777216 + net.ipv4.tcp_mem = 262144 524288 1572864 + net.ipv4.tcp_max_syn_backlog = 131072 net.ipv4.ip_local_port_range = 10000 65535 - net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1 - net.core.rmem_max = 4194304 - net.core.wmem_max = 4194304 - net.ipv4.tcp_mem = 262144 524288 1572864 - net.ipv4.tcp_rmem = 16384 131072 4194304 - net.ipv4.tcp_wmem = 16384 131072 4194304 + net.ipv4.ip_forward = 1 + net.ipv4.conf.all.rp_filter = 0 net.ipv4.neigh.default.gc_thresh2 = 4096 net.ipv4.neigh.default.gc_thresh3 = 32768 EOT + cat < /etc/security/limits.d/999-limits.conf + * soft nproc 131072 + * hard nproc 262144 + * soft nofile 131072 + * hard nofile 262144 + LIMITS + image: testground_2023-05-04 + machineType: ${MASTER_NODE_TYPE} + maxSize: 3 + minSize: 1 cloudLabels: - testground.node.role.plan: "true" - image: 909427826938/testground_2020-10-30 - machineType: ${WORKER_NODE_TYPE} - maxSize: ${WORKER_NODES} - minSize: ${WORKER_NODES} + k8s.io/cluster-autoscaler/${CLUSTER_NAME}: "owned" + k8s.io/cluster-autoscaler/enabled: "" + k8s.io/cluster-autoscaler/node-template/label: "" nodeLabels: - kops.k8s.io/instancegroup: nodes - testground.node.role.plan: "true" - role: Node + kops.k8s.io/instancegroup: master-${ZONE_A}1 + role: Master subnets: - ${ZONE_A} - - ${ZONE_B} --- - apiVersion: kops.k8s.io/v1alpha2 kind: InstanceGroup metadata: creationTimestamp: null labels: kops.k8s.io/cluster: ${CLUSTER_NAME} - name: tginfra + name: master-${ZONE}2 spec: additionalUserData: - name: myscript.sh @@ -214,31 +271,203 @@ spec: content: | #!/bin/sh cat <> /etc/sysctl.d/999-testground.conf + fs.file-max = 3178504 net.core.somaxconn = 131072 net.netfilter.nf_conntrack_max = 1048576 - net.ipv4.tcp_max_syn_backlog = 131072 net.core.netdev_max_backlog = 524288 + net.core.rmem_max = 16777216 + net.core.wmem_max = 16777216 + net.ipv4.tcp_rmem = 16384 131072 16777216 + net.ipv4.tcp_wmem = 16384 131072 16777216 + net.ipv4.tcp_mem = 262144 524288 1572864 + net.ipv4.tcp_max_syn_backlog = 131072 net.ipv4.ip_local_port_range = 10000 65535 - net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1 - net.core.rmem_max = 4194304 - net.core.wmem_max = 4194304 - net.ipv4.tcp_mem = 262144 524288 1572864 - net.ipv4.tcp_rmem = 16384 131072 4194304 - net.ipv4.tcp_wmem = 16384 131072 4194304 + net.ipv4.ip_forward = 1 + net.ipv4.conf.all.rp_filter = 0 net.ipv4.neigh.default.gc_thresh2 = 4096 net.ipv4.neigh.default.gc_thresh3 = 32768 EOT + cat < /etc/security/limits.d/999-limits.conf + * soft nproc 131072 + * hard nproc 262144 + * soft nofile 131072 + * hard nofile 262144 + LIMITS + image: testground_2023-05-04 + machineType: ${MASTER_NODE_TYPE} + maxSize: 3 + minSize: 1 + cloudLabels: + k8s.io/cluster-autoscaler/${CLUSTER_NAME}: "owned" + k8s.io/cluster-autoscaler/enabled: "" + k8s.io/cluster-autoscaler/node-template/label: "" + nodeLabels: + kops.k8s.io/instancegroup: master-${ZONE_A}2 + role: Master + subnets: + - ${ZONE_A} + +--- +apiVersion: kops.k8s.io/v1alpha2 +kind: InstanceGroup +metadata: + creationTimestamp: null + labels: + kops.k8s.io/cluster: ${CLUSTER_NAME} + name: nodes +spec: + kubelet: + maxPods: 117 # as we reduced the resources to the half + allowed-unsafe-sysctls: net.core.somaxconn + use-max-pods: false + sysctlParameters: + - fs.file-max = 3178504 + - net.core.somaxconn = 131072 + - net.netfilter.nf_conntrack_max = 1048576 + - net.core.netdev_max_backlog = 524288 + - net.core.rmem_max = 16777216 + - net.core.wmem_max = 16777216 + - net.ipv4.tcp_rmem = 16384 131072 16777216 + - net.ipv4.tcp_wmem = 16384 131072 16777216 + - net.ipv4.tcp_mem = 262144 524288 1572864 + - net.ipv4.tcp_max_syn_backlog = 131072 + - net.ipv4.ip_local_port_range = 10000 65535 + - net.ipv4.tcp_tw_reuse = 1 + - net.ipv4.ip_forward = 1 + - net.ipv4.conf.all.rp_filter = 0 + - net.ipv4.neigh.default.gc_thresh2 = 4096 + - net.ipv4.neigh.default.gc_thresh3 = 32768 + additionalUserData: + - name: myscript.sh + type: text/x-shellscript + content: | + #!/bin/sh + #cat <> /etc/sysctl.d/999-testground.conf + #fs.file-max = 3178504 + #net.core.somaxconn = 131072 + #net.netfilter.nf_conntrack_max = 1048576 + #net.core.netdev_max_backlog = 524288 + #net.core.rmem_max = 16777216 + #net.core.wmem_max = 16777216 + #net.ipv4.tcp_rmem = 16384 131072 16777216 + #net.ipv4.tcp_wmem = 16384 131072 16777216 + #net.ipv4.tcp_mem = 262144 524288 1572864 + #net.ipv4.tcp_max_syn_backlog = 131072 + #net.ipv4.ip_local_port_range = 10000 65535 + #net.ipv4.tcp_tw_reuse = 1 + #net.ipv4.ip_forward = 1 + #net.ipv4.conf.all.rp_filter = 0 + #net.ipv4.neigh.default.gc_thresh2 = 4096 + #net.ipv4.neigh.default.gc_thresh3 = 32768 + #EOT + cat < /etc/security/limits.d/999-limits.conf + * soft nproc 131072 + * hard nproc 262144 + * soft nofile 131072 + * hard nofile 262144 + LIMITS + cloudLabels: + testground.node.role.plan: "true" + image: testground_2023-05-04 + machineType: ${WORKER_NODE_TYPE} + ##mixedInstancesPolicy: + ## instances: + ## - c5.9xlarge + ## onDemandAboveBase: 0 + ## onDemandBase: 0 + ## spotAllocationStrategy: capacity-optimized + maxSize: ${MAX_WORKER_NODES} + minSize: ${MIN_WORKER_NODES} + cloudLabels: + k8s.io/cluster-autoscaler/${CLUSTER_NAME}: "owned" + k8s.io/cluster-autoscaler/enabled: "" + k8s.io/cluster-autoscaler/node-template/label: "" + nodeLabels: + kops.k8s.io/instancegroup: nodes + testground.node.role.plan: "true" + role: Node + subnets: + - ${ZONE_A} + +--- +apiVersion: kops.k8s.io/v1alpha2 +kind: InstanceGroup +metadata: + creationTimestamp: null + labels: + kops.k8s.io/cluster: ${CLUSTER_NAME} + name: tginfra +spec: + kubelet: + maxPods: 58 + use-max-pods: false + sysctlParameters: + - fs.file-max = 3178504 + - net.core.somaxconn = 131072 + - net.netfilter.nf_conntrack_max = 1048576 + - net.core.netdev_max_backlog = 524288 + - net.core.rmem_max = 16777216 + - net.core.wmem_max = 16777216 + - net.ipv4.tcp_rmem = 16384 131072 16777216 + - net.ipv4.tcp_wmem = 16384 131072 16777216 + - net.ipv4.tcp_mem = 262144 524288 1572864 + - net.ipv4.tcp_max_syn_backlog = 131072 + - net.ipv4.ip_local_port_range = 10000 65535 + - net.ipv4.tcp_tw_reuse = 1 + - net.ipv4.ip_forward = 1 + - net.ipv4.conf.all.rp_filter = 0 + - net.ipv4.neigh.default.gc_thresh2 = 4096 + - net.ipv4.neigh.default.gc_thresh3 = 32768 + additionalUserData: + - name: myscript.sh + type: text/x-shellscript + content: | + #!/bin/sh + #cat <> /etc/sysctl.d/999-testground.conf + #fs.file-max = 3178504 + #net.core.somaxconn = 131072 + #net.netfilter.nf_conntrack_max = 1048576 + #net.core.netdev_max_backlog = 524288 + #net.core.rmem_max = 16777216 + #net.core.wmem_max = 16777216 + #net.ipv4.tcp_rmem = 16384 131072 16777216 + #net.ipv4.tcp_wmem = 16384 131072 16777216 + #net.ipv4.tcp_mem = 262144 524288 1572864 + #net.ipv4.tcp_max_syn_backlog = 131072 + #net.ipv4.ip_local_port_range = 10000 65535 + #net.ipv4.tcp_tw_reuse = 1 + #net.ipv4.ip_forward = 1 + #net.ipv4.conf.all.rp_filter = 0 + #net.ipv4.neigh.default.gc_thresh2 = 4096 + #net.ipv4.neigh.default.gc_thresh3 = 32768 + #EOT + cat < /etc/security/limits.d/999-limits.conf + * soft nproc 131072 + * hard nproc 262144 + * soft nofile 131072 + * hard nofile 262144 + LIMITS cloudLabels: testground.node.role.infra: "true" - image: 909427826938/testground_2020-10-30 - machineType: c5.2xlarge - maxSize: 2 + image: testground_2023-05-04 + #machineType: c5.4xlarge + machineType: c5.9xlarge + #mixedInstancesPolicy: + # instances: + # - c5.9xlarge + # onDemandAboveBase: 0 + # onDemandBase: 0 + # spotAllocationStrategy: capacity-optimized minSize: 2 + maxSize: 10 + cloudLabels: + k8s.io/cluster-autoscaler/${CLUSTER_NAME}: "owned" + k8s.io/cluster-autoscaler/enabled: "" + k8s.io/cluster-autoscaler/node-template/label: "" nodeLabels: kops.k8s.io/instancegroup: tginfra testground.node.role.infra: "true" role: Node subnets: - ${ZONE_A} - - ${ZONE_B} diff --git a/k8s/kops/delete_ebs.sh b/k8s/kops/delete_ebs.sh deleted file mode 100755 index 6b1563e4..00000000 --- a/k8s/kops/delete_ebs.sh +++ /dev/null @@ -1,25 +0,0 @@ -#!/bin/bash - -set -o errexit -set -o pipefail -set -x -set -e - -err_report() { - echo "Error on line $1" -} - -trap 'err_report $LINENO' ERR - -pushd ebs-terraform - -# extract s3 bucket from kops state store -S3_BUCKET="${KOPS_STATE_STORE:5:100}" - -terraform init -backend-config=bucket=$S3_BUCKET \ - -backend-config=key=${DEPLOYMENT_NAME}-ebs \ - -backend-config=region=$AWS_REGION - -terraform destroy -var aws_region=$AWS_REGION -var aws_availability_zone=${AWS_REGION}a -auto-approve - -popd diff --git a/k8s/kops/delete_efs.sh b/k8s/kops/delete_efs.sh deleted file mode 100755 index 3322f4c0..00000000 --- a/k8s/kops/delete_efs.sh +++ /dev/null @@ -1,49 +0,0 @@ -#!/bin/bash - -set -o errexit -set -o pipefail -set -x -set -e - -err_report() { - echo "Error on line $1" -} - -trap 'err_report $LINENO' ERR - -vpcId=`aws ec2 describe-vpcs --filters Name=tag:Name,Values=$CLUSTER_NAME --output text | awk '/VPCS/ { print $8 }'` - -if [[ -z ${vpcId} ]]; then - echo "Couldn't detect AWS VPC created by kops" - exit 1 -fi - -echo "Detected VPC: $vpcId" - -securityGroupId=`aws ec2 describe-security-groups --output text | awk '/nodes.'$CLUSTER_NAME'/ && /SECURITYGROUPS/ { print $6 };'` - -if [[ -z ${securityGroupId} ]]; then - echo "Couldn't detect AWS Security Group created by kops" - exit 1 -fi - -echo "Detected Security Group ID: $securityGroupId" - -subnetIdZoneA=`aws ec2 describe-subnets --region=$AWS_REGION --output text | awk '/'$vpcId'/ { print $13 }' | sort | head -1` -subnetIdZoneB=`aws ec2 describe-subnets --region=$AWS_REGION --output text | awk '/'$vpcId'/ { print $13 }' | sort | tail -1` - -echo "Detected Subnet: $subnetIdZoneA" -echo "Detected Subnet: $subnetIdZoneB" - -pushd efs-terraform - -# extract s3 bucket from kops state store -S3_BUCKET="${KOPS_STATE_STORE:5:100}" - -terraform init -backend-config=bucket=$S3_BUCKET \ - -backend-config=key=${DEPLOYMENT_NAME}-efs \ - -backend-config=region=$AWS_REGION - -terraform destroy -var aws_region=$AWS_REGION -var fs_subnet_id_zone_a=$subnetIdZoneA -var fs_subnet_id_zone_b=$subnetIdZoneB -var fs_sg_id=$securityGroupId -auto-approve - -popd diff --git a/k8s/kops/delete_kops.sh b/k8s/kops/delete_kops.sh index 33310f6e..29a1c83b 100755 --- a/k8s/kops/delete_kops.sh +++ b/k8s/kops/delete_kops.sh @@ -5,10 +5,19 @@ set -o pipefail set -x set -e +TF_RESOURCES="../../terraform/kops-resources/" + err_report() { echo "Error on line $1" } trap 'err_report $LINENO' ERR +# ======================================================= +echo "Create EFS resources & storageclasses..." +cd ${TF_RESOURCES} +terraform destroy -auto-approve &&\ +cd - +pwd +# ======================================================= kops delete cluster $CLUSTER_NAME --yes diff --git a/k8s/kops/ebs-terraform/backend.tf b/k8s/kops/ebs-terraform/backend.tf deleted file mode 100644 index 12c0dbe5..00000000 --- a/k8s/kops/ebs-terraform/backend.tf +++ /dev/null @@ -1,3 +0,0 @@ -terraform { - backend "s3" {} -} diff --git a/k8s/kops/ebs-terraform/ebs.tf b/k8s/kops/ebs-terraform/ebs.tf deleted file mode 100644 index a19dc19a..00000000 --- a/k8s/kops/ebs-terraform/ebs.tf +++ /dev/null @@ -1,13 +0,0 @@ -provider "aws" { - region = var.aws_region - version = "~> 3.70.0" -} - -# EBS for Testground daemon datadir -resource "aws_ebs_volume" "testground-daemon-datadir" { - availability_zone = var.aws_availability_zone - size = 10 - type = "gp2" - - tags = "${merge(var.default_tags)}" -} diff --git a/k8s/kops/ebs-terraform/outputs.tf b/k8s/kops/ebs-terraform/outputs.tf deleted file mode 100644 index 255c8e20..00000000 --- a/k8s/kops/ebs-terraform/outputs.tf +++ /dev/null @@ -1,8 +0,0 @@ -output "volume_id" { - value = aws_ebs_volume.testground-daemon-datadir.id -} - -output "availability_zone" { - value = var.aws_availability_zone -} - diff --git a/k8s/kops/ebs-terraform/variables.tf b/k8s/kops/ebs-terraform/variables.tf deleted file mode 100644 index 2cc27833..00000000 --- a/k8s/kops/ebs-terraform/variables.tf +++ /dev/null @@ -1,13 +0,0 @@ -variable "aws_region" {} - -variable "aws_availability_zone" {} - -variable "default_tags" { - type = map - description = "change KubernetesCluster to fit your cluster name" - - default = { - Name = "taas-daemon-datadir-volume" - KubernetesCluster = "anton-kops.k8s.local" - } -} diff --git a/k8s/kops/ebs/pv.yml.spec b/k8s/kops/ebs/pv.yml.spec deleted file mode 100644 index cf00476b..00000000 --- a/k8s/kops/ebs/pv.yml.spec +++ /dev/null @@ -1,15 +0,0 @@ -kind: PersistentVolume -apiVersion: v1 -metadata: - name: testground-daemon-datadir-pv - labels: - type: aws-ebs -spec: - capacity: - storage: 10Gi - accessModes: - - ReadWriteOnce - storageClassName: "gp2-retain" - awsElasticBlockStore: - volumeID: ${TG_EBS_DATADIR_VOLUME_ID} - fsType: ext4 diff --git a/k8s/kops/ebs/storageclass.yml b/k8s/kops/ebs/storageclass.yml deleted file mode 100644 index e712ccd2..00000000 --- a/k8s/kops/ebs/storageclass.yml +++ /dev/null @@ -1,11 +0,0 @@ -kind: StorageClass -apiVersion: storage.k8s.io/v1 -metadata: - name: gp2-retain -provisioner: kubernetes.io/aws-ebs -parameters: - type: gp2 -reclaimPolicy: Retain -mountOptions: - - debug -volumeBindingMode: Immediate diff --git a/k8s/kops/efs-terraform/backend.tf b/k8s/kops/efs-terraform/backend.tf deleted file mode 100644 index 12c0dbe5..00000000 --- a/k8s/kops/efs-terraform/backend.tf +++ /dev/null @@ -1,3 +0,0 @@ -terraform { - backend "s3" {} -} diff --git a/k8s/kops/efs-terraform/efs.tf b/k8s/kops/efs-terraform/efs.tf deleted file mode 100644 index 9f7b543d..00000000 --- a/k8s/kops/efs-terraform/efs.tf +++ /dev/null @@ -1,23 +0,0 @@ -provider "aws" { - region = var.aws_region - version = "~> 3.70.0" -} - -# EFS for Testground outputs -resource "aws_efs_file_system" "default" { - count = 1 -} - -resource "aws_efs_mount_target" "target_subnet_zone_a" { - count = 1 - file_system_id = join("", aws_efs_file_system.default.*.id) - subnet_id = var.fs_subnet_id_zone_a - security_groups = [var.fs_sg_id] -} - -resource "aws_efs_mount_target" "target_subnet_zone_b" { - count = 1 - file_system_id = join("", aws_efs_file_system.default.*.id) - subnet_id = var.fs_subnet_id_zone_b - security_groups = [var.fs_sg_id] -} diff --git a/k8s/kops/efs-terraform/outputs.tf b/k8s/kops/efs-terraform/outputs.tf deleted file mode 100644 index eb2ef29e..00000000 --- a/k8s/kops/efs-terraform/outputs.tf +++ /dev/null @@ -1,23 +0,0 @@ -output "filesystem_id" { - value = join("", aws_efs_file_system.default.*.id) -} - -output "dns_name" { - value = join("", aws_efs_file_system.default.*.dns_name) -} - -output "mount_target_dns_names" { - value = coalescelist(aws_efs_mount_target.target_subnet_zone_a.*.dns_name, [""]) -} - -output "mount_target_ids" { - value = coalescelist(aws_efs_mount_target.target_subnet_zone_a.*.id, [""]) -} - -output "mount_target_ips" { - value = coalescelist(aws_efs_mount_target.target_subnet_zone_a.*.ip_address, [""]) -} - -output "network_interface_ids" { - value = coalescelist(aws_efs_mount_target.target_subnet_zone_a.*.network_interface_id, [""]) -} diff --git a/k8s/kops/efs-terraform/variables.tf b/k8s/kops/efs-terraform/variables.tf deleted file mode 100644 index 1c17fe66..00000000 --- a/k8s/kops/efs-terraform/variables.tf +++ /dev/null @@ -1,11 +0,0 @@ -variable "aws_region" { -} - -variable "fs_subnet_id_zone_a" { -} - -variable "fs_subnet_id_zone_b" { -} - -variable "fs_sg_id" { -} diff --git a/k8s/kops/efs/manifest.yaml.spec b/k8s/kops/efs/manifest.yaml.spec deleted file mode 100644 index 90058670..00000000 --- a/k8s/kops/efs/manifest.yaml.spec +++ /dev/null @@ -1,74 +0,0 @@ ---- -apiVersion: v1 -kind: ServiceAccount -metadata: - name: efs-provisioner ---- -kind: Deployment -apiVersion: apps/v1 -metadata: - name: efs-provisioner -spec: - replicas: 1 - selector: - matchLabels: - app: efs-provisioner - strategy: - type: Recreate - template: - metadata: - labels: - app: efs-provisioner - spec: - serviceAccountName: efs-provisioner - nodeSelector: - testground.node.role.infra: "true" - containers: - - name: efs-provisioner - image: quay.io/external_storage/efs-provisioner:v2.4.0 - env: - - name: FILE_SYSTEM_ID - valueFrom: - configMapKeyRef: - name: efs-provisioner - key: file.system.id - - name: AWS_REGION - valueFrom: - configMapKeyRef: - name: efs-provisioner - key: aws.region - - name: PROVISIONER_NAME - valueFrom: - configMapKeyRef: - name: efs-provisioner - key: provisioner.name - volumeMounts: - - name: pv-volume - mountPath: /persistentvolumes - volumes: - - name: pv-volume - nfs: - server: ${EFS_DNSNAME} - path: / ---- -kind: StorageClass -apiVersion: storage.k8s.io/v1 -metadata: - name: aws-efs -provisioner: testground.io/aws-efs ---- -kind: PersistentVolumeClaim -apiVersion: v1 -metadata: - name: efs - annotations: - volume.beta.kubernetes.io/storage-class: "aws-efs" -spec: - accessModes: - - ReadOnlyMany - - ReadWriteOnce - storageClassName: aws-efs - resources: - requests: - storage: 1000Mi ---- diff --git a/k8s/kops/efs/rbac.yaml b/k8s/kops/efs/rbac.yaml deleted file mode 100644 index 74d76f05..00000000 --- a/k8s/kops/efs/rbac.yaml +++ /dev/null @@ -1,54 +0,0 @@ -kind: ClusterRole -apiVersion: rbac.authorization.k8s.io/v1 -metadata: - name: efs-provisioner-runner -rules: - - apiGroups: [""] - resources: ["persistentvolumes"] - verbs: ["get", "list", "watch", "create", "delete"] - - apiGroups: [""] - resources: ["persistentvolumeclaims"] - verbs: ["get", "list", "watch", "update"] - - apiGroups: ["storage.k8s.io"] - resources: ["storageclasses"] - verbs: ["get", "list", "watch"] - - apiGroups: [""] - resources: ["events"] - verbs: ["create", "update", "patch"] ---- -kind: ClusterRoleBinding -apiVersion: rbac.authorization.k8s.io/v1 -metadata: - name: efs-provisioner -subjects: - - kind: ServiceAccount - name: efs-provisioner - # replace with namespace where provisioner is deployed - namespace: default -roleRef: - kind: ClusterRole - name: efs-provisioner-runner - apiGroup: rbac.authorization.k8s.io ---- -kind: Role -apiVersion: rbac.authorization.k8s.io/v1 -metadata: - name: efs-provisioner -rules: - - apiGroups: [""] - resources: ["endpoints"] - verbs: ["get", "list", "watch", "create", "update", "patch"] ---- -kind: RoleBinding -apiVersion: rbac.authorization.k8s.io/v1 -metadata: - name: efs-provisioner -subjects: - - kind: ServiceAccount - name: efs-provisioner - # replace with namespace where provisioner is deployed - namespace: default -roleRef: - kind: Role - name: efs-provisioner - apiGroup: rbac.authorization.k8s.io diff --git a/k8s/kops/influxdb/values.yaml b/k8s/kops/influxdb/values.yaml deleted file mode 100644 index 2c52abe3..00000000 --- a/k8s/kops/influxdb/values.yaml +++ /dev/null @@ -1,7 +0,0 @@ -authEnabled: false -influxdb: - nodeSelector: - testground.node.role.infra: "true" - service: - type: ClusterIP - clusterIP: None diff --git a/k8s/kops/install-playbook/validation.sh b/k8s/kops/install-playbook/validation.sh index f0d8a361..3caf3284 100644 --- a/k8s/kops/install-playbook/validation.sh +++ b/k8s/kops/install-playbook/validation.sh @@ -54,9 +54,14 @@ then echo -e "Environment variable MASTER_NODE_TYPE must be set." exit 2 fi -if [ -z "$WORKER_NODES" ] +if [ -z "$MAX_WORKER_NODES" ] then - echo -e "Environment variable WORKER_NODES must be set." + echo -e "Environment variable MAX_WORKER_NODES must be set." + exit 2 +fi +if [ -z "$MIN_WORKER_NODES" ] +then + echo -e "Environment variable MIN_WORKER_NODES must be set." exit 2 fi if [ -z "$PUBKEY" ] diff --git a/k8s/kops/install_k8s.sh b/k8s/kops/install_k8s.sh new file mode 100755 index 00000000..224ed61f --- /dev/null +++ b/k8s/kops/install_k8s.sh @@ -0,0 +1,144 @@ +#!/bin/bash + +# ======================================================= +# Description: +# This script is used to spin up a new Kubernetes cluster using Kops. +# Also, the tool creates an ArgoCD app to provision the cluster. +# +# The following variables are required to make it work +# CLUSTER_NAME= +# DEPLOYMENT_NAME= +# WORKER_NODE_TYPE= +# MASTER_NODE_TYPE= +# MIN_WORKER_NODES= +# MAX_WORKER_NODES= +# TEAM= +# PROJECT= +# AWS_REGION= +# KOPS_STATE_STORE= +# ZONE_A= +# ZONE_B= +# PUBKEY= +# AWS_ACCESS_KEY_ID=$(aws configure get aws_access_key_id) +# AWS_SECRET_ACCESS_KEY=$(aws configure get aws_secret_access_key) +# ======================================================= + +set -o errexit +set -o pipefail +set -e + +# ======================================================= +TF_RESOURCES="../../terraform/kops-resources/" +ARGOCD_VERSION="5.31.0" +# ======================================================= +err_report() { + echo "Error on line $1" +} +# ======================================================= +trap 'err_report $LINENO' ERR +# ======================================================= +START_TIME=`date +%s` +# ======================================================= +echo "Creating cluster for Testground..." +echo +CLUSTER_SPEC_TEMPLATE=$1 + +my_dir="$(dirname "$0")" +source "$my_dir/install-playbook/validation.sh" + +echo "Required arguments" +echo "------------------" +echo "Deployment name (DEPLOYMENT_NAME): $DEPLOYMENT_NAME" +echo "Cluster name (CLUSTER_NAME): $CLUSTER_NAME" +echo "Kops state store (KOPS_STATE_STORE): $KOPS_STATE_STORE" +echo "AWS availability zone A (ZONE_A): $ZONE_A" +echo "AWS availability zone B (ZONE_B): $ZONE_B" +echo "AWS region (AWS_REGION): $AWS_REGION" +echo "AWS worker node type (WORKER_NODE_TYPE): $WORKER_NODE_TYPE" +echo "AWS master node type (MASTER_NODE_TYPE): $MASTER_NODE_TYPE" +echo "Min number of Worker nodes (MIN_WORKER_NODES): $MIN_WORKER_NODES" +echo "Max number of Worker nodes (MAX_WORKER_NODES): $MAX_WORKER_NODES" +echo "Public key (PUBKEY): $PUBKEY" +echo + +CLUSTER_SPEC=$(mktemp) +envsubst <$CLUSTER_SPEC_TEMPLATE >$CLUSTER_SPEC + +# ======================================================= +# Verify with the user before continuing. +echo +echo "The cluster will be built based on the params above." +echo -n "Do they look right to you? [y/n]: " +read response +if [ "$response" != "y" ] +then + echo "Canceling ." + exit 2 +fi + +# ======================================================= +# The remainder of this script creates the cluster using the generated template +kops create -f $CLUSTER_SPEC +kops create secret --name $CLUSTER_NAME sshpublickey admin -i $PUBKEY +# The following command updates the cluster and updates the kubeconfig +kops update cluster $CLUSTER_NAME --admin --yes +# Wait for worker nodes and master to be ready +kops validate cluster --wait 20m +echo "Cluster nodes are Ready" +echo + +# ======================================================= +echo "Create EFS resources & storageclasses..." +cd ${TF_RESOURCES} +# we used the && to be sure that the previous command was executed properly +terraform init &&\ +terraform plan &&\ +terraform apply -auto-approve &&\ +cd - +pwd + +# ======================================================= +echo "Installing ArgoCD" +kubectl create namespace argocd + +helm repo add argo https://argoproj.github.io/argo-helm +helm install argocd argo/argo-cd \ + --version ${ARGOCD_VERSION} \ + --namespace=argocd \ + -f ./argocd/values.yaml + +echo "Giving some seconds to ArgoCD to be operative..." +sleep 20 + +# ======================================================= +echo "Installing root app..." +kubectl apply -f - < - while [ "$FLANNEL_ROUTES" = "" ]; do export FLANNEL_ROUTES=$(ip route | grep flannel.1); echo "Got FLANNEL_ROUTES: $FLANNEL_ROUTES"; sleep 5; done; - containers: - # Create a container with install.sh that - # Installs required 00-genie.conf and genie binary - # on slave node. - - name: install-cni - image: quay.io/huawei-cni-genie/genie-plugin:1382 - command: ["/launch.sh"] - env: - - name: CNI_NETWORK_CONFIG - valueFrom: - configMapKeyRef: - name: genie-config - key: cni_genie_network_config - - name: KUBERNETES_NODE_NAME - valueFrom: - fieldRef: - fieldPath: spec.nodeName - volumeMounts: - - mountPath: /host/opt/cni/bin - name: cni-bin-dir - - mountPath: /host/etc/cni/net.d - name: cni-net-dir - tolerations: - - key: CriticalAddonsOnly - operator: Exists - - key: node.kubernetes.io/not-ready - effect: NoSchedule - operator: Exists - volumes: - - name: cni-bin-dir - hostPath: - path: /opt/cni/bin - - name: cni-net-dir - hostPath: - path: /etc/cni/net.d - ---- -# Genie network admission controller daemonset configuration -# Genie network admission controller pods will run only in master nodes -kind: DaemonSet -apiVersion: apps/v1 -metadata: - name: genie-network-admission-controller - namespace: kube-system - labels: - k8s-app: genie-network-admission -spec: - selector: - matchLabels: - k8s-app: genie-network-admission - template: - metadata: - labels: - k8s-app: genie-network-admission - role: genie-network-admission-controller - annotations: - scheduler.alpha.kubernetes.io/critical-pod: '' - spec: - tolerations: - - key: node-role.kubernetes.io/master - effect: NoSchedule - - key: node.kubernetes.io/not-ready - effect: NoSchedule - operator: Exists - - key: CriticalAddonsOnly - operator: Exists - nodeSelector: - node-role.kubernetes.io/master: "" - hostNetwork: true - serviceAccountName: genie-plugin - containers: - - name: genie-network-admission-controller - image: quay.io/huawei-cni-genie/genie-admission-controller:1382 - ports: - - containerPort: 8000 ---- -# Genie network admission controller service -apiVersion: v1 -kind: Service -metadata: - labels: - role: genie-network-admission-controller - name: genie-network-admission-controller - namespace: kube-system -spec: - ports: - - port: 443 - targetPort: 8000 - selector: - role: genie-network-admission-controller diff --git a/k8s/kops/kops-weave/weave-metrics-service.yml b/k8s/kops/kops-weave/weave-metrics-service.yml deleted file mode 100644 index b5cd8ced..00000000 --- a/k8s/kops/kops-weave/weave-metrics-service.yml +++ /dev/null @@ -1,15 +0,0 @@ -apiVersion: v1 -kind: Service -metadata: - labels: - k8s-app: weave-net - name: weave-net - namespace: kube-system -spec: - clusterIP: None - ports: - - name: weave-metrics - port: 6782 - targetPort: 6782 - selector: - name: weave-net diff --git a/k8s/kops/kops-weave/weave-service-monitor.yml b/k8s/kops/kops-weave/weave-service-monitor.yml deleted file mode 100644 index 804e5384..00000000 --- a/k8s/kops/kops-weave/weave-service-monitor.yml +++ /dev/null @@ -1,19 +0,0 @@ -apiVersion: monitoring.coreos.com/v1 -kind: ServiceMonitor -metadata: - name: weave-net - labels: - k8s-app: weave-net - namespace: default -spec: - jobLabel: k8s-app - selector: - matchLabels: - k8s-app: weave-net - namespaceSelector: - matchNames: - - kube-system - endpoints: - - port: weave-metrics - path: /metrics - interval: 5s diff --git a/k8s/kops/kops-weave/weave.yml b/k8s/kops/kops-weave/weave.yml deleted file mode 100644 index 715db5ef..00000000 --- a/k8s/kops/kops-weave/weave.yml +++ /dev/null @@ -1,260 +0,0 @@ -apiVersion: v1 -kind: List -items: - - apiVersion: v1 - kind: ServiceAccount - metadata: - name: weave-net - annotations: - cloud.weave.works/launcher-info: |- - { - "original-request": { - "url": "/k8s/v1.10/net.yaml?k8s-version=Q2xpZW50IFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxNyIsIEdpdFZlcnNpb246InYxLjE3LjAiLCBHaXRDb21taXQ6IjcwMTMyYjBmMTMwYWNjMGJlZDE5M2Q5YmE1OWRkMTg2ZjBlNjM0Y2YiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE5LTEyLTEzVDExOjUyOjMyWiIsIEdvVmVyc2lvbjoiZ28xLjEzLjQiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToiZGFyd2luL2FtZDY0In0KU2VydmVyIFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxNyIsIEdpdFZlcnNpb246InYxLjE3LjAiLCBHaXRDb21taXQ6IjcwMTMyYjBmMTMwYWNjMGJlZDE5M2Q5YmE1OWRkMTg2ZjBlNjM0Y2YiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE5LTEyLTA3VDIxOjEyOjE3WiIsIEdvVmVyc2lvbjoiZ28xLjEzLjQiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToibGludXgvYW1kNjQifQo=", - "date": "Tue Dec 24 2019 11:21:46 GMT+0000 (UTC)" - }, - "email-address": "support@weave.works" - } - labels: - name: weave-net - namespace: kube-system - - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole - metadata: - name: weave-net - annotations: - cloud.weave.works/launcher-info: |- - { - "original-request": { - "url": "/k8s/v1.10/net.yaml?k8s-version=Q2xpZW50IFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxNyIsIEdpdFZlcnNpb246InYxLjE3LjAiLCBHaXRDb21taXQ6IjcwMTMyYjBmMTMwYWNjMGJlZDE5M2Q5YmE1OWRkMTg2ZjBlNjM0Y2YiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE5LTEyLTEzVDExOjUyOjMyWiIsIEdvVmVyc2lvbjoiZ28xLjEzLjQiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToiZGFyd2luL2FtZDY0In0KU2VydmVyIFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxNyIsIEdpdFZlcnNpb246InYxLjE3LjAiLCBHaXRDb21taXQ6IjcwMTMyYjBmMTMwYWNjMGJlZDE5M2Q5YmE1OWRkMTg2ZjBlNjM0Y2YiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE5LTEyLTA3VDIxOjEyOjE3WiIsIEdvVmVyc2lvbjoiZ28xLjEzLjQiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToibGludXgvYW1kNjQifQo=", - "date": "Tue Dec 24 2019 11:21:46 GMT+0000 (UTC)" - }, - "email-address": "support@weave.works" - } - labels: - name: weave-net - rules: - - apiGroups: - - '' - resources: - - pods - - namespaces - - nodes - verbs: - - get - - list - - watch - - apiGroups: - - networking.k8s.io - resources: - - networkpolicies - verbs: - - get - - list - - watch - - apiGroups: - - '' - resources: - - nodes/status - verbs: - - patch - - update - - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: weave-net - annotations: - cloud.weave.works/launcher-info: |- - { - "original-request": { - "url": "/k8s/v1.10/net.yaml?k8s-version=Q2xpZW50IFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxNyIsIEdpdFZlcnNpb246InYxLjE3LjAiLCBHaXRDb21taXQ6IjcwMTMyYjBmMTMwYWNjMGJlZDE5M2Q5YmE1OWRkMTg2ZjBlNjM0Y2YiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE5LTEyLTEzVDExOjUyOjMyWiIsIEdvVmVyc2lvbjoiZ28xLjEzLjQiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToiZGFyd2luL2FtZDY0In0KU2VydmVyIFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxNyIsIEdpdFZlcnNpb246InYxLjE3LjAiLCBHaXRDb21taXQ6IjcwMTMyYjBmMTMwYWNjMGJlZDE5M2Q5YmE1OWRkMTg2ZjBlNjM0Y2YiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE5LTEyLTA3VDIxOjEyOjE3WiIsIEdvVmVyc2lvbjoiZ28xLjEzLjQiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToibGludXgvYW1kNjQifQo=", - "date": "Tue Dec 24 2019 11:21:46 GMT+0000 (UTC)" - }, - "email-address": "support@weave.works" - } - labels: - name: weave-net - roleRef: - kind: ClusterRole - name: weave-net - apiGroup: rbac.authorization.k8s.io - subjects: - - kind: ServiceAccount - name: weave-net - namespace: kube-system - - apiVersion: rbac.authorization.k8s.io/v1 - kind: Role - metadata: - name: weave-net - annotations: - cloud.weave.works/launcher-info: |- - { - "original-request": { - "url": "/k8s/v1.10/net.yaml?k8s-version=Q2xpZW50IFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxNyIsIEdpdFZlcnNpb246InYxLjE3LjAiLCBHaXRDb21taXQ6IjcwMTMyYjBmMTMwYWNjMGJlZDE5M2Q5YmE1OWRkMTg2ZjBlNjM0Y2YiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE5LTEyLTEzVDExOjUyOjMyWiIsIEdvVmVyc2lvbjoiZ28xLjEzLjQiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToiZGFyd2luL2FtZDY0In0KU2VydmVyIFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxNyIsIEdpdFZlcnNpb246InYxLjE3LjAiLCBHaXRDb21taXQ6IjcwMTMyYjBmMTMwYWNjMGJlZDE5M2Q5YmE1OWRkMTg2ZjBlNjM0Y2YiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE5LTEyLTA3VDIxOjEyOjE3WiIsIEdvVmVyc2lvbjoiZ28xLjEzLjQiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToibGludXgvYW1kNjQifQo=", - "date": "Tue Dec 24 2019 11:21:46 GMT+0000 (UTC)" - }, - "email-address": "support@weave.works" - } - labels: - name: weave-net - namespace: kube-system - rules: - - apiGroups: - - '' - resourceNames: - - weave-net - resources: - - configmaps - verbs: - - get - - update - - apiGroups: - - '' - resources: - - configmaps - verbs: - - create - - apiVersion: rbac.authorization.k8s.io/v1 - kind: RoleBinding - metadata: - name: weave-net - annotations: - cloud.weave.works/launcher-info: |- - { - "original-request": { - "url": "/k8s/v1.10/net.yaml?k8s-version=Q2xpZW50IFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxNyIsIEdpdFZlcnNpb246InYxLjE3LjAiLCBHaXRDb21taXQ6IjcwMTMyYjBmMTMwYWNjMGJlZDE5M2Q5YmE1OWRkMTg2ZjBlNjM0Y2YiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE5LTEyLTEzVDExOjUyOjMyWiIsIEdvVmVyc2lvbjoiZ28xLjEzLjQiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToiZGFyd2luL2FtZDY0In0KU2VydmVyIFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxNyIsIEdpdFZlcnNpb246InYxLjE3LjAiLCBHaXRDb21taXQ6IjcwMTMyYjBmMTMwYWNjMGJlZDE5M2Q5YmE1OWRkMTg2ZjBlNjM0Y2YiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE5LTEyLTA3VDIxOjEyOjE3WiIsIEdvVmVyc2lvbjoiZ28xLjEzLjQiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToibGludXgvYW1kNjQifQo=", - "date": "Tue Dec 24 2019 11:21:46 GMT+0000 (UTC)" - }, - "email-address": "support@weave.works" - } - labels: - name: weave-net - namespace: kube-system - roleRef: - kind: Role - name: weave-net - apiGroup: rbac.authorization.k8s.io - subjects: - - kind: ServiceAccount - name: weave-net - namespace: kube-system - - apiVersion: apps/v1 - kind: DaemonSet - metadata: - name: weave-net - annotations: - cloud.weave.works/launcher-info: |- - { - "original-request": { - "url": "/k8s/v1.10/net.yaml?k8s-version=Q2xpZW50IFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxNyIsIEdpdFZlcnNpb246InYxLjE3LjAiLCBHaXRDb21taXQ6IjcwMTMyYjBmMTMwYWNjMGJlZDE5M2Q5YmE1OWRkMTg2ZjBlNjM0Y2YiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE5LTEyLTEzVDExOjUyOjMyWiIsIEdvVmVyc2lvbjoiZ28xLjEzLjQiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToiZGFyd2luL2FtZDY0In0KU2VydmVyIFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxNyIsIEdpdFZlcnNpb246InYxLjE3LjAiLCBHaXRDb21taXQ6IjcwMTMyYjBmMTMwYWNjMGJlZDE5M2Q5YmE1OWRkMTg2ZjBlNjM0Y2YiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE5LTEyLTA3VDIxOjEyOjE3WiIsIEdvVmVyc2lvbjoiZ28xLjEzLjQiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToibGludXgvYW1kNjQifQo=", - "date": "Tue Dec 24 2019 11:21:46 GMT+0000 (UTC)" - }, - "email-address": "support@weave.works" - } - labels: - name: weave-net - namespace: kube-system - spec: - minReadySeconds: 5 - selector: - matchLabels: - name: weave-net - template: - metadata: - labels: - name: weave-net - spec: - priorityClassName: system-node-critical - initContainers: - - name: wait-for-flannel - image: busybox:1.31.1 - securityContext: - privileged: true - command: - - sh - - -ac - - > - while [ "$FLANNEL_ROUTES" = "" ]; do export FLANNEL_ROUTES=$(ip route | grep flannel.1); echo "Got FLANNEL_ROUTES: $FLANNEL_ROUTES"; sleep 5; done; - containers: - - name: weave - command: - - /home/weave/launch.sh - env: - - name: WEAVE_MTU - value: "8912" - - name: CONN_LIMIT - value: "250" - - name: EXTRA_ARGS - value: --log-level=info - - name: EXPECT_NPC - value: "0" - - name: IPALLOC_RANGE - value: "16.0.0.0/4" - - name: HOSTNAME - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: spec.nodeName - image: 'iptestground/weave-kube:0.0.2-v2.7.0-63821434' - readinessProbe: - httpGet: - host: 127.0.0.1 - path: /status - port: 6784 - timeoutSeconds: 5 - resources: - requests: - cpu: 100m - memory: 1536Mi - limits: - memory: 1536Mi - securityContext: - privileged: true - volumeMounts: - - name: weavedb - mountPath: /weavedb - - name: cni-bin - mountPath: /host/opt - - name: cni-bin2 - mountPath: /host/home - - name: cni-conf - mountPath: /host/etc - - name: dbus - mountPath: /host/var/lib/dbus - - name: lib-modules - mountPath: /lib/modules - - name: xtables-lock - mountPath: /run/xtables.lock - hostNetwork: true - hostPID: true - restartPolicy: Always - securityContext: - seLinuxOptions: {} - serviceAccountName: weave-net - tolerations: - - effect: NoSchedule - operator: Exists - volumes: - - name: weavedb - hostPath: - path: /var/lib/weave - - name: cni-bin - hostPath: - path: /opt - - name: cni-bin2 - hostPath: - path: /home - - name: cni-conf - hostPath: - path: /etc - - name: dbus - hostPath: - path: /var/lib/dbus - - name: lib-modules - hostPath: - path: /lib/modules - - name: xtables-lock - hostPath: - path: /run/xtables.lock - type: FileOrCreate - updateStrategy: - type: RollingUpdate diff --git a/k8s/kops/limit-range/limit-range.yaml b/k8s/kops/limit-range/limit-range.yaml deleted file mode 100644 index e8acaf6f..00000000 --- a/k8s/kops/limit-range/limit-range.yaml +++ /dev/null @@ -1,12 +0,0 @@ -apiVersion: v1 -kind: LimitRange -metadata: - name: default-pod-limits -spec: - limits: - - default: - memory: 256Mi - defaultRequest: - memory: 256Mi - cpu: 100m - type: Container diff --git a/k8s/kops/packer/Makefile b/k8s/kops/packer/Makefile index e76de25d..62bf82bb 100644 --- a/k8s/kops/packer/Makefile +++ b/k8s/kops/packer/Makefile @@ -1,2 +1,3 @@ build-ami-image: - packer build -var 'source_ami=ami-0ed4a9453b39ea8c1' -var 'aws_region=eu-west-2' testground-ami.json + #packer build -var 'source_ami=ami-00aa9d3df94c6c354' -var 'aws_region=eu-west-1' testground-ami.json + packer build -var 'source_ami=ami-05147510eb2885c80' -var 'aws_region=eu-west-1' testground-ami.json diff --git a/k8s/kops/packer/docker-pull-images.sh b/k8s/kops/packer/docker-pull-images.sh index c3deeece..43720cc5 100644 --- a/k8s/kops/packer/docker-pull-images.sh +++ b/k8s/kops/packer/docker-pull-images.sh @@ -2,6 +2,12 @@ sudo apt update sudo apt install -y docker.io +# Required to have NFS client available +sudo apt-get update -y +sudo apt-get install -y libnfsidmap2 +sudo apt-get install -y rpcbind +sudo apt-get install -y libkeyutils1 +sudo apt-get install -y nfs-common sudo docker pull bitnami/redis:5.0.8-debian-10-r39 sudo docker pull bitnami/redis-exporter:1.5.2-debian-10-r27 diff --git a/k8s/kops/prometheus-operator/values.yaml b/k8s/kops/prometheus-operator/values.yaml deleted file mode 100644 index 5f73e111..00000000 --- a/k8s/kops/prometheus-operator/values.yaml +++ /dev/null @@ -1,155 +0,0 @@ ---- -# Testground values overrides: - -# Changes from defaults: -# * Disabled the alertmanager. -# * We do not manage certificates. In order to prevent scrape errors caused by -# certificate malfunction, skip TLS checks or disable https. -# * The grafana username and password are stored in a kubernetes secret called -# -grafana. Changed from the default (prom-operator) to testground. -# * Enable the grafana sidecar. This will watch for secrets or configmaps which -# have the label grafana_dashboard or grafana_datasource, and adds them -# automatically to the grafana dashboard. -# * I configured the scraper to look for ServiceMonitors in any namespace. This -# allows a single prometheus to scrape plan as well as kube-system metrics. -# * createCustomResource is configured to false. For helm v3, custom resources -# are still created. By including this option, users won't see a confusing -# error message about CRDs not being created. -# See the following for a description of this option: -# https://github.com/helm/charts/blob/master/stable/prometheus-operator/ -# for an explanation of this option. -prometheusOperator: - createCustomResource: false - admissionWebooks: - patch: - affinity: - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - redis - topologyKey: "kubernetes.io/hostname" - nodeSelector: - testground.node.role.infra: "true" - resources: - requests: - memory: 512Mi - cpu: 500m - limits: - memory: 512Mi -alertmanager: - enabled: false -grafana: - defaultDashboardsEnabled: false - grafana.ini: - users: - viewers_can_edit: false - auth: - disable_login_form: false - disable_signout_menu: false - auth.anonymous: - enabled: true - org_role: Viewer - adminPassword: admin - affinity: - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - redis - topologyKey: "kubernetes.io/hostname" - podAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - prometheus-operator-operator - topologyKey: "kubernetes.io/hostname" - nodeSelector: - testground.node.role.infra: "true" - sidecar: - dashboards: - enabled: true - datasources: - enabled: true - additionalDataSources: - - name: influxdb - access: proxy - database: "testground" - basicAuth: false - editable: true - jsonData: - tlsSkipVerify: true - orgId: 1 - type: influxdb - url: http://influxdb:8086 - version: 1 - affinity: - podAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - prometheus-operator-operator - topologyKey: "kubernetes.io/hostname" -kubeProxy: - serviceMonitor: - https: false -kubelet: - serviceMonitor: - https: false -kubeApiServer: - tlsConfig: - insecureSkipVerify: true - serviceMonitor: - https: false -kubeControllerManager: - serviceMonitor: - insecureSkipVerify: true - https: false -kubeEtcd: - serviceMonitor: - insecureSkipVerify: true - https: false -kubeStateMetrics: - enabled: false -prometheus: - prometheusSpec: - serviceMonitorSelectorNilUsesHelmValues: false - affinity: - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - redis - topologyKey: "kubernetes.io/hostname" - podAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - prometheus-operator-operator - topologyKey: "kubernetes.io/hostname" - nodeSelector: - testground.node.role.infra: "true" - resources: - requests: - memory: 6000Mi - cpu: 1000m - limits: - memory: 6000Mi diff --git a/k8s/kops/sidecar.yaml b/k8s/kops/sidecar.yaml deleted file mode 100644 index 39fa0c1c..00000000 --- a/k8s/kops/sidecar.yaml +++ /dev/null @@ -1,73 +0,0 @@ -apiVersion: apps/v1 -kind: DaemonSet -metadata: - name: testground-sidecar -spec: - selector: - matchLabels: - name: testground-sidecar - template: - metadata: - labels: - name: testground-sidecar - spec: - terminationGracePeriodSeconds: 10 - hostPID: true - hostNetwork: true - dnsPolicy: ClusterFirstWithHostNet - priorityClassName: system-node-critical - initContainers: - - name: iproute-add - image: busybox:1.31.1 - securityContext: - privileged: true - command: - - sh - - -ac - - > - while [ "$GW" = "" ]; do export GW=$(ip route | grep cni0 | awk '{print $7}'); echo "Got GW: $GW"; sleep 5; done; - echo $GW && - ip route && - ip route add 100.64.0.0/16 via $GW && - ip route || true; - nslookup testground-infra-redis-headless; - while [ $? -ne 0 ]; do echo "redis-headless doesn't resolve."; sleep 5; nslookup testground-infra-redis-headless; done; - echo "redis-headless resolved." - containers: - - name: testground-sidecar - image: iptestground/sidecar:edge - imagePullPolicy: Always - command: ["testground"] - args: ["sidecar", "--runner", "k8s"] - securityContext: - capabilities: - add: ["NET_ADMIN", "SYS_ADMIN", "SYS_TIME"] - privileged: true - env: - - name: REDIS_HOST - value: "testground-infra-redis-headless" - - name: INFLUXDB_HOST - value: "influxdb" - ports: - - name: sidecarhttp - containerPort: 6060 - resources: - limits: - memory: 256Mi - requests: - cpu: 200m - memory: 256Mi - volumeMounts: - - name: dockersock - mountPath: "/var/run/docker.sock" - - name: cnibin - mountPath: "/host/opt/cni/bin" - volumes: - - name: dockersock - hostPath: - path: /var/run/docker.sock - - name: cnibin - hostPath: - path: /opt/cni/bin - nodeSelector: - testground.node.role.plan: "true" diff --git a/k8s/kops/testground-daemon/service.yml b/k8s/kops/testground-daemon/service.yml deleted file mode 100644 index 75471b63..00000000 --- a/k8s/kops/testground-daemon/service.yml +++ /dev/null @@ -1,12 +0,0 @@ -apiVersion: v1 -kind: Service -metadata: - name: testground-daemon - labels: - app: testground-daemon -spec: - ports: - - port: 8042 - protocol: TCP - selector: - app: testground-daemon diff --git a/k8s/kops/testground-infra/Chart.yaml b/k8s/kops/testground-infra/Chart.yaml deleted file mode 100644 index 17656d24..00000000 --- a/k8s/kops/testground-infra/Chart.yaml +++ /dev/null @@ -1,8 +0,0 @@ -apiVersion: v1 -appVersion: 0.0.1 -description: Testground Monitoring -engine: gotpl -home: https://github.com/ipfs/testground -name: testground-infra -sources: -version: 0.0.1 diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/Chart.yaml b/k8s/kops/testground-infra/charts/testground-dashboards/Chart.yaml deleted file mode 100644 index 44bc6c3a..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/Chart.yaml +++ /dev/null @@ -1,8 +0,0 @@ -apiVersion: v1 -appVersion: 0.0.1 -description: Testground Dashboards -engine: gotpl -home: https://github.com/ipfs/testground -name: testground-dashboards -sources: -version: 0.0.1 diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/README.md b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/README.md deleted file mode 100644 index 51cdd5af..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/README.md +++ /dev/null @@ -1,59 +0,0 @@ -# Dashboards on Testground - -Testground has grafana! - -## Get access to the grafana dashboard from your k8s cluster. - -Prometheus and Grafana are running in the `monitoring` kubernetes namespace. Get access to -grafana locally and direct your web browser to `http://localhost:3000` - -``` -kubectl -n monitoring port-forward service/grafana 3000:3000 - -xdg-open http://localhost:3000 || open http://localhost:3000 -``` - -Log in and change your password in the UI. - -## Temporary dashboards -Any dashboard you create for yourself remains local to your copy of grafana. If you destroy your k8s -cluster, you will destroy the kubernetes dashboard as well. That's no good! Once you have your -dashboard just the way you like it, share it with the rest of the testground users by backing it up -into this dashboards directory! - - - -## Saving a dashboard to git, for all to enjoy. -1. Tag your dashboard with the `testground` tag. - - If you have additional useful tags, you can add those as well. -2. Run the dashboard code (in this directory) -```bash -$ export GRAFANA_API_KEY="XXX" -$ go run main.go -``` -or -```bash -$ go run main.go -apikey "XXX" -``` - - -## Import dashboards to your grafana instance -(same as saving, but adding the -import flag) - -```bash -$ export GRAFANA_API_KEY="XXX" -$ go run main.go -import -``` -or -```bash -$ go run main.go -apikey "XXX" -import -``` - - -# BUGS -Users need to create an API key through the web UI, which prevents this process from being easily -automated. -Maybe this will be less of a problem when this is run as a service. - -I really *ought* to be able to upload these just by editing kubernetes configmaps -- this is of -course how the default prometheus operator dashboards are created. diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/coredns.json b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/coredns.json deleted file mode 100644 index 19e10b1d..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/coredns.json +++ /dev/null @@ -1,1400 +0,0 @@ -{ - "annotations": { - "list": [ - { - "builtIn": 1, - "datasource": "-- Grafana --", - "enable": true, - "hide": true, - "iconColor": "rgba(0, 211, 255, 1)", - "name": "Annotations & Alerts", - "type": "dashboard" - } - ] - }, - "description": "A dashboard for the CoreDNS DNS server.", - "editable": true, - "gnetId": 7279, - "graphTooltip": 0, - "id": 14, - "iteration": 1591021729968, - "links": [ - { - "icon": "external link", - "tags": [], - "targetBlank": true, - "title": "CoreDNS.io", - "type": "link", - "url": "https://coredns.io" - } - ], - "panels": [ - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "Prometheus", - "editable": true, - "error": false, - "fill": 1, - "fillGradient": 0, - "grid": {}, - "gridPos": { - "h": 7, - "w": 8, - "x": 0, - "y": 0 - }, - "hiddenSeries": false, - "id": 1, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [ - { - "alias": "total", - "yaxis": 2 - } - ], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(coredns_dns_request_count_total{instance=~\"$instance\"}[5m])) by (proto)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{proto}}", - "refId": "A", - "step": 60 - }, - { - "expr": "sum(rate(coredns_dns_request_count_total{instance=~\"$instance\"}[5m]))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "total", - "refId": "B", - "step": 60 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Requests (total)", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "cumulative" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "Prometheus", - "editable": true, - "error": false, - "fill": 1, - "fillGradient": 0, - "grid": {}, - "gridPos": { - "h": 7, - "w": 8, - "x": 8, - "y": 0 - }, - "hiddenSeries": false, - "id": 12, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [ - { - "alias": "total", - "yaxis": 2 - }, - { - "alias": "other", - "yaxis": 2 - } - ], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(coredns_dns_request_type_count_total{instance=~\"$instance\"}[5m])) by (type)", - "intervalFactor": 2, - "legendFormat": "{{type}}", - "refId": "A", - "step": 60 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Requests (by qtype)", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "cumulative" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "Prometheus", - "editable": true, - "error": false, - "fill": 1, - "fillGradient": 0, - "grid": {}, - "gridPos": { - "h": 7, - "w": 8, - "x": 16, - "y": 0 - }, - "hiddenSeries": false, - "id": 2, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [ - { - "alias": "total", - "yaxis": 2 - } - ], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(coredns_dns_request_count_total{instance=~\"$instance\"}[5m])) by (zone)", - "intervalFactor": 2, - "legendFormat": "{{zone}}", - "refId": "A", - "step": 60 - }, - { - "expr": "sum(rate(coredns_dns_request_count_total{instance=~\"$instance\"}[5m]))", - "intervalFactor": 2, - "legendFormat": "total", - "refId": "B", - "step": 60 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Requests (by zone)", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "cumulative" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "Prometheus", - "editable": true, - "error": false, - "fill": 1, - "fillGradient": 0, - "grid": {}, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 7 - }, - "hiddenSeries": false, - "id": 10, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [ - { - "alias": "total", - "yaxis": 2 - } - ], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(coredns_dns_request_do_count_total{instance=~\"$instance\"}[5m]))", - "intervalFactor": 2, - "legendFormat": "DO", - "refId": "A", - "step": 40 - }, - { - "expr": "sum(rate(coredns_dns_request_count_total{instance=~\"$instance\"}[5m]))", - "intervalFactor": 2, - "legendFormat": "total", - "refId": "B", - "step": 40 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Requests (DO bit)", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "cumulative" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "Prometheus", - "editable": true, - "error": false, - "fill": 1, - "fillGradient": 0, - "grid": {}, - "gridPos": { - "h": 7, - "w": 6, - "x": 12, - "y": 7 - }, - "hiddenSeries": false, - "id": 9, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [ - { - "alias": "tcp:90", - "yaxis": 2 - }, - { - "alias": "tcp:99 ", - "yaxis": 2 - }, - { - "alias": "tcp:50", - "yaxis": 2 - } - ], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "histogram_quantile(0.99, sum(rate(coredns_dns_request_size_bytes_bucket{instance=~\"$instance\",proto=\"udp\"}[5m])) by (le,proto))", - "intervalFactor": 2, - "legendFormat": "{{proto}}:99 ", - "refId": "A", - "step": 60 - }, - { - "expr": "histogram_quantile(0.90, sum(rate(coredns_dns_request_size_bytes_bucket{instance=~\"$instance\",proto=\"udp\"}[5m])) by (le,proto))", - "intervalFactor": 2, - "legendFormat": "{{proto}}:90", - "refId": "B", - "step": 60 - }, - { - "expr": "histogram_quantile(0.50, sum(rate(coredns_dns_request_size_bytes_bucket{instance=~\"$instance\",proto=\"udp\"}[5m])) by (le,proto))", - "intervalFactor": 2, - "legendFormat": "{{proto}}:50", - "refId": "C", - "step": 60 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Requests (size, udp)", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "cumulative" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "bytes", - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "Prometheus", - "editable": true, - "error": false, - "fill": 1, - "fillGradient": 0, - "grid": {}, - "gridPos": { - "h": 7, - "w": 6, - "x": 18, - "y": 7 - }, - "hiddenSeries": false, - "id": 14, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [ - { - "alias": "tcp:90", - "yaxis": 1 - }, - { - "alias": "tcp:99 ", - "yaxis": 1 - }, - { - "alias": "tcp:50", - "yaxis": 1 - } - ], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "histogram_quantile(0.99, sum(rate(coredns_dns_request_size_bytes_bucket{instance=~\"$instance\",proto=\"tcp\"}[5m])) by (le,proto))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{proto}}:99 ", - "refId": "A", - "step": 60 - }, - { - "expr": "histogram_quantile(0.90, sum(rate(coredns_dns_request_size_bytes_bucket{instance=~\"$instance\",proto=\"tcp\"}[5m])) by (le,proto))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{proto}}:90", - "refId": "B", - "step": 60 - }, - { - "expr": "histogram_quantile(0.50, sum(rate(coredns_dns_request_size_bytes_bucket{instance=~\"$instance\",proto=\"tcp\"}[5m])) by (le,proto))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{proto}}:50", - "refId": "C", - "step": 60 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Requests (size,tcp)", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "cumulative" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "bytes", - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "Prometheus", - "editable": true, - "error": false, - "fill": 1, - "fillGradient": 0, - "grid": {}, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 14 - }, - "hiddenSeries": false, - "id": 5, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(coredns_dns_response_rcode_count_total{instance=~\"$instance\"}[5m])) by (rcode)", - "intervalFactor": 2, - "legendFormat": "{{rcode}}", - "refId": "A", - "step": 40 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Responses (by rcode)", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "cumulative" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "Prometheus", - "editable": true, - "error": false, - "fill": 1, - "fillGradient": 0, - "grid": {}, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 14 - }, - "hiddenSeries": false, - "id": 3, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "histogram_quantile(0.99, sum(rate(coredns_dns_request_duration_seconds_bucket{instance=~\"$instance\"}[5m])) by (le, job))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "99%", - "refId": "A", - "step": 40 - }, - { - "expr": "histogram_quantile(0.90, sum(rate(coredns_dns_request_duration_seconds_bucket{instance=~\"$instance\"}[5m])) by (le))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "90%", - "refId": "B", - "step": 40 - }, - { - "expr": "histogram_quantile(0.50, sum(rate(coredns_dns_request_duration_seconds_bucket{instance=~\"$instance\"}[5m])) by (le))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "50%", - "refId": "C", - "step": 40 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Responses (duration)", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "cumulative" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "s", - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "Prometheus", - "editable": true, - "error": false, - "fill": 1, - "fillGradient": 0, - "grid": {}, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 21 - }, - "hiddenSeries": false, - "id": 8, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [ - { - "alias": "udp:50%", - "yaxis": 1 - }, - { - "alias": "tcp:50%", - "yaxis": 2 - }, - { - "alias": "tcp:90%", - "yaxis": 2 - }, - { - "alias": "tcp:99%", - "yaxis": 2 - } - ], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "histogram_quantile(0.99, sum(rate(coredns_dns_response_size_bytes_bucket{instance=~\"$instance\",proto=\"udp\"}[5m])) by (le,proto)) ", - "intervalFactor": 2, - "legendFormat": "{{proto}}:99%", - "refId": "A", - "step": 40 - }, - { - "expr": "histogram_quantile(0.90, sum(rate(coredns_dns_response_size_bytes_bucket{instance=\"$instance\",proto=\"udp\"}[5m])) by (le,proto)) ", - "intervalFactor": 2, - "legendFormat": "{{proto}}:90%", - "refId": "B", - "step": 40 - }, - { - "expr": "histogram_quantile(0.50, sum(rate(coredns_dns_response_size_bytes_bucket{instance=~\"$instance\",proto=\"udp\"}[5m])) by (le,proto)) ", - "intervalFactor": 2, - "legendFormat": "{{proto}}:50%", - "metric": "", - "refId": "C", - "step": 40 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Responses (size, udp)", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "cumulative" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "bytes", - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "Prometheus", - "editable": true, - "error": false, - "fill": 1, - "fillGradient": 0, - "grid": {}, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 21 - }, - "hiddenSeries": false, - "id": 13, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [ - { - "alias": "udp:50%", - "yaxis": 1 - }, - { - "alias": "tcp:50%", - "yaxis": 1 - }, - { - "alias": "tcp:90%", - "yaxis": 1 - }, - { - "alias": "tcp:99%", - "yaxis": 1 - } - ], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "histogram_quantile(0.99, sum(rate(coredns_dns_response_size_bytes_bucket{instance=~\"$instance\",proto=\"tcp\"}[5m])) by (le,proto)) ", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{proto}}:99%", - "refId": "A", - "step": 40 - }, - { - "expr": "histogram_quantile(0.90, sum(rate(coredns_dns_response_size_bytes_bucket{instance=~\"$instance\",proto=\"tcp\"}[5m])) by (le,proto)) ", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{proto}}:90%", - "refId": "B", - "step": 40 - }, - { - "expr": "histogram_quantile(0.50, sum(rate(coredns_dns_response_size_bytes_bucket{instance=~\"$instance\",proto=\"tcp\"}[5m])) by (le, proto)) ", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{proto}}:50%", - "metric": "", - "refId": "C", - "step": 40 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Responses (size, tcp)", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "cumulative" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "bytes", - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "Prometheus", - "editable": true, - "error": false, - "fill": 1, - "fillGradient": 0, - "grid": {}, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 28 - }, - "hiddenSeries": false, - "id": 15, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(coredns_cache_size{instance=~\"$instance\"}) by (type)", - "intervalFactor": 2, - "legendFormat": "{{type}}", - "refId": "A", - "step": 40 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Cache (size)", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "cumulative" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "Prometheus", - "editable": true, - "error": false, - "fill": 1, - "fillGradient": 0, - "grid": {}, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 28 - }, - "hiddenSeries": false, - "id": 16, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [ - { - "alias": "misses", - "yaxis": 2 - } - ], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(coredns_cache_hits_total{instance=~\"$instance\"}[5m])) by (type)", - "intervalFactor": 2, - "legendFormat": "hits:{{type}}", - "refId": "A", - "step": 40 - }, - { - "expr": "sum(rate(coredns_cache_misses_total{instance=~\"$instance\"}[5m])) by (type)", - "intervalFactor": 2, - "legendFormat": "misses", - "refId": "B", - "step": 40 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Cache (hitrate)", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "cumulative" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - } - ], - "refresh": false, - "schemaVersion": 22, - "style": "dark", - "tags": [ - "dns", - "coredns" - ], - "templating": { - "list": [ - { - "allValue": ".*", - "current": { - "selected": false, - "text": "All", - "value": "$__all" - }, - "datasource": "Prometheus", - "definition": "", - "hide": 0, - "includeAll": true, - "index": -1, - "label": "Instance", - "multi": false, - "name": "instance", - "options": [], - "query": "label_values(up{k8s_app=\"coredns\"}, instance)", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "sort": 3, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - } - ] - }, - "time": { - "from": "now-5m", - "to": "now" - }, - "timepicker": { - "now": true, - "refresh_intervals": [ - "5s", - "10s", - "30s", - "1m", - "5m", - "15m", - "30m", - "1h", - "2h", - "1d" - ], - "time_options": [ - "5m", - "15m", - "1h", - "6h", - "12h", - "24h", - "2d", - "7d", - "30d" - ] - }, - "timezone": "utc", - "title": "CoreDNS", - "uid": "2XyV79Kiz", - "variables": { - "list": [] - }, - "version": 1 -} diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-api-server.json b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-api-server.json deleted file mode 100644 index 85c41d21..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-api-server.json +++ /dev/null @@ -1,2193 +0,0 @@ -{ - "annotations": { - "list": [ - { - "builtIn": 1, - "datasource": "-- Grafana --", - "enable": true, - "hide": true, - "iconColor": "rgba(0, 211, 255, 1)", - "name": "Annotations & Alerts", - "type": "dashboard" - } - ] - }, - "editable": false, - "gnetId": null, - "graphTooltip": 0, - "id": 4, - "iteration": 1588347039689, - "links": [], - "panels": [ - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "decimals": 3, - "format": "percentunit", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 7, - "w": 8, - "x": 0, - "y": 0 - }, - "id": 2, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "apiserver_request:availability30d{verb=\"all\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "", - "title": "Availability (30d) > 99.000", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "decimals": 3, - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 16, - "x": 8, - "y": 0 - }, - "hiddenSeries": false, - "id": 3, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "100 * (apiserver_request:availability30d{verb=\"all\"} - 0.990000)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "errorbudget", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "ErrorBudget (30d) > 99.000", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "decimals": 3, - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "decimals": 3, - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "decimals": 3, - "format": "percentunit", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 7, - "w": 6, - "x": 0, - "y": 7 - }, - "id": 4, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "apiserver_request:availability30d{verb=\"read\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "", - "title": "Read Availability (30d)", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 6, - "x": 6, - "y": 7 - }, - "hiddenSeries": false, - "id": 5, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(code_resource:apiserver_request_total:rate5m{verb=\"read\"})", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Read SLI - Requests", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "reqps", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "reqps", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 6, - "x": 12, - "y": 7 - }, - "hiddenSeries": false, - "id": 6, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum by (resource) (code_resource:apiserver_request_total:rate5m{verb=\"read\",code=~\"5..\"}) / sum by (resource) (code_resource:apiserver_request_total:rate5m{verb=\"read\"})", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{ resource }}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Read SLI - Errors", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 6, - "x": 18, - "y": 7 - }, - "hiddenSeries": false, - "id": 7, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile{verb=\"read\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{ resource }}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Read SLI - Duration", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "decimals": 3, - "format": "percentunit", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 7, - "w": 6, - "x": 0, - "y": 14 - }, - "id": 8, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "apiserver_request:availability30d{verb=\"write\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "", - "title": "Write Availability (30d)", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 6, - "x": 6, - "y": 14 - }, - "hiddenSeries": false, - "id": 9, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(code_resource:apiserver_request_total:rate5m{verb=\"write\"})", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Write SLI - Requests", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "reqps", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "reqps", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 6, - "x": 12, - "y": 14 - }, - "hiddenSeries": false, - "id": 10, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum by (resource) (code_resource:apiserver_request_total:rate5m{verb=\"write\",code=~\"5..\"}) / sum by (resource) (code_resource:apiserver_request_total:rate5m{verb=\"write\"})", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{ resource }}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Write SLI - Errors", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 6, - "x": 18, - "y": 14 - }, - "hiddenSeries": false, - "id": 11, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile{verb=\"write\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{ resource }}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Write SLI - Duration", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 7, - "w": 4, - "x": 0, - "y": 21 - }, - "id": 12, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "sum(up{job=\"apiserver\", cluster=\"$cluster\"})", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "", - "title": "Up", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "min" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 10, - "x": 4, - "y": 21 - }, - "hiddenSeries": false, - "id": 13, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(apiserver_request_total{job=\"apiserver\", instance=~\"$instance\",code=~\"2..\", cluster=\"$cluster\"}[5m]))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "2xx", - "refId": "A" - }, - { - "expr": "sum(rate(apiserver_request_total{job=\"apiserver\", instance=~\"$instance\",code=~\"3..\", cluster=\"$cluster\"}[5m]))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "3xx", - "refId": "B" - }, - { - "expr": "sum(rate(apiserver_request_total{job=\"apiserver\", instance=~\"$instance\",code=~\"4..\", cluster=\"$cluster\"}[5m]))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "4xx", - "refId": "C" - }, - { - "expr": "sum(rate(apiserver_request_total{job=\"apiserver\", instance=~\"$instance\",code=~\"5..\", cluster=\"$cluster\"}[5m]))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "5xx", - "refId": "D" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "RPC Rate", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 10, - "x": 14, - "y": 21 - }, - "hiddenSeries": false, - "id": 14, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\", instance=~\"$instance\", verb!=\"WATCH\", cluster=\"$cluster\"}[5m])) by (verb, le))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{verb}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Request duration 99th quantile", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 28 - }, - "hiddenSeries": false, - "id": 15, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(workqueue_adds_total{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance, name)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} {{name}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Work Queue Add Rate", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 28 - }, - "hiddenSeries": false, - "id": 16, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(workqueue_depth{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance, name)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} {{name}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Work Queue Depth", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 24, - "x": 0, - "y": 35 - }, - "hiddenSeries": false, - "id": 17, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "histogram_quantile(0.99, sum(rate(workqueue_queue_duration_seconds_bucket{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance, name, le))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} {{name}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Work Queue Latency", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 8, - "x": 0, - "y": 42 - }, - "hiddenSeries": false, - "id": 18, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "etcd_helper_cache_entry_total{job=\"apiserver\", instance=~\"$instance\", cluster=\"$cluster\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "ETCD Cache Entry Total", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 8, - "x": 8, - "y": 42 - }, - "hiddenSeries": false, - "id": 19, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(etcd_helper_cache_hit_total{job=\"apiserver\",instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} hit", - "refId": "A" - }, - { - "expr": "sum(rate(etcd_helper_cache_miss_total{job=\"apiserver\",instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} miss", - "refId": "B" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "ETCD Cache Hit/Miss Rate", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 8, - "x": 16, - "y": 42 - }, - "hiddenSeries": false, - "id": 20, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "histogram_quantile(0.99,sum(rate(etcd_request_cache_get_duration_seconds_bucket{job=\"apiserver\",instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance, le))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} get", - "refId": "A" - }, - { - "expr": "histogram_quantile(0.99,sum(rate(etcd_request_cache_add_duration_seconds_bucket{job=\"apiserver\",instance=~\"$instance\", cluster=\"$cluster\"}[5m])) by (instance, le))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} miss", - "refId": "B" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "ETCD Cache Duration 99th Quantile", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 8, - "x": 0, - "y": 49 - }, - "hiddenSeries": false, - "id": 21, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "process_resident_memory_bytes{job=\"apiserver\",instance=~\"$instance\", cluster=\"$cluster\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Memory", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "bytes", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "bytes", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 8, - "x": 8, - "y": 49 - }, - "hiddenSeries": false, - "id": 22, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "rate(process_cpu_seconds_total{job=\"apiserver\",instance=~\"$instance\", cluster=\"$cluster\"}[5m])", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "CPU usage", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 8, - "x": 16, - "y": 49 - }, - "hiddenSeries": false, - "id": 23, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "go_goroutines{job=\"apiserver\",instance=~\"$instance\", cluster=\"$cluster\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Goroutines", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - } - ], - "refresh": "10s", - "schemaVersion": 22, - "style": "dark", - "tags": [ - "kubernetes-mixin" - ], - "templating": { - "list": [ - { - "current": { - "selected": false, - "text": "default", - "value": "default" - }, - "hide": 0, - "includeAll": false, - "label": null, - "multi": false, - "name": "datasource", - "options": [], - "query": "prometheus", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "type": "datasource" - }, - { - "allValue": null, - "current": { - "isNone": true, - "selected": false, - "text": "None", - "value": "" - }, - "datasource": "$datasource", - "definition": "", - "hide": 2, - "includeAll": false, - "index": -1, - "label": null, - "multi": false, - "name": "cluster", - "options": [], - "query": "label_values(apiserver_request_total, cluster)", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - }, - { - "allValue": null, - "current": { - "selected": false, - "text": "All", - "value": "$__all" - }, - "datasource": "$datasource", - "definition": "", - "hide": 0, - "includeAll": true, - "index": -1, - "label": null, - "multi": false, - "name": "instance", - "options": [], - "query": "label_values(apiserver_request_total{job=\"apiserver\", cluster=\"$cluster\"}, instance)", - "refresh": 2, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - } - ] - }, - "time": { - "from": "now-15m", - "to": "now" - }, - "timepicker": { - "refresh_intervals": [ - "5s", - "10s", - "30s", - "1m", - "5m", - "15m", - "30m", - "1h", - "2h", - "1d" - ], - "time_options": [ - "5m", - "15m", - "1h", - "6h", - "12h", - "24h", - "2d", - "7d", - "30d" - ] - }, - "timezone": "", - "title": "Kubernetes / API server", - "uid": "09ec8aa1e996d6ffcd6817bbaff4db1b", - "variables": { - "list": [] - }, - "version": 1 -} diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-kubelet.json b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-kubelet.json deleted file mode 100644 index e32f985d..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-kubelet.json +++ /dev/null @@ -1,2322 +0,0 @@ -{ - "annotations": { - "list": [ - { - "builtIn": 1, - "datasource": "-- Grafana --", - "enable": true, - "hide": true, - "iconColor": "rgba(0, 211, 255, 1)", - "name": "Annotations & Alerts", - "type": "dashboard" - } - ] - }, - "editable": false, - "gnetId": null, - "graphTooltip": 0, - "id": 5, - "iteration": 1588347138910, - "links": [], - "panels": [ - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 7, - "w": 4, - "x": 0, - "y": 0 - }, - "id": 2, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "sum(up{cluster=\"$cluster\", job=\"kubelet\", metrics_path=\"/metrics\"})", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "", - "title": "Up", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "min" - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 7, - "w": 4, - "x": 4, - "y": 0 - }, - "id": 3, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "sum(kubelet_running_pod_count{cluster=\"$cluster\", job=\"kubelet\", metrics_path=\"/metrics\", instance=~\"$instance\"})", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "refId": "A" - } - ], - "thresholds": "", - "title": "Running Pods", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "min" - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 7, - "w": 4, - "x": 8, - "y": 0 - }, - "id": 4, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "sum(kubelet_running_container_count{cluster=\"$cluster\", job=\"kubelet\", metrics_path=\"/metrics\", instance=~\"$instance\"})", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "refId": "A" - } - ], - "thresholds": "", - "title": "Running Container", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "min" - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 7, - "w": 4, - "x": 12, - "y": 0 - }, - "id": 5, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "sum(volume_manager_total_volumes{cluster=\"$cluster\", job=\"kubelet\", metrics_path=\"/metrics\", instance=~\"$instance\", state=\"actual_state_of_world\"})", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "refId": "A" - } - ], - "thresholds": "", - "title": "Actual Volume Count", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "min" - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 7, - "w": 4, - "x": 16, - "y": 0 - }, - "id": 6, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "sum(volume_manager_total_volumes{cluster=\"$cluster\", job=\"kubelet\", metrics_path=\"/metrics\", instance=~\"$instance\",state=\"desired_state_of_world\"})", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "refId": "A" - } - ], - "thresholds": "", - "title": "Desired Volume Count", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "min" - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 7, - "w": 4, - "x": 20, - "y": 0 - }, - "id": 7, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "sum(rate(kubelet_node_config_error{cluster=\"$cluster\", job=\"kubelet\", metrics_path=\"/metrics\", instance=~\"$instance\"}[5m]))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "refId": "A" - } - ], - "thresholds": "", - "title": "Config Error Count", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "min" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 7 - }, - "hiddenSeries": false, - "id": 8, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(kubelet_runtime_operations_total{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\",instance=~\"$instance\"}[5m])) by (operation_type, instance)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} {{operation_type}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Operation Rate", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 7 - }, - "hiddenSeries": false, - "id": 9, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(kubelet_runtime_operations_errors_total{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\",instance=~\"$instance\"}[5m])) by (instance, operation_type)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} {{operation_type}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Operation Error Rate", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 24, - "x": 0, - "y": 14 - }, - "hiddenSeries": false, - "id": 10, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "histogram_quantile(0.99, sum(rate(kubelet_runtime_operations_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\",instance=~\"$instance\"}[5m])) by (instance, operation_type, le))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} {{operation_type}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Operation duration 99th quantile", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 21 - }, - "hiddenSeries": false, - "id": 11, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(kubelet_pod_start_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\",instance=~\"$instance\"}[5m])) by (instance)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} pod", - "refId": "A" - }, - { - "expr": "sum(rate(kubelet_pod_worker_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\",instance=~\"$instance\"}[5m])) by (instance)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} worker", - "refId": "B" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Pod Start Rate", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 21 - }, - "hiddenSeries": false, - "id": 12, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "histogram_quantile(0.99, sum(rate(kubelet_pod_start_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\",instance=~\"$instance\"}[5m])) by (instance, le))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} pod", - "refId": "A" - }, - { - "expr": "histogram_quantile(0.99, sum(rate(kubelet_pod_worker_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\",instance=~\"$instance\"}[5m])) by (instance, le))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} worker", - "refId": "B" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Pod Start Duration", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 28 - }, - "hiddenSeries": false, - "id": 13, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(storage_operation_duration_seconds_count{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\",instance=~\"$instance\"}[5m])) by (instance, operation_name, volume_plugin)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} {{operation_name}} {{volume_plugin}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Storage Operation Rate", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 28 - }, - "hiddenSeries": false, - "id": 14, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(storage_operation_errors_total{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\",instance=~\"$instance\"}[5m])) by (instance, operation_name, volume_plugin)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} {{operation_name}} {{volume_plugin}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Storage Operation Error Rate", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 24, - "x": 0, - "y": 35 - }, - "hiddenSeries": false, - "id": 15, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "histogram_quantile(0.99, sum(rate(storage_operation_duration_seconds_bucket{cluster=\"$cluster\", job=\"kubelet\", metrics_path=\"/metrics\", instance=~\"$instance\"}[5m])) by (instance, operation_name, volume_plugin, le))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} {{operation_name}} {{volume_plugin}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Storage Operation Duration 99th quantile", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 42 - }, - "hiddenSeries": false, - "id": 16, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(kubelet_cgroup_manager_duration_seconds_count{cluster=\"$cluster\", job=\"kubelet\", metrics_path=\"/metrics\", instance=~\"$instance\"}[5m])) by (instance, operation_type)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{operation_type}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Cgroup manager operation rate", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 42 - }, - "hiddenSeries": false, - "id": 17, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "histogram_quantile(0.99, sum(rate(kubelet_cgroup_manager_duration_seconds_bucket{cluster=\"$cluster\", job=\"kubelet\", metrics_path=\"/metrics\", instance=~\"$instance\"}[5m])) by (instance, operation_type, le))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} {{operation_type}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Cgroup manager 99th quantile", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "Pod lifecycle event generator", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 49 - }, - "hiddenSeries": false, - "id": 18, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(kubelet_pleg_relist_duration_seconds_count{cluster=\"$cluster\", job=\"kubelet\", metrics_path=\"/metrics\", instance=~\"$instance\"}[5m])) by (instance)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "PLEG relist rate", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 49 - }, - "hiddenSeries": false, - "id": 19, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "histogram_quantile(0.99, sum(rate(kubelet_pleg_relist_interval_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\",instance=~\"$instance\"}[5m])) by (instance, le))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "PLEG relist interval", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 24, - "x": 0, - "y": 56 - }, - "hiddenSeries": false, - "id": 20, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "histogram_quantile(0.99, sum(rate(kubelet_pleg_relist_duration_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\",instance=~\"$instance\"}[5m])) by (instance, le))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "PLEG relist duration", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 24, - "x": 0, - "y": 63 - }, - "hiddenSeries": false, - "id": 21, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\", instance=~\"$instance\",code=~\"2..\"}[5m]))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "2xx", - "refId": "A" - }, - { - "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\", instance=~\"$instance\",code=~\"3..\"}[5m]))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "3xx", - "refId": "B" - }, - { - "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\", instance=~\"$instance\",code=~\"4..\"}[5m]))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "4xx", - "refId": "C" - }, - { - "expr": "sum(rate(rest_client_requests_total{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\", instance=~\"$instance\",code=~\"5..\"}[5m]))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "5xx", - "refId": "D" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "RPC Rate", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "ops", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "gridPos": { - "h": 7, - "w": 24, - "x": 0, - "y": 70 - }, - "id": 22, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "histogram_quantile(0.99, sum(rate(rest_client_request_latency_seconds_bucket{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\", instance=~\"$instance\"}[5m])) by (instance, verb, url, le))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} {{verb}} {{url}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Request duration 99th quantile", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "gridPos": { - "h": 7, - "w": 8, - "x": 0, - "y": 77 - }, - "id": 23, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "process_resident_memory_bytes{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\",instance=~\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Memory", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "bytes", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "bytes", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "gridPos": { - "h": 7, - "w": 8, - "x": 8, - "y": 77 - }, - "id": 24, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "rate(process_cpu_seconds_total{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\",instance=~\"$instance\"}[5m])", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "CPU usage", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "gridPos": { - "h": 7, - "w": 8, - "x": 16, - "y": 77 - }, - "id": 25, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "go_goroutines{cluster=\"$cluster\",job=\"kubelet\", metrics_path=\"/metrics\",instance=~\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Goroutines", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ] - } - ], - "refresh": "10s", - "schemaVersion": 22, - "style": "dark", - "tags": [ - "kubernetes-mixin" - ], - "templating": { - "list": [ - { - "current": { - "selected": false, - "text": "default", - "value": "default" - }, - "hide": 0, - "includeAll": false, - "label": null, - "multi": false, - "name": "datasource", - "options": [], - "query": "prometheus", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "type": "datasource" - }, - { - "allValue": null, - "current": { - "isNone": true, - "selected": false, - "text": "None", - "value": "" - }, - "datasource": "$datasource", - "definition": "", - "hide": 2, - "includeAll": false, - "index": -1, - "label": "cluster", - "multi": false, - "name": "cluster", - "options": [], - "query": "label_values(kube_pod_info, cluster)", - "refresh": 2, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - }, - { - "allValue": null, - "current": { - "selected": false, - "text": "All", - "value": "$__all" - }, - "datasource": "$datasource", - "definition": "", - "hide": 0, - "includeAll": true, - "index": -1, - "label": null, - "multi": false, - "name": "instance", - "options": [], - "query": "label_values(kubelet_runtime_operations_total{cluster=\"$cluster\", job=\"kubelet\", metrics_path=\"/metrics\"}, instance)", - "refresh": 2, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - } - ] - }, - "time": { - "from": "now-15m", - "to": "now" - }, - "timepicker": { - "refresh_intervals": [ - "5s", - "10s", - "30s", - "1m", - "5m", - "15m", - "30m", - "1h", - "2h", - "1d" - ], - "time_options": [ - "5m", - "15m", - "1h", - "6h", - "12h", - "24h", - "2d", - "7d", - "30d" - ] - }, - "timezone": "", - "title": "Kubernetes / Kubelet", - "uid": "3138fa155d5915769fbded898ac09fd9", - "variables": { - "list": [] - }, - "version": 1 -} diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-networking-cluster.json b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-networking-cluster.json deleted file mode 100644 index b4b0035b..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-networking-cluster.json +++ /dev/null @@ -1,1703 +0,0 @@ -{ - "annotations": { - "list": [ - { - "$$hashKey": "object:16", - "builtIn": 1, - "datasource": "-- Grafana --", - "enable": true, - "hide": true, - "iconColor": "rgba(0, 211, 255, 1)", - "name": "Annotations & Alerts", - "type": "dashboard" - } - ] - }, - "editable": true, - "gnetId": null, - "graphTooltip": 0, - "iteration": 1588347242015, - "links": [], - "panels": [ - { - "collapse": false, - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 0 - }, - "id": 2, - "panels": [], - "repeat": null, - "repeatIteration": null, - "repeatRowId": null, - "showTitle": true, - "title": "Current Bandwidth", - "titleSize": "h6", - "type": "row" - }, - { - "aliasColors": {}, - "bars": true, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "fillGradient": 0, - "gridPos": { - "h": 9, - "w": 12, - "x": 0, - "y": 1 - }, - "hiddenSeries": false, - "id": 3, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "sort": "current", - "sortDesc": true, - "total": false, - "values": true - }, - "lines": false, - "linewidth": 1, - "links": [], - "minSpan": 24, - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 24, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{namespace}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Current Rate of Bytes Received", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "series", - "name": null, - "show": false, - "values": [ - "current" - ] - }, - "yaxes": [ - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": true, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "fillGradient": 0, - "gridPos": { - "h": 9, - "w": 12, - "x": 12, - "y": 1 - }, - "hiddenSeries": false, - "id": 4, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "sort": "current", - "sortDesc": true, - "total": false, - "values": true - }, - "lines": false, - "linewidth": 1, - "links": [], - "minSpan": 24, - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 24, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{namespace}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Current Rate of Bytes Transmitted", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "series", - "name": null, - "show": false, - "values": [ - "current" - ] - }, - "yaxes": [ - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "columns": [ - { - "text": "Time", - "value": "Time" - }, - { - "text": "Value #A", - "value": "Value #A" - }, - { - "text": "Value #B", - "value": "Value #B" - }, - { - "text": "Value #C", - "value": "Value #C" - }, - { - "text": "Value #D", - "value": "Value #D" - }, - { - "text": "Value #E", - "value": "Value #E" - }, - { - "text": "Value #F", - "value": "Value #F" - }, - { - "text": "Value #G", - "value": "Value #G" - }, - { - "text": "Value #H", - "value": "Value #H" - }, - { - "text": "namespace", - "value": "namespace" - } - ], - "datasource": "$datasource", - "fill": 1, - "fontSize": "90%", - "gridPos": { - "h": 9, - "w": 24, - "x": 0, - "y": 10 - }, - "id": 5, - "lines": true, - "linewidth": 1, - "minSpan": 24, - "nullPointMode": "null as zero", - "pageSize": null, - "renderer": "flot", - "scroll": true, - "showHeader": true, - "sort": { - "col": 0, - "desc": false - }, - "spaceLength": 10, - "span": 24, - "styles": [ - { - "alias": "Time", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Time", - "thresholds": [], - "type": "hidden", - "unit": "short" - }, - { - "alias": "Current Bandwidth Received", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Value #A", - "thresholds": [], - "type": "number", - "unit": "Bps" - }, - { - "alias": "Current Bandwidth Transmitted", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Value #B", - "thresholds": [], - "type": "number", - "unit": "Bps" - }, - { - "alias": "Average Bandwidth Received", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Value #C", - "thresholds": [], - "type": "number", - "unit": "Bps" - }, - { - "alias": "Average Bandwidth Transmitted", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Value #D", - "thresholds": [], - "type": "number", - "unit": "Bps" - }, - { - "alias": "Rate of Received Packets", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Value #E", - "thresholds": [], - "type": "number", - "unit": "pps" - }, - { - "alias": "Rate of Transmitted Packets", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Value #F", - "thresholds": [], - "type": "number", - "unit": "pps" - }, - { - "alias": "Rate of Received Packets Dropped", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Value #G", - "thresholds": [], - "type": "number", - "unit": "pps" - }, - { - "alias": "Rate of Transmitted Packets Dropped", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Value #H", - "thresholds": [], - "type": "number", - "unit": "pps" - }, - { - "alias": "Namespace", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": true, - "linkTooltip": "Drill down", - "linkUrl": "d/8b7a8b326d7a6f1f04244066368c67af/kubernetes-networking-namespace-pods?orgId=1&refresh=30s&var-namespace=$__cell", - "pattern": "namespace", - "thresholds": [], - "type": "number", - "unit": "short" - } - ], - "targets": [ - { - "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "table", - "instant": true, - "intervalFactor": 2, - "legendFormat": "", - "refId": "A", - "step": 10 - }, - { - "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "table", - "instant": true, - "intervalFactor": 2, - "legendFormat": "", - "refId": "B", - "step": 10 - }, - { - "expr": "sort_desc(avg(irate(container_network_receive_bytes_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "table", - "instant": true, - "intervalFactor": 2, - "legendFormat": "", - "refId": "C", - "step": 10 - }, - { - "expr": "sort_desc(avg(irate(container_network_transmit_bytes_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "table", - "instant": true, - "intervalFactor": 2, - "legendFormat": "", - "refId": "D", - "step": 10 - }, - { - "expr": "sort_desc(sum(irate(container_network_receive_packets_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "table", - "instant": true, - "intervalFactor": 2, - "legendFormat": "", - "refId": "E", - "step": 10 - }, - { - "expr": "sort_desc(sum(irate(container_network_transmit_packets_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "table", - "instant": true, - "intervalFactor": 2, - "legendFormat": "", - "refId": "F", - "step": 10 - }, - { - "expr": "sort_desc(sum(irate(container_network_receive_packets_dropped_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "table", - "instant": true, - "intervalFactor": 2, - "legendFormat": "", - "refId": "G", - "step": 10 - }, - { - "expr": "sort_desc(sum(irate(container_network_transmit_packets_dropped_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "table", - "instant": true, - "intervalFactor": 2, - "legendFormat": "", - "refId": "H", - "step": 10 - } - ], - "timeFrom": null, - "timeShift": null, - "title": "Current Status", - "transform": "table", - "type": "table" - }, - { - "collapse": true, - "collapsed": true, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 19 - }, - "id": 6, - "panels": [ - { - "aliasColors": {}, - "bars": true, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "gridPos": { - "h": 9, - "w": 12, - "x": 0, - "y": 11 - }, - "id": 7, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "sort": "current", - "sortDesc": true, - "total": false, - "values": true - }, - "lines": false, - "linewidth": 1, - "links": [], - "minSpan": 24, - "nullPointMode": "null", - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 24, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(avg(irate(container_network_receive_bytes_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{namespace}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Average Rate of Bytes Received", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "series", - "name": null, - "show": false, - "values": [ - "current" - ] - }, - "yaxes": [ - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - }, - { - "aliasColors": {}, - "bars": true, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "gridPos": { - "h": 9, - "w": 12, - "x": 12, - "y": 11 - }, - "id": 8, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "sort": "current", - "sortDesc": true, - "total": false, - "values": true - }, - "lines": false, - "linewidth": 1, - "links": [], - "minSpan": 24, - "nullPointMode": "null", - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 24, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(avg(irate(container_network_transmit_bytes_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{namespace}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Average Rate of Bytes Transmitted", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "series", - "name": null, - "show": false, - "values": [ - "current" - ] - }, - "yaxes": [ - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - } - ], - "repeat": null, - "repeatIteration": null, - "repeatRowId": null, - "showTitle": true, - "title": "Average Bandwidth", - "titleSize": "h6", - "type": "row" - }, - { - "collapse": false, - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 20 - }, - "id": 9, - "panels": [], - "repeat": null, - "repeatIteration": null, - "repeatRowId": null, - "showTitle": true, - "title": "Bandwidth History", - "titleSize": "h6", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "fillGradient": 0, - "gridPos": { - "h": 9, - "w": 24, - "x": 0, - "y": 21 - }, - "hiddenSeries": false, - "id": 10, - "legend": { - "alignAsTable": true, - "avg": true, - "current": true, - "hideEmpty": true, - "hideZero": true, - "max": true, - "min": true, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 24, - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 24, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(sum(irate(container_network_receive_bytes_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{namespace}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Receive Bandwidth", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "fillGradient": 0, - "gridPos": { - "h": 9, - "w": 24, - "x": 0, - "y": 30 - }, - "hiddenSeries": false, - "id": 11, - "legend": { - "alignAsTable": true, - "avg": true, - "current": true, - "hideEmpty": true, - "hideZero": true, - "max": true, - "min": true, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 24, - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 24, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(sum(irate(container_network_transmit_bytes_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{namespace}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Transmit Bandwidth", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "collapse": true, - "collapsed": true, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 39 - }, - "id": 12, - "panels": [ - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "gridPos": { - "h": 9, - "w": 24, - "x": 0, - "y": 31 - }, - "id": 13, - "legend": { - "alignAsTable": true, - "avg": true, - "current": true, - "hideEmpty": true, - "hideZero": true, - "max": true, - "min": true, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 24, - "nullPointMode": "connected", - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 24, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(sum(irate(container_network_receive_packets_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{namespace}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Rate of Received Packets", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "gridPos": { - "h": 9, - "w": 24, - "x": 0, - "y": 40 - }, - "id": 14, - "legend": { - "alignAsTable": true, - "avg": true, - "current": true, - "hideEmpty": true, - "hideZero": true, - "max": true, - "min": true, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 24, - "nullPointMode": "connected", - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 24, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(sum(irate(container_network_transmit_packets_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{namespace}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Rate of Transmitted Packets", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - } - ], - "repeat": null, - "repeatIteration": null, - "repeatRowId": null, - "showTitle": true, - "title": "Packets", - "titleSize": "h6", - "type": "row" - }, - { - "collapse": true, - "collapsed": true, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 40 - }, - "id": 15, - "panels": [ - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "gridPos": { - "h": 9, - "w": 24, - "x": 0, - "y": 50 - }, - "id": 16, - "legend": { - "alignAsTable": true, - "avg": true, - "current": true, - "hideEmpty": true, - "hideZero": true, - "max": true, - "min": true, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 24, - "nullPointMode": "connected", - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 24, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(sum(irate(container_network_receive_packets_dropped_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{namespace}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Rate of Received Packets Dropped", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "gridPos": { - "h": 9, - "w": 24, - "x": 0, - "y": 59 - }, - "id": 17, - "legend": { - "alignAsTable": true, - "avg": true, - "current": true, - "hideEmpty": true, - "hideZero": true, - "max": true, - "min": true, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 24, - "nullPointMode": "connected", - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 24, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(sum(irate(container_network_transmit_packets_dropped_total{namespace=~\".+\"}[$interval:$resolution])) by (namespace))", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{namespace}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Rate of Transmitted Packets Dropped", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "gridPos": { - "h": 9, - "w": 24, - "x": 0, - "y": 59 - }, - "id": 18, - "legend": { - "alignAsTable": true, - "avg": true, - "current": true, - "hideEmpty": true, - "hideZero": true, - "max": true, - "min": true, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 2, - "links": [ - { - "targetBlank": true, - "title": "What is TCP Retransmit?", - "url": "https://accedian.com/enterprises/blog/network-packet-loss-retransmissions-and-duplicate-acknowledgements/" - } - ], - "minSpan": 24, - "nullPointMode": "connected", - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 24, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(sum(rate(node_netstat_Tcp_RetransSegs[$interval:$resolution]) / rate(node_netstat_Tcp_OutSegs[$interval:$resolution])) by (instance))", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{instance}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Rate of TCP Retransmits out of all sent segments", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "gridPos": { - "h": 9, - "w": 24, - "x": 0, - "y": 59 - }, - "id": 19, - "legend": { - "alignAsTable": true, - "avg": true, - "current": true, - "hideEmpty": true, - "hideZero": true, - "max": true, - "min": true, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 2, - "links": [ - { - "targetBlank": true, - "title": "Why monitor SYN retransmits?", - "url": "https://github.com/prometheus/node_exporter/issues/1023#issuecomment-408128365" - } - ], - "minSpan": 24, - "nullPointMode": "connected", - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 24, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(sum(rate(node_netstat_TcpExt_TCPSynRetrans[$interval:$resolution]) / rate(node_netstat_Tcp_RetransSegs[$interval:$resolution])) by (instance))", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{instance}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Rate of TCP SYN Retransmits out of all retransmits", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - } - ], - "repeat": null, - "repeatIteration": null, - "repeatRowId": null, - "showTitle": true, - "title": "Errors", - "titleSize": "h6", - "type": "row" - } - ], - "refresh": "10s", - "schemaVersion": 22, - "style": "dark", - "tags": [ - "kubernetes-mixin" - ], - "templating": { - "list": [ - { - "allValue": null, - "auto": false, - "auto_count": 30, - "auto_min": "10s", - "current": { - "selected": false, - "text": "5m", - "value": "5m" - }, - "datasource": "$datasource", - "hide": 0, - "includeAll": false, - "label": null, - "multi": false, - "name": "resolution", - "options": [ - { - "selected": false, - "text": "30s", - "value": "30s" - }, - { - "selected": true, - "text": "5m", - "value": "5m" - }, - { - "selected": false, - "text": "1h", - "value": "1h" - } - ], - "query": "30s,5m,1h", - "refresh": 2, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "interval", - "useTags": false - }, - { - "allValue": null, - "auto": false, - "auto_count": 30, - "auto_min": "10s", - "current": { - "selected": false, - "text": "4h", - "value": "4h" - }, - "datasource": "$datasource", - "hide": 2, - "includeAll": false, - "label": null, - "multi": false, - "name": "interval", - "options": [ - { - "selected": true, - "text": "4h", - "value": "4h" - } - ], - "query": "4h", - "refresh": 2, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "interval", - "useTags": false - }, - { - "current": { - "selected": false, - "text": "default", - "value": "default" - }, - "hide": 0, - "includeAll": false, - "label": null, - "multi": false, - "name": "datasource", - "options": [], - "query": "prometheus", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "type": "datasource" - } - ] - }, - "time": { - "from": "now-1h", - "to": "now" - }, - "timepicker": { - "refresh_intervals": [ - "5s", - "10s", - "30s", - "1m", - "5m", - "15m", - "30m", - "1h", - "2h", - "1d" - ], - "time_options": [ - "5m", - "15m", - "1h", - "6h", - "12h", - "24h", - "2d", - "7d", - "30d" - ] - }, - "timezone": "", - "title": "Kubernetes / Networking / Cluster", - "uid": "ff635a025bcfea7bc3dd4f508990a3e9", - "variables": { - "list": [] - }, - "version": 1 -} diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-networking-namespace-pods.json b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-networking-namespace-pods.json deleted file mode 100644 index 73353429..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-networking-namespace-pods.json +++ /dev/null @@ -1,1369 +0,0 @@ -{ - "annotations": { - "list": [ - { - "$$hashKey": "object:19", - "builtIn": 1, - "datasource": "-- Grafana --", - "enable": true, - "hide": true, - "iconColor": "rgba(0, 211, 255, 1)", - "name": "Annotations & Alerts", - "type": "dashboard" - } - ] - }, - "editable": true, - "gnetId": null, - "graphTooltip": 0, - "iteration": 1588347345377, - "links": [], - "panels": [ - { - "collapse": false, - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 0 - }, - "id": 2, - "panels": [], - "repeat": null, - "repeatIteration": null, - "repeatRowId": null, - "showTitle": true, - "title": "Current Bandwidth", - "titleSize": "h6", - "type": "row" - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "decimals": 0, - "format": "time_series", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 9, - "w": 12, - "x": 0, - "y": 1 - }, - "height": 9, - "id": 3, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "minSpan": 12, - "nullPointMode": "connected", - "nullText": null, - "options": { - "fieldOptions": { - "calcs": [ - "last" - ], - "defaults": { - "color": { - "mode": "thresholds" - }, - "mappings": [], - "max": 10000000000, - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "dark-green", - "index": 0, - "value": null - }, - { - "color": "dark-yellow", - "index": 1, - "value": 5000000000 - }, - { - "color": "dark-red", - "index": 2, - "value": 7000000000 - } - ] - }, - "title": "$namespace", - "unit": "Bps" - }, - "overrides": [], - "thresholds": [ - { - "color": "dark-green", - "index": 0, - "value": null - }, - { - "color": "dark-yellow", - "index": 1, - "value": 5000000000 - }, - { - "color": "dark-red", - "index": 2, - "value": 7000000000 - } - ], - "values": false - }, - "orientation": "auto", - "showThresholdLabels": false, - "showThresholdMarkers": true - }, - "pluginVersion": "6.7.3", - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "span": 12, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "sum(irate(container_network_receive_bytes_total{namespace=~\"$namespace\"}[$interval:$resolution]))", - "format": "time_series", - "instant": null, - "intervalFactor": 1, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, - "title": "Current Rate of Bytes Received", - "type": "gauge", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "decimals": 0, - "format": "time_series", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 9, - "w": 12, - "x": 12, - "y": 1 - }, - "height": 9, - "id": 4, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "minSpan": 12, - "nullPointMode": "connected", - "nullText": null, - "options": { - "fieldOptions": { - "calcs": [ - "last" - ], - "defaults": { - "color": { - "mode": "thresholds" - }, - "mappings": [], - "max": 10000000000, - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "dark-green", - "index": 0, - "value": null - }, - { - "color": "dark-yellow", - "index": 1, - "value": 5000000000 - }, - { - "color": "dark-red", - "index": 2, - "value": 7000000000 - } - ] - }, - "title": "$namespace", - "unit": "Bps" - }, - "overrides": [], - "thresholds": [ - { - "color": "dark-green", - "index": 0, - "value": null - }, - { - "color": "dark-yellow", - "index": 1, - "value": 5000000000 - }, - { - "color": "dark-red", - "index": 2, - "value": 7000000000 - } - ], - "values": false - }, - "orientation": "auto", - "showThresholdLabels": false, - "showThresholdMarkers": true - }, - "pluginVersion": "6.7.3", - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "span": 12, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "sum(irate(container_network_transmit_bytes_total{namespace=~\"$namespace\"}[$interval:$resolution]))", - "format": "time_series", - "instant": null, - "intervalFactor": 1, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, - "title": "Current Rate of Bytes Transmitted", - "type": "gauge", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" - }, - { - "columns": [ - { - "text": "Time", - "value": "Time" - }, - { - "text": "Value #A", - "value": "Value #A" - }, - { - "text": "Value #B", - "value": "Value #B" - }, - { - "text": "Value #C", - "value": "Value #C" - }, - { - "text": "Value #D", - "value": "Value #D" - }, - { - "text": "Value #E", - "value": "Value #E" - }, - { - "text": "Value #F", - "value": "Value #F" - }, - { - "text": "pod", - "value": "pod" - } - ], - "datasource": "$datasource", - "fill": 1, - "fontSize": "100%", - "gridPos": { - "h": 9, - "w": 24, - "x": 0, - "y": 10 - }, - "id": 5, - "lines": true, - "linewidth": 1, - "minSpan": 24, - "nullPointMode": "null as zero", - "pageSize": null, - "renderer": "flot", - "scroll": true, - "showHeader": true, - "sort": { - "col": 0, - "desc": false - }, - "spaceLength": 10, - "span": 24, - "styles": [ - { - "alias": "Time", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Time", - "thresholds": [], - "type": "hidden", - "unit": "short" - }, - { - "alias": "Bandwidth Received", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Value #A", - "thresholds": [], - "type": "number", - "unit": "Bps" - }, - { - "alias": "Bandwidth Transmitted", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Value #B", - "thresholds": [], - "type": "number", - "unit": "Bps" - }, - { - "alias": "Rate of Received Packets", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Value #C", - "thresholds": [], - "type": "number", - "unit": "pps" - }, - { - "alias": "Rate of Transmitted Packets", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Value #D", - "thresholds": [], - "type": "number", - "unit": "pps" - }, - { - "alias": "Rate of Received Packets Dropped", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Value #E", - "thresholds": [], - "type": "number", - "unit": "pps" - }, - { - "alias": "Rate of Transmitted Packets Dropped", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Value #F", - "thresholds": [], - "type": "number", - "unit": "pps" - }, - { - "alias": "Pod", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": true, - "linkTooltip": "Drill down", - "linkUrl": "d/7a18067ce943a40ae25454675c19ff5c/kubernetes-networking-pod?orgId=1&refresh=30s&var-namespace=$namespace&var-pod=$__cell", - "pattern": "pod", - "thresholds": [], - "type": "number", - "unit": "short" - } - ], - "targets": [ - { - "expr": "sum(irate(container_network_receive_bytes_total{namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)", - "format": "table", - "instant": true, - "intervalFactor": 2, - "legendFormat": "", - "refId": "A", - "step": 10 - }, - { - "expr": "sum(irate(container_network_transmit_bytes_total{namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)", - "format": "table", - "instant": true, - "intervalFactor": 2, - "legendFormat": "", - "refId": "B", - "step": 10 - }, - { - "expr": "sum(irate(container_network_receive_packets_total{namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)", - "format": "table", - "instant": true, - "intervalFactor": 2, - "legendFormat": "", - "refId": "C", - "step": 10 - }, - { - "expr": "sum(irate(container_network_transmit_packets_total{namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)", - "format": "table", - "instant": true, - "intervalFactor": 2, - "legendFormat": "", - "refId": "D", - "step": 10 - }, - { - "expr": "sum(irate(container_network_receive_packets_dropped_total{namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)", - "format": "table", - "instant": true, - "intervalFactor": 2, - "legendFormat": "", - "refId": "E", - "step": 10 - }, - { - "expr": "sum(irate(container_network_transmit_packets_dropped_total{namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)", - "format": "table", - "instant": true, - "intervalFactor": 2, - "legendFormat": "", - "refId": "F", - "step": 10 - } - ], - "timeFrom": null, - "timeShift": null, - "title": "Current Status", - "transform": "table", - "type": "table" - }, - { - "collapse": false, - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 19 - }, - "id": 6, - "panels": [], - "repeat": null, - "repeatIteration": null, - "repeatRowId": null, - "showTitle": true, - "title": "Bandwidth", - "titleSize": "h6", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "fillGradient": 0, - "gridPos": { - "h": 9, - "w": 12, - "x": 0, - "y": 20 - }, - "hiddenSeries": false, - "id": 7, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 12, - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 12, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sum(irate(container_network_receive_bytes_total{namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{pod}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Receive Bandwidth", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "fillGradient": 0, - "gridPos": { - "h": 9, - "w": 12, - "x": 12, - "y": 20 - }, - "hiddenSeries": false, - "id": 8, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 12, - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 12, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sum(irate(container_network_transmit_bytes_total{namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{pod}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Transmit Bandwidth", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "collapse": true, - "collapsed": true, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 29 - }, - "id": 9, - "panels": [ - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "gridPos": { - "h": 10, - "w": 12, - "x": 0, - "y": 30 - }, - "id": 10, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 12, - "nullPointMode": "connected", - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 12, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sum(irate(container_network_receive_packets_total{namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{pod}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Rate of Received Packets", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "gridPos": { - "h": 10, - "w": 12, - "x": 12, - "y": 30 - }, - "id": 11, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 12, - "nullPointMode": "connected", - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 12, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sum(irate(container_network_transmit_packets_total{namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{pod}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Rate of Transmitted Packets", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - } - ], - "repeat": null, - "repeatIteration": null, - "repeatRowId": null, - "showTitle": true, - "title": "Packets", - "titleSize": "h6", - "type": "row" - }, - { - "collapse": true, - "collapsed": true, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 30 - }, - "id": 12, - "panels": [ - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "gridPos": { - "h": 10, - "w": 12, - "x": 0, - "y": 40 - }, - "id": 13, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 12, - "nullPointMode": "connected", - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 12, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sum(irate(container_network_receive_packets_dropped_total{namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{pod}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Rate of Received Packets Dropped", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "gridPos": { - "h": 10, - "w": 12, - "x": 12, - "y": 40 - }, - "id": 14, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 12, - "nullPointMode": "connected", - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 12, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sum(irate(container_network_transmit_packets_dropped_total{namespace=~\"$namespace\"}[$interval:$resolution])) by (pod)", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{pod}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Rate of Transmitted Packets Dropped", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - } - ], - "repeat": null, - "repeatIteration": null, - "repeatRowId": null, - "showTitle": true, - "title": "Errors", - "titleSize": "h6", - "type": "row" - } - ], - "refresh": "10s", - "schemaVersion": 22, - "style": "dark", - "tags": [ - "kubernetes-mixin" - ], - "templating": { - "list": [ - { - "current": { - "selected": false, - "text": "default", - "value": "default" - }, - "hide": 0, - "includeAll": false, - "label": null, - "multi": false, - "name": "datasource", - "options": [], - "query": "prometheus", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "type": "datasource" - }, - { - "allValue": ".+", - "auto": false, - "auto_count": 30, - "auto_min": "10s", - "current": { - "text": "kube-system", - "value": "kube-system" - }, - "datasource": "$datasource", - "definition": "label_values(container_network_receive_packets_total, namespace)", - "hide": 0, - "includeAll": true, - "index": -1, - "label": null, - "multi": false, - "name": "namespace", - "options": [], - "query": "label_values(container_network_receive_packets_total, namespace)", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - }, - { - "allValue": null, - "auto": false, - "auto_count": 30, - "auto_min": "10s", - "current": { - "selected": false, - "text": "5m", - "value": "5m" - }, - "datasource": "$datasource", - "hide": 0, - "includeAll": false, - "label": null, - "multi": false, - "name": "resolution", - "options": [ - { - "selected": false, - "text": "30s", - "value": "30s" - }, - { - "selected": true, - "text": "5m", - "value": "5m" - }, - { - "selected": false, - "text": "1h", - "value": "1h" - } - ], - "query": "30s,5m,1h", - "refresh": 2, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "interval", - "useTags": false - }, - { - "allValue": null, - "auto": false, - "auto_count": 30, - "auto_min": "10s", - "current": { - "selected": false, - "text": "4h", - "value": "4h" - }, - "datasource": "$datasource", - "hide": 2, - "includeAll": false, - "label": null, - "multi": false, - "name": "interval", - "options": [ - { - "selected": true, - "text": "4h", - "value": "4h" - } - ], - "query": "4h", - "refresh": 2, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "interval", - "useTags": false - } - ] - }, - "time": { - "from": "now-1h", - "to": "now" - }, - "timepicker": { - "refresh_intervals": [ - "5s", - "10s", - "30s", - "1m", - "5m", - "15m", - "30m", - "1h", - "2h", - "1d" - ], - "time_options": [ - "5m", - "15m", - "1h", - "6h", - "12h", - "24h", - "2d", - "7d", - "30d" - ] - }, - "timezone": "", - "title": "Kubernetes / Networking / Namespace (Pods)", - "uid": "8b7a8b326d7a6f1f04244066368c67af", - "variables": { - "list": [] - }, - "version": 1 -} diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-networking-pod.json b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-networking-pod.json deleted file mode 100644 index fb6b05f3..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-networking-pod.json +++ /dev/null @@ -1,1155 +0,0 @@ -{ - "annotations": { - "list": [ - { - "$$hashKey": "object:22", - "builtIn": 1, - "datasource": "-- Grafana --", - "enable": true, - "hide": true, - "iconColor": "rgba(0, 211, 255, 1)", - "name": "Annotations & Alerts", - "type": "dashboard" - } - ] - }, - "editable": true, - "gnetId": null, - "graphTooltip": 0, - "iteration": 1588347348337, - "links": [], - "panels": [ - { - "collapse": false, - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 0 - }, - "id": 2, - "panels": [], - "repeat": null, - "repeatIteration": null, - "repeatRowId": null, - "showTitle": true, - "title": "Current Bandwidth", - "titleSize": "h6", - "type": "row" - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "decimals": 0, - "format": "time_series", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 9, - "w": 12, - "x": 0, - "y": 1 - }, - "height": 9, - "id": 3, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "minSpan": 12, - "nullPointMode": "connected", - "nullText": null, - "options": { - "fieldOptions": { - "calcs": [ - "last" - ], - "defaults": { - "color": { - "mode": "thresholds" - }, - "mappings": [], - "max": 10000000000, - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "dark-green", - "index": 0, - "value": null - }, - { - "color": "dark-yellow", - "index": 1, - "value": 5000000000 - }, - { - "color": "dark-red", - "index": 2, - "value": 7000000000 - } - ] - }, - "title": "$namespace: $pod", - "unit": "Bps" - }, - "overrides": [], - "thresholds": [ - { - "color": "dark-green", - "index": 0, - "value": null - }, - { - "color": "dark-yellow", - "index": 1, - "value": 5000000000 - }, - { - "color": "dark-red", - "index": 2, - "value": 7000000000 - } - ], - "values": false - }, - "orientation": "auto", - "showThresholdLabels": false, - "showThresholdMarkers": true - }, - "pluginVersion": "6.7.3", - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "span": 12, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "sum(irate(container_network_receive_bytes_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution]))", - "format": "time_series", - "instant": null, - "intervalFactor": 1, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, - "title": "Current Rate of Bytes Received", - "type": "gauge", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "decimals": 0, - "format": "time_series", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 9, - "w": 12, - "x": 12, - "y": 1 - }, - "height": 9, - "id": 4, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "minSpan": 12, - "nullPointMode": "connected", - "nullText": null, - "options": { - "fieldOptions": { - "calcs": [ - "last" - ], - "defaults": { - "color": { - "mode": "thresholds" - }, - "mappings": [], - "max": 10000000000, - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "dark-green", - "index": 0, - "value": null - }, - { - "color": "dark-yellow", - "index": 1, - "value": 5000000000 - }, - { - "color": "dark-red", - "index": 2, - "value": 7000000000 - } - ] - }, - "title": "$namespace: $pod", - "unit": "Bps" - }, - "overrides": [], - "thresholds": [ - { - "color": "dark-green", - "index": 0, - "value": null - }, - { - "color": "dark-yellow", - "index": 1, - "value": 5000000000 - }, - { - "color": "dark-red", - "index": 2, - "value": 7000000000 - } - ], - "values": false - }, - "orientation": "auto", - "showThresholdLabels": false, - "showThresholdMarkers": true - }, - "pluginVersion": "6.7.3", - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "span": 12, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "sum(irate(container_network_transmit_bytes_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution]))", - "format": "time_series", - "instant": null, - "intervalFactor": 1, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, - "title": "Current Rate of Bytes Transmitted", - "type": "gauge", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" - }, - { - "collapse": false, - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 10 - }, - "id": 5, - "panels": [], - "repeat": null, - "repeatIteration": null, - "repeatRowId": null, - "showTitle": true, - "title": "Bandwidth", - "titleSize": "h6", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "fillGradient": 0, - "gridPos": { - "h": 9, - "w": 12, - "x": 0, - "y": 11 - }, - "hiddenSeries": false, - "id": 6, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 12, - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 12, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sum(irate(container_network_receive_bytes_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{pod}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Receive Bandwidth", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "fillGradient": 0, - "gridPos": { - "h": 9, - "w": 12, - "x": 12, - "y": 11 - }, - "hiddenSeries": false, - "id": 7, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 12, - "nullPointMode": "connected", - "options": { - "dataLinks": [] - }, - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 12, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sum(irate(container_network_transmit_bytes_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{pod}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Transmit Bandwidth", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "collapse": true, - "collapsed": true, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 20 - }, - "id": 8, - "panels": [ - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "gridPos": { - "h": 10, - "w": 12, - "x": 0, - "y": 21 - }, - "id": 9, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 12, - "nullPointMode": "connected", - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 12, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sum(irate(container_network_receive_packets_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{pod}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Rate of Received Packets", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "gridPos": { - "h": 10, - "w": 12, - "x": 12, - "y": 21 - }, - "id": 10, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 12, - "nullPointMode": "connected", - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 12, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sum(irate(container_network_transmit_packets_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{pod}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Rate of Transmitted Packets", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - } - ], - "repeat": null, - "repeatIteration": null, - "repeatRowId": null, - "showTitle": true, - "title": "Packets", - "titleSize": "h6", - "type": "row" - }, - { - "collapse": true, - "collapsed": true, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 21 - }, - "id": 11, - "panels": [ - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "gridPos": { - "h": 10, - "w": 12, - "x": 0, - "y": 32 - }, - "id": 12, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 12, - "nullPointMode": "connected", - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 12, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sum(irate(container_network_receive_packets_dropped_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{pod}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Rate of Received Packets Dropped", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 2, - "gridPos": { - "h": 10, - "w": 12, - "x": 12, - "y": 32 - }, - "id": 13, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": true, - "hideZero": true, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "links": [], - "minSpan": 12, - "nullPointMode": "connected", - "paceLength": 10, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "span": 12, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sum(irate(container_network_transmit_packets_dropped_total{namespace=~\"$namespace\", pod=~\"$pod\"}[$interval:$resolution])) by (pod)", - "format": "time_series", - "intervalFactor": 1, - "legendFormat": "{{pod}}", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Rate of Transmitted Packets Dropped", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "pps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ] - } - ], - "repeat": null, - "repeatIteration": null, - "repeatRowId": null, - "showTitle": true, - "title": "Errors", - "titleSize": "h6", - "type": "row" - } - ], - "refresh": "10s", - "schemaVersion": 22, - "style": "dark", - "tags": [ - "kubernetes-mixin" - ], - "templating": { - "list": [ - { - "current": { - "selected": false, - "text": "default", - "value": "default" - }, - "hide": 0, - "includeAll": false, - "label": null, - "multi": false, - "name": "datasource", - "options": [], - "query": "prometheus", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "type": "datasource" - }, - { - "allValue": ".+", - "auto": false, - "auto_count": 30, - "auto_min": "10s", - "current": { - "text": "kube-system", - "value": "kube-system" - }, - "datasource": "$datasource", - "definition": "label_values(container_network_receive_packets_total, namespace)", - "hide": 0, - "includeAll": true, - "index": -1, - "label": null, - "multi": false, - "name": "namespace", - "options": [], - "query": "label_values(container_network_receive_packets_total, namespace)", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - }, - { - "allValue": ".+", - "auto": false, - "auto_count": 30, - "auto_min": "10s", - "current": { - "text": "dns-controller-9cf777c54-9dwcc", - "value": "dns-controller-9cf777c54-9dwcc" - }, - "datasource": "$datasource", - "definition": "label_values(container_network_receive_packets_total{namespace=~\"$namespace\"}, pod)", - "hide": 0, - "includeAll": false, - "index": -1, - "label": null, - "multi": false, - "name": "pod", - "options": [], - "query": "label_values(container_network_receive_packets_total{namespace=~\"$namespace\"}, pod)", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - }, - { - "allValue": null, - "auto": false, - "auto_count": 30, - "auto_min": "10s", - "current": { - "selected": false, - "text": "5m", - "value": "5m" - }, - "datasource": "$datasource", - "hide": 0, - "includeAll": false, - "label": null, - "multi": false, - "name": "resolution", - "options": [ - { - "selected": false, - "text": "30s", - "value": "30s" - }, - { - "selected": true, - "text": "5m", - "value": "5m" - }, - { - "selected": false, - "text": "1h", - "value": "1h" - } - ], - "query": "30s,5m,1h", - "refresh": 2, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "interval", - "useTags": false - }, - { - "allValue": null, - "auto": false, - "auto_count": 30, - "auto_min": "10s", - "current": { - "selected": false, - "text": "4h", - "value": "4h" - }, - "datasource": "$datasource", - "hide": 2, - "includeAll": false, - "label": null, - "multi": false, - "name": "interval", - "options": [ - { - "selected": true, - "text": "4h", - "value": "4h" - } - ], - "query": "4h", - "refresh": 2, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "interval", - "useTags": false - } - ] - }, - "time": { - "from": "now-1h", - "to": "now" - }, - "timepicker": { - "refresh_intervals": [ - "5s", - "10s", - "30s", - "1m", - "5m", - "15m", - "30m", - "1h", - "2h", - "1d" - ], - "time_options": [ - "5m", - "15m", - "1h", - "6h", - "12h", - "24h", - "2d", - "7d", - "30d" - ] - }, - "timezone": "", - "title": "Kubernetes / Networking / Pod", - "uid": "7a18067ce943a40ae25454675c19ff5c", - "variables": { - "list": [] - }, - "version": 1 -} diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-stateful-sets.json b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-stateful-sets.json deleted file mode 100644 index 085d8b12..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/k8s-stateful-sets.json +++ /dev/null @@ -1,873 +0,0 @@ -{ - "annotations": { - "list": [ - { - "builtIn": 1, - "datasource": "-- Grafana --", - "enable": true, - "hide": true, - "iconColor": "rgba(0, 211, 255, 1)", - "name": "Annotations & Alerts", - "type": "dashboard" - } - ] - }, - "editable": false, - "gnetId": null, - "graphTooltip": 0, - "id": 3, - "iteration": 1588347434847, - "links": [], - "panels": [ - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 7, - "w": 8, - "x": 0, - "y": 0 - }, - "id": 2, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "cores", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "lineColor": "rgb(31, 120, 193)", - "show": true - }, - "tableColumn": "", - "targets": [ - { - "expr": "sum(rate(container_cpu_usage_seconds_total{job=\"kubelet\", metrics_path=\"/metrics/cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$statefulset.*\"}[3m]))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "", - "title": "CPU", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "0", - "value": "null" - } - ], - "valueName": "current" - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 7, - "w": 8, - "x": 8, - "y": 0 - }, - "id": 3, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "GB", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "lineColor": "rgb(31, 120, 193)", - "show": true - }, - "tableColumn": "", - "targets": [ - { - "expr": "sum(container_memory_usage_bytes{job=\"kubelet\", metrics_path=\"/metrics/cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$statefulset.*\"}) / 1024^3", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "", - "title": "Memory", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "0", - "value": "null" - } - ], - "valueName": "current" - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 7, - "w": 8, - "x": 16, - "y": 0 - }, - "id": 4, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "Bps", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "lineColor": "rgb(31, 120, 193)", - "show": true - }, - "tableColumn": "", - "targets": [ - { - "expr": "sum(rate(container_network_transmit_bytes_total{job=\"kubelet\", metrics_path=\"/metrics/cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$statefulset.*\"}[3m])) + sum(rate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=\"$namespace\",pod=~\"$statefulset.*\"}[3m]))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "", - "title": "Network", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "0", - "value": "null" - } - ], - "valueName": "current" - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 6, - "x": 0, - "y": 7 - }, - "id": 5, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "max(kube_statefulset_replicas{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", statefulset=\"$statefulset\"}) without (instance, pod)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "", - "title": "Desired Replicas", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "0", - "value": "null" - } - ], - "valueName": "current" - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 6, - "x": 6, - "y": 7 - }, - "id": 6, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "min(kube_statefulset_status_replicas_current{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", statefulset=\"$statefulset\"}) without (instance, pod)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "", - "title": "Replicas of current version", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "0", - "value": "null" - } - ], - "valueName": "current" - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 6, - "x": 12, - "y": 7 - }, - "id": 7, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "max(kube_statefulset_status_observed_generation{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", statefulset=\"$statefulset\"}) without (instance, pod)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "", - "title": "Observed Generation", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "0", - "value": "null" - } - ], - "valueName": "current" - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "$datasource", - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 6, - "x": 18, - "y": 7 - }, - "id": 8, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "max(kube_statefulset_metadata_generation{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "", - "title": "Metadata Generation", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "0", - "value": "null" - } - ], - "valueName": "current" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 8, - "x": 0, - "y": 10 - }, - "hiddenSeries": false, - "id": 9, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "max(kube_statefulset_replicas{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "replicas specified", - "refId": "A" - }, - { - "expr": "max(kube_statefulset_status_replicas{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "replicas created", - "refId": "B" - }, - { - "expr": "min(kube_statefulset_status_replicas_ready{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "ready", - "refId": "C" - }, - { - "expr": "min(kube_statefulset_status_replicas_current{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "replicas of current version", - "refId": "D" - }, - { - "expr": "min(kube_statefulset_status_replicas_updated{job=\"kube-state-metrics\", statefulset=\"$statefulset\", cluster=\"$cluster\", namespace=\"$namespace\"}) without (instance, pod)", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "updated", - "refId": "E" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Replicas", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - } - ], - "refresh": "", - "schemaVersion": 22, - "style": "dark", - "tags": [ - "kubernetes-mixin" - ], - "templating": { - "list": [ - { - "current": { - "selected": false, - "text": "default", - "value": "default" - }, - "hide": 0, - "includeAll": false, - "label": null, - "multi": false, - "name": "datasource", - "options": [], - "query": "prometheus", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "type": "datasource" - }, - { - "allValue": null, - "current": { - "isNone": true, - "selected": false, - "text": "None", - "value": "" - }, - "datasource": "$datasource", - "definition": "", - "hide": 2, - "includeAll": false, - "index": -1, - "label": "cluster", - "multi": false, - "name": "cluster", - "options": [], - "query": "label_values(kube_statefulset_metadata_generation, cluster)", - "refresh": 2, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - }, - { - "allValue": null, - "current": { - "text": "default", - "value": "default" - }, - "datasource": "$datasource", - "definition": "", - "hide": 0, - "includeAll": false, - "index": -1, - "label": "Namespace", - "multi": false, - "name": "namespace", - "options": [], - "query": "label_values(kube_statefulset_metadata_generation{job=\"kube-state-metrics\", cluster=\"$cluster\"}, namespace)", - "refresh": 2, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - }, - { - "allValue": null, - "current": { - "text": "influxdb", - "value": "influxdb" - }, - "datasource": "$datasource", - "definition": "", - "hide": 0, - "includeAll": false, - "index": -1, - "label": "Name", - "multi": false, - "name": "statefulset", - "options": [], - "query": "label_values(kube_statefulset_metadata_generation{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\"}, statefulset)", - "refresh": 2, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - } - ] - }, - "time": { - "from": "now-1h", - "to": "now" - }, - "timepicker": { - "refresh_intervals": [ - "5s", - "10s", - "30s", - "1m", - "5m", - "15m", - "30m", - "1h", - "2h", - "1d" - ], - "time_options": [ - "5m", - "15m", - "1h", - "6h", - "12h", - "24h", - "2d", - "7d", - "30d" - ] - }, - "timezone": "", - "title": "Kubernetes / StatefulSets", - "uid": "a31c1f46e6f727cb37c0d731a7245005", - "variables": { - "list": [] - }, - "version": 1 -} diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/nodes.json b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/nodes.json deleted file mode 100644 index 717bd39d..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/nodes.json +++ /dev/null @@ -1,933 +0,0 @@ -{ - "annotations": { - "list": [ - { - "builtIn": 1, - "datasource": "-- Grafana --", - "enable": true, - "hide": true, - "iconColor": "rgba(0, 211, 255, 1)", - "name": "Annotations & Alerts", - "type": "dashboard" - } - ] - }, - "editable": false, - "gnetId": null, - "graphTooltip": 0, - "id": 6, - "iteration": 1588347461367, - "links": [], - "panels": [ - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 0 - }, - "hiddenSeries": false, - "id": 2, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "(\n (1 - rate(node_cpu_seconds_total{job=\"node-exporter\", mode=\"idle\", instance=\"$instance\"}[$__interval]))\n/ ignoring(cpu) group_left\n count without (cpu)( node_cpu_seconds_total{job=\"node-exporter\", mode=\"idle\", instance=\"$instance\"})\n)\n", - "format": "time_series", - "interval": "1m", - "intervalFactor": 5, - "legendFormat": "{{cpu}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "CPU Usage", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": 1, - "min": 0, - "show": true - }, - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": 1, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 0, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 0 - }, - "hiddenSeries": false, - "id": 3, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "node_load1{job=\"node-exporter\", instance=\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "1m load average", - "refId": "A" - }, - { - "expr": "node_load5{job=\"node-exporter\", instance=\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "5m load average", - "refId": "B" - }, - { - "expr": "node_load15{job=\"node-exporter\", instance=\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "15m load average", - "refId": "C" - }, - { - "expr": "count(node_cpu_seconds_total{job=\"node-exporter\", instance=\"$instance\", mode=\"idle\"})", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "logical cores", - "refId": "D" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Load Average", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 18, - "x": 0, - "y": 7 - }, - "hiddenSeries": false, - "id": 4, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "(\n node_memory_MemTotal_bytes{job=\"node-exporter\", instance=\"$instance\"}\n-\n node_memory_MemFree_bytes{job=\"node-exporter\", instance=\"$instance\"}\n-\n node_memory_Buffers_bytes{job=\"node-exporter\", instance=\"$instance\"}\n-\n node_memory_Cached_bytes{job=\"node-exporter\", instance=\"$instance\"}\n)\n", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "memory used", - "refId": "A" - }, - { - "expr": "node_memory_Buffers_bytes{job=\"node-exporter\", instance=\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "memory buffers", - "refId": "B" - }, - { - "expr": "node_memory_Cached_bytes{job=\"node-exporter\", instance=\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "memory cached", - "refId": "C" - }, - { - "expr": "node_memory_MemFree_bytes{job=\"node-exporter\", instance=\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "memory free", - "refId": "D" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Memory Usage", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "bytes", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "bytes", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "rgba(50, 172, 45, 0.97)", - "rgba(237, 129, 40, 0.89)", - "rgba(245, 54, 54, 0.9)" - ], - "datasource": "$datasource", - "format": "percent", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": true, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 7, - "w": 6, - "x": 18, - "y": 7 - }, - "id": 5, - "interval": null, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false - }, - "tableColumn": "", - "targets": [ - { - "expr": "100 -\n(\n node_memory_MemAvailable_bytes{job=\"node-exporter\", instance=\"$instance\"}\n/\n node_memory_MemTotal_bytes{job=\"node-exporter\", instance=\"$instance\"}\n* 100\n)\n", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "", - "refId": "A" - } - ], - "thresholds": "80, 90", - "title": "Memory Usage", - "tooltip": { - "shared": false - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 0, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 14 - }, - "hiddenSeries": false, - "id": 6, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [ - { - "alias": "/ read| written/", - "yaxis": 1 - }, - { - "alias": "/ io time/", - "yaxis": 2 - } - ], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "rate(node_disk_read_bytes_total{job=\"node-exporter\", instance=\"$instance\", device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\"}[$__interval])", - "format": "time_series", - "interval": "1m", - "intervalFactor": 2, - "legendFormat": "{{device}} read", - "refId": "A" - }, - { - "expr": "rate(node_disk_written_bytes_total{job=\"node-exporter\", instance=\"$instance\", device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\"}[$__interval])", - "format": "time_series", - "interval": "1m", - "intervalFactor": 2, - "legendFormat": "{{device}} written", - "refId": "B" - }, - { - "expr": "rate(node_disk_io_time_seconds_total{job=\"node-exporter\", instance=\"$instance\", device=~\"nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+\"}[$__interval])", - "format": "time_series", - "interval": "1m", - "intervalFactor": 2, - "legendFormat": "{{device}} io time", - "refId": "C" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Disk I/O", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "bytes", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "s", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 14 - }, - "hiddenSeries": false, - "id": 7, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [ - { - "alias": "used", - "color": "#E0B400" - }, - { - "alias": "available", - "color": "#73BF69" - } - ], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sum(\n max by (device) (\n node_filesystem_size_bytes{job=\"node-exporter\", instance=\"$instance\", fstype!=\"\"}\n -\n node_filesystem_avail_bytes{job=\"node-exporter\", instance=\"$instance\", fstype!=\"\"}\n )\n)\n", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "used", - "refId": "A" - }, - { - "expr": "sum(\n max by (device) (\n node_filesystem_avail_bytes{job=\"node-exporter\", instance=\"$instance\", fstype!=\"\"}\n )\n)\n", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "available", - "refId": "B" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Disk Space Usage", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "bytes", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "bytes", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 0, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 21 - }, - "hiddenSeries": false, - "id": 8, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "rate(node_network_receive_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!=\"lo\"}[$__interval])", - "format": "time_series", - "interval": "1m", - "intervalFactor": 2, - "legendFormat": "{{device}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Network Received", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "bytes", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "bytes", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 0, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 21 - }, - "hiddenSeries": false, - "id": 9, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "repeat": null, - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "rate(node_network_transmit_bytes_total{job=\"node-exporter\", instance=\"$instance\", device!=\"lo\"}[$__interval])", - "format": "time_series", - "interval": "1m", - "intervalFactor": 2, - "legendFormat": "{{device}}", - "refId": "A" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Network Transmitted", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "bytes", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "bytes", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - } - ], - "refresh": "", - "schemaVersion": 22, - "style": "dark", - "tags": [], - "templating": { - "list": [ - { - "current": { - "selected": false, - "text": "Prometheus", - "value": "Prometheus" - }, - "hide": 0, - "includeAll": false, - "label": null, - "multi": false, - "name": "datasource", - "options": [], - "query": "prometheus", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "type": "datasource" - }, - { - "allValue": null, - "current": { - "text": "172.20.93.188:9100", - "value": "172.20.93.188:9100" - }, - "datasource": "$datasource", - "definition": "", - "hide": 0, - "includeAll": false, - "index": -1, - "label": null, - "multi": false, - "name": "instance", - "options": [], - "query": "label_values(node_exporter_build_info{job=\"node-exporter\"}, instance)", - "refresh": 2, - "regex": "", - "skipUrlSync": false, - "sort": 0, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - } - ] - }, - "time": { - "from": "now-1h", - "to": "now" - }, - "timepicker": { - "refresh_intervals": [ - "5s", - "10s", - "30s", - "1m", - "5m", - "15m", - "30m", - "1h", - "2h", - "1d" - ], - "time_options": [ - "5m", - "15m", - "1h", - "6h", - "12h", - "24h", - "2d", - "7d", - "30d" - ] - }, - "timezone": "", - "title": "Nodes", - "uid": "fa49a4706d07a042595b664c87fb33ea", - "variables": { - "list": [] - }, - "version": 1 -} diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/prometheus.json b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/prometheus.json deleted file mode 100644 index 2b6d27a5..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/prometheus.json +++ /dev/null @@ -1,1265 +0,0 @@ -{ - "annotations": { - "list": [ - { - "builtIn": 1, - "datasource": "-- Grafana --", - "enable": true, - "hide": true, - "iconColor": "rgba(0, 211, 255, 1)", - "name": "Annotations & Alerts", - "type": "dashboard" - } - ] - }, - "editable": true, - "gnetId": null, - "graphTooltip": 0, - "id": 18, - "iteration": 1588347462345, - "links": [], - "panels": [ - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 0 - }, - "id": 11, - "panels": [], - "repeat": null, - "title": "Prometheus Stats", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "columns": [], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fontSize": "100%", - "gridPos": { - "h": 7, - "w": 24, - "x": 0, - "y": 1 - }, - "id": 1, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null as zero", - "pageSize": null, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "showHeader": true, - "sort": { - "col": 0, - "desc": true - }, - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "styles": [ - { - "alias": "Time", - "align": "auto", - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "pattern": "Time", - "type": "hidden" - }, - { - "alias": "Count", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Value #A", - "thresholds": [], - "type": "hidden", - "unit": "short" - }, - { - "alias": "Uptime", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "Value #B", - "thresholds": [], - "type": "number", - "unit": "short" - }, - { - "alias": "Instance", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "instance", - "thresholds": [], - "type": "number", - "unit": "short" - }, - { - "alias": "Job", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "job", - "thresholds": [], - "type": "number", - "unit": "short" - }, - { - "alias": "Version", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "link": false, - "linkTooltip": "Drill down", - "linkUrl": "", - "pattern": "version", - "thresholds": [], - "type": "number", - "unit": "short" - }, - { - "alias": "", - "align": "auto", - "colorMode": null, - "colors": [], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "pattern": "/.*/", - "thresholds": [], - "type": "string", - "unit": "short" - } - ], - "targets": [ - { - "expr": "count by (job, instance, version) (prometheus_build_info{job=~\"$job\", instance=~\"$instance\"})", - "format": "table", - "instant": true, - "intervalFactor": 2, - "legendFormat": "", - "refId": "A", - "step": 10 - }, - { - "expr": "max by (job, instance) (time() - process_start_time_seconds{job=~\"$job\", instance=~\"$instance\"})", - "format": "table", - "instant": true, - "intervalFactor": 2, - "legendFormat": "", - "refId": "B", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeShift": null, - "title": "Prometheus Stats", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "individual" - }, - "transform": "table", - "type": "table", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 8 - }, - "id": 12, - "panels": [], - "repeat": null, - "title": "Discovery", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 9 - }, - "hiddenSeries": false, - "id": 2, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(rate(prometheus_target_sync_length_seconds_sum{job=~\"$job\",instance=~\"$instance\"}[5m])) by (scrape_job) * 1e3", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{scrape_job}}", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Target Sync", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "ms", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 10, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 9 - }, - "hiddenSeries": false, - "id": 3, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 0, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sum(prometheus_sd_discovered_targets{job=~\"$job\",instance=~\"$instance\"})", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "Targets", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Targets", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 16 - }, - "id": 13, - "panels": [], - "repeat": null, - "title": "Retrieval", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 8, - "x": 0, - "y": 17 - }, - "hiddenSeries": false, - "id": 4, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "rate(prometheus_target_interval_length_seconds_sum{job=~\"$job\",instance=~\"$instance\"}[5m]) / rate(prometheus_target_interval_length_seconds_count{job=~\"$job\",instance=~\"$instance\"}[5m]) * 1e3", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{interval}} configured", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Average Scrape Interval Duration", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "ms", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 10, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 8, - "x": 8, - "y": 17 - }, - "hiddenSeries": false, - "id": 5, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 0, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sum by (job) (rate(prometheus_target_scrapes_exceeded_sample_limit_total[1m]))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "exceeded sample limit: {{job}}", - "legendLink": null, - "refId": "A", - "step": 10 - }, - { - "expr": "sum by (job) (rate(prometheus_target_scrapes_sample_duplicate_timestamp_total[1m]))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "duplicate timestamp: {{job}}", - "legendLink": null, - "refId": "B", - "step": 10 - }, - { - "expr": "sum by (job) (rate(prometheus_target_scrapes_sample_out_of_bounds_total[1m]))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "out of bounds: {{job}}", - "legendLink": null, - "refId": "C", - "step": 10 - }, - { - "expr": "sum by (job) (rate(prometheus_target_scrapes_sample_out_of_order_total[1m]))", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "out of order: {{job}}", - "legendLink": null, - "refId": "D", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Scrape failures", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 10, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 8, - "x": 16, - "y": 17 - }, - "hiddenSeries": false, - "id": 6, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 0, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "rate(prometheus_tsdb_head_samples_appended_total{job=~\"$job\",instance=~\"$instance\"}[5m])", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{job}} {{instance}}", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Appended Samples", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 24 - }, - "id": 14, - "panels": [], - "repeat": null, - "title": "Storage", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 10, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 25 - }, - "hiddenSeries": false, - "id": 7, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 0, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "prometheus_tsdb_head_series{job=~\"$job\",instance=~\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{job}} {{instance}} head series", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Head Series", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 10, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 25 - }, - "hiddenSeries": false, - "id": 8, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 0, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "prometheus_tsdb_head_chunks{job=~\"$job\",instance=~\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{job}} {{instance}} head chunks", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Head Chunks", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 32 - }, - "id": 15, - "panels": [], - "repeat": null, - "title": "Query", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 10, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 33 - }, - "hiddenSeries": false, - "id": 9, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 0, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "rate(prometheus_engine_query_duration_seconds_count{job=~\"$job\",instance=~\"$instance\",slice=\"inner_eval\"}[5m])", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{job}} {{instance}}", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Query Rate", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 10, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 33 - }, - "hiddenSeries": false, - "id": 10, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 0, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "max by (slice) (prometheus_engine_query_duration_seconds{quantile=\"0.9\",job=~\"$job\",instance=~\"$instance\"}) * 1e3", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{slice}}", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Stage Duration", - "tooltip": { - "shared": true, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "ms", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - } - ], - "refresh": "10s", - "schemaVersion": 22, - "style": "dark", - "tags": [], - "templating": { - "list": [ - { - "current": { - "selected": false, - "text": "default", - "value": "default" - }, - "hide": 0, - "includeAll": false, - "label": null, - "multi": false, - "name": "datasource", - "options": [], - "query": "prometheus", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "type": "datasource" - }, - { - "allValue": null, - "current": { - "selected": false, - "text": "All", - "value": "$__all" - }, - "datasource": "$datasource", - "definition": "", - "hide": 0, - "includeAll": true, - "index": -1, - "label": "job", - "multi": true, - "name": "job", - "options": [], - "query": "label_values(prometheus_build_info, job)", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "sort": 2, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - }, - { - "allValue": null, - "current": { - "selected": false, - "text": "All", - "value": "$__all" - }, - "datasource": "$datasource", - "definition": "", - "hide": 0, - "includeAll": true, - "index": -1, - "label": "instance", - "multi": true, - "name": "instance", - "options": [], - "query": "label_values(prometheus_build_info, instance)", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "sort": 2, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - } - ] - }, - "time": { - "from": "now-1h", - "to": "now" - }, - "timepicker": { - "refresh_intervals": [ - "5s", - "10s", - "30s", - "1m", - "5m", - "15m", - "30m", - "1h", - "2h", - "1d" - ], - "time_options": [ - "5m", - "15m", - "1h", - "6h", - "12h", - "24h", - "2d", - "7d", - "30d" - ] - }, - "timezone": "utc", - "title": "Prometheus", - "uid": "kp5tnFeWz", - "variables": { - "list": [] - }, - "version": 1 -} diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/redis.json b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/redis.json deleted file mode 100644 index 94654e35..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/redis.json +++ /dev/null @@ -1,1168 +0,0 @@ -{ - "id": 23, - "uid": "2LtOhWXZk", - "slug": "redis", - "title": "Redis", - "originalTitle": "", - "tags": [ - "testground" - ], - "style": "dark", - "timezone": "browser", - "editable": true, - "hideControls": false, - "sharedCrosshair": false, - "panels": [ - { - "datasource": "Prometheus", - "editable": true, - "error": false, - "gridPos": { - "h": 7, - "w": 2, - "x": 0, - "y": 0 - }, - "id": 9, - "isNew": true, - "span": 0, - "title": "Uptime", - "transparent": false, - "type": "singlestat", - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "colorValue": false, - "colorBackground": false, - "decimals": 0, - "format": "s", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "lineColor": "rgb(31, 120, 193)", - "show": true - }, - "targets": [ - { - "refId": "A", - "expr": "max(max_over_time(redis_uptime_in_seconds{instance=~\"$instance\"}[$__interval]))", - "intervalFactor": 2, - "step": 1800, - "format": "time_series" - } - ], - "thresholds": "", - "valueFontSize": "70%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" - }, - { - "datasource": "Prometheus", - "editable": true, - "error": false, - "gridPos": { - "h": 7, - "w": 2, - "x": 2, - "y": 0 - }, - "hideTimeOverride": true, - "id": 12, - "isNew": true, - "span": 0, - "title": "Clients", - "transparent": false, - "type": "singlestat", - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "colorValue": false, - "colorBackground": false, - "decimals": 0, - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "lineColor": "rgb(31, 120, 193)", - "show": true - }, - "targets": [ - { - "refId": "A", - "expr": "redis_connected_clients{instance=~\"$instance\"}", - "intervalFactor": 2, - "step": 2, - "format": "time_series" - } - ], - "thresholds": "", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" - }, - { - "datasource": "Prometheus", - "editable": true, - "error": false, - "gridPos": { - "h": 7, - "w": 4, - "x": 4, - "y": 0 - }, - "hideTimeOverride": true, - "id": 11, - "isNew": true, - "span": 0, - "title": "Memory Usage", - "transparent": false, - "type": "singlestat", - "colors": [ - "rgba(50, 172, 45, 0.97)", - "rgba(237, 129, 40, 0.89)", - "rgba(245, 54, 54, 0.9)" - ], - "colorValue": false, - "colorBackground": false, - "decimals": 0, - "format": "percent", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": true, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "connected", - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "lineColor": "rgb(31, 120, 193)", - "show": true - }, - "targets": [ - { - "refId": "A", - "expr": "100 * (redis_memory_used_bytes{instance=~\"$instance\"} / redis_memory_max_bytes{instance=~\"$instance\"})", - "intervalFactor": 2, - "step": 2, - "format": "time_series" - } - ], - "thresholds": "80,95", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" - }, - { - "datasource": "Prometheus", - "editable": true, - "error": false, - "gridPos": { - "h": 7, - "w": 8, - "x": 8, - "y": 0 - }, - "id": 18, - "isNew": true, - "renderer": "flot", - "span": 0, - "title": "Total Commands / sec", - "transparent": false, - "type": "graph", - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "fill": 8, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": false, - "hideZero": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "nullPointMode": "connected", - "percentage": false, - "pointradius": 5, - "points": false, - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "refId": "A", - "expr": "sum(topk(10, irate(redis_commands_total{instance=~\"$instance\"} [1m]))) by (cmd)", - "intervalFactor": 2, - "step": 240, - "legendFormat": "{{ cmd }}", - "format": "time_series" - } - ], - "tooltip": { - "shared": true, - "value_type": "individual", - "msResolution": true, - "sort": 2 - }, - "xaxis": { - "format": "", - "logBase": 0, - "show": true - }, - "yaxes": [ - { - "format": "short", - "logBase": 1, - "show": true - }, - { - "format": "short", - "logBase": 1, - "show": true - } - ] - }, - { - "datasource": "Prometheus", - "editable": true, - "error": false, - "gridPos": { - "h": 7, - "w": 8, - "x": 16, - "y": 0 - }, - "id": 1, - "isNew": true, - "renderer": "flot", - "span": 0, - "title": "Hits / Misses per Sec", - "transparent": false, - "type": "graph", - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "decimals": 2, - "fill": 1, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": false, - "hideZero": false, - "max": false, - "min": false, - "rightSide": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "nullPointMode": "connected", - "percentage": true, - "pointradius": 5, - "points": false, - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "refId": "A", - "expr": "irate(redis_keyspace_hits_total{instance=~\"$instance\"}[5m])", - "intervalFactor": 2, - "step": 240, - "legendFormat": "hits", - "format": "time_series" - }, - { - "refId": "B", - "expr": "irate(redis_keyspace_misses_total{instance=~\"$instance\"}[5m])", - "intervalFactor": 2, - "step": 240, - "legendFormat": "misses", - "format": "time_series" - } - ], - "tooltip": { - "shared": true, - "value_type": "individual" - }, - "xaxis": { - "format": "", - "logBase": 0, - "show": true - }, - "yaxes": [ - { - "format": "short", - "logBase": 1, - "min": 0, - "show": true - }, - { - "format": "short", - "logBase": 1, - "show": true - } - ] - }, - { - "datasource": "Prometheus", - "editable": true, - "error": false, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 7 - }, - "id": 7, - "isNew": true, - "renderer": "flot", - "span": 0, - "title": "Total Memory Usage", - "transparent": false, - "type": "graph", - "aliasColors": { - "max": "#BF1B00" - }, - "bars": false, - "dashLength": 10, - "dashes": false, - "fill": 1, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": false, - "hideZero": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "nullPointMode": "null as zero", - "percentage": false, - "pointradius": 5, - "points": false, - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "refId": "A", - "expr": "redis_memory_used_bytes{instance=~\"$instance\"} ", - "intervalFactor": 2, - "step": 240, - "legendFormat": "used", - "format": "time_series" - }, - { - "refId": "B", - "expr": "redis_memory_max_bytes{instance=~\"$instance\"} ", - "intervalFactor": 2, - "step": 240, - "legendFormat": "max", - "format": "time_series" - } - ], - "tooltip": { - "shared": true, - "value_type": "cumulative" - }, - "xaxis": { - "format": "", - "logBase": 0, - "show": true - }, - "yaxes": [ - { - "format": "bytes", - "logBase": 1, - "min": 0, - "show": true - }, - { - "format": "short", - "logBase": 1, - "show": true - } - ] - }, - { - "datasource": "Prometheus", - "editable": true, - "error": false, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 7 - }, - "id": 10, - "isNew": true, - "renderer": "flot", - "span": 0, - "title": "Network I/O", - "transparent": false, - "type": "graph", - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "fill": 1, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": false, - "hideZero": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "nullPointMode": "connected", - "percentage": false, - "pointradius": 5, - "points": false, - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "refId": "A", - "expr": "rate(redis_net_input_bytes_total{instance=~\"$instance\"}[5m])", - "intervalFactor": 2, - "step": 240, - "legendFormat": "{{ input }}", - "format": "time_series" - }, - { - "refId": "B", - "expr": "rate(redis_net_output_bytes_total{instance=~\"$instance\"}[5m])", - "intervalFactor": 2, - "step": 240, - "legendFormat": "{{ output }}", - "format": "time_series" - } - ], - "tooltip": { - "shared": true, - "value_type": "cumulative", - "msResolution": true - }, - "xaxis": { - "format": "", - "logBase": 0, - "show": true - }, - "yaxes": [ - { - "format": "bytes", - "logBase": 1, - "show": true - }, - { - "format": "short", - "logBase": 1, - "show": true - } - ] - }, - { - "datasource": "Prometheus", - "editable": true, - "error": false, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 14 - }, - "id": 5, - "isNew": true, - "renderer": "flot", - "span": 0, - "title": "Total Items per DB", - "transparent": false, - "type": "graph", - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "fill": 7, - "legend": { - "alignAsTable": true, - "avg": false, - "current": true, - "hideEmpty": false, - "hideZero": false, - "max": false, - "min": false, - "rightSide": true, - "show": true, - "total": false, - "values": true - }, - "lines": true, - "linewidth": 2, - "nullPointMode": "connected", - "percentage": false, - "pointradius": 5, - "points": false, - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "refId": "A", - "expr": "sum (redis_db_keys{instance=~\"$instance\"}) by (db)", - "intervalFactor": 2, - "step": 240, - "legendFormat": "{{ db }} ", - "format": "time_series" - } - ], - "tooltip": { - "shared": true, - "value_type": "individual" - }, - "xaxis": { - "format": "", - "logBase": 0, - "show": true - }, - "yaxes": [ - { - "format": "none", - "logBase": 1, - "show": true - }, - { - "format": "short", - "logBase": 1, - "show": true - } - ] - }, - { - "datasource": "Prometheus", - "editable": true, - "error": false, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 14 - }, - "id": 13, - "isNew": true, - "renderer": "flot", - "span": 0, - "title": "Expiring vs Not-Expiring Keys", - "transparent": false, - "type": "graph", - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "fill": 7, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": false, - "hideZero": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "nullPointMode": "connected", - "percentage": false, - "pointradius": 5, - "points": false, - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "refId": "A", - "expr": "sum (redis_db_keys{instance=~\"$instance\"}) - sum (redis_db_keys_expiring{instance=~\"$instance\"}) ", - "intervalFactor": 2, - "step": 240, - "legendFormat": "not expiring", - "format": "time_series" - }, - { - "refId": "B", - "expr": "sum (redis_db_keys_expiring{instance=~\"$instance\"})", - "intervalFactor": 2, - "step": 240, - "legendFormat": "expiring", - "format": "time_series" - } - ], - "tooltip": { - "shared": true, - "value_type": "individual" - }, - "xaxis": { - "format": "", - "logBase": 0, - "show": true - }, - "yaxes": [ - { - "format": "short", - "logBase": 1, - "show": true - }, - { - "format": "short", - "logBase": 1, - "show": true - } - ] - }, - { - "datasource": "Prometheus", - "editable": true, - "error": false, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 21 - }, - "id": 8, - "isNew": true, - "renderer": "flot", - "span": 0, - "title": "Expired/Evicted Keys", - "transparent": false, - "type": "graph", - "aliasColors": { - "evicts": "#890F02", - "memcached_items_evicted_total{instance=\"172.17.0.1:9150\",job=\"prometheus\"}": "#890F02", - "reclaims": "#3F6833" - }, - "bars": false, - "dashLength": 10, - "dashes": false, - "fill": 1, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": false, - "hideZero": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 2, - "nullPointMode": "connected", - "percentage": false, - "pointradius": 5, - "points": false, - "seriesOverrides": [ - { - "alias": "reclaims", - "yaxis": 2 - } - ], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "refId": "A", - "expr": "sum(rate(redis_expired_keys_total{instance=~\"$instance\"}[5m])) by (instance)", - "intervalFactor": 2, - "step": 240, - "legendFormat": "expired", - "format": "time_series" - }, - { - "refId": "B", - "expr": "sum(rate(redis_evicted_keys_total{instance=~\"$instance\"}[5m])) by (instance)", - "intervalFactor": 2, - "step": 240, - "legendFormat": "evicted", - "format": "time_series" - } - ], - "tooltip": { - "shared": true, - "value_type": "cumulative" - }, - "xaxis": { - "format": "", - "logBase": 0, - "show": true - }, - "yaxes": [ - { - "format": "short", - "logBase": 1, - "show": true - }, - { - "format": "short", - "logBase": 1, - "show": true - } - ] - }, - { - "datasource": "Prometheus", - "editable": true, - "error": false, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 21 - }, - "id": 16, - "isNew": false, - "renderer": "flot", - "span": 0, - "title": "Connected/Blocked Clients", - "transparent": false, - "type": "graph", - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "fill": 1, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": false, - "hideZero": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "nullPointMode": "null", - "percentage": false, - "pointradius": 2, - "points": false, - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "refId": "A", - "expr": "redis_connected_clients{instance=\"$instance\"}", - "intervalFactor": 1, - "legendFormat": "connected", - "format": "time_series" - }, - { - "refId": "B", - "expr": "redis_blocked_clients{instance=\"$instance\"}", - "intervalFactor": 1, - "legendFormat": "blocked", - "format": "time_series" - } - ], - "tooltip": { - "shared": true, - "value_type": "individual" - }, - "xaxis": { - "format": "", - "logBase": 0, - "show": true - }, - "yaxes": [ - { - "format": "short", - "logBase": 1, - "show": true - }, - { - "format": "short", - "logBase": 1, - "show": true - } - ] - }, - { - "datasource": "Prometheus", - "editable": true, - "error": false, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 28 - }, - "id": 20, - "isNew": true, - "renderer": "flot", - "span": 0, - "title": "Average Time Spent by Command / sec", - "transparent": false, - "type": "graph", - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "fill": 1, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": false, - "hideZero": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "nullPointMode": "connected", - "percentage": false, - "pointradius": 5, - "points": false, - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "refId": "A", - "expr": "topk(10,\n sum(irate(redis_commands_duration_seconds_total{instance = \"$instance\"}[1m])) by (cmd)\n /\n sum(irate(redis_commands_total{instance = \"$instance\"}[1m])) by (cmd)\n)", - "intervalFactor": 2, - "step": 240, - "legendFormat": "{{ cmd }}", - "format": "time_series" - } - ], - "tooltip": { - "shared": true, - "value_type": "individual", - "msResolution": true, - "sort": 2 - }, - "xaxis": { - "format": "", - "logBase": 0, - "show": true - }, - "yaxes": [ - { - "format": "s", - "logBase": 1, - "show": true - }, - { - "format": "short", - "logBase": 1, - "show": true - } - ] - }, - { - "datasource": "Prometheus", - "editable": true, - "error": false, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 28 - }, - "id": 14, - "isNew": true, - "renderer": "flot", - "span": 0, - "title": "Total Time Spent by Command / sec", - "transparent": false, - "type": "graph", - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "fill": 8, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "hideEmpty": false, - "hideZero": false, - "max": false, - "min": false, - "rightSide": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "nullPointMode": "connected", - "percentage": false, - "pointradius": 5, - "points": false, - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "refId": "A", - "expr": "topk(10, sum(irate(redis_commands_duration_seconds_total{instance=~\"$instance\"}[1m])) by (cmd))", - "intervalFactor": 2, - "step": 240, - "legendFormat": "{{ cmd }}", - "format": "time_series" - } - ], - "tooltip": { - "shared": true, - "value_type": "individual", - "msResolution": true, - "sort": 2 - }, - "xaxis": { - "format": "", - "logBase": 0, - "show": true - }, - "yaxes": [ - { - "format": "s", - "logBase": 1, - "show": true - }, - { - "format": "short", - "logBase": 1, - "show": true - } - ] - } - ], - "rows": null, - "templating": { - "list": [ - { - "name": "instance", - "type": "query", - "datasource": "Prometheus", - "refresh": 2, - "options": [], - "includeAll": false, - "allFormat": "", - "allValue": "", - "multi": false, - "multiFormat": "", - "query": "label_values(redis_up, instance)", - "regex": "", - "current": { - "text": "", - "value": null - }, - "label": "", - "hide": 0, - "sort": 1 - } - ] - }, - "annotations": { - "list": [ - { - "name": "Annotations \u0026 Alerts", - "datasource": "-- Grafana --", - "showLine": false, - "iconColor": "rgba(0, 211, 255, 1)", - "lineColor": "", - "iconSize": 0, - "enable": true, - "query": "", - "textField": "", - "tagsField": "", - "tags": null, - "type": "dashboard" - } - ] - }, - "refresh": "10s", - "schemaVersion": 22, - "version": 1, - "links": [], - "time": { - "from": "now-24h", - "to": "now" - }, - "timepicker": { - "refresh_intervals": [ - "5s", - "10s", - "30s", - "1m", - "5m", - "15m", - "30m", - "1h", - "2h", - "1d" - ], - "time_options": [ - "5m", - "15m", - "1h", - "6h", - "12h", - "24h", - "2d", - "7d", - "30d" - ] - }, - "graphTooltip": 1 -} diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/use-method-cluster.json b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/use-method-cluster.json deleted file mode 100644 index b391533e..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/use-method-cluster.json +++ /dev/null @@ -1,1026 +0,0 @@ -{ - "annotations": { - "list": [ - { - "$$hashKey": "object:10", - "builtIn": 1, - "datasource": "-- Grafana --", - "enable": true, - "hide": true, - "iconColor": "rgba(0, 211, 255, 1)", - "name": "Annotations & Alerts", - "type": "dashboard" - } - ] - }, - "editable": true, - "gnetId": null, - "graphTooltip": 0, - "iteration": 1588346975414, - "links": [], - "panels": [ - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 0 - }, - "id": 10, - "panels": [], - "repeat": null, - "title": "CPU", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 10, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 1 - }, - "hiddenSeries": false, - "id": 1, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 0, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "(\n instance:node_cpu_utilisation:rate1m{job=\"node-exporter\"}\n*\n instance:node_num_cpu:sum{job=\"node-exporter\"}\n)\n/ scalar(sum(instance:node_num_cpu:sum{job=\"node-exporter\"}))\n", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "legendLink": "/dashboard/file/node-rsrc-use.json", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "CPU Utilisation", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": 1, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 10, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 1 - }, - "hiddenSeries": false, - "id": 2, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 0, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "instance:node_load1_per_cpu:ratio{job=\"node-exporter\"}\n/ scalar(count(instance:node_load1_per_cpu:ratio{job=\"node-exporter\"}))\n", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "legendLink": "/dashboard/file/node-rsrc-use.json", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "CPU Saturation (load1 per CPU)", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": 1, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 8 - }, - "id": 11, - "panels": [], - "repeat": null, - "title": "Memory", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 10, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 9 - }, - "hiddenSeries": false, - "id": 3, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 0, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "instance:node_memory_utilisation:ratio{job=\"node-exporter\"}\n/ scalar(count(instance:node_memory_utilisation:ratio{job=\"node-exporter\"}))\n", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "legendLink": "/dashboard/file/node-rsrc-use.json", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Memory Utilisation", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": 1, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 10, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 9 - }, - "hiddenSeries": false, - "id": 4, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 0, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "instance:node_vmstat_pgmajfault:rate1m{job=\"node-exporter\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "legendLink": "/dashboard/file/node-rsrc-use.json", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Memory Saturation (Major Page Faults)", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "rps", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 16 - }, - "id": 12, - "panels": [], - "repeat": null, - "title": "Network", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 10, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 17 - }, - "hiddenSeries": false, - "id": 5, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 0, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [ - { - "alias": "/ Receive/", - "stack": "A" - }, - { - "alias": "/ Transmit/", - "stack": "B", - "transform": "negative-Y" - } - ], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "instance:node_network_receive_bytes_excluding_lo:rate1m{job=\"node-exporter\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} Receive", - "legendLink": "/dashboard/file/node-rsrc-use.json", - "refId": "A", - "step": 10 - }, - { - "expr": "instance:node_network_transmit_bytes_excluding_lo:rate1m{job=\"node-exporter\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} Transmit", - "legendLink": "/dashboard/file/node-rsrc-use.json", - "refId": "B", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Net Utilisation (Bytes Receive/Transmit)", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 10, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 17 - }, - "hiddenSeries": false, - "id": 6, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 0, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [ - { - "alias": "/ Receive/", - "stack": "A" - }, - { - "alias": "/ Transmit/", - "stack": "B", - "transform": "negative-Y" - } - ], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "instance:node_network_receive_drop_excluding_lo:rate1m{job=\"node-exporter\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} Receive", - "legendLink": "/dashboard/file/node-rsrc-use.json", - "refId": "A", - "step": 10 - }, - { - "expr": "instance:node_network_transmit_drop_excluding_lo:rate1m{job=\"node-exporter\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} Transmit", - "legendLink": "/dashboard/file/node-rsrc-use.json", - "refId": "B", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Net Saturation (Drops Receive/Transmit)", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "rps", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 24 - }, - "id": 13, - "panels": [], - "repeat": null, - "title": "Disk IO", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 10, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 25 - }, - "hiddenSeries": false, - "id": 7, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 0, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "instance_device:node_disk_io_time_seconds:rate1m{job=\"node-exporter\"}\n/ scalar(count(instance_device:node_disk_io_time_seconds:rate1m{job=\"node-exporter\"}))\n", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} {{device}}", - "legendLink": "/dashboard/file/node-rsrc-use.json", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Disk IO Utilisation", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": 1, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 10, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 25 - }, - "hiddenSeries": false, - "id": 8, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 0, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "instance_device:node_disk_io_time_weighted_seconds:rate1m{job=\"node-exporter\"}\n/ scalar(count(instance_device:node_disk_io_time_weighted_seconds:rate1m{job=\"node-exporter\"}))\n", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}} {{device}}", - "legendLink": "/dashboard/file/node-rsrc-use.json", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Disk IO Saturation", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": 1, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 32 - }, - "id": 14, - "panels": [], - "repeat": null, - "title": "Disk Space", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 10, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 24, - "x": 0, - "y": 33 - }, - "hiddenSeries": false, - "id": 9, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 0, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "sum without (device) (\n max without (fstype, mountpoint) (\n node_filesystem_size_bytes{job=\"node-exporter\", fstype!=\"\"} - node_filesystem_avail_bytes{job=\"node-exporter\", fstype!=\"\"}\n )\n) \n/ scalar(sum(max without (fstype, mountpoint) (node_filesystem_size_bytes{job=\"node-exporter\", fstype!=\"\"})))\n", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{instance}}", - "legendLink": "/dashboard/file/node-rsrc-use.json", - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Disk Space Utilisation", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": 1, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - } - ], - "refresh": "10s", - "schemaVersion": 22, - "style": "dark", - "tags": [], - "templating": { - "list": [ - { - "current": { - "selected": false, - "text": "default", - "value": "default" - }, - "hide": 0, - "includeAll": false, - "label": null, - "multi": false, - "name": "datasource", - "options": [], - "query": "prometheus", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "type": "datasource" - } - ] - }, - "time": { - "from": "now-15m", - "to": "now" - }, - "timepicker": { - "refresh_intervals": [ - "5s", - "10s", - "30s", - "1m", - "5m", - "15m", - "30m", - "1h", - "2h", - "1d" - ], - "time_options": [ - "5m", - "15m", - "1h", - "6h", - "12h", - "24h", - "2d", - "7d", - "30d" - ] - }, - "timezone": "", - "title": "USE Method / Cluster", - "uid": "3e97d1d02672cdd0861f4c97c64f89b2", - "variables": { - "list": [] - }, - "version": 1 -} diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/use-method-node.json b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/use-method-node.json deleted file mode 100644 index c971b251..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/use-method-node.json +++ /dev/null @@ -1,1052 +0,0 @@ -{ - "annotations": { - "list": [ - { - "$$hashKey": "object:13", - "builtIn": 1, - "datasource": "-- Grafana --", - "enable": true, - "hide": true, - "iconColor": "rgba(0, 211, 255, 1)", - "name": "Annotations & Alerts", - "type": "dashboard" - } - ] - }, - "editable": true, - "gnetId": null, - "graphTooltip": 0, - "iteration": 1588346976497, - "links": [], - "panels": [ - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 0 - }, - "id": 10, - "panels": [], - "repeat": null, - "title": "CPU", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 1 - }, - "hiddenSeries": false, - "id": 1, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "instance:node_cpu_utilisation:rate1m{job=\"node-exporter\", instance=\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "Utilisation", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "CPU Utilisation", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 1 - }, - "hiddenSeries": false, - "id": 2, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "instance:node_load1_per_cpu:ratio{job=\"node-exporter\", instance=\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "Saturation", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "CPU Saturation (Load1 per CPU)", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 8 - }, - "id": 11, - "panels": [], - "repeat": null, - "title": "Memory", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 9 - }, - "hiddenSeries": false, - "id": 3, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "instance:node_memory_utilisation:ratio{job=\"node-exporter\", job=\"node-exporter\", instance=\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "Memory", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Memory Utilisation", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 9 - }, - "hiddenSeries": false, - "id": 4, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "instance:node_vmstat_pgmajfault:rate1m{job=\"node-exporter\", instance=\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "Major page faults", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Memory Saturation (Major Page Faults)", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 16 - }, - "id": 12, - "panels": [], - "repeat": null, - "title": "Net", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 17 - }, - "hiddenSeries": false, - "id": 5, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [ - { - "alias": "/Receive/", - "stack": "A" - }, - { - "alias": "/Transmit/", - "stack": "B", - "transform": "negative-Y" - } - ], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "instance:node_network_receive_bytes_excluding_lo:rate1m{job=\"node-exporter\", instance=\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "Receive", - "legendLink": null, - "refId": "A", - "step": 10 - }, - { - "expr": "instance:node_network_transmit_bytes_excluding_lo:rate1m{job=\"node-exporter\", instance=\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "Transmit", - "legendLink": null, - "refId": "B", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Net Utilisation (Bytes Receive/Transmit)", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "Bps", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 17 - }, - "hiddenSeries": false, - "id": 6, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [ - { - "alias": "/Receive/", - "stack": "A" - }, - { - "alias": "/Transmit/", - "stack": "B", - "transform": "negative-Y" - } - ], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "instance:node_network_receive_drop_excluding_lo:rate1m{job=\"node-exporter\", instance=\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "Receive drops", - "legendLink": null, - "refId": "A", - "step": 10 - }, - { - "expr": "instance:node_network_transmit_drop_excluding_lo:rate1m{job=\"node-exporter\", instance=\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "Transmit drops", - "legendLink": null, - "refId": "B", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Net Saturation (Drops Receive/Transmit)", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "rps", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 24 - }, - "id": 13, - "panels": [], - "repeat": null, - "title": "Disk IO", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 0, - "y": 25 - }, - "hiddenSeries": false, - "id": 7, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "instance_device:node_disk_io_time_seconds:rate1m{job=\"node-exporter\", instance=\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{device}}", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Disk IO Utilisation", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 12, - "x": 12, - "y": 25 - }, - "hiddenSeries": false, - "id": 8, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "instance_device:node_disk_io_time_weighted_seconds:rate1m{job=\"node-exporter\", instance=\"$instance\"}", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{device}}", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Disk IO Saturation", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 32 - }, - "id": 14, - "panels": [], - "repeat": null, - "title": "Disk Space", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 7, - "w": 24, - "x": 0, - "y": 33 - }, - "hiddenSeries": false, - "id": 9, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "1 -\n(\n max without (mountpoint, fstype) (node_filesystem_avail_bytes{job=\"node-exporter\", fstype!=\"\", instance=\"$instance\"})\n/\n max without (mountpoint, fstype) (node_filesystem_size_bytes{job=\"node-exporter\", fstype!=\"\", instance=\"$instance\"})\n)\n", - "format": "time_series", - "intervalFactor": 2, - "legendFormat": "{{device}}", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Disk Space Utilisation", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - } - ], - "refresh": "10s", - "schemaVersion": 22, - "style": "dark", - "tags": [], - "templating": { - "list": [ - { - "current": { - "selected": false, - "text": "default", - "value": "default" - }, - "hide": 0, - "includeAll": false, - "label": null, - "multi": false, - "name": "datasource", - "options": [], - "query": "prometheus", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "type": "datasource" - }, - { - "allValue": null, - "current": { - "text": "172.20.94.202:9100", - "value": "172.20.94.202:9100" - }, - "datasource": "$datasource", - "definition": "", - "hide": 0, - "includeAll": false, - "index": -1, - "label": "instance", - "multi": false, - "name": "instance", - "options": [], - "query": "label_values(up{job=\"node-exporter\"}, instance)", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "sort": 2, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - } - ] - }, - "time": { - "from": "now-15m", - "to": "now" - }, - "timepicker": { - "refresh_intervals": [ - "5s", - "10s", - "30s", - "1m", - "5m", - "15m", - "30m", - "1h", - "2h", - "1d" - ], - "time_options": [ - "5m", - "15m", - "1h", - "6h", - "12h", - "24h", - "2d", - "7d", - "30d" - ] - }, - "timezone": "", - "title": "USE Method / Node", - "uid": "fac67cfbe174d3ef53eb473d73d9212f", - "variables": { - "list": [] - }, - "version": 1 -} diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/weavenet-cluster.json b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/weavenet-cluster.json deleted file mode 100644 index c35b0638..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/weavenet-cluster.json +++ /dev/null @@ -1,3347 +0,0 @@ -{ - "__inputs": [], - "__requires": [ - { - "type": "grafana", - "id": "grafana", - "name": "Grafana", - "version": "6.6.0" - }, - { - "type": "panel", - "id": "graph", - "name": "Graph", - "version": "" - }, - { - "type": "datasource", - "id": "prometheus", - "name": "Prometheus", - "version": "1.0.0" - }, - { - "type": "panel", - "id": "singlestat", - "name": "Singlestat", - "version": "" - }, - { - "type": "panel", - "id": "table", - "name": "Table", - "version": "" - } - ], - "annotations": { - "list": [ - { - "builtIn": 1, - "datasource": "-- Grafana --", - "enable": true, - "hide": true, - "iconColor": "rgba(0, 211, 255, 1)", - "name": "Annotations & Alerts", - "type": "dashboard" - } - ] - }, - "description": "WeaveNet metrics at the cluster level. It was made on top of the weavenet prometheus metrics. Please check this for more details https://www.weave.works/docs/net/latest/tasks/manage/metrics", - "editable": true, - "gnetId": 11804, - "graphTooltip": 0, - "id": null, - "iteration": 1582826890106, - "links": [], - "panels": [ - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 0 - }, - "id": 20, - "panels": [], - "repeat": null, - "title": "Headlines", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorPostfix": false, - "colorPrefix": false, - "colorValue": false, - "colors": [ - "#d44a3a", - "rgba(237, 129, 40, 0.89)", - "#299c46" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 0, - "y": 1 - }, - "id": 56, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sum(weave_flows)", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1", - "timeFrom": null, - "timeShift": null, - "title": "Fast DP flows", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": true, - "colors": [ - "#299c46", - "#FADE2A", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 4, - "y": 1 - }, - "id": 5, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sum(weave_flows == bool 0)", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1,2", - "timeFrom": null, - "timeShift": null, - "title": "Fast DP off", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": true, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 8, - "y": 1 - }, - "id": 3, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sum(weave_ipam_pending_allocates)", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1,10", - "timeFrom": null, - "timeShift": null, - "title": "IPAM Pending Allocates", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": true, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 12, - "y": 1 - }, - "id": 4, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sum(weave_ipam_pending_claims)", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1,10", - "timeFrom": null, - "timeShift": null, - "title": "IPAM Pending Claims", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": true, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "format": "cpm", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 16, - "y": 1 - }, - "id": 61, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sum(increase(weave_connection_terminations_total[1m]))", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1", - "timeFrom": null, - "timeShift": null, - "title": "Terminations", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 20, - "y": 1 - }, - "id": 1, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "count(central_nodeagent:node_route_unhealthy_count)", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "70,80", - "timeFrom": null, - "timeShift": null, - "title": "Weave Pods", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorPostfix": false, - "colorValue": false, - "colors": [ - "#d44a3a", - "rgba(237, 129, 40, 0.89)", - "#5794F2" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 0, - "y": 4 - }, - "id": 57, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sum(weave_connections{state=\"established\"})", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1,", - "timeFrom": null, - "timeShift": null, - "title": "P2P Established", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": true, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 4, - "y": 4 - }, - "id": 62, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sum(weave_connections{state=\"connecting\"})", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1,10", - "timeFrom": null, - "timeShift": null, - "title": "P2P Connecting", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": true, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 8, - "y": 4 - }, - "id": 59, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sum(weave_connections{state=\"pending\"})", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1,10", - "timeFrom": null, - "timeShift": null, - "title": "P2P Pending", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": true, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 12, - "y": 4 - }, - "id": 58, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sum(weave_connections{state=\"failed\"})", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1,10", - "timeFrom": null, - "timeShift": null, - "title": "P2P Failed", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorPrefix": false, - "colorValue": true, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 16, - "y": 4 - }, - "id": 60, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sum(weave_connections{state=\"retrying\"})", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1,10", - "timeFrom": null, - "timeShift": null, - "title": "P2P Retrying", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 7 - }, - "id": 21, - "panels": [], - "repeat": null, - "title": "IPAM", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "Percentage of all IP addresses owned by unreachable peers.", - "fill": 0, - "fillGradient": 9, - "gridPos": { - "h": 5, - "w": 8, - "x": 0, - "y": 8 - }, - "hiddenSeries": false, - "id": 7, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "label_replace(weave_ipam_unreachable_percentage, \"node\", \"$1\", \"instance\", \"(.*):.*\")", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "{{node}}", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "IPAM Unreachable Percentage", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percent", - "label": "", - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "Number of unreachable peers that own IPAM addresses.", - "fill": 0, - "fillGradient": 0, - "gridPos": { - "h": 5, - "w": 8, - "x": 8, - "y": 8 - }, - "hiddenSeries": false, - "id": 35, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "label_replace(weave_ipam_unreachable_count, \"node\", \"$1\", \"instance\", \"(.*):.*\")", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "{{node}}", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "IPAM Unreachable Count", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": " Number of pending claims.", - "fill": 1, - "fillGradient": 0, - "gridPos": { - "h": 5, - "w": 8, - "x": 16, - "y": 8 - }, - "hiddenSeries": false, - "id": 36, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(weave_ipam_pending_allocates)", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "pending allocates", - "legendLink": null, - "refId": "A", - "step": 10 - }, - { - "expr": "sum(weave_ipam_pending_claims)", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "pending claims", - "legendLink": null, - "refId": "B", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "IPAM Pending", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "decimals": null, - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 13 - }, - "id": 41, - "panels": [], - "title": "Connections", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "fillGradient": 4, - "gridPos": { - "h": 7, - "w": 13, - "x": 0, - "y": 14 - }, - "hiddenSeries": false, - "id": 50, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "label_replace(weave_flows, \"node\", \"$1\", \"instance\", \"(.*):.*\")", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "{{node}}", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Fast DP Flows", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "fillGradient": 4, - "gridPos": { - "h": 7, - "w": 11, - "x": 13, - "y": 14 - }, - "hiddenSeries": false, - "id": 51, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null as zero", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "label_replace(rate(weave_connection_terminations_total[5m]), \"node\", \"$1\", \"instance\", \"(.*):.*\") > 0", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "{{node}}", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Rate of Terminated Connections", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "fillGradient": 4, - "gridPos": { - "h": 7, - "w": 8, - "x": 0, - "y": 21 - }, - "hiddenSeries": false, - "id": 55, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(weave_connections{state=\"established\"})", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "connections established", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "P2P / Established", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "fillGradient": 4, - "gridPos": { - "h": 7, - "w": 8, - "x": 8, - "y": 21 - }, - "hiddenSeries": false, - "id": 52, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(weave_connections{state=\"connecting\"})", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "connections connecting", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "P2P / Connecting", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "fillGradient": 4, - "gridPos": { - "h": 7, - "w": 8, - "x": 16, - "y": 21 - }, - "hiddenSeries": false, - "id": 54, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(weave_connections{state=\"pending\"})", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "pending connections", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "P2P / Pending", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "fillGradient": 4, - "gridPos": { - "h": 7, - "w": 8, - "x": 0, - "y": 28 - }, - "hiddenSeries": false, - "id": 67, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(weave_connections{state=\"retrying\"})", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "retrying connections", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "P2P / Retrying", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "fillGradient": 4, - "gridPos": { - "h": 7, - "w": 16, - "x": 8, - "y": 28 - }, - "hiddenSeries": false, - "id": 53, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sum(weave_connections{state=\"failed\"})", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "failed connections", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "P2P / Failed", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "columns": [], - "datasource": "$datasource", - "description": "", - "fontSize": "100%", - "gridPos": { - "h": 10, - "w": 6, - "x": 0, - "y": 35 - }, - "id": 46, - "interval": "", - "options": {}, - "pageSize": null, - "pluginVersion": "6.6.0", - "showHeader": true, - "sort": { - "col": 0, - "desc": true - }, - "styles": [ - { - "alias": "Time", - "align": "auto", - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "pattern": "Time", - "type": "hidden" - }, - { - "alias": "Node", - "align": "left", - "colorMode": null, - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "mappingType": 1, - "pattern": "Metric", - "thresholds": [], - "type": "number", - "unit": "short" - }, - { - "alias": "Established", - "align": "left", - "colorMode": null, - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "decimals": 0, - "pattern": "Value", - "thresholds": [], - "type": "number", - "unit": "short" - } - ], - "targets": [ - { - "expr": "sort(label_replace(weave_connections{state=\"established\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "instant": true, - "legendFormat": "{{node_ip}}", - "refId": "A" - } - ], - "timeFrom": null, - "timeShift": null, - "title": "P2P / Established", - "transform": "timeseries_to_rows", - "type": "table" - }, - { - "columns": [], - "datasource": "$datasource", - "description": "", - "fontSize": "100%", - "gridPos": { - "h": 10, - "w": 6, - "x": 6, - "y": 35 - }, - "id": 63, - "interval": "", - "options": {}, - "pageSize": null, - "pluginVersion": "6.6.0", - "showHeader": true, - "sort": { - "col": 0, - "desc": true - }, - "styles": [ - { - "alias": "Time", - "align": "auto", - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "pattern": "Time", - "type": "hidden" - }, - { - "alias": "Node", - "align": "left", - "colorMode": null, - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "mappingType": 1, - "pattern": "Metric", - "thresholds": [], - "type": "number", - "unit": "short" - }, - { - "alias": "Connecting", - "align": "left", - "colorMode": null, - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "decimals": 0, - "pattern": "Value", - "thresholds": [], - "type": "number", - "unit": "short" - } - ], - "targets": [ - { - "expr": "sort(label_replace(weave_connections{state=\"connecting\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "instant": true, - "legendFormat": "{{node_ip}}", - "refId": "A" - } - ], - "timeFrom": null, - "timeShift": null, - "title": "P2P / Connecting", - "transform": "timeseries_to_rows", - "type": "table" - }, - { - "columns": [], - "datasource": "$datasource", - "description": "", - "fontSize": "100%", - "gridPos": { - "h": 10, - "w": 6, - "x": 12, - "y": 35 - }, - "id": 64, - "interval": "", - "options": {}, - "pageSize": null, - "pluginVersion": "6.6.0", - "showHeader": true, - "sort": { - "col": 0, - "desc": true - }, - "styles": [ - { - "alias": "Time", - "align": "auto", - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "pattern": "Time", - "type": "hidden" - }, - { - "alias": "Node", - "align": "left", - "colorMode": null, - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "mappingType": 1, - "pattern": "Metric", - "thresholds": [], - "type": "number", - "unit": "short" - }, - { - "alias": "Pending", - "align": "left", - "colorMode": null, - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "decimals": 0, - "pattern": "Value", - "thresholds": [], - "type": "number", - "unit": "short" - } - ], - "targets": [ - { - "expr": "sort(label_replace(weave_connections{state=\"pending\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "instant": true, - "legendFormat": "{{node_ip}}", - "refId": "A" - } - ], - "timeFrom": null, - "timeShift": null, - "title": "P2P / Pending", - "transform": "timeseries_to_rows", - "type": "table" - }, - { - "columns": [], - "datasource": "$datasource", - "description": "", - "fontSize": "100%", - "gridPos": { - "h": 10, - "w": 6, - "x": 18, - "y": 35 - }, - "id": 65, - "interval": "", - "options": {}, - "pageSize": null, - "pluginVersion": "6.6.0", - "showHeader": true, - "sort": { - "col": 0, - "desc": true - }, - "styles": [ - { - "alias": "Time", - "align": "auto", - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "pattern": "Time", - "type": "hidden" - }, - { - "alias": "Node", - "align": "left", - "colorMode": null, - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "mappingType": 1, - "pattern": "Metric", - "thresholds": [], - "type": "number", - "unit": "short" - }, - { - "alias": "Failed", - "align": "left", - "colorMode": null, - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "decimals": 0, - "pattern": "Value", - "thresholds": [], - "type": "number", - "unit": "short" - } - ], - "targets": [ - { - "expr": "sort(label_replace(weave_connections{state=\"failed\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "instant": true, - "legendFormat": "{{node_ip}}", - "refId": "A" - } - ], - "timeFrom": null, - "timeShift": null, - "title": "P2P / Failed", - "transform": "timeseries_to_rows", - "type": "table" - }, - { - "columns": [], - "datasource": "$datasource", - "description": "", - "fontSize": "100%", - "gridPos": { - "h": 10, - "w": 6, - "x": 0, - "y": 45 - }, - "id": 66, - "interval": "", - "options": {}, - "pageSize": null, - "pluginVersion": "6.6.0", - "showHeader": true, - "sort": { - "col": 0, - "desc": true - }, - "styles": [ - { - "alias": "Time", - "align": "auto", - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "pattern": "Time", - "type": "hidden" - }, - { - "alias": "Node", - "align": "left", - "colorMode": null, - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "mappingType": 1, - "pattern": "Metric", - "thresholds": [], - "type": "number", - "unit": "short" - }, - { - "alias": "Retrying", - "align": "left", - "colorMode": null, - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "decimals": 0, - "pattern": "Value", - "thresholds": [], - "type": "number", - "unit": "short" - } - ], - "targets": [ - { - "expr": "sort_desc(label_replace(weave_connections{state=\"retrying\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "instant": true, - "legendFormat": "{{node_ip}}", - "refId": "A" - } - ], - "timeFrom": null, - "timeShift": null, - "title": "P2P / Retrying", - "transform": "timeseries_to_rows", - "type": "table" - }, - { - "columns": [], - "datasource": "$datasource", - "description": "", - "fontSize": "100%", - "gridPos": { - "h": 10, - "w": 6, - "x": 6, - "y": 45 - }, - "id": 44, - "interval": "", - "options": {}, - "pageSize": null, - "pluginVersion": "6.6.0", - "showHeader": true, - "sort": { - "col": 2, - "desc": false - }, - "styles": [ - { - "alias": "Time", - "align": "auto", - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "pattern": "Time", - "type": "hidden" - }, - { - "alias": "Node", - "align": "left", - "colorMode": null, - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "mappingType": 1, - "pattern": "Metric", - "thresholds": [], - "type": "number", - "unit": "short" - }, - { - "alias": "Fast DP Flows", - "align": "left", - "colorMode": null, - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "decimals": 2, - "pattern": "Value", - "thresholds": [], - "type": "number", - "unit": "short" - } - ], - "targets": [ - { - "expr": "sort_desc(label_replace(weave_flows, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "instant": true, - "legendFormat": "{{node_ip}}", - "refId": "A" - } - ], - "timeFrom": null, - "timeShift": null, - "title": "Fast DP Flows (Current)", - "transform": "timeseries_to_rows", - "type": "table" - }, - { - "columns": [], - "datasource": "$datasource", - "description": "", - "fontSize": "100%", - "gridPos": { - "h": 10, - "w": 6, - "x": 12, - "y": 45 - }, - "id": 45, - "interval": "", - "options": {}, - "pageSize": null, - "pluginVersion": "6.6.0", - "showHeader": true, - "sort": { - "col": 0, - "desc": true - }, - "styles": [ - { - "alias": "Time", - "align": "auto", - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "pattern": "Time", - "type": "hidden" - }, - { - "alias": "Node", - "align": "left", - "colorMode": null, - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "mappingType": 1, - "pattern": "Metric", - "thresholds": [], - "type": "number", - "unit": "short" - }, - { - "alias": "No. of Pods", - "align": "left", - "colorMode": null, - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "decimals": 0, - "pattern": "Value", - "thresholds": [], - "type": "number", - "unit": "short" - } - ], - "targets": [ - { - "expr": "sort_desc(floor(label_replace(max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"}) * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}) / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"}) , \"node_ip\", \"$1.$2.$3.$4\", \"node\", \"^ip-([0-9]+)-([0-9]+)-([0-9]+)-([0-9]+).*$\") * 100))", - "format": "time_series", - "hide": false, - "instant": true, - "legendFormat": "{{node_ip}}", - "refId": "A" - } - ], - "timeFrom": null, - "timeShift": null, - "title": "Pods Per Node (Current)", - "transform": "timeseries_to_rows", - "type": "table" - }, - { - "columns": [], - "datasource": "$datasource", - "description": "", - "fontSize": "100%", - "gridPos": { - "h": 10, - "w": 6, - "x": 18, - "y": 45 - }, - "id": 47, - "interval": "", - "options": {}, - "pageSize": null, - "pluginVersion": "6.6.0", - "showHeader": true, - "sort": { - "col": 2, - "desc": false - }, - "styles": [ - { - "alias": "Time", - "align": "auto", - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "pattern": "Time", - "type": "hidden" - }, - { - "alias": "Node", - "align": "left", - "colorMode": null, - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "dateFormat": "YYYY-MM-DD HH:mm:ss", - "decimals": 2, - "mappingType": 1, - "pattern": "Metric", - "thresholds": [], - "type": "number", - "unit": "short" - }, - { - "alias": "Terminated", - "align": "left", - "colorMode": null, - "colors": [ - "rgba(245, 54, 54, 0.9)", - "rgba(237, 129, 40, 0.89)", - "rgba(50, 172, 45, 0.97)" - ], - "decimals": 0, - "pattern": "Value", - "thresholds": [], - "type": "number", - "unit": "short" - } - ], - "targets": [ - { - "expr": "sort(label_replace(increase(weave_connection_terminations_total[1m]), \"node_ip\", \"$1\", \"instance\", \"(.*):.*\")) != 0", - "format": "time_series", - "hide": false, - "instant": true, - "legendFormat": "{{node_ip}}", - "refId": "A" - } - ], - "timeFrom": null, - "timeShift": null, - "title": "Peer To Peer Connections Terminated", - "transform": "timeseries_to_rows", - "type": "table" - } - ], - "refresh": "10s", - "schemaVersion": 22, - "style": "dark", - "tags": [ - "practo-managed" - ], - "templating": { - "list": [ - { - "current": { - "text": "prometheus", - "value": "prometheus" - }, - "hide": 0, - "includeAll": false, - "label": null, - "multi": false, - "name": "datasource", - "options": [], - "query": "prometheus", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "type": "datasource" - }, - { - "allValue": null, - "current": {}, - "datasource": "$datasource", - "definition": "", - "hide": 2, - "includeAll": false, - "label": "cluster", - "multi": false, - "name": "cluster", - "options": [], - "query": "label_values(node_cpu_seconds_total, cluster)", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "sort": 2, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - }, - { - "allValue": null, - "auto": false, - "auto_count": 30, - "auto_min": "10s", - "current": { - "selected": false, - "text": "4h", - "value": "4h" - }, - "datasource": "$datasource", - "hide": 2, - "includeAll": false, - "label": null, - "multi": false, - "name": "interval", - "options": [ - { - "selected": true, - "text": "4h", - "value": "4h" - } - ], - "query": "4h", - "refresh": 2, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "interval", - "useTags": false - } - ] - }, - "time": { - "from": "now-12h", - "to": "now" - }, - "timepicker": { - "refresh_intervals": [ - "5s", - "10s", - "30s", - "1m", - "5m", - "15m", - "30m", - "1h", - "2h", - "1d" - ], - "time_options": [ - "5m", - "15m", - "1h", - "6h", - "12h", - "24h", - "2d", - "7d", - "30d" - ] - }, - "timezone": "", - "title": "WeaveNet (Cluster)", - "uid": "voS3tW_Zk", - "version": 9 -} \ No newline at end of file diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/weavenet.json b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/weavenet.json deleted file mode 100644 index 3fad6520..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/dashboards/weavenet.json +++ /dev/null @@ -1,2605 +0,0 @@ -{ - "__inputs": [], - "__requires": [ - { - "type": "grafana", - "id": "grafana", - "name": "Grafana", - "version": "6.6.0" - }, - { - "type": "panel", - "id": "graph", - "name": "Graph", - "version": "" - }, - { - "type": "datasource", - "id": "prometheus", - "name": "Prometheus", - "version": "1.0.0" - }, - { - "type": "panel", - "id": "singlestat", - "name": "Singlestat", - "version": "" - } - ], - "annotations": { - "list": [ - { - "builtIn": 1, - "datasource": "-- Grafana --", - "enable": true, - "hide": true, - "iconColor": "rgba(0, 211, 255, 1)", - "name": "Annotations & Alerts", - "type": "dashboard" - } - ] - }, - "description": "WeaveNet metrics. It was made on top of the weavenet prometheus metrics. Please check this for more details https://www.weave.works/docs/net/latest/tasks/manage/metrics", - "editable": true, - "gnetId": 11789, - "graphTooltip": 0, - "id": null, - "iteration": 1582791388040, - "links": [], - "panels": [ - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 0 - }, - "id": 20, - "panels": [], - "repeat": null, - "title": "Headlines", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#d44a3a", - "rgba(237, 129, 40, 0.89)", - "#299c46" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 0, - "y": 1 - }, - "id": 48, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sort_desc(label_replace(weave_flows{instance=\"$instance\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1,50", - "timeFrom": null, - "timeShift": null, - "title": "Fast DP Flows", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": true, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 4, - "y": 1 - }, - "id": 3, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sum(weave_ipam_pending_allocates{instance=\"$instance\"})", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1,10", - "timeFrom": null, - "timeShift": null, - "title": "IPAM Pending Allocates", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": true, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 8, - "y": 1 - }, - "id": 4, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sum(weave_ipam_pending_claims{instance=\"$instance\"})", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1,10", - "timeFrom": null, - "timeShift": null, - "title": "IPAM Pending Claims", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 12, - "y": 1 - }, - "id": 62, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sort(label_replace(weave_max_ips{instance=\"$instance\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, - "title": "Max IPs", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": true, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "format": "cpm", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 16, - "y": 1 - }, - "id": 54, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "increase(weave_connection_terminations_total{instance=\"$instance\"}[1m])", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1,2", - "timeFrom": null, - "timeShift": null, - "title": "Terminations", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#d44a3a", - "rgba(237, 129, 40, 0.89)", - "#299c46" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 0, - "y": 4 - }, - "id": 50, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sort_desc(label_replace(weave_connections{instance=\"$instance\", state=\"established\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1,10", - "timeFrom": null, - "timeShift": null, - "title": "P2P Established", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": true, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 4, - "y": 4 - }, - "id": 53, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sort_desc(label_replace(weave_connections{instance=\"$instance\", state=\"connecting\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1,10", - "timeFrom": null, - "timeShift": null, - "title": "P2P Connecting", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": true, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 8, - "y": 4 - }, - "id": 52, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sort_desc(label_replace(weave_connections{instance=\"$instance\", state=\"pending\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1,10", - "timeFrom": null, - "timeShift": null, - "title": "P2P Pending", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": true, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 12, - "y": 4 - }, - "id": 51, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sort_desc(label_replace(weave_connections{instance=\"$instance\", state=\"failed\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1, 5", - "timeFrom": null, - "timeShift": null, - "title": "P2P Failed", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "aliasColors": {}, - "bars": false, - "cacheTimeout": null, - "colorBackground": false, - "colorValue": true, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "format": "short", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, - "gridPos": { - "h": 3, - "w": 4, - "x": 16, - "y": 4 - }, - "id": 49, - "interval": null, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], - "maxDataPoints": 100, - "nullPointMode": "null as zero", - "nullText": null, - "options": {}, - "percentage": false, - "pointradius": 5, - "points": false, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "stack": false, - "steppedLine": false, - "tableColumn": "", - "targets": [ - { - "expr": "sort_desc(label_replace(weave_connections{instance=\"$instance\", state=\"retrying\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "instant": true, - "intervalFactor": 2, - "refId": "A" - } - ], - "thresholds": "1, 5", - "timeFrom": null, - "timeShift": null, - "title": "P2P Retrying", - "tooltip": { - "shared": false, - "sort": 0, - "value_type": "individual" - }, - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "avg", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": 0, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ] - }, - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 7 - }, - "id": 21, - "panels": [], - "repeat": null, - "title": "IPAM", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "Percentage of all IP addresses owned by unreachable peers.", - "fill": 1, - "fillGradient": 9, - "gridPos": { - "h": 5, - "w": 8, - "x": 0, - "y": 8 - }, - "hiddenSeries": false, - "id": 7, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "label_replace(weave_ipam_unreachable_percentage{instance=\"$instance\"}, \"node\", \"$1\", \"instance\", \"(.*):.*\")", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "IPAM unreachable percentage", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "IPAM Unreachable Percentage", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "percent", - "label": "", - "logBase": 1, - "max": null, - "min": null, - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "Number of unreachable peers that own IPAM addresses.", - "fill": 1, - "fillGradient": 4, - "gridPos": { - "h": 5, - "w": 8, - "x": 8, - "y": 8 - }, - "hiddenSeries": false, - "id": 35, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "label_replace(weave_ipam_unreachable_count{instance=\"$instance\"}, \"node\", \"$1\", \"instance\", \"(.*):.*\")", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "IPAM unreachable count", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "IPAM Unreachable Count", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": " Number of pending allocates and claims.", - "fill": 5, - "fillGradient": 0, - "gridPos": { - "h": 5, - "w": 8, - "x": 16, - "y": 8 - }, - "hiddenSeries": false, - "id": 34, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [ - { - "alias": "pending allocates", - "color": "#FF9830" - }, - { - "alias": "pending claims", - "color": "#F2495C" - } - ], - "spaceLength": 10, - "stack": true, - "steppedLine": false, - "targets": [ - { - "expr": "label_replace(weave_ipam_pending_allocates{instance=\"$instance\"}, \"node\", \"$1\", \"instance\", \"(.*):.*\")", - "format": "time_series", - "hide": false, - "intervalFactor": 1, - "legendFormat": "pending allocates", - "legendLink": null, - "refId": "A", - "step": 10 - }, - { - "expr": "label_replace(weave_ipam_pending_claims{instance=\"$instance\"}, \"node\", \"$1\", \"instance\", \"(.*):.*\")", - "legendFormat": "pending claims", - "refId": "B" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "IPAM Pending", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "collapsed": false, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 13 - }, - "id": 41, - "panels": [], - "title": "Connections", - "type": "row" - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "https://github.com/weaveworks/weave/blob/master/site/concepts/fastdp-how-it-works.md", - "fill": 1, - "fillGradient": 4, - "gridPos": { - "h": 7, - "w": 13, - "x": 0, - "y": 14 - }, - "hiddenSeries": false, - "id": 55, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "label_replace(weave_flows{instance=\"$instance\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\")", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "weave fast dp flows", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Fast DP Flows", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "https://weave-community.slack.com/archives/C2ND76PAA/p1582641797016700?thread_ts=1582613239.010900&cid=C2ND76PAA", - "fill": 1, - "fillGradient": 4, - "gridPos": { - "h": 7, - "w": 11, - "x": 13, - "y": 14 - }, - "hiddenSeries": false, - "id": 56, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": true, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "label_replace(increase(weave_connection_terminations_total{instance=\"$instance\"}[1m]), \"node\", \"$1\", \"instance\", \"(.*):.*\")", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "rate of terminated connections", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "Rate of Terminated Connections", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "cpm", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "fillGradient": 4, - "gridPos": { - "h": 7, - "w": 8, - "x": 0, - "y": 21 - }, - "hiddenSeries": false, - "id": 59, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(label_replace(weave_connections{instance=\"$instance\", state=\"established\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "connections established", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "P2P / Established", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "fillGradient": 4, - "gridPos": { - "h": 7, - "w": 8, - "x": 8, - "y": 21 - }, - "hiddenSeries": false, - "id": 57, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(label_replace(weave_connections{instance=\"$instance\", state=\"connecting\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "connections connecting", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "P2P / Connecting", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "fillGradient": 4, - "gridPos": { - "h": 7, - "w": 8, - "x": 16, - "y": 21 - }, - "hiddenSeries": false, - "id": 58, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(label_replace(weave_connections{instance=\"$instance\", state=\"pending\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "connections pending", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "P2P / Pending", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "fillGradient": 4, - "gridPos": { - "h": 7, - "w": 8, - "x": 0, - "y": 28 - }, - "hiddenSeries": false, - "id": 61, - "legend": { - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(label_replace(weave_connections{instance=\"$instance\", state=\"retrying\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "retrying connections", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "P2P / Retrying", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - }, - { - "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, - "datasource": "$datasource", - "description": "", - "fill": 1, - "fillGradient": 4, - "gridPos": { - "h": 7, - "w": 16, - "x": 8, - "y": 28 - }, - "hiddenSeries": false, - "id": 60, - "legend": { - "alignAsTable": false, - "avg": false, - "current": false, - "max": false, - "min": false, - "show": false, - "total": false, - "values": false - }, - "lines": true, - "linewidth": 1, - "links": [], - "nullPointMode": "null", - "options": { - "dataLinks": [] - }, - "percentage": false, - "pointradius": 5, - "points": false, - "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, - "stack": false, - "steppedLine": false, - "targets": [ - { - "expr": "sort_desc(label_replace(weave_connections{instance=\"$instance\", state=\"failed\"}, \"node_ip\", \"$1\", \"instance\", \"(.*):.*\"))", - "format": "time_series", - "hide": false, - "intervalFactor": 2, - "legendFormat": "failed connections", - "legendLink": null, - "refId": "A", - "step": 10 - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "P2P / Failed", - "tooltip": { - "shared": true, - "sort": 2, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ - { - "format": "short", - "label": "", - "logBase": 1, - "max": null, - "min": "0", - "show": true - }, - { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false - } - ], - "yaxis": { - "align": false, - "alignLevel": null - } - } - ], - "refresh": "10s", - "schemaVersion": 22, - "style": "dark", - "tags": [ - "practo-managed" - ], - "templating": { - "list": [ - { - "current": { - "text": "prometheus", - "value": "prometheus" - }, - "hide": 0, - "includeAll": false, - "label": null, - "multi": false, - "name": "datasource", - "options": [], - "query": "prometheus", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "type": "datasource" - }, - { - "allValue": null, - "current": {}, - "datasource": "$datasource", - "definition": "", - "hide": 2, - "includeAll": false, - "label": "cluster", - "multi": false, - "name": "cluster", - "options": [], - "query": "label_values(node_cpu_seconds_total, cluster)", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "sort": 2, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - }, - { - "allValue": null, - "auto": false, - "auto_count": 30, - "auto_min": "10s", - "current": { - "selected": false, - "text": "4h", - "value": "4h" - }, - "datasource": "$datasource", - "hide": 2, - "includeAll": false, - "label": null, - "multi": false, - "name": "interval", - "options": [ - { - "selected": true, - "text": "4h", - "value": "4h" - } - ], - "query": "4h", - "refresh": 2, - "regex": "", - "skipUrlSync": false, - "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "interval", - "useTags": false - }, - { - "allValue": null, - "current": {}, - "datasource": "$datasource", - "definition": "label_values(weave_ipam_unreachable_percentage, instance)", - "hide": 0, - "includeAll": false, - "label": "instance", - "multi": false, - "name": "instance", - "options": [], - "query": "label_values(weave_ipam_unreachable_percentage, instance)", - "refresh": 1, - "regex": "", - "skipUrlSync": false, - "sort": 0, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", - "type": "query", - "useTags": false - } - ] - }, - "time": { - "from": "now-1h", - "to": "now" - }, - "timepicker": { - "refresh_intervals": [ - "5s", - "10s", - "30s", - "1m", - "5m", - "15m", - "30m", - "1h", - "2h", - "1d" - ], - "time_options": [ - "5m", - "15m", - "1h", - "6h", - "12h", - "24h", - "2d", - "7d", - "30d" - ] - }, - "timezone": "", - "title": "WeaveNet", - "uid": "GzIXGqwZz", - "version": 5 -} \ No newline at end of file diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/datasources/prometheus.json b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/datasources/prometheus.json deleted file mode 100644 index 7a1e14ea..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/datasources/prometheus.json +++ /dev/null @@ -1 +0,0 @@ -{"id":1,"orgId":1,"name":"Prometheus","type":"prometheus","access":"proxy","url":" http://prometheus-operated:9090/","password":"","user":"","database":"","basicAuth":false,"isDefault":true,"jsonData":{},"secureJsonData":null} \ No newline at end of file diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/main.go b/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/main.go deleted file mode 100644 index e3cb2057..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/dashboards/main.go +++ /dev/null @@ -1,194 +0,0 @@ -// This code is heavily --inspired by-- **stolen from** the grafana API examples. -// https://github.com/grafana-tools/sdk/blob/master/cmd/ -// Rather than separate commands for dashboard and datasources, I have combined the logic into a single -// file with appropriate changes. -// dataset files are stored in ./datasources -// dashboards are stored in ./dashboards -package main - -import ( - "encoding/json" - "flag" - "fmt" - "io/ioutil" - "log" - "os" - "strings" - - "github.com/gosimple/slug" - "github.com/grafana-tools/sdk" -) - -func main() { - imode := flag.Bool("import", false, "import dashboards into grafana. (default: false)") - grafana := flag.String("grafana", "http://localhost:3000", "Grafana HTTP endpoint") - apikey := flag.String("apikey", os.Getenv("GRAFANA_API_KEY"), "Grafana API key") - flag.Parse() - - if *grafana == "" { - log.Fatal("Unknown grafana API key. Please set GRAFANA_API_KEY or use the -apikey flag") - } - - client := sdk.NewClient(*grafana, *apikey, sdk.DefaultHTTPClient) - - if *imode { - // import mode - log.Println("importing datasources") - ImportDatasources(client) - log.Println("importing dashboards") - ImportDashboards(client) - } else { - // backup mode - log.Println("backing up datasources") - BackupDatasources(client) - log.Println("backing up dashboards") - BackupDashboards(client) - } - -} - -// ImportDatasources imports datasource json files from ./datasources and uploads them to the -// grafana api -// Credit: -// https://raw.githubusercontent.com/grafana-tools/sdk/master/cmd/import-datasources/main.go -func ImportDatasources(c *sdk.Client) { - var ( - datasources []sdk.Datasource - filesInDir []os.FileInfo - rawDS []byte - status sdk.StatusMessage - err error - ) - if datasources, err = c.GetAllDatasources(); err != nil { - fmt.Fprintf(os.Stderr, "%s\n", err) - os.Exit(1) - } - filesInDir, err = ioutil.ReadDir("./datasources") - if err != nil { - fmt.Fprintf(os.Stderr, "%s\n", err) - } - for _, file := range filesInDir { - if strings.HasSuffix(file.Name(), ".json") { - if rawDS, err = ioutil.ReadFile("./datasources/" + file.Name()); err != nil { - fmt.Fprintf(os.Stderr, "%s\n", err) - continue - } - var newDS sdk.Datasource - if err = json.Unmarshal(rawDS, &newDS); err != nil { - fmt.Fprintf(os.Stderr, "%s\n", err) - continue - } - for _, existingDS := range datasources { - if existingDS.Name == newDS.Name { - sm, err := c.DeleteDatasource(existingDS.ID) - if err != nil { - log.Print(sm.Message) - } - break - } - } - if status, err = c.CreateDatasource(newDS); err != nil { - fmt.Fprintf(os.Stderr, "error on importing datasource %s with %s (%s)", newDS.Name, err, *status.Message) - } - } - } -} - -// ImportDashboards loads dashboard json files from ./dashboards and uplaods them to the grafana api -// Credit: -// https://raw.githubusercontent.com/grafana-tools/sdk/master/cmd/import-dashboards/main.go -func ImportDashboards(c *sdk.Client) { - var ( - filesInDir []os.FileInfo - rawBoard []byte - err error - ) - filesInDir, err = ioutil.ReadDir("./dashboards") - if err != nil { - log.Fatal(err) - } - for _, file := range filesInDir { - if strings.HasSuffix(file.Name(), ".json") { - if rawBoard, err = ioutil.ReadFile("./dashboards/" + file.Name()); err != nil { - log.Println(err) - continue - } - var board sdk.Board - if err = json.Unmarshal(rawBoard, &board); err != nil { - log.Println(err) - continue - } - sm, err := c.DeleteDashboard(board.UpdateSlug()) - if err != nil { - log.Print(sm.Message) - } - _, err = c.SetDashboard(board, false) - if err != nil { - log.Printf("error on importing dashboard %s", board.Title) - continue - } - } - } -} - -// BackupDatasources searches a grafana endpoint for datasets and saves them to ./datasets -// Credit: -// https://raw.githubusercontent.com/grafana-tools/sdk/master/cmd/backup-datasources/main.go -func BackupDatasources(c *sdk.Client) { - var ( - datasources []sdk.Datasource - dsPacked []byte - meta sdk.BoardProperties - err error - ) - if datasources, err = c.GetAllDatasources(); err != nil { - fmt.Fprintf(os.Stderr, "%s\n", err) - os.Exit(1) - } - for _, ds := range datasources { - if dsPacked, err = json.Marshal(ds); err != nil { - fmt.Fprintf(os.Stderr, "%s for %s\n", err, ds.Name) - continue - } - if err = ioutil.WriteFile(fmt.Sprintf("./datasources/%s.json", slug.Make(ds.Name)), dsPacked, os.FileMode(int(0666))); err != nil { - fmt.Fprintf(os.Stderr, "%s for %s\n", err, meta.Slug) - } - } -} - -// BackupDashboards searches a grafana endpoint for dashboards with the "testground" tag -// and saves them into ./dashboards -// Credit: -// https://raw.githubusercontent.com/grafana-tools/sdk/master/cmd/backup-dashboards/main.go -func BackupDashboards(c *sdk.Client) { - var ( - boardLinks []sdk.FoundBoard - rawBoard []byte - meta sdk.BoardProperties - err error - ) - if boardLinks, err = c.SearchDashboards("", false, "testground"); err != nil { - fmt.Fprintf(os.Stderr, "%s\n", err) - os.Exit(1) - } - var board sdk.Board - for _, link := range boardLinks { - if rawBoard, meta, err = c.GetRawDashboardBySlug(link.URI); err != nil { - fmt.Fprintf(os.Stderr, "%s for %s\n", err, link.URI) - continue - } - err := json.Unmarshal(rawBoard, &board) - if err != nil { - log.Println(err) - continue - } - pretty, err := json.MarshalIndent(board, "", " ") - if err != nil { - log.Printf("couldn't pretty print %s. continuing anyway.", meta.Slug) - pretty = rawBoard - } - if err = ioutil.WriteFile(fmt.Sprintf("./dashboards/%s.json", meta.Slug), pretty, os.FileMode(int(0666))); err != nil { - fmt.Fprintf(os.Stderr, "%s for %s\n", err, meta.Slug) - } - } -} diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/templates/dashboard.yaml b/k8s/kops/testground-infra/charts/testground-dashboards/templates/dashboard.yaml deleted file mode 100644 index 1a7b45c9..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/templates/dashboard.yaml +++ /dev/null @@ -1,8 +0,0 @@ -apiVersion: v1 -kind: ConfigMap -metadata: - name: {{ .Release.Name }}-dashboards - labels: - grafana_dashboard: "1" -data: -{{ (.Files.Glob "dashboards/dashboards/*").AsConfig | indent 2 }} diff --git a/k8s/kops/testground-infra/charts/testground-dashboards/templates/datasource.yaml b/k8s/kops/testground-infra/charts/testground-dashboards/templates/datasource.yaml deleted file mode 100644 index eb780e11..00000000 --- a/k8s/kops/testground-infra/charts/testground-dashboards/templates/datasource.yaml +++ /dev/null @@ -1,8 +0,0 @@ -apiVersion: v1 -kind: ConfigMap -metadata: - name: {{ .Release.Name }}-datasources - labels: - grafana_datasource: "1" -data: -{{ (.Files.Glob "dashboards/datasources/*").AsConfig | indent 2 }} diff --git a/k8s/kops/testground-infra/requirements.lock b/k8s/kops/testground-infra/requirements.lock deleted file mode 100644 index 294a3aaf..00000000 --- a/k8s/kops/testground-infra/requirements.lock +++ /dev/null @@ -1,9 +0,0 @@ -dependencies: -- name: redis - repository: https://charts.bitnami.com/bitnami - version: 10.6.6 -- name: testground-dashboards - repository: "" - version: 0.0.* -digest: sha256:2a9e152a6d30e2aca28f44c2e5131469c014b505888c5e3a6eeac2af94ba0915 -generated: "2020-04-14T14:57:04.250358476-07:00" diff --git a/k8s/kops/testground-infra/requirements.yaml b/k8s/kops/testground-infra/requirements.yaml deleted file mode 100644 index f72253e8..00000000 --- a/k8s/kops/testground-infra/requirements.yaml +++ /dev/null @@ -1,9 +0,0 @@ ---- -dependencies: - - - name: redis - repository: https://charts.bitnami.com/bitnami - version: "*" - - - name: testground-dashboards - version: "0.0.*" diff --git a/k8s/kops/testground-infra/values.yaml b/k8s/kops/testground-infra/values.yaml deleted file mode 100644 index ca21b4db..00000000 --- a/k8s/kops/testground-infra/values.yaml +++ /dev/null @@ -1,44 +0,0 @@ ---- -# Changes from defaults: -# * enable redis exporter -# * enable the serviceMonitor so it will be picked up by prometheus -# * Disable cluster mode -- there are failure modes which could result in -# dataloss after acknowledgement. -# * SecurityContext and sysctlImage provides tuning for redis to provide for 10k -# * Master resources is setup to be a large value so redis will be mostly -# singele-tenant. -# * podAnnotations is set to flannel to force it to be on the control network. -redis: - metrics: - enabled: true - serviceMonitor: - enabled: true - namespace: default - resources: - requests: - memory: 256Mi - cpu: 200m - limits: - memory: 256Mi - cluster: - enabled: false - usePassword: false - securityContext: - sysctls: - - name: net.core.somaxconn - value: "131072" - - name: net.netfilter.nf_conntrack_max - value: "1048576" - master: - persistence: - enabled: false - extraFlags: - - "--maxclients 131072" - nodeSelector: - testground.node.role.infra: "true" - resources: - requests: - memory: 10000Mi - cpu: 4000m - limits: - memory: 10000Mi diff --git a/k8s/kops/tools/README.md b/k8s/kops/tools/README.md deleted file mode 100644 index b6958451..00000000 --- a/k8s/kops/tools/README.md +++ /dev/null @@ -1,9 +0,0 @@ -# Tools and commands that are helpful during running Testground on Kubernetes - -1. `confirm_dns_route.sh` - -A command which makes sure that a DNS route has been setup on all `host` machines. Number of lines returned should be the number of nodes in the cluster. - -``` -./confirm_dns_route.sh | wc -l -``` diff --git a/k8s/kops/tools/authorize-19999.sh b/k8s/kops/tools/authorize-19999.sh deleted file mode 100755 index 1d007466..00000000 --- a/k8s/kops/tools/authorize-19999.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/bin/sh - -securityGroupId=`aws ec2 describe-security-groups --region=$AWS_REGION --output text | awk '/nodes.'$NAME'/ && /SECURITYGROUPS/ { print $6 };'` - -if [[ -z ${securityGroupId} ]]; then - echo "Couldn't detect AWS Security Group created by `kops`" - exit 1 -fi - -aws ec2 authorize-security-group-ingress \ - --group-id $securityGroupId \ - --ip-permissions IpProtocol=tcp,FromPort=19999,ToPort=19999,IpRanges='[{CidrIp=0.0.0.0/0}]' diff --git a/k8s/kops/tools/confirm_dns_route.sh b/k8s/kops/tools/confirm_dns_route.sh deleted file mode 100755 index 26d6d114..00000000 --- a/k8s/kops/tools/confirm_dns_route.sh +++ /dev/null @@ -1,6 +0,0 @@ -#!/bin/zsh - -IFS=$'\n' sidecars=( $(kubectl get pods | grep sidecar | awk '{print $1}') ) -for i in "${sidecars[@]}"; do - kubectl logs $i -c iproute-add | grep 100.64 -done diff --git a/k8s/kops/tools/redeploy-daemon.sh b/k8s/kops/tools/redeploy-daemon.sh deleted file mode 100755 index c80fa95f..00000000 --- a/k8s/kops/tools/redeploy-daemon.sh +++ /dev/null @@ -1,19 +0,0 @@ -#!/bin/bash - -set -o errexit -set -o pipefail - -set -e - -err_report() { - echo "Error on line $1" -} - -trap 'err_report $LINENO' ERR - -kubectl delete deployment testground-daemon || true -kubectl delete service testground-daemon || true -kubectl apply -f config-map-env-toml.yml -kubectl apply -f service-account.yml -kubectl apply -f role-binding.yml -kubectl apply -f deployment.yml -f service.yml diff --git a/manifests/aws/ebs/kustomization.yaml b/manifests/aws/ebs/kustomization.yaml new file mode 100644 index 00000000..b4a9df97 --- /dev/null +++ b/manifests/aws/ebs/kustomization.yaml @@ -0,0 +1,5 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: +- pvc.yaml diff --git a/k8s/kops/ebs/pvc.yml b/manifests/aws/ebs/pvc.yaml similarity index 100% rename from k8s/kops/ebs/pvc.yml rename to manifests/aws/ebs/pvc.yaml diff --git a/manifests/aws/efs/charts/aws-efs-csi-driver/.helmignore b/manifests/aws/efs/charts/aws-efs-csi-driver/.helmignore new file mode 100644 index 00000000..50af0317 --- /dev/null +++ b/manifests/aws/efs/charts/aws-efs-csi-driver/.helmignore @@ -0,0 +1,22 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*~ +# Various IDEs +.project +.idea/ +*.tmproj +.vscode/ diff --git a/manifests/aws/efs/charts/aws-efs-csi-driver/CHANGELOG.md b/manifests/aws/efs/charts/aws-efs-csi-driver/CHANGELOG.md new file mode 100644 index 00000000..afcece46 --- /dev/null +++ b/manifests/aws/efs/charts/aws-efs-csi-driver/CHANGELOG.md @@ -0,0 +1,181 @@ +# Helm chart +# v2.4.1 +* Bump app/driver version to `v1.5.4` +# v2.4.0 +* Bump app/driver version to `v1.5.3` +# v2.3.9 +* Bump app/driver version to `v1.5.2` +# v2.3.8 +* Bump app/driver version to `v1.5.1` +# v2.3.7 +* Bump app/driver version to `v1.5.0` +# v2.3.6 +* Bump app/driver version to `v1.4.9` +# v2.3.5 +* Bump app/driver version to `v1.4.8` + +# v2.3.4 +* Bump app/driver version to `v1.4.7` + +# v2.3.3 +* Bump app/driver version to `v1.4.6` + +# v2.3.2 +* Bump app/driver version to `v1.4.5` + +# v2.3.1 +* Bump app/driver version to `v1.4.4` + +# v2.3.0 +* Bump app/driver version to `v1.4.3` + +# v2.2.9 +* Bump app/driver version to `v1.4.2` + +# v2.2.8 +* Bump app/driver version to `v1.4.1` + +# v2.2.7 +* Bump app/driver version to `v1.4.0` +# v2.2.6 +* Bump app/driver version to `v1.3.8` + +# v2.2.5 +* Bump app/driver version to `v1.3.7` + +# v2.2.4 +* Add STS regional endpoints flag to fix PV creation on private EKS + +# v2.2.3 +* Bump app/driver version to `v1.3.6` + +# v2.2.2 +* Add controller.volMetricsOptIn for emitting volume metrics +* Update ECR sidecars to 1-18-13 + +# v2.2.1 +* Bump app/driver version to `v1.3.5` + +# v2.2.0 +* Allow health ports to be configured +* Add Missing "patch" permission for "events" + +# v2.1.6 +* Bump app/driver version to `v1.3.4` + +# v2.1.5 +* Bump app/driver version to `v1.3.3` + +# v2.1.4 +* Add node.serviceAccount values for creating and/or specifying daemonset service account + +# v2.1.3 +* Bump app/driver version to `v1.3.2` + +# v2.1.2 +* Add extra-create-metadata + +# v2.1.1 +* Update app/driver version to `v1.3.1` + +# v2.1.0 + +## New features +* Update app/driver version to `v1.3.0` + +## Bug fixes +* Put comments back in place inside the values file ([#475](https://github.com/kubernetes-sigs/aws-efs-csi-driver/pull/475), [@pierluigilenoci](https://github.com/pierluigilenoci)) + +# v2.0.1 + +## Bug fixes +* Helm chart: fix reclaimPolicy and volumeBindingMode ([#464](https://github.com/kubernetes-sigs/aws-efs-csi-driver/pull/464), [@devinsmith911](https://github.com/devinsmith911)) + + +# v2.0.0 + +## Breaking changes + +Multiple changes in values file at `sidecars`, `controller` and `node` + +--- +```yaml +sidecars: + xxxxxxxxx: + repository: + tag: +``` + +Moving to + +```yaml +sidecars: + xxxxxxxxx: + image: + repository: + tag: +``` + +--- +```yaml +podAnnotations: +resources: +nodeSelector: +tolerations: +affinity: +``` + +Moving to + +```yaml +controller: + podAnnotations: + resources: + nodeSelector: + tolerations: + affinity: +``` + +--- +```yaml +hostAliases: +dnsPolicy: +dnsConfig: +``` + +Moving to + +```yaml +node: + hostAliases: + dnsPolicy: + dnsConfig: +``` + +--- +```yaml +serviceAccount: + controller: +``` + +Moving to + +```yaml +controller: + serviceAccount: +``` + +## New features + +* Chart API `v2` (requires Helm 3) +* Set `resources` and `imagePullPolicy` fields independently for containers +* Set `logLevel`, `affinity`, `nodeSelector`, `podAnnotations` and `tolerations` fields independently +for Controller deployment and Node daemonset +* Set `reclaimPolicy` and `volumeBindingMode` fields in storage class + +## Fixes + +* Fixing Controller deployment using `podAnnotations` and `tolerations` values from Node daemonset +* Let the user define the whole `tolerations` array, default to `- operator: Exists` +* Default `logLevel` lowered from `5` to `2` +* Default `imagePullPolicy` everywhere set to `IfNotPresent` diff --git a/manifests/aws/efs/charts/aws-efs-csi-driver/Chart.yaml b/manifests/aws/efs/charts/aws-efs-csi-driver/Chart.yaml new file mode 100644 index 00000000..8620cb58 --- /dev/null +++ b/manifests/aws/efs/charts/aws-efs-csi-driver/Chart.yaml @@ -0,0 +1,18 @@ +apiVersion: v2 +appVersion: 1.5.4 +description: A Helm chart for AWS EFS CSI Driver +home: https://github.com/kubernetes-sigs/aws-efs-csi-driver +keywords: +- aws +- efs +- csi +kubeVersion: '>=1.17.0-0' +maintainers: +- name: leakingtapan + url: https://github.com/leakingtapan +- name: krmichel + url: https://github.com/krmichel +name: aws-efs-csi-driver +sources: +- https://github.com/kubernetes-sigs/aws-efs-csi-driver +version: 2.4.1 diff --git a/manifests/aws/efs/charts/aws-efs-csi-driver/templates/NOTES.txt b/manifests/aws/efs/charts/aws-efs-csi-driver/templates/NOTES.txt new file mode 100644 index 00000000..66f4d12d --- /dev/null +++ b/manifests/aws/efs/charts/aws-efs-csi-driver/templates/NOTES.txt @@ -0,0 +1,3 @@ +To verify that aws-efs-csi-driver has started, run: + + kubectl get pod -n {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "aws-efs-csi-driver.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" diff --git a/manifests/aws/efs/charts/aws-efs-csi-driver/templates/_helpers.tpl b/manifests/aws/efs/charts/aws-efs-csi-driver/templates/_helpers.tpl new file mode 100644 index 00000000..ab6b4e0e --- /dev/null +++ b/manifests/aws/efs/charts/aws-efs-csi-driver/templates/_helpers.tpl @@ -0,0 +1,56 @@ +{{/* vim: set filetype=mustache: */}} +{{/* +Expand the name of the chart. +*/}} +{{- define "aws-efs-csi-driver.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +If release name contains chart name it will be used as a full name. +*/}} +{{- define "aws-efs-csi-driver.fullname" -}} +{{- if .Values.fullnameOverride -}} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- $name := default .Chart.Name .Values.nameOverride -}} +{{- if contains $name .Release.Name -}} +{{- .Release.Name | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} +{{- end -}} +{{- end -}} +{{- end -}} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "aws-efs-csi-driver.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{/* +Common labels +*/}} +{{- define "aws-efs-csi-driver.labels" -}} +app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} +helm.sh/chart: {{ include "aws-efs-csi-driver.chart" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end -}} + +{{/* +Create a string out of the map for controller tags flag +*/}} +{{- define "aws-efs-csi-driver.tags" -}} +{{- $tags := list -}} +{{ range $key, $val := . }} +{{- $tags = print $key ":" $val | append $tags -}} +{{- end -}} +{{- join " " $tags -}} +{{- end -}} diff --git a/manifests/aws/efs/charts/aws-efs-csi-driver/templates/controller-deployment.yaml b/manifests/aws/efs/charts/aws-efs-csi-driver/templates/controller-deployment.yaml new file mode 100644 index 00000000..6f8b2ca5 --- /dev/null +++ b/manifests/aws/efs/charts/aws-efs-csi-driver/templates/controller-deployment.yaml @@ -0,0 +1,132 @@ +{{- if .Values.controller.create }} +# Controller Service +kind: Deployment +apiVersion: apps/v1 +metadata: + name: efs-csi-controller + labels: + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: + app: efs-csi-controller + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} + app.kubernetes.io/instance: {{ .Release.Name }} + {{- with .Values.controller.updateStrategy }} + strategy: + {{ toYaml . | nindent 4 }} + {{- end }} + template: + metadata: + labels: + app: efs-csi-controller + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} + app.kubernetes.io/instance: {{ .Release.Name }} + {{- with .Values.controller.podAnnotations }} + annotations: {{- toYaml . | nindent 8 }} + {{- end }} + spec: + {{- if .Values.imagePullSecrets }} + imagePullSecrets: + {{- range .Values.imagePullSecrets }} + - name: {{ . }} + {{- end }} + {{- end }} + nodeSelector: + kubernetes.io/os: linux + {{- with .Values.controller.nodeSelector }} + {{- toYaml . | nindent 8 }} + {{- end }} + serviceAccountName: {{ .Values.controller.serviceAccount.name }} + priorityClassName: system-cluster-critical + {{- with .Values.controller.tolerations }} + tolerations: {{- toYaml . | nindent 8 }} + {{- end }} + containers: + - name: efs-plugin + securityContext: + privileged: true + image: {{ printf "%s:%s" .Values.image.repository (default (printf "v%s" .Chart.AppVersion) (toString .Values.image.tag)) }} + imagePullPolicy: {{ .Values.image.pullPolicy }} + args: + - --endpoint=$(CSI_ENDPOINT) + - --logtostderr + {{- if .Values.controller.tags }} + - --tags={{ include "aws-efs-csi-driver.tags" .Values.controller.tags }} + {{- end }} + - --v={{ .Values.controller.logLevel }} + - --delete-access-point-root-dir={{ hasKey .Values.controller "deleteAccessPointRootDir" | ternary .Values.controller.deleteAccessPointRootDir false }} + - --vol-metrics-opt-in={{ hasKey .Values.controller "volMetricsOptIn" | ternary .Values.controller.volMetricsOptIn false }} + env: + - name: CSI_ENDPOINT + value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock + {{- if .Values.controller.regionalStsEndpoints }} + - name: AWS_STS_REGIONAL_ENDPOINTS + value: regional + {{- end }} + - name: CSI_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + {{- if .Values.useFIPS }} + - name: AWS_USE_FIPS_ENDPOINT + value: "true" + {{- end }} + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/sockets/pluginproxy/ + ports: + - name: healthz + containerPort: {{ .Values.controller.healthPort }} + protocol: TCP + livenessProbe: + httpGet: + path: /healthz + port: healthz + initialDelaySeconds: 10 + timeoutSeconds: 3 + periodSeconds: 10 + failureThreshold: 5 + {{- with .Values.controller.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + - name: csi-provisioner + image: {{ printf "%s:%s" .Values.sidecars.csiProvisioner.image.repository .Values.sidecars.csiProvisioner.image.tag }} + imagePullPolicy: {{ .Values.sidecars.csiProvisioner.image.pullPolicy }} + args: + - --csi-address=$(ADDRESS) + - --v={{ .Values.controller.logLevel }} + - --feature-gates=Topology=true + {{- if .Values.controller.extraCreateMetadata }} + - --extra-create-metadata + {{- end }} + - --leader-election + env: + - name: ADDRESS + value: /var/lib/csi/sockets/pluginproxy/csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/sockets/pluginproxy/ + {{- with .Values.sidecars.csiProvisioner.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + - name: liveness-probe + image: {{ printf "%s:%s" .Values.sidecars.livenessProbe.image.repository .Values.sidecars.livenessProbe.image.tag }} + imagePullPolicy: {{ .Values.sidecars.livenessProbe.image.pullPolicy }} + args: + - --csi-address=/csi/csi.sock + - --health-port={{ .Values.controller.healthPort }} + volumeMounts: + - name: socket-dir + mountPath: /csi + {{- with .Values.sidecars.livenessProbe.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + volumes: + - name: socket-dir + emptyDir: {} + {{- with .Values.controller.affinity }} + affinity: {{- toYaml . | nindent 8 }} + {{- end }} +{{- end }} diff --git a/manifests/aws/efs/charts/aws-efs-csi-driver/templates/controller-serviceaccount.yaml b/manifests/aws/efs/charts/aws-efs-csi-driver/templates/controller-serviceaccount.yaml new file mode 100644 index 00000000..bc1d11d1 --- /dev/null +++ b/manifests/aws/efs/charts/aws-efs-csi-driver/templates/controller-serviceaccount.yaml @@ -0,0 +1,62 @@ +{{- if .Values.controller.serviceAccount.create }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ .Values.controller.serviceAccount.name }} + labels: + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} + {{- with .Values.controller.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- end }} +--- + +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: efs-csi-external-provisioner-role + labels: + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} +rules: + - apiGroups: [""] + resources: ["persistentvolumes"] + verbs: ["get", "list", "watch", "create", "delete"] + - apiGroups: [""] + resources: ["persistentvolumeclaims"] + verbs: ["get", "list", "watch", "update"] + - apiGroups: ["storage.k8s.io"] + resources: ["storageclasses"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["events"] + verbs: ["list", "watch", "create", "patch"] + - apiGroups: ["storage.k8s.io"] + resources: ["csinodes"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "list", "watch"] + - apiGroups: ["coordination.k8s.io"] + resources: ["leases"] + verbs: ["get", "watch", "list", "delete", "update", "create"] + # - apiGroups: [ "" ] + # resources: [ "secrets" ] + # verbs: [ "get", "watch", "list" ] + +--- + +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: efs-csi-provisioner-binding + labels: + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} +subjects: + - kind: ServiceAccount + name: {{ .Values.controller.serviceAccount.name }} + namespace: {{ .Release.Namespace }} +roleRef: + kind: ClusterRole + name: efs-csi-external-provisioner-role + apiGroup: rbac.authorization.k8s.io diff --git a/manifests/aws/efs/charts/aws-efs-csi-driver/templates/csidriver.yaml b/manifests/aws/efs/charts/aws-efs-csi-driver/templates/csidriver.yaml new file mode 100644 index 00000000..e6b4d419 --- /dev/null +++ b/manifests/aws/efs/charts/aws-efs-csi-driver/templates/csidriver.yaml @@ -0,0 +1,10 @@ +apiVersion: {{ ternary "storage.k8s.io/v1" "storage.k8s.io/v1beta1" (semverCompare ">=1.18.0-0" .Capabilities.KubeVersion.Version) }} +kind: CSIDriver +metadata: + name: efs.csi.aws.com + annotations: + "helm.sh/hook": pre-install, pre-upgrade + "helm.sh/hook-delete-policy": before-hook-creation + "helm.sh/resource-policy": keep +spec: + attachRequired: false diff --git a/manifests/aws/efs/charts/aws-efs-csi-driver/templates/node-daemonset.yaml b/manifests/aws/efs/charts/aws-efs-csi-driver/templates/node-daemonset.yaml new file mode 100644 index 00000000..3d321fe0 --- /dev/null +++ b/manifests/aws/efs/charts/aws-efs-csi-driver/templates/node-daemonset.yaml @@ -0,0 +1,171 @@ +# Node Service +kind: DaemonSet +apiVersion: apps/v1 +metadata: + name: efs-csi-node + labels: + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} +spec: + selector: + matchLabels: + app: efs-csi-node + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} + app.kubernetes.io/instance: {{ .Release.Name }} + {{- with .Values.node.updateStrategy }} + updateStrategy: + {{ toYaml . | nindent 4 }} + {{- end }} + template: + metadata: + labels: + app: efs-csi-node + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} + app.kubernetes.io/instance: {{ .Release.Name }} + {{- if .Values.node.podAnnotations }} + annotations: {{ toYaml .Values.node.podAnnotations | nindent 8 }} + {{- end }} + spec: + {{- with .Values.node.hostAliases }} + hostAliases: + {{- range $k, $v := . }} + - ip: {{ $v.ip }} + hostnames: + - {{ $k }}.efs.{{ $v.region }}.amazonaws.com + {{- end }} + {{- end }} + {{- if .Values.imagePullSecrets }} + imagePullSecrets: + {{- range .Values.imagePullSecrets }} + - name: {{ . }} + {{- end }} + {{- end }} + nodeSelector: + kubernetes.io/os: linux + {{- with .Values.node.nodeSelector }} + {{- toYaml . | nindent 8 }} + {{- end }} + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: eks.amazonaws.com/compute-type + operator: NotIn + values: + - fargate + hostNetwork: true + dnsPolicy: {{ .Values.node.dnsPolicy }} + {{- with .Values.node.dnsConfig }} + dnsConfig: {{- toYaml . | nindent 8 }} + {{- end }} + serviceAccountName: {{ .Values.node.serviceAccount.name }} + priorityClassName: system-node-critical + {{- with .Values.node.tolerations }} + tolerations: {{- toYaml . | nindent 8 }} + {{- end }} + containers: + - name: efs-plugin + securityContext: + privileged: true + image: {{ printf "%s:%s" .Values.image.repository (default (printf "v%s" .Chart.AppVersion) (toString .Values.image.tag)) }} + imagePullPolicy: {{ .Values.image.pullPolicy }} + args: + - --endpoint=$(CSI_ENDPOINT) + - --logtostderr + - --v={{ .Values.node.logLevel }} + env: + - name: CSI_ENDPOINT + value: unix:/csi/csi.sock + {{- if .Values.useFIPS }} + - name: AWS_USE_FIPS_ENDPOINT + value: "true" + {{- end }} + volumeMounts: + - name: kubelet-dir + mountPath: /var/lib/kubelet + mountPropagation: "Bidirectional" + - name: plugin-dir + mountPath: /csi + - name: efs-state-dir + mountPath: /var/run/efs + - name: efs-utils-config + mountPath: /var/amazon/efs + - name: efs-utils-config-legacy + mountPath: /etc/amazon/efs-legacy + ports: + - name: healthz + containerPort: {{ .Values.node.healthPort }} + protocol: TCP + livenessProbe: + httpGet: + path: /healthz + port: healthz + initialDelaySeconds: 10 + timeoutSeconds: 3 + periodSeconds: 2 + failureThreshold: 5 + {{- with .Values.node.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + - name: csi-driver-registrar + image: {{ printf "%s:%s" .Values.sidecars.nodeDriverRegistrar.image.repository .Values.sidecars.nodeDriverRegistrar.image.tag }} + imagePullPolicy: {{ .Values.sidecars.nodeDriverRegistrar.image.pullPolicy }} + args: + - --csi-address=$(ADDRESS) + - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH) + - --v={{ .Values.node.logLevel }} + env: + - name: ADDRESS + value: /csi/csi.sock + - name: DRIVER_REG_SOCK_PATH + value: /var/lib/kubelet/plugins/efs.csi.aws.com/csi.sock + - name: KUBE_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + volumeMounts: + - name: plugin-dir + mountPath: /csi + - name: registration-dir + mountPath: /registration + {{- with .Values.sidecars.nodeDriverRegistrar.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + - name: liveness-probe + image: {{ printf "%s:%s" .Values.sidecars.livenessProbe.image.repository .Values.sidecars.livenessProbe.image.tag }} + imagePullPolicy: {{ .Values.sidecars.livenessProbe.image.pullPolicy }} + args: + - --csi-address=/csi/csi.sock + - --health-port={{ .Values.node.healthPort }} + - --v={{ .Values.node.logLevel }} + volumeMounts: + - name: plugin-dir + mountPath: /csi + {{- with .Values.sidecars.livenessProbe.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + volumes: + - name: kubelet-dir + hostPath: + path: /var/lib/kubelet + type: Directory + - name: plugin-dir + hostPath: + path: /var/lib/kubelet/plugins/efs.csi.aws.com/ + type: DirectoryOrCreate + - name: registration-dir + hostPath: + path: /var/lib/kubelet/plugins_registry/ + type: Directory + - name: efs-state-dir + hostPath: + path: /var/run/efs + type: DirectoryOrCreate + - name: efs-utils-config + hostPath: + path: /var/amazon/efs + type: DirectoryOrCreate + - name: efs-utils-config-legacy + hostPath: + path: /etc/amazon/efs + type: DirectoryOrCreate diff --git a/manifests/aws/efs/charts/aws-efs-csi-driver/templates/node-serviceaccount.yaml b/manifests/aws/efs/charts/aws-efs-csi-driver/templates/node-serviceaccount.yaml new file mode 100644 index 00000000..9fd3c7a0 --- /dev/null +++ b/manifests/aws/efs/charts/aws-efs-csi-driver/templates/node-serviceaccount.yaml @@ -0,0 +1,12 @@ +{{- if .Values.node.serviceAccount.create }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ .Values.node.serviceAccount.name }} + labels: + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} + {{- with .Values.node.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- end }} diff --git a/manifests/aws/efs/charts/aws-efs-csi-driver/templates/storageclass.yaml b/manifests/aws/efs/charts/aws-efs-csi-driver/templates/storageclass.yaml new file mode 100644 index 00000000..8ce1ec87 --- /dev/null +++ b/manifests/aws/efs/charts/aws-efs-csi-driver/templates/storageclass.yaml @@ -0,0 +1,26 @@ +{{- range .Values.storageClasses }} +kind: StorageClass +apiVersion: storage.k8s.io/v1 +metadata: + name: {{ .name }} + {{- with .annotations }} + annotations: + {{ toYaml . | indent 4 }} + {{- end }} +provisioner: efs.csi.aws.com +{{- with .mountOptions }} +mountOptions: +{{ toYaml . }} +{{- end }} +{{- with .parameters }} +parameters: +{{ toYaml . | indent 2 }} +{{- end }} +{{- with .reclaimPolicy }} +reclaimPolicy: {{ . }} +{{- end }} +{{- with .volumeBindingMode }} +volumeBindingMode: {{ . }} +{{- end }} +--- +{{- end }} \ No newline at end of file diff --git a/manifests/aws/efs/charts/aws-efs-csi-driver/values.yaml b/manifests/aws/efs/charts/aws-efs-csi-driver/values.yaml new file mode 100644 index 00000000..79c6943d --- /dev/null +++ b/manifests/aws/efs/charts/aws-efs-csi-driver/values.yaml @@ -0,0 +1,147 @@ +# Default values for aws-efs-csi-driver. +# This is a YAML-formatted file. +# Declare variables to be passed into your templates. + +nameOverride: "" +fullnameOverride: "" + +replicaCount: 2 + +useFIPS: false + +image: + repository: amazon/aws-efs-csi-driver + tag: "v1.5.4" + pullPolicy: IfNotPresent + +sidecars: + livenessProbe: + image: + repository: public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe + tag: v2.8.0-eks-1-25-latest + pullPolicy: IfNotPresent + resources: {} + nodeDriverRegistrar: + image: + repository: public.ecr.aws/eks-distro/kubernetes-csi/node-driver-registrar + tag: v2.6.2-eks-1-25-latest + pullPolicy: IfNotPresent + resources: {} + csiProvisioner: + image: + repository: public.ecr.aws/eks-distro/kubernetes-csi/external-provisioner + tag: v3.3.0-eks-1-25-latest + pullPolicy: IfNotPresent + resources: {} + +imagePullSecrets: [] + +## Controller deployment variables + +controller: + # Specifies whether a deployment should be created + create: true + # Number for the log level verbosity + logLevel: 2 + # If set, add pv/pvc metadata to plugin create requests as parameters. + extraCreateMetadata: true + # Add additional tags to access points + tags: + {} + # environment: prod + # region: us-east-1 + # Enable if you want the controller to also delete the + # path on efs when deleteing an access point + deleteAccessPointRootDir: false + volMetricsOptIn: false + podAnnotations: {} + resources: + {} + # We usually recommend not to specify default resources and to leave this as a conscious + # choice for the user. This also increases chances charts run on environments with little + # resources, such as Minikube. If you do want to specify resources, uncomment the following + # lines, adjust them as necessary, and remove the curly braces after 'resources:'. + # limits: + # cpu: 100m + # memory: 128Mi + # requests: + # cpu: 100m + # memory: 128Mi + nodeSelector: {} + updateStrategy: {} + tolerations: [] + affinity: {} + # Specifies whether a service account should be created + serviceAccount: + create: true + name: efs-csi-controller-sa + annotations: {} + ## Enable if EKS IAM for SA is used + # eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/efs-csi-role + healthPort: 9909 + regionalStsEndpoints: false +## Node daemonset variables + +node: + # Number for the log level verbosity + logLevel: 2 + hostAliases: + {} + # For cross VPC EFS, you need to poison or overwrite the DNS for the efs volume as per + # https://docs.aws.amazon.com/efs/latest/ug/efs-different-vpc.html#wt6-efs-utils-step3 + # implementing the suggested solution found here: + # https://github.com/kubernetes-sigs/aws-efs-csi-driver/issues/240#issuecomment-676849346 + # EFS Vol ID, IP, Region + # "fs-01234567": + # ip: 10.10.2.2 + # region: us-east-2 + dnsPolicy: ClusterFirst + dnsConfig: + {} + # Example config which uses the AWS nameservers + # dnsPolicy: "None" + # dnsConfig: + # nameservers: + # - 169.254.169.253 + podAnnotations: {} + resources: + {} + # limits: + # cpu: 100m + # memory: 128Mi + # requests: + # cpu: 100m + # memory: 128Mi + nodeSelector: {} + updateStrategy: {} + # Override default strategy (RollingUpdate) to speed up deployment. + # This can be useful if helm timeouts are observed. + # type: OnDelete + tolerations: + - operator: Exists + # Specifies whether a service account should be created + serviceAccount: + create: true + name: efs-csi-node-sa + annotations: {} + ## Enable if EKS IAM for SA is used + # eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/efs-csi-role + healthPort: 9809 + +storageClasses: [] +# Add StorageClass resources like: +# - name: efs-sc +# annotations: +# # Use that annotation if you want this to your default storageclass +# storageclass.kubernetes.io/is-default-class: "true" +# mountOptions: +# - tls +# parameters: +# provisioningMode: efs-ap +# fileSystemId: fs-1122aabb +# directoryPerms: "700" +# gidRangeStart: "1000" +# gidRangeEnd: "2000" +# basePath: "/dynamic_provisioning" +# reclaimPolicy: Delete +# volumeBindingMode: Immediate diff --git a/manifests/aws/efs/kustomization.yaml b/manifests/aws/efs/kustomization.yaml new file mode 100644 index 00000000..f41f5b5f --- /dev/null +++ b/manifests/aws/efs/kustomization.yaml @@ -0,0 +1,17 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: default + +# https://github.com/kubernetes-sigs/aws-efs-csi-driver +helmCharts: +- name: aws-efs-csi-driver + releaseName: aws-efs-csi-driver + repo: https://kubernetes-sigs.github.io/aws-efs-csi-driver/ + version: 2.4.1 + valuesFile: values.yaml + namespace: kube-system + +resources: +- pvc.yaml +- rbac.yaml diff --git a/manifests/aws/efs/pvc.yaml b/manifests/aws/efs/pvc.yaml new file mode 100644 index 00000000..34428067 --- /dev/null +++ b/manifests/aws/efs/pvc.yaml @@ -0,0 +1,11 @@ +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: efs +spec: + accessModes: + - ReadWriteOnce + storageClassName: aws-efs + resources: + requests: + storage: 1000Mi diff --git a/manifests/aws/efs/rbac.yaml b/manifests/aws/efs/rbac.yaml new file mode 100644 index 00000000..c9af882d --- /dev/null +++ b/manifests/aws/efs/rbac.yaml @@ -0,0 +1,100 @@ +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: efs-csi-controller-sa +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: efs-csi-controller-sa +rules: +- apiGroups: [""] + resources: + - endpoints + verbs: + - get + - list + - watch + - create + - update + - patch +- apiGroups: ["coordination.k8s.io" ] + resources: ["leases"] + verbs: + - get + - list + - watch + - create + - update + - patch +--- +kind: RoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: efs-csi-controller-sa-rolebinding +subjects: +- kind: ServiceAccount + name: efs-csi-controller-sa +roleRef: + kind: Role + name: efs-csi-controller-sa + apiGroup: rbac.authorization.k8s.io +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: efs-csi-controller-sa-cluster-role +rules: +- apiGroups: + - "" + resources: + - persistentvolumes + verbs: + - get + - list + - watch + - create + - delete +- apiGroups: + - "" + resources: + - persistentvolumeclaims + verbs: + - get + - list + - watch + - create + - delete +- apiGroups: + - "storage.k8s.io" + resources: + - storageclasses + verbs: + - get + - list + - watch +- apiGroups: + - "" + resources: + - events + verbs: + - get + - list + - create + - delete + - patch + - update + - watch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: efs-csi-controller-sa-cluster-rolebinding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: efs-csi-controller-sa-cluster-role +subjects: +- kind: ServiceAccount + name: efs-csi-controller-sa diff --git a/manifests/aws/efs/values.yaml b/manifests/aws/efs/values.yaml new file mode 100644 index 00000000..7710a5dd --- /dev/null +++ b/manifests/aws/efs/values.yaml @@ -0,0 +1,32 @@ +# https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/charts/aws-efs-csi-driver/values.yaml +replicaCount: 1 + +image: + repository: amazon/aws-efs-csi-driver + tag: "v1.5.4" + +sidecars: + csiProvisioner: + image: + repository: public.ecr.aws/eks-distro/kubernetes-csi/external-provisioner + tag: v3.4.1-eks-1-22-latest + pullPolicy: IfNotPresent + resources: {} + +## Controller deployment variables +controller: + # Number for the log level verbosity + logLevel: 2 + # Specifies whether a service account should be created + serviceAccount: + create: false + name: efs-csi-controller-sa + +## Node daemonset variables +node: + # Number for the log level verbosity + logLevel: 2 + # Specifies whether a service account should be created + serviceAccount: + create: true + name: efs-csi-node-sa diff --git a/manifests/aws/kustomization.yaml b/manifests/aws/kustomization.yaml new file mode 100644 index 00000000..5001d7aa --- /dev/null +++ b/manifests/aws/kustomization.yaml @@ -0,0 +1,7 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: +- ebs +- efs + diff --git a/manifests/kustomization.yaml b/manifests/kustomization.yaml new file mode 100644 index 00000000..a5d83819 --- /dev/null +++ b/manifests/kustomization.yaml @@ -0,0 +1,11 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: +- aws +- networking +- observability +- testground +- tooling +- olm +- operators diff --git a/manifests/networking/kustomization.yaml b/manifests/networking/kustomization.yaml new file mode 100644 index 00000000..585cc271 --- /dev/null +++ b/manifests/networking/kustomization.yaml @@ -0,0 +1,7 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: +- multus +- weave + diff --git a/manifests/networking/multus/kustomization.yaml b/manifests/networking/multus/kustomization.yaml new file mode 100644 index 00000000..292aec6c --- /dev/null +++ b/manifests/networking/multus/kustomization.yaml @@ -0,0 +1,7 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: +- multus-daemonset.yaml +- softlink-cm.yaml +- softlink-ds.yaml diff --git a/manifests/networking/multus/multus-daemonset.yaml b/manifests/networking/multus/multus-daemonset.yaml new file mode 100644 index 00000000..8c301715 --- /dev/null +++ b/manifests/networking/multus/multus-daemonset.yaml @@ -0,0 +1,275 @@ +# Forked from here: +# https://github.com/k8snetworkplumbingwg/multus-cni/blob/f4c0adf54c99d7395d30a050a8d29b674b99d700/deployments/multus-daemonset.yml +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: network-attachment-definitions.k8s.cni.cncf.io +spec: + group: k8s.cni.cncf.io + scope: Namespaced + names: + plural: network-attachment-definitions + singular: network-attachment-definition + kind: NetworkAttachmentDefinition + shortNames: + - net-attach-def + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + description: 'NetworkAttachmentDefinition is a CRD schema specified by the Network Plumbing + Working Group to express the intent for attaching pods to one or more logical or physical + networks. More information available at: https://github.com/k8snetworkplumbingwg/multi-net-spec' + type: object + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this represen + tation of an object. Servers should convert recognized schemas to the + latest internal value, and may reject unrecognized values. More info: + https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: 'NetworkAttachmentDefinition spec defines the desired state of a network attachment' + type: object + properties: + config: + description: 'NetworkAttachmentDefinition config is a JSON-formatted CNI configuration' + type: string +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: multus +rules: + - apiGroups: ["k8s.cni.cncf.io"] + resources: + - '*' + verbs: + - '*' + - apiGroups: + - "" + resources: + - pods + - pods/status + verbs: + - get + - update + - apiGroups: + - "" + - events.k8s.io + resources: + - events + verbs: + - create + - patch + - update +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: multus +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: multus +subjects: +- kind: ServiceAccount + name: multus + namespace: kube-system +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: multus + namespace: kube-system +--- +kind: ConfigMap +apiVersion: v1 +metadata: + name: multus-cni-config + namespace: kube-system + labels: + tier: node + app: multus +data: + cni-conf.json: | + { + "name": "multus-cni-network", + "cniVersion": "0.3.1", + "type": "multus", + "capabilities": { + "portMappings": true + }, + "delegates": [ + { + "cniVersion": "0.3.1", + "name": "aws-cni", + "plugins": [ + { + "name": "aws-cni", + "type": "aws-cni", + "vethPrefix": "eni", + "mtu": "9001", + "pluginLogFile": "/var/log/aws-routed-eni/plugin.log", + "pluginLogLevel": "DEBUG" + }, + { + "type": "portmap", + "capabilities": { + "portMappings": true + }, + "snat": true + } + ] + } + ], + "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig" + } +--- +apiVersion: apps/v1 +kind: DaemonSet +metadata: + labels: + app: multus + name: multus + tier: node + name: kube-multus-ds + namespace: kube-system +spec: + revisionHistoryLimit: 10 + selector: + matchLabels: + name: multus + template: + metadata: + labels: + app: multus + name: multus + tier: node + spec: + containers: + - args: + - -cni-version=0.3.1 + - -cni-config-dir=/host/etc/cni/net.d + - -multus-autoconfig-dir=/host/etc/cni/net.d + - -multus-log-to-stderr=true + - -multus-log-level=verbose + command: + - /bin/bash + - -cex + - | + #!/bin/bash + sed "s|__KUBERNETES_NODE_NAME__|${KUBERNETES_NODE_NAME}|g" /tmp/multus-conf/00-multus.conf.template > /tmp/multus-conf/00-multus.conf + /entrypoint.sh \ + --multus-conf-file=/tmp/multus-conf/00-multus.conf + - /usr/src/multus-cni/bin/multus-daemon + image: ghcr.io/k8snetworkplumbingwg/multus-cni:v3.9.2 + imagePullPolicy: IfNotPresent + name: kube-multus + resources: + limits: + cpu: 250m + memory: 350Mi + requests: + cpu: 100m + memory: 50Mi + securityContext: + privileged: true + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + volumeMounts: + - mountPath: /host/etc/cni/net.d + name: cni + - mountPath: /host/opt/cni/bin + name: cnibin + - mountPath: /tmp/multus-conf/00-multus.conf.template + name: multus-cfg + subPath: "cni-conf.json" + dnsPolicy: ClusterFirst + hostNetwork: true + initContainers: + - command: + - cp + - /usr/src/multus-cni/bin/multus + - /host/opt/cni/bin/multus + image: ghcr.io/k8snetworkplumbingwg/multus-cni:v3.9.2 + imagePullPolicy: IfNotPresent + name: install-multus-binary + resources: + requests: + cpu: 10m + memory: 15Mi + securityContext: + privileged: true + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + volumeMounts: + - mountPath: /host/opt/cni/bin + mountPropagation: Bidirectional + name: cnibin + - args: + - -k8s-service-host=$(KUBERNETES_SERVICE_HOST) + - -k8s-service-port=$(KUBERNETES_SERVICE_PORT) + command: + - /usr/src/multus-cni/bin/generate-kubeconfig + image: ghcr.io/k8snetworkplumbingwg/multus-cni:v3.9.2 + imagePullPolicy: IfNotPresent + name: generate-kubeconfig + resources: + requests: + cpu: 10m + memory: 15Mi + securityContext: + privileged: true + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + volumeMounts: + - mountPath: /host/etc/cni/net.d + mountPropagation: Bidirectional + name: cni + restartPolicy: Always + schedulerName: default-scheduler + securityContext: {} + serviceAccount: multus + serviceAccountName: multus + terminationGracePeriodSeconds: 10 + tolerations: + - effect: NoSchedule + operator: Exists + - effect: NoExecute + operator: Exists + volumes: + - hostPath: + path: /etc/cni/net.d + type: "" + name: cni + - hostPath: + path: /opt/cni/bin + type: "" + name: cnibin + - name: multus-cfg + configMap: + name: multus-cni-config + updateStrategy: + rollingUpdate: + maxSurge: 0 + maxUnavailable: 1 + type: RollingUpdate + +# Notes: +# multus and weave were 'fighting' over eth0; happening because multus writes the weave config inside 00-multus.conf, instead of the primary CNI config. errors like: +# could not create veth pair vethwepl7524981-vethwepg7524981: file exists +# Initially solved by restarting the 'kube-multus-ds' in '-n kube-system', but later introduced a custom configmap and editing the multus daemonset ("cniVersion": "0.3.1") +# so it always includes the primary CNI config (AWS VPC CNI instead of weave) +# Issue referenced here: +# https://github.com/testground/testground/issues/1459 diff --git a/manifests/networking/multus/softlink-cm.yaml b/manifests/networking/multus/softlink-cm.yaml new file mode 100644 index 00000000..fddaaf3b --- /dev/null +++ b/manifests/networking/multus/softlink-cm.yaml @@ -0,0 +1,27 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: softlink-cm + labels: + app: softlink-cm +data: + softlink-cm.sh: | + #!/bin/bash + cat <> /tmp/link_script.sh + sudo mkdir /etc/cni/multus + sudo ln -s /etc/cni/net.d /etc/cni/multus/net.d + sudo rm /etc/cron.d/cron_link + EOT + chmod +x /tmp/link_script.sh + cat <> /etc/cron.d/cron_link + SHELL=/bin/bash + PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin + * * * * * root /tmp/link_script.sh + EOT + +# Notes: +# multus was not linking to weave; pods receiving errors like 'err in GetCNIConfigFromFile: No networks found in /etc/cni/multus/net.d>' +# or \"weave-net\": error in getting result from DelNetwork: invalid version \"\": the version is empty" +# The solution was to create a configmap + daemonset that will execute on the worker nodes, creating a softlink for multus configs +# Issue referenced here: +# https://github.com/testground/testground/issues/1452 diff --git a/manifests/networking/multus/softlink-ds.yaml b/manifests/networking/multus/softlink-ds.yaml new file mode 100644 index 00000000..a513f3c4 --- /dev/null +++ b/manifests/networking/multus/softlink-ds.yaml @@ -0,0 +1,48 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + labels: + k8s-app: softlink-cm + name: softlink-cm + namespace: default +spec: + selector: + matchLabels: + k8s-app: softlink-cm + template: + metadata: + labels: + k8s-app: softlink-cm + spec: + initContainers: + - image: amazonlinux + imagePullPolicy: Always + name: softlink-cm + command: ["/scripts/softlink-cm.sh"] + securityContext: + allowPrivilegeEscalation: true + volumeMounts: + - mountPath: /tmp + name: output + - mountPath: /etc/cron.d + name: cron + - name: softlink-cm + mountPath: /scripts + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + volumes: + - name: output + hostPath: + path: /tmp + type: Directory + - name: cron + hostPath: + path: /etc/cron.d + type: Directory + - name: softlink-cm + configMap: + name: softlink-cm + defaultMode: 0744 + containers: + - image: "gcr.io/google-containers/pause:2.0" + name: pause diff --git a/manifests/networking/weave/clusterrolebinding.yaml b/manifests/networking/weave/clusterrolebinding.yaml new file mode 100644 index 00000000..20afdf94 --- /dev/null +++ b/manifests/networking/weave/clusterrolebinding.yaml @@ -0,0 +1,69 @@ +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: service-reader +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: service-reader +subjects: +- kind: ServiceAccount + name: default + namespace: default + +--- +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + annotations: + kubernetes.io/description: "Allow net sysctls for Testground test pods." + seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*' + labels: + eks.amazonaws.com/component: pod-security-policy + kubernetes.io/cluster-service: "true" + name: allowunsafesysctls +spec: + allowedUnsafeSysctls: + - net.core.somaxconn + allowedCapabilities: + - '*' + allowPrivilegeEscalation: true + privileged: true + fsGroup: + rule: RunAsAny + runAsUser: + rule: RunAsAny + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + volumes: + - '*' + +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + namespace: default + name: service-reader +rules: +- apiGroups: ["*"] # "" indicates the core API group + resources: ["*"] + verbs: ["*"] + +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: allow-sysctls + namespace: default +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: service-reader +subjects: +- apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + namespace: default diff --git a/manifests/networking/weave/drop-weave-cm.yaml b/manifests/networking/weave/drop-weave-cm.yaml new file mode 100644 index 00000000..df75746b --- /dev/null +++ b/manifests/networking/weave/drop-weave-cm.yaml @@ -0,0 +1,29 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: entrypoint + labels: + app: drop-weave +data: + entrypoint.sh: | + #!/bin/bash + cat <> /tmp/script.sh + sudo iptables -D FORWARD -o weave -j DROP &> /tmp/info.txt + sudo iptables -t nat -I POSTROUTING 2 -m comment --comment "AWS SNAT CHAIN" ! -o weave -j AWS-SNAT-CHAIN-0 &> /tmp/info.txt + sudo iptables -t nat -D POSTROUTING -m comment --comment "AWS SNAT CHAIN" -j AWS-SNAT-CHAIN-0 &> /tmp/info.txt + sudo rm /etc/cron.d/cron + EOT + chmod +x /tmp/script.sh + cat <> /etc/cron.d/cron + SHELL=/bin/bash + PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin + * * * * * root /tmp/script.sh + EOT + +# Notes: +# ping between weave interfaces wasn't working; ARP resolution works, ICMP doesn't +# The solution was to DROP the above iptables rules; easiest way to achieve this was to deploy a configmap and a daemonset that would +# pull from this CM and execute the script +# Issues referenced here: +# https://github.com/testground/testground/issues/1455 +# https://github.com/testground/testground/issues/1456 diff --git a/manifests/networking/weave/drop-weave-ds.yaml b/manifests/networking/weave/drop-weave-ds.yaml new file mode 100644 index 00000000..b9c2b60b --- /dev/null +++ b/manifests/networking/weave/drop-weave-ds.yaml @@ -0,0 +1,48 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + labels: + k8s-app: drop-weave-iptables-rule + name: drop-weave-iptables-rule + namespace: default +spec: + selector: + matchLabels: + k8s-app: drop-weave-iptables-rule + template: + metadata: + labels: + k8s-app: drop-weave-iptables-rule + spec: + initContainers: + - image: amazonlinux + imagePullPolicy: Always + name: drop-iptables + command: ["/scripts/entrypoint.sh"] + securityContext: + allowPrivilegeEscalation: true + volumeMounts: + - mountPath: /tmp + name: output + - mountPath: /etc/cron.d + name: cron + - name: entrypoint + mountPath: /scripts + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + volumes: + - name: output + hostPath: + path: /tmp + type: Directory + - name: cron + hostPath: + path: /etc/cron.d + type: Directory + - name: entrypoint + configMap: + name: entrypoint + defaultMode: 0744 + containers: + - image: "gcr.io/google-containers/pause:2.0" + name: pause diff --git a/manifests/networking/weave/kustomization.yaml b/manifests/networking/weave/kustomization.yaml new file mode 100644 index 00000000..5dc7cf7c --- /dev/null +++ b/manifests/networking/weave/kustomization.yaml @@ -0,0 +1,11 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: +- weave.yaml +- drop-weave-cm.yaml +- drop-weave-ds.yaml +- networkattachmentdefinition.yaml +- clusterrolebinding.yaml +#- multi-weave.yaml +#- https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s-1.11.yaml diff --git a/manifests/networking/weave/networkattachmentdefinition.yaml b/manifests/networking/weave/networkattachmentdefinition.yaml new file mode 100644 index 00000000..bf7ae422 --- /dev/null +++ b/manifests/networking/weave/networkattachmentdefinition.yaml @@ -0,0 +1,4 @@ +apiVersion: "k8s.cni.cncf.io/v1" +kind: NetworkAttachmentDefinition +metadata: + name: weave diff --git a/manifests/networking/weave/weave.yaml b/manifests/networking/weave/weave.yaml new file mode 100644 index 00000000..73a73e13 --- /dev/null +++ b/manifests/networking/weave/weave.yaml @@ -0,0 +1,234 @@ +apiVersion: v1 +kind: List +items: + - apiVersion: v1 + kind: ServiceAccount + metadata: + name: weave-net + labels: + name: weave-net + namespace: kube-system + - apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: weave-net + labels: + name: weave-net + rules: + - apiGroups: + - '' + resources: + - pods + - namespaces + - nodes + verbs: + - get + - list + - watch + - apiGroups: + - extensions + resources: + - networkpolicies + verbs: + - get + - list + - watch + - apiGroups: + - 'networking.k8s.io' + resources: + - networkpolicies + verbs: + - get + - list + - watch + - apiGroups: + - '' + resources: + - nodes/status + verbs: + - patch + - update + - apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: weave-net + labels: + name: weave-net + roleRef: + kind: ClusterRole + name: weave-net + apiGroup: rbac.authorization.k8s.io + subjects: + - kind: ServiceAccount + name: weave-net + namespace: kube-system + - apiVersion: rbac.authorization.k8s.io/v1 + kind: Role + metadata: + name: weave-net + namespace: kube-system + labels: + name: weave-net + rules: + - apiGroups: + - '' + resources: + - configmaps + resourceNames: + - weave-net + verbs: + - get + - update + - apiGroups: + - '' + resources: + - configmaps + verbs: + - create + - apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: weave-net + namespace: kube-system + labels: + name: weave-net + roleRef: + kind: Role + name: weave-net + apiGroup: rbac.authorization.k8s.io + subjects: + - kind: ServiceAccount + name: weave-net + namespace: kube-system + - apiVersion: apps/v1 + kind: DaemonSet + metadata: + name: weave-net + labels: + name: weave-net + namespace: kube-system + spec: + # Wait 5 seconds to let pod connect before rolling next pod + selector: + matchLabels: + name: weave-net + minReadySeconds: 5 + template: + metadata: + labels: + name: weave-net + spec: + initContainers: + - name: weave-init + image: 'weaveworks/weave-kube:2.8.1' + command: + - /home/weave/init.sh + env: + securityContext: + privileged: true + volumeMounts: + - name: cni-bin + mountPath: /host/opt + - name: cni-bin2 + mountPath: /host/home + - name: cni-conf + mountPath: /host/etc + - name: lib-modules + mountPath: /lib/modules + - name: xtables-lock + mountPath: /run/xtables.lock + readOnly: false + containers: + - name: weave + command: + - /home/weave/launch.sh + env: + - name: INIT_CONTAINER + value: "true" + - name: HOSTNAME + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: spec.nodeName + image: 'weaveworks/weave-kube:2.8.1' + readinessProbe: + httpGet: + host: 127.0.0.1 + path: /status + port: 6784 + resources: + requests: + cpu: 50m + securityContext: + privileged: true + volumeMounts: + - name: weavedb + mountPath: /weavedb + - name: dbus + mountPath: /host/var/lib/dbus + readOnly: true + - mountPath: /host/etc/machine-id + name: cni-machine-id + readOnly: true + - name: xtables-lock + mountPath: /run/xtables.lock + readOnly: false + - name: weave-npc + env: + - name: HOSTNAME + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: spec.nodeName + image: 'weaveworks/weave-npc:2.8.1' +#npc-args + resources: + requests: + cpu: 50m + securityContext: + privileged: true + volumeMounts: + - name: xtables-lock + mountPath: /run/xtables.lock + readOnly: false + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + hostPID: false + restartPolicy: Always + securityContext: + seLinuxOptions: {} + serviceAccountName: weave-net + tolerations: + - effect: NoSchedule + operator: Exists + - effect: NoExecute + operator: Exists + volumes: + - name: weavedb + hostPath: + path: /var/lib/weave + - name: cni-bin + hostPath: + path: /opt + - name: cni-bin2 + hostPath: + path: /home + - name: cni-conf + hostPath: + path: /etc + - name: cni-machine-id + hostPath: + path: /etc/machine-id + - name: dbus + hostPath: + path: /var/lib/dbus + - name: lib-modules + hostPath: + path: /lib/modules + - name: xtables-lock + hostPath: + path: /run/xtables.lock + type: FileOrCreate + priorityClassName: system-node-critical + updateStrategy: + type: RollingUpdate diff --git a/manifests/observability/grafana-agent-operator/kustomization.yaml b/manifests/observability/grafana-agent-operator/kustomization.yaml new file mode 100644 index 00000000..7f1a6945 --- /dev/null +++ b/manifests/observability/grafana-agent-operator/kustomization.yaml @@ -0,0 +1,14 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: default + +# https://github.com/grafana-operator/grafana-operator +helmCharts: +- name: grafana-agent-operator + releaseName: grafana-agent-operator + repo: https://grafana.github.io/helm-charts + version: "0.2.12" + +resources: + - https://raw.githubusercontent.com/grafana-operator/grafana-operator/master/deploy/kustomize/base/crds.yaml diff --git a/manifests/observability/grafana/base/grafana.yaml b/manifests/observability/grafana/base/grafana.yaml new file mode 100644 index 00000000..3e93e407 --- /dev/null +++ b/manifests/observability/grafana/base/grafana.yaml @@ -0,0 +1,18 @@ +apiVersion: integreatly.org/v1alpha1 +kind: Grafana +metadata: + name: grafana +spec: + config: + auth.anonymous: + enabled: True + dashboardLabelSelector: + - matchExpressions: + - { key: app, operator: In, values: [ grafana ] } + resources: + # Optionally specify container resources + requests: + cpu: 800m + memory: 400Mi + limits: + memory: 800Mi diff --git a/manifests/observability/grafana/base/kustomization.yaml b/manifests/observability/grafana/base/kustomization.yaml new file mode 100644 index 00000000..f82d7535 --- /dev/null +++ b/manifests/observability/grafana/base/kustomization.yaml @@ -0,0 +1,10 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: default + +resources: +- monitoring +- grafana.yaml +- prometheus_datasource.yaml +- loki_datasource.yaml diff --git a/manifests/observability/grafana/base/loki_datasource.yaml b/manifests/observability/grafana/base/loki_datasource.yaml new file mode 100644 index 00000000..1cf32095 --- /dev/null +++ b/manifests/observability/grafana/base/loki_datasource.yaml @@ -0,0 +1,17 @@ +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDataSource +metadata: + name: loki-k8s +spec: + name: loki-k8s + datasources: + - name: Loki + type: loki + access: proxy + url: http://loki-headless.default.svc.cluster.local:3100 + isDefault: false + version: 1 + editable: true + jsonData: + tlsSkipVerify: true + timeInterval: "5s" diff --git a/manifests/observability/grafana/base/monitoring/README.md b/manifests/observability/grafana/base/monitoring/README.md new file mode 100644 index 00000000..e3aaa930 --- /dev/null +++ b/manifests/observability/grafana/base/monitoring/README.md @@ -0,0 +1 @@ +The `dashboard.sh` script creates `GrafanaDashboard` resources from [kube-prometheus](https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/grafana-dashboardDefinitions.yaml) default dashboards. diff --git a/manifests/observability/grafana/base/monitoring/dashboards.sh b/manifests/observability/grafana/base/monitoring/dashboards.sh new file mode 100755 index 00000000..07f00e53 --- /dev/null +++ b/manifests/observability/grafana/base/monitoring/dashboards.sh @@ -0,0 +1,31 @@ +#!/bin/bash + +cd $(dirname "$0") + +mkdir -p tmp +curl -o tmp/dashboards.yaml https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/grafana-dashboardDefinitions.yaml + +names=$(yq '.items[].metadata.name' tmp/dashboards.yaml) +files=$(yq '.items[].data | keys | .[]' tmp/dashboards.yaml) + +readarray -t arr_name <<<"$names" +readarray -t arr_file <<<"$files" + +size=${#arr_name[@]} + +echo "" > dashboards.yaml + +for (( i=0; i<=$size-1; i++ )) +do + echo $i + name=${arr_name[$i]} + file=${arr_file[$i]} + echo $name + echo $file + + export name file + + cat dashboards.yaml.tmpl | envsubst >> dashboards.yaml +done + +rm -r tmp diff --git a/manifests/observability/grafana/base/monitoring/dashboards.yaml b/manifests/observability/grafana/base/monitoring/dashboards.yaml new file mode 100644 index 00000000..4f462b0e --- /dev/null +++ b/manifests/observability/grafana/base/monitoring/dashboards.yaml @@ -0,0 +1,301 @@ + +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-alertmanager-overview + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-alertmanager-overview + key: alertmanager-overview.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-apiserver + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-apiserver + key: apiserver.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-cluster-total + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-cluster-total + key: cluster-total.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-controller-manager + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-controller-manager + key: controller-manager.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-grafana-overview + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-grafana-overview + key: grafana-overview.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-k8s-resources-cluster + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-k8s-resources-cluster + key: k8s-resources-cluster.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-k8s-resources-namespace + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-k8s-resources-namespace + key: k8s-resources-namespace.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-k8s-resources-node + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-k8s-resources-node + key: k8s-resources-node.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-k8s-resources-pod + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-k8s-resources-pod + key: k8s-resources-pod.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-k8s-resources-workload + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-k8s-resources-workload + key: k8s-resources-workload.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-k8s-resources-workloads-namespace + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-k8s-resources-workloads-namespace + key: k8s-resources-workloads-namespace.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-kubelet + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-kubelet + key: kubelet.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-namespace-by-pod + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-namespace-by-pod + key: namespace-by-pod.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-namespace-by-workload + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-namespace-by-workload + key: namespace-by-workload.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-node-cluster-rsrc-use + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-node-cluster-rsrc-use + key: node-cluster-rsrc-use.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-node-rsrc-use + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-node-rsrc-use + key: node-rsrc-use.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-nodes-darwin + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-nodes-darwin + key: nodes-darwin.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-nodes + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-nodes + key: nodes.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-persistentvolumesusage + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-persistentvolumesusage + key: persistentvolumesusage.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-pod-total + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-pod-total + key: pod-total.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-prometheus-remote-write + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-prometheus-remote-write + key: prometheus-remote-write.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-prometheus + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-prometheus + key: prometheus.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-proxy + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-proxy + key: proxy.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-scheduler + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-scheduler + key: scheduler.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: grafana-dashboard-workload-total + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: grafana-dashboard-workload-total + key: workload-total.json diff --git a/manifests/observability/grafana/base/monitoring/dashboards.yaml.tmpl b/manifests/observability/grafana/base/monitoring/dashboards.yaml.tmpl new file mode 100644 index 00000000..6d52ab54 --- /dev/null +++ b/manifests/observability/grafana/base/monitoring/dashboards.yaml.tmpl @@ -0,0 +1,12 @@ +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: ${name} + labels: + app: grafana +spec: + customFolderName: default + configMapRef: + name: ${name} + key: ${file} diff --git a/manifests/observability/grafana/base/monitoring/kustomization.yaml b/manifests/observability/grafana/base/monitoring/kustomization.yaml new file mode 100644 index 00000000..66904bb1 --- /dev/null +++ b/manifests/observability/grafana/base/monitoring/kustomization.yaml @@ -0,0 +1,9 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: +- https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/v0.12.0/manifests/grafana-dashboardDefinitions.yaml +- dashboards.yaml + +configurations: +- kustomizeconfig.yaml diff --git a/manifests/observability/grafana/base/monitoring/kustomizeconfig.yaml b/manifests/observability/grafana/base/monitoring/kustomizeconfig.yaml new file mode 100644 index 00000000..55e98ff8 --- /dev/null +++ b/manifests/observability/grafana/base/monitoring/kustomizeconfig.yaml @@ -0,0 +1,5 @@ +nameReference: +- kind: ConfigMap + fieldSpecs: + - path: spec/configMapRef/name + kind: GrafanaDashboard diff --git a/manifests/observability/grafana/base/prometheus_datasource.yaml b/manifests/observability/grafana/base/prometheus_datasource.yaml new file mode 100644 index 00000000..e87e5f0e --- /dev/null +++ b/manifests/observability/grafana/base/prometheus_datasource.yaml @@ -0,0 +1,17 @@ +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDataSource +metadata: + name: prometheus-k8s +spec: + name: prometheus-k8s + datasources: + - name: Prometheus + type: prometheus + access: proxy + url: http://prometheus-k8s.default.svc.cluster.local:9090 + isDefault: true + version: 1 + editable: true + jsonData: + tlsSkipVerify: true + timeInterval: "5s" diff --git a/manifests/observability/grafana/dashboards.yaml b/manifests/observability/grafana/dashboards.yaml new file mode 100644 index 00000000..18e2e396 --- /dev/null +++ b/manifests/observability/grafana/dashboards.yaml @@ -0,0 +1,24 @@ +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: testground-base-monitoring + labels: + app: grafana +spec: + customFolderName: TestGround + configMapRef: + name: testground + key: testground-base-monitoring.json +--- +apiVersion: integreatly.org/v1alpha1 +kind: GrafanaDashboard +metadata: + name: testground-base + labels: + app: grafana +spec: + customFolderName: TestGround + configMapRef: + name: testground + key: testground-base.json diff --git a/manifests/observability/grafana/dashboards/base-logs.json b/manifests/observability/grafana/dashboards/base-logs.json new file mode 100644 index 00000000..e22aacd8 --- /dev/null +++ b/manifests/observability/grafana/dashboards/base-logs.json @@ -0,0 +1,652 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": "-- Grafana --", + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "gnetId": null, + "graphTooltip": 0, + "id": 38, + "links": [], + "panels": [ + { + "aliasColors": {}, + "bars": false, + "dashLength": 10, + "dashes": false, + "datasource": "Loki", + "description": "Number of messages in the namespace", + "fieldConfig": { + "defaults": { + "unit": "short" + }, + "overrides": [] + }, + "fill": 1, + "fillGradient": 0, + "gridPos": { + "h": 9, + "w": 24, + "x": 0, + "y": 0 + }, + "hiddenSeries": false, + "id": 16, + "legend": { + "alignAsTable": false, + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "nullPointMode": "null", + "options": { + "alertThreshold": true + }, + "percentage": false, + "pluginVersion": "7.5.17", + "pointradius": 2, + "points": false, + "renderer": "flot", + "seriesOverrides": [], + "spaceLength": 10, + "stack": false, + "steppedLine": false, + "targets": [ + { + "expr": "rate({namespace=\"testground-base\"}[$__interval])", + "refId": "A" + } + ], + "thresholds": [], + "timeFrom": null, + "timeRegions": [], + "timeShift": null, + "title": "testground-base - NameSpace - Number of msg", + "tooltip": { + "shared": true, + "sort": 0, + "value_type": "individual" + }, + "type": "graph", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "$$hashKey": "object:283", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + }, + { + "$$hashKey": "object:284", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + } + ], + "yaxis": { + "align": false, + "alignLevel": null + } + }, + { + "datasource": null, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 9 + }, + "id": 13, + "title": "testground - Consensus", + "type": "row" + }, + { + "datasource": "Loki", + "description": "", + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 24, + "x": 0, + "y": 10 + }, + "id": 2, + "options": { + "dedupStrategy": "none", + "showLabels": false, + "showTime": false, + "sortOrder": "Descending", + "wrapLogMessage": false + }, + "pluginVersion": "7.5.17", + "targets": [ + { + "expr": "{namespace=~\"testground-base\"}", + "refId": "A" + } + ], + "timeFrom": null, + "timeShift": null, + "title": "testground-base-consensus - NameSpace", + "type": "logs" + }, + { + "datasource": "Loki", + "description": "", + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 17 + }, + "id": 14, + "options": { + "dedupStrategy": "none", + "showLabels": false, + "showTime": false, + "sortOrder": "Descending", + "wrapLogMessage": false + }, + "pluginVersion": "7.5.17", + "targets": [ + { + "expr": "{app=~\"consensus-full-*.\",namespace=\"testground-base\"}", + "legendFormat": "", + "refId": "A" + } + ], + "timeFrom": null, + "timeShift": null, + "title": "testground-base-consensus - Full", + "type": "logs" + }, + { + "datasource": "Loki", + "description": "", + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 17 + }, + "id": 15, + "options": { + "dedupStrategy": "none", + "showLabels": false, + "showTime": false, + "sortOrder": "Descending", + "wrapLogMessage": false + }, + "pluginVersion": "7.5.17", + "targets": [ + { + "expr": "{app=~\"consensus-validator-*.\",namespace=\"testground-base\"}", + "legendFormat": "", + "refId": "A" + } + ], + "timeFrom": null, + "timeShift": null, + "title": "testground-base-consensus - Validator", + "type": "logs" + }, + { + "collapsed": false, + "datasource": null, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 27 + }, + "id": 11, + "panels": [], + "title": "testground - Data Availability", + "type": "row" + }, + { + "aliasColors": {}, + "bars": false, + "dashLength": 10, + "dashes": false, + "datasource": "Loki", + "description": "Number of messages in the namespace", + "fieldConfig": { + "defaults": { + "unit": "short" + }, + "overrides": [] + }, + "fill": 1, + "fillGradient": 0, + "gridPos": { + "h": 9, + "w": 24, + "x": 0, + "y": 28 + }, + "hiddenSeries": false, + "id": 17, + "legend": { + "alignAsTable": false, + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "nullPointMode": "null", + "options": { + "alertThreshold": true + }, + "percentage": false, + "pluginVersion": "7.5.17", + "pointradius": 2, + "points": false, + "renderer": "flot", + "seriesOverrides": [], + "spaceLength": 10, + "stack": false, + "steppedLine": false, + "targets": [ + { + "expr": "rate({namespace=\"testground-base\"}[$__interval])", + "refId": "A" + } + ], + "thresholds": [], + "timeFrom": null, + "timeRegions": [], + "timeShift": null, + "title": "testground-base-da - NameSpace - Number of msg", + "tooltip": { + "shared": true, + "sort": 0, + "value_type": "individual" + }, + "type": "graph", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "$$hashKey": "object:283", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + }, + { + "$$hashKey": "object:284", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + } + ], + "yaxis": { + "align": false, + "alignLevel": null + } + }, + { + "datasource": "Loki", + "description": "", + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 21, + "x": 0, + "y": 37 + }, + "id": 24, + "options": { + "dedupStrategy": "none", + "showLabels": false, + "showTime": false, + "sortOrder": "Descending", + "wrapLogMessage": false + }, + "pluginVersion": "7.5.17", + "targets": [ + { + "expr": "{namespace=\"testground-base\"}|=\"ERROR\"", + "legendFormat": "", + "refId": "A" + } + ], + "timeFrom": null, + "timeShift": null, + "title": "testground-base-da - Errors in the Namespace", + "type": "logs" + }, + { + "datasource": "Loki", + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 3, + "x": 21, + "y": 37 + }, + "id": 6, + "options": { + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "text": {} + }, + "pluginVersion": "7.5.17", + "targets": [ + { + "expr": "sum(count_over_time({namespace=\"testground-base\"}|=\"ERROR\"[5m]))", + "legendFormat": "", + "refId": "A" + } + ], + "timeFrom": null, + "timeShift": null, + "title": "testground-base-da - Errors in the Namespace", + "type": "gauge" + }, + { + "datasource": "Loki", + "description": "", + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 24, + "x": 0, + "y": 43 + }, + "id": 3, + "options": { + "dedupStrategy": "none", + "showLabels": false, + "showTime": false, + "sortOrder": "Descending", + "wrapLogMessage": false + }, + "pluginVersion": "7.5.17", + "targets": [ + { + "expr": "{namespace=\"testground-base\"}", + "legendFormat": "", + "refId": "A" + } + ], + "timeFrom": null, + "timeShift": null, + "title": "testground-base-da - NameSpace", + "type": "logs" + }, + { + "datasource": "Loki", + "description": "", + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 52 + }, + "id": 20, + "options": { + "dedupStrategy": "none", + "showLabels": false, + "showTime": false, + "sortOrder": "Descending", + "wrapLogMessage": false + }, + "pluginVersion": "7.5.17", + "targets": [ + { + "expr": "{app=~\"da-bridge-1\",namespace=\"testground-base\"}", + "legendFormat": "", + "refId": "A" + } + ], + "timeFrom": null, + "timeShift": null, + "title": "testground-base-da - Bridge-1", + "type": "logs" + }, + { + "datasource": "Loki", + "description": "", + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 52 + }, + "id": 7, + "options": { + "dedupStrategy": "none", + "showLabels": false, + "showTime": false, + "sortOrder": "Descending", + "wrapLogMessage": false + }, + "pluginVersion": "7.5.17", + "targets": [ + { + "expr": "{app=~\"da-bridge-2\",namespace=\"testground-base\"}", + "legendFormat": "", + "refId": "A" + } + ], + "timeFrom": null, + "timeShift": null, + "title": "testground-base-da - Bridge-2", + "type": "logs" + }, + { + "datasource": "Loki", + "description": "", + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 0, + "y": 60 + }, + "id": 18, + "options": { + "dedupStrategy": "none", + "showLabels": false, + "showTime": false, + "sortOrder": "Descending", + "wrapLogMessage": false + }, + "pluginVersion": "7.5.17", + "targets": [ + { + "expr": "{app=~\"da-full-1\",namespace=\"testground-base\"}", + "legendFormat": "", + "refId": "A" + } + ], + "timeFrom": null, + "timeShift": null, + "title": "testground-base-da - Full-1", + "type": "logs" + }, + { + "datasource": "Loki", + "description": "", + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 60 + }, + "id": 19, + "options": { + "dedupStrategy": "none", + "showLabels": false, + "showTime": false, + "sortOrder": "Descending", + "wrapLogMessage": false + }, + "pluginVersion": "7.5.17", + "targets": [ + { + "expr": "{app=\"da-full\",namespace=\"testground-base\"}", + "legendFormat": "", + "refId": "A" + } + ], + "timeFrom": null, + "timeShift": null, + "title": "testground-base-da - Full-2", + "type": "logs" + }, + { + "datasource": "Loki", + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 11, + "w": 24, + "x": 0, + "y": 67 + }, + "id": 22, + "options": { + "dedupStrategy": "none", + "showLabels": false, + "showTime": false, + "sortOrder": "Descending", + "wrapLogMessage": false + }, + "pluginVersion": "7.5.17", + "targets": [ + { + "expr": "{app=\"external-dns\",namespace=\"testground-base\"}", + "refId": "A" + } + ], + "title": "external-dns", + "type": "logs" + } + ], + "refresh": "10s", + "schemaVersion": 27, + "style": "dark", + "tags": [], + "templating": { + "list": [] + }, + "time": { + "from": "now-5m", + "to": "now" + }, + "timepicker": {}, + "timezone": "", + "title": "Logs: testground-base", + "uid": "testground-base", + "version": 5 +} diff --git a/manifests/observability/grafana/dashboards/base-monitoring.json b/manifests/observability/grafana/dashboards/base-monitoring.json new file mode 100644 index 00000000..5d88eb16 --- /dev/null +++ b/manifests/observability/grafana/dashboards/base-monitoring.json @@ -0,0 +1,812 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": "-- Grafana --", + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "gnetId": null, + "graphTooltip": 0, + "id": 32, + "links": [], + "panels": [ + { + "aliasColors": {}, + "bars": false, + "dashLength": 10, + "dashes": false, + "datasource": null, + "fieldConfig": { + "defaults": { + "unit": "short" + }, + "overrides": [] + }, + "fill": 1, + "fillGradient": 0, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 0 + }, + "hiddenSeries": false, + "id": 8, + "legend": { + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "nullPointMode": "null", + "options": { + "alertThreshold": true + }, + "percentage": false, + "pluginVersion": "7.5.17", + "pointradius": 2, + "points": false, + "renderer": "flot", + "seriesOverrides": [], + "spaceLength": 10, + "stack": false, + "steppedLine": false, + "targets": [ + { + "exemplar": true, + "expr": "count by (pod)(rate(container_cpu_user_seconds_total{\n namespace=\"testground-base\"\n}[5m]))", + "interval": "", + "legendFormat": "", + "refId": "A" + } + ], + "thresholds": [], + "timeFrom": null, + "timeRegions": [], + "timeShift": null, + "title": "CPU usage: namespace: testground-base", + "tooltip": { + "shared": true, + "sort": 0, + "value_type": "individual" + }, + "type": "graph", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "$$hashKey": "object:139", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + }, + { + "$$hashKey": "object:140", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + } + ], + "yaxis": { + "align": false, + "alignLevel": null + } + }, + { + "aliasColors": {}, + "bars": false, + "dashLength": 10, + "dashes": false, + "datasource": null, + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "fill": 1, + "fillGradient": 0, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 0 + }, + "hiddenSeries": false, + "id": 11, + "legend": { + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "nullPointMode": "null", + "options": { + "alertThreshold": true + }, + "percentage": false, + "pluginVersion": "7.5.17", + "pointradius": 2, + "points": false, + "renderer": "flot", + "seriesOverrides": [], + "spaceLength": 10, + "stack": false, + "steppedLine": false, + "targets": [ + { + "exemplar": true, + "expr": "round(max by (pod)(max_over_time(container_memory_usage_bytes{\n namespace=\"testground-base\",\n}[5m]))/ on (pod) (max by (pod) (kube_pod_container_resource_limits)) * 100,0.01)", + "interval": "", + "legendFormat": "", + "refId": "A" + } + ], + "thresholds": [], + "timeFrom": null, + "timeRegions": [], + "timeShift": null, + "title": "Memory usage: namespace: testground-base", + "tooltip": { + "shared": true, + "sort": 0, + "value_type": "individual" + }, + "type": "graph", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "$$hashKey": "object:139", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + }, + { + "$$hashKey": "object:140", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + } + ], + "yaxis": { + "align": false, + "alignLevel": null + } + }, + { + "aliasColors": {}, + "bars": false, + "dashLength": 10, + "dashes": false, + "datasource": null, + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "fill": 1, + "fillGradient": 0, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 12 + }, + "hiddenSeries": false, + "id": 5, + "legend": { + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "nullPointMode": "null", + "options": { + "alertThreshold": true + }, + "percentage": false, + "pluginVersion": "7.5.17", + "pointradius": 2, + "points": false, + "renderer": "flot", + "seriesOverrides": [], + "spaceLength": 10, + "stack": false, + "steppedLine": false, + "targets": [ + { + "exemplar": true, + "expr": "rate(container_cpu_user_seconds_total{\n namespace=\"testground-base\",\n pod=~\"consensus.+\",\n container=\"consensus\",\n}[5m])", + "interval": "", + "legendFormat": "", + "refId": "A" + } + ], + "thresholds": [], + "timeFrom": null, + "timeRegions": [], + "timeShift": null, + "title": "CPU usage in Consensus", + "tooltip": { + "shared": true, + "sort": 0, + "value_type": "individual" + }, + "type": "graph", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "$$hashKey": "object:139", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + }, + { + "$$hashKey": "object:140", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + } + ], + "yaxis": { + "align": false, + "alignLevel": null + } + }, + { + "aliasColors": {}, + "bars": false, + "dashLength": 10, + "dashes": false, + "datasource": null, + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "fill": 1, + "fillGradient": 0, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 12 + }, + "hiddenSeries": false, + "id": 7, + "legend": { + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "nullPointMode": "null", + "options": { + "alertThreshold": true + }, + "percentage": false, + "pluginVersion": "7.5.17", + "pointradius": 2, + "points": false, + "renderer": "flot", + "seriesOverrides": [], + "spaceLength": 10, + "stack": false, + "steppedLine": false, + "targets": [ + { + "exemplar": true, + "expr": "rate(container_cpu_user_seconds_total{\n namespace=\"testground-base\",\n pod=~\"consensus.+\",\n container=\"consensus\",\n}[5m])", + "interval": "", + "legendFormat": "", + "refId": "A" + } + ], + "thresholds": [], + "timeFrom": null, + "timeRegions": [], + "timeShift": null, + "title": "CPU usage in Consensus", + "tooltip": { + "shared": true, + "sort": 0, + "value_type": "individual" + }, + "type": "graph", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "$$hashKey": "object:139", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + }, + { + "$$hashKey": "object:140", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + } + ], + "yaxis": { + "align": false, + "alignLevel": null + } + }, + { + "aliasColors": {}, + "bars": false, + "dashLength": 10, + "dashes": false, + "datasource": null, + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "fill": 1, + "fillGradient": 0, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 20 + }, + "hiddenSeries": false, + "id": 6, + "legend": { + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "nullPointMode": "null", + "options": { + "alertThreshold": true + }, + "percentage": false, + "pluginVersion": "7.5.17", + "pointradius": 2, + "points": false, + "renderer": "flot", + "seriesOverrides": [], + "spaceLength": 10, + "stack": false, + "steppedLine": false, + "targets": [ + { + "exemplar": true, + "expr": "round(max by (pod)(max_over_time(container_memory_usage_bytes{\n namespace=\"testground-base\",\n pod=~\"consensus.+\",\n container=\"consensus\",\n}[5m]))/ on (pod) (max by (pod) (kube_pod_container_resource_limits)) * 100,0.01)", + "interval": "", + "legendFormat": "", + "refId": "A" + } + ], + "thresholds": [], + "timeFrom": null, + "timeRegions": [], + "timeShift": null, + "title": "Memory usage in Consensus", + "tooltip": { + "shared": true, + "sort": 0, + "value_type": "individual" + }, + "type": "graph", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "$$hashKey": "object:139", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + }, + { + "$$hashKey": "object:140", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + } + ], + "yaxis": { + "align": false, + "alignLevel": null + } + }, + { + "aliasColors": {}, + "bars": false, + "dashLength": 10, + "dashes": false, + "datasource": null, + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "fill": 1, + "fillGradient": 0, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 20 + }, + "hiddenSeries": false, + "id": 10, + "legend": { + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "nullPointMode": "null", + "options": { + "alertThreshold": true + }, + "percentage": false, + "pluginVersion": "7.5.17", + "pointradius": 2, + "points": false, + "renderer": "flot", + "seriesOverrides": [], + "spaceLength": 10, + "stack": false, + "steppedLine": false, + "targets": [ + { + "exemplar": true, + "expr": "round(max by (pod)(max_over_time(container_memory_usage_bytes{\n namespace=\"testground-base\",\n pod=~\"da.+\",\n container=\"da\",\n}[5m]))/ on (pod) (max by (pod) (kube_pod_container_resource_limits)) * 100,0.01)", + "interval": "", + "legendFormat": "", + "refId": "A" + } + ], + "thresholds": [], + "timeFrom": null, + "timeRegions": [], + "timeShift": null, + "title": "Memory usage in DA", + "tooltip": { + "shared": true, + "sort": 0, + "value_type": "individual" + }, + "type": "graph", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "$$hashKey": "object:139", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + }, + { + "$$hashKey": "object:140", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + } + ], + "yaxis": { + "align": false, + "alignLevel": null + } + }, + { + "aliasColors": {}, + "bars": false, + "dashLength": 10, + "dashes": false, + "datasource": null, + "fieldConfig": { + "defaults": { + "unit": "short" + }, + "overrides": [] + }, + "fill": 1, + "fillGradient": 0, + "gridPos": { + "h": 14, + "w": 12, + "x": 0, + "y": 30 + }, + "hiddenSeries": false, + "id": 4, + "legend": { + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "nullPointMode": "null", + "options": { + "alertThreshold": true + }, + "percentage": false, + "pluginVersion": "7.5.17", + "pointradius": 2, + "points": false, + "renderer": "flot", + "seriesOverrides": [], + "spaceLength": 10, + "stack": false, + "steppedLine": false, + "targets": [ + { + "exemplar": true, + "expr": "100 * (\n instance:node_memory_utilisation:ratio\n)", + "interval": "", + "legendFormat": "", + "refId": "A" + } + ], + "thresholds": [], + "timeFrom": null, + "timeRegions": [], + "timeShift": null, + "title": "K8S Memory Nodes Usage", + "tooltip": { + "shared": true, + "sort": 0, + "value_type": "individual" + }, + "type": "graph", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "$$hashKey": "object:139", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + }, + { + "$$hashKey": "object:140", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + } + ], + "yaxis": { + "align": false, + "alignLevel": null + } + }, + { + "aliasColors": {}, + "bars": false, + "dashLength": 10, + "dashes": false, + "datasource": null, + "fieldConfig": { + "defaults": { + "unit": "short" + }, + "overrides": [] + }, + "fill": 1, + "fillGradient": 0, + "gridPos": { + "h": 14, + "w": 12, + "x": 12, + "y": 30 + }, + "hiddenSeries": false, + "id": 9, + "legend": { + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "nullPointMode": "null", + "options": { + "alertThreshold": true + }, + "percentage": false, + "pluginVersion": "7.5.17", + "pointradius": 2, + "points": false, + "renderer": "flot", + "seriesOverrides": [], + "spaceLength": 10, + "stack": false, + "steppedLine": false, + "targets": [ + { + "exemplar": true, + "expr": "100 * (\n instance:node_cpu_utilisation:rate5m\n)", + "interval": "", + "legendFormat": "", + "refId": "A" + } + ], + "thresholds": [], + "timeFrom": null, + "timeRegions": [], + "timeShift": null, + "title": "K8S CPU Nodes Usage", + "tooltip": { + "shared": true, + "sort": 0, + "value_type": "individual" + }, + "type": "graph", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "$$hashKey": "object:139", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + }, + { + "$$hashKey": "object:140", + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": true + } + ], + "yaxis": { + "align": false, + "alignLevel": null + } + } + ], + "refresh": "5s", + "schemaVersion": 27, + "style": "dark", + "tags": [], + "templating": { + "list": [] + }, + "time": { + "from": "now-15m", + "to": "now" + }, + "timepicker": {}, + "timezone": "", + "title": "Monitoring: testground-base", + "uid": "testground-base", + "version": 23 + } diff --git a/manifests/observability/grafana/kustomization.yaml b/manifests/observability/grafana/kustomization.yaml new file mode 100644 index 00000000..4b8824d3 --- /dev/null +++ b/manifests/observability/grafana/kustomization.yaml @@ -0,0 +1,17 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: default + +resources: + - ./base + - dashboards.yaml + +configMapGenerator: + - name: testground + options: + labels: + grafana_dashboard: "1" + files: + - dashboards/base-logs.json + - dashboards/base-monitoring.json diff --git a/manifests/observability/kustomization.yaml b/manifests/observability/kustomization.yaml new file mode 100644 index 00000000..652547c9 --- /dev/null +++ b/manifests/observability/kustomization.yaml @@ -0,0 +1,12 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: +- system-tools +- grafana +- grafana-agent-operator +- loki +- otel +- prometheus +- prometheus-crds +- promtail diff --git a/manifests/observability/loki/kustomization.yaml b/manifests/observability/loki/kustomization.yaml new file mode 100644 index 00000000..50f45213 --- /dev/null +++ b/manifests/observability/loki/kustomization.yaml @@ -0,0 +1,11 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: default + +helmCharts: +- name: loki + releaseName: grafana-loki + repo: https://grafana.github.io/helm-charts + version: "4.8.0" + valuesFile: values.yaml diff --git a/manifests/observability/loki/values.yaml b/manifests/observability/loki/values.yaml new file mode 100644 index 00000000..da117d5b --- /dev/null +++ b/manifests/observability/loki/values.yaml @@ -0,0 +1,36 @@ +backend: + replicas: 0 +write: + replicas: 0 +read: + replicas: 0 +loki: + auth_enabled: false + limits_config: + split_queries_by_interval: 24h + query_scheduler: + max_outstanding_requests_per_tenant: 4096 + commonConfig: + replication_factor: 1 + storage: + type: 'filesystem' + rulerConfig: + storage: + type: local + query_range: + align_queries_with_step: true + split_queries_by_interval: 0 # 720h # 30d + parallelise_shardable_queries: false + querier: + max_concurrent: 2048 + frontend: + max_outstanding_per_tenant: 4096 + compress_responses: true +global: + dnsService: "coredns" + # Configuration for the backend pod(s) + backend: + persistence: + storageClass: gp2 + # -- Size of persistent disk + size: 30Gi diff --git a/manifests/observability/otel/configmap-collector.yaml b/manifests/observability/otel/configmap-collector.yaml new file mode 100644 index 00000000..06e58509 --- /dev/null +++ b/manifests/observability/otel/configmap-collector.yaml @@ -0,0 +1,37 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: otel-collector-conf + labels: + app: opentelemetry + component: otel-collector-conf +data: + otel-collector-config: | + receivers: + otlp: + protocols: + grpc: + http: + exporters: + influxdb: + endpoint: http://influxdb:8086 + timeout: 500ms + org: default + bucket: testground + metrics_schema: telegraf-prometheus-v1 + + sending_queue: + enabled: true + num_consumers: 3 + queue_size: 10 + + retry_on_failure: + enabled: true + initial_interval: 1s + max_interval: 3s + max_elapsed_time: 10s + service: + pipelines: + metrics: + receivers: [otlp] + exporters: [influxdb] \ No newline at end of file diff --git a/manifests/observability/otel/deployment.yaml b/manifests/observability/otel/deployment.yaml new file mode 100644 index 00000000..13733651 --- /dev/null +++ b/manifests/observability/otel/deployment.yaml @@ -0,0 +1,47 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: otel-collector + labels: + app: opentelemetry + component: otel-collector +spec: + selector: + matchLabels: + app: opentelemetry + component: otel-collector + minReadySeconds: 5 + progressDeadlineSeconds: 120 + replicas: 3 + template: + metadata: + labels: + app: opentelemetry + component: otel-collector + spec: + containers: + - command: + - "/otelcol-contrib" + - "--config=/conf/otel-collector-config.yaml" + image: otel/opentelemetry-collector-contrib:0.70.0 + name: otel-collector + resources: + limits: + cpu: 2 + memory: 4Gi + requests: + cpu: 200m + memory: 400Mi + ports: + - containerPort: 4317 # Default endpoint for OpenTelemetry receiver. + - containerPort: 4318 + volumeMounts: + - name: otel-collector-config-vol + mountPath: /conf + volumes: + - configMap: + name: otel-collector-conf + items: + - key: otel-collector-config + path: otel-collector-config.yaml + name: otel-collector-config-vol diff --git a/manifests/observability/otel/kustomization.yaml b/manifests/observability/otel/kustomization.yaml new file mode 100644 index 00000000..a8dfe931 --- /dev/null +++ b/manifests/observability/otel/kustomization.yaml @@ -0,0 +1,9 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: default + +resources: +- configmap-collector.yaml +- deployment.yaml +- service.yaml diff --git a/manifests/observability/otel/service.yaml b/manifests/observability/otel/service.yaml new file mode 100644 index 00000000..5f40a4c4 --- /dev/null +++ b/manifests/observability/otel/service.yaml @@ -0,0 +1,26 @@ +apiVersion: v1 +kind: Service +metadata: + name: otel-collector + labels: + app: opentelemetry + component: otel-collector +spec: + selector: + component: otel-collector + type: LoadBalancer + ports: + - name: otlp-grpc # Default endpoint for OpenTelemetry gRPC receiver. + port: 4317 + protocol: TCP + targetPort: 4317 + - name: otlp-http # Default endpoint for OpenTelemetry HTTP receiver. + port: 4318 + protocol: TCP + targetPort: 4318 + - name: metrics # Default endpoint for querying metrics. + port: 8888 + - name: otlp-exporter + port: 8889 + protocol: TCP + targetPort: 8889 diff --git a/manifests/observability/prometheus-crds/kustomization.yaml b/manifests/observability/prometheus-crds/kustomization.yaml new file mode 100644 index 00000000..46ebff8d --- /dev/null +++ b/manifests/observability/prometheus-crds/kustomization.yaml @@ -0,0 +1,21 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: default + +helmCharts: +- name: prometheus-operator-crds + releaseName: tg-monitoring + repo: https://prometheus-community.github.io/helm-charts + version: 3.0.0 + namespace: default + +# resources: +# - https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.63.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +# - https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.63.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +# - https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.63.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml +# - https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.63.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +# - https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.63.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +# - https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.63.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml +# - https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.63.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml +# - https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.63.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/.helmignore b/manifests/observability/prometheus/charts/kube-prometheus-stack/.helmignore new file mode 100644 index 00000000..1937f42c --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/.helmignore @@ -0,0 +1,28 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*~ +# Various IDEs +.project +.idea/ +*.tmproj +# helm/charts +OWNERS +hack/ +ci/ +kube-prometheus-*.tgz + +unittests/ diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/CONTRIBUTING.md b/manifests/observability/prometheus/charts/kube-prometheus-stack/CONTRIBUTING.md new file mode 100644 index 00000000..f6ce2a32 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/CONTRIBUTING.md @@ -0,0 +1,12 @@ +# Contributing Guidelines + +## How to contribute to this chart + +1. Fork this repository, develop and test your Chart. +1. Bump the chart version for every change. +1. Ensure PR title has the prefix `[kube-prometheus-stack]` +1. When making changes to rules or dashboards, see the README.md section on how to sync data from upstream repositories +1. Check the `hack/minikube` folder has scripts to set up minikube and components of this chart that will allow all components to be scraped. You can use this configuration when validating your changes. +1. Check for changes of RBAC rules. +1. Check for changes in CRD specs. +1. PR must pass the linter (`helm lint`) diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/Chart.lock b/manifests/observability/prometheus/charts/kube-prometheus-stack/Chart.lock new file mode 100644 index 00000000..056a85ec --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/Chart.lock @@ -0,0 +1,12 @@ +dependencies: +- name: kube-state-metrics + repository: https://prometheus-community.github.io/helm-charts + version: 4.24.0 +- name: prometheus-node-exporter + repository: https://prometheus-community.github.io/helm-charts + version: 4.8.1 +- name: grafana + repository: https://grafana.github.io/helm-charts + version: 6.48.2 +digest: sha256:0d3ceb56c869c5666de120d6c230eea5eb513b9aaa1f267719efdce351b04868 +generated: "2022-12-28T14:24:08.786548-07:00" diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/Chart.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/Chart.yaml new file mode 100644 index 00000000..4708a3c9 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/Chart.yaml @@ -0,0 +1,55 @@ +annotations: + artifacthub.io/license: Apache-2.0 + artifacthub.io/links: | + - name: Chart Source + url: https://github.com/prometheus-community/helm-charts + - name: Upstream Project + url: https://github.com/prometheus-operator/kube-prometheus + artifacthub.io/operator: "true" +apiVersion: v2 +appVersion: 0.61.1 +dependencies: +- condition: kubeStateMetrics.enabled + name: kube-state-metrics + repository: https://prometheus-community.github.io/helm-charts + version: 4.24.* +- condition: nodeExporter.enabled + name: prometheus-node-exporter + repository: https://prometheus-community.github.io/helm-charts + version: 4.8.* +- condition: grafana.enabled + name: grafana + repository: https://grafana.github.io/helm-charts + version: 6.48.* +description: kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, + and Prometheus rules combined with documentation and scripts to provide easy to + operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus + Operator. +home: https://github.com/prometheus-operator/kube-prometheus +icon: https://raw.githubusercontent.com/prometheus/prometheus.github.io/master/assets/prometheus_logo-cb55bb5c346.png +keywords: +- operator +- prometheus +- kube-prometheus +kubeVersion: '>=1.16.0-0' +maintainers: +- email: andrew@quadcorps.co.uk + name: andrewgkew +- email: gianrubio@gmail.com + name: gianrubio +- email: github.gkarthiks@gmail.com + name: gkarthiks +- email: kube-prometheus-stack@sisti.pt + name: GMartinez-Sisti +- email: scott@r6by.com + name: scottrigby +- email: miroslav.hadzhiev@gmail.com + name: Xtigyro +- email: quentin.bisson@gmail.com + name: QuentinBisson +name: kube-prometheus-stack +sources: +- https://github.com/prometheus-community/helm-charts +- https://github.com/prometheus-operator/kube-prometheus +type: application +version: 43.2.1 diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/README.md b/manifests/observability/prometheus/charts/kube-prometheus-stack/README.md new file mode 100644 index 00000000..e32bba5c --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/README.md @@ -0,0 +1,768 @@ +# kube-prometheus-stack + +Installs the [kube-prometheus stack](https://github.com/prometheus-operator/kube-prometheus), a collection of Kubernetes manifests, [Grafana](http://grafana.com/) dashboards, and [Prometheus rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with [Prometheus](https://prometheus.io/) using the [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator). + +See the [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) README for details about components, dashboards, and alerts. + +_Note: This chart was formerly named `prometheus-operator` chart, now renamed to more clearly reflect that it installs the `kube-prometheus` project stack, within which Prometheus Operator is only one component._ + +## Prerequisites + +- Kubernetes 1.16+ +- Helm 3+ + +## Get Helm Repository Info + +```console +helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +helm repo update +``` + +_See [`helm repo`](https://helm.sh/docs/helm/helm_repo/) for command documentation._ + +## Install Helm Chart + +```console +helm install [RELEASE_NAME] prometheus-community/kube-prometheus-stack +``` + +_See [configuration](#configuration) below._ + +_See [helm install](https://helm.sh/docs/helm/helm_install/) for command documentation._ + +## Dependencies + +By default this chart installs additional, dependent charts: + +- [prometheus-community/kube-state-metrics](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-state-metrics) +- [prometheus-community/prometheus-node-exporter](https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-node-exporter) +- [grafana/grafana](https://github.com/grafana/helm-charts/tree/main/charts/grafana) + +To disable dependencies during installation, see [multiple releases](#multiple-releases) below. + +_See [helm dependency](https://helm.sh/docs/helm/helm_dependency/) for command documentation._ + +## Uninstall Helm Chart + +```console +helm uninstall [RELEASE_NAME] +``` + +This removes all the Kubernetes components associated with the chart and deletes the release. + +_See [helm uninstall](https://helm.sh/docs/helm/helm_uninstall/) for command documentation._ + +CRDs created by this chart are not removed by default and should be manually cleaned up: + +```console +kubectl delete crd alertmanagerconfigs.monitoring.coreos.com +kubectl delete crd alertmanagers.monitoring.coreos.com +kubectl delete crd podmonitors.monitoring.coreos.com +kubectl delete crd probes.monitoring.coreos.com +kubectl delete crd prometheuses.monitoring.coreos.com +kubectl delete crd prometheusrules.monitoring.coreos.com +kubectl delete crd servicemonitors.monitoring.coreos.com +kubectl delete crd thanosrulers.monitoring.coreos.com +``` + +## Upgrading Chart + +```console +helm upgrade [RELEASE_NAME] prometheus-community/kube-prometheus-stack +``` + +With Helm v3, CRDs created by this chart are not updated by default and should be manually updated. +Consult also the [Helm Documentation on CRDs](https://helm.sh/docs/chart_best_practices/custom_resource_definitions). + +_See [helm upgrade](https://helm.sh/docs/helm/helm_upgrade/) for command documentation._ + +### Upgrading an existing Release to a new major version + +A major chart version change (like v1.2.3 -> v2.0.0) indicates that there is an incompatible breaking change needing manual actions. + +### From 42.x to 43.x + +This version upgrades Prometheus-Operator to v0.61.1, Prometheus to v2.40.5 and Thanos to v0.29.0. + +Run these commands to update the CRDs before applying the upgrade. + +```console +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.61.1/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.61.1/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.61.1/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.61.1/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.61.1/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.61.1/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.61.1/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.61.1/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml +``` + +### From 41.x to 42.x + +This includes the overridability of container registry for all containers at the global level using `global.imageRegistry` or per container image. The defaults have not changed but if you were using a custom image, you will have to override the registry of said custom container image before you upgrade. + +For instance, the prometheus-config-reloader used to be configured as follow: + +```yaml + image: + repository: quay.io/prometheus-operator/prometheus-config-reloader + tag: v0.60.1 + sha: "" +``` + +But it now moved to: + +```yaml + image: + registry: quay.io + repository: prometheus-operator/prometheus-config-reloader + tag: v0.60.1 + sha: "" +``` + +### From 40.x to 41.x + +This version upgrades Prometheus-Operator to v0.60.1, Prometheus to v2.39.1 and Thanos to v0.28.1. +This version also upgrades the Helm charts of kube-state-metrics to 4.20.2, prometheus-node-exporter to 4.3.0 and Grafana to 6.40.4. + +Run these commands to update the CRDs before applying the upgrade. + +```console +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.60.1/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.60.1/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.60.1/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.60.1/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.60.1/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.60.1/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.60.1/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.60.1/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml +``` + +This version splits kubeScheduler recording and altering rules in separate config values. +Instead of `defaultRules.rules.kubeScheduler` the 2 new variables `defaultRules.rules.kubeSchedulerAlerting` and `defaultRules.rules.kubeSchedulerRecording` are used. + +### From 39.x to 40.x + +This version upgrades Prometheus-Operator to v0.59.1, Prometheus to v2.38.0, kube-state-metrics to v2.6.0 and Thanos to v0.28.0. +This version also upgrades the Helm charts of kube-state-metrics to 4.18.0 and prometheus-node-exporter to 4.2.0. + +Run these commands to update the CRDs before applying the upgrade. + +```console +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.59.1/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.59.1/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.59.1/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.59.1/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.59.1/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.59.1/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.59.1/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.59.1/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml +``` + +Starting from prometheus-node-exporter version 4.0.0, the `node exporter` chart is using the [Kubernetes recommended labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/). Therefore you have to delete the daemonset before you upgrade. + +```console +kubectl delete daemonset -l app=prometheus-node-exporter +helm upgrade -i kube-prometheus-stack prometheus-community/kube-prometheus-stack +``` + +If you use your own custom [ServiceMonitor](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#servicemonitor) or [PodMonitor](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#podmonitor), please ensure to upgrade their `selector` fields accordingly to the new labels. + +### From 38.x to 39.x + +This upgraded prometheus-operator to v0.58.0 and prometheus to v2.37.0 + +Run these commands to update the CRDs before applying the upgrade. + +```console +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.58.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.58.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.58.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.58.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.58.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.58.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.58.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.58.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml +``` + +### From 37.x to 38.x + +Reverted one of the default metrics relabelings for cAdvisor added in 36.x, due to it breaking container_network_* and various other statistics. If you do not want this change, you will need to override the `kubelet.cAdvisorMetricRelabelings`. + +### From 36.x to 37.x + +This includes some default metric relabelings for cAdvisor and apiserver metrics to reduce cardinality. If you do not want these defaults, you will need to override the `kubeApiServer.metricRelabelings` and or `kubelet.cAdvisorMetricRelabelings`. + +### From 35.x to 36.x + +This upgraded prometheus-operator to v0.57.0 and prometheus to v2.36.1 + +Run these commands to update the CRDs before applying the upgrade. + +```console +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.57.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.57.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.57.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.57.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.57.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.57.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.57.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.57.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml +``` + +### From 34.x to 35.x + +This upgraded prometheus-operator to v0.56.0 and prometheus to v2.35.0 + +Run these commands to update the CRDs before applying the upgrade. + +```console +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.56.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.56.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.56.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.56.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.56.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.56.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.56.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.56.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml +``` + +### From 33.x to 34.x + +This upgrades to prometheus-operator to v0.55.0 and prometheus to v2.33.5. + +Run these commands to update the CRDs before applying the upgrade. + +```console +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.55.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.55.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.55.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.55.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.55.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.55.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.55.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.55.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml +``` + +### From 32.x to 33.x + +This upgrades the prometheus-node-exporter Chart to v3.0.0. Please review the changes to this subchart if you make customizations to hostMountPropagation. + +### From 31.x to 32.x + +This upgrades to prometheus-operator to v0.54.0 and prometheus to v2.33.1. It also changes the default for `grafana.serviceMonitor.enabled` to `true. + +Run these commands to update the CRDs before applying the upgrade. + +```console +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.54.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.54.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.54.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.54.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.54.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.54.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.54.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.54.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml +``` + +### From 30.x to 31.x + +This version removes the built-in grafana ServiceMonitor and instead relies on the ServiceMonitor of the sub-chart. +`grafana.serviceMonitor.enabled` must be set instead of `grafana.serviceMonitor.selfMonitor` and the old ServiceMonitor may +need to be manually cleaned up after deploying the new release. + +### From 29.x to 30.x + +This version updates kube-state-metrics to 4.3.0 and uses the new option `kube-state-metrics.releaseLabel=true` which adds the "release" label to kube-state-metrics labels, making scraping of the metrics by kube-prometheus-stack work out of the box again, independent of the used kube-prometheus-stack release name. If you already set the "release" label via `kube-state-metrics.customLabels` you might have to remove that and use it via the new option. + +### From 28.x to 29.x + +This version makes scraping port for kube-controller-manager and kube-scheduler dynamic to reflect changes to default serving ports +for those components in Kubernetes versions v1.22 and v1.23 respectively. + +If you deploy on clusters using version v1.22+, kube-controller-manager will be scraped over HTTPS on port 10257. + +If you deploy on clusters running version v1.23+, kube-scheduler will be scraped over HTTPS on port 10259. + +### From 27.x to 28.x + +This version disables PodSecurityPolicies by default because they are deprecated in Kubernetes 1.21 and will be removed in Kubernetes 1.25. + +If you are using PodSecurityPolicies you can enable the previous behaviour by setting `kube-state-metrics.podSecurityPolicy.enabled`, `prometheus-node-exporter.rbac.pspEnabled`, `grafana.rbac.pspEnabled` and `global.rbac.pspEnabled` to `true`. + +### From 26.x to 27.x + +This version splits prometheus-node-exporter chart recording and altering rules in separate config values. +Instead of `defaultRules.rules.node` the 2 new variables `defaultRules.rules.nodeExporterAlerting` and `defaultRules.rules.nodeExporterRecording` are used. + +Also the following defaultRules.rules has been removed as they had no effect: `kubeApiserverError`, `kubePrometheusNodeAlerting`, `kubernetesAbsent`, `time`. + +The ability to set a rubookUrl via `defaultRules.rules.rubookUrl` was reintroduced. + +### From 25.x to 26.x + +This version enables the prometheus-node-exporter subchart servicemonitor by default again, by setting `prometheus-node-exporter.prometheus.monitor.enabled` to `true`. + +### From 24.x to 25.x + +This version upgrade to prometheus-operator v0.53.1. It removes support for setting a runbookUrl, since the upstream format for runbooks changed. + +```console +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.53.1/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.53.1/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.53.1/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.53.1/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.53.1/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.53.1/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.53.1/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml +kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.53.1/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml +``` + +### From 23.x to 24.x + +The custom `ServiceMonitor` for the _kube-state-metrics_ & _prometheus-node-exporter_ charts have been removed in favour of the built-in sub-chart `ServiceMonitor`; for both sub-charts this means that `ServiceMonitor` customisations happen via the values passed to the chart. If you haven't directly customised this behaviour then there are no changes required to upgrade, but if you have please read the following. + +For _kube-state-metrics_ the `ServiceMonitor` customisation is now set via `kube-state-metrics.prometheus.monitor` and the `kubeStateMetrics.serviceMonitor.selfMonitor.enabled` value has moved to `kube-state-metrics.selfMonitor.enabled`. + +For _prometheus-node-exporter_ the `ServiceMonitor` customisation is now set via `prometheus-node-exporter.prometheus.monitor` and the `nodeExporter.jobLabel` values has moved to `prometheus-node-exporter.prometheus.monitor.jobLabel`. + +### From 22.x to 23.x + +Port names have been renamed for Istio's +[explicit protocol selection](https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/#explicit-protocol-selection). + +| | old value | new value | +|-|-----------|-----------| +| `alertmanager.alertmanagerSpec.portName` | `web` | `http-web` | +| `grafana.service.portName` | `service` | `http-web` | +| `prometheus-node-exporter.service.portName` | `metrics` (hardcoded) | `http-metrics` | +| `prometheus.prometheusSpec.portName` | `web` | `http-web` | + +### From 21.x to 22.x + +Due to the upgrade of the `kube-state-metrics` chart, removal of its deployment/stateful needs to done manually prior to upgrading: + +```console +kubectl delete deployments.apps -l app.kubernetes.io/instance=prometheus-operator,app.kubernetes.io/name=kube-state-metrics --cascade=orphan +``` + +or if you use autosharding: + +```console +kubectl delete statefulsets.apps -l app.kubernetes.io/instance=prometheus-operator,app.kubernetes.io/name=kube-state-metrics --cascade=orphan +``` + +### From 20.x to 21.x + +The config reloader values have been refactored. All the values have been moved to the key `prometheusConfigReloader` and the limits and requests can now be set separately. + +### From 19.x to 20.x + +Version 20 upgrades prometheus-operator from 0.50.x to 0.52.x. Helm does not automatically upgrade or install new CRDs on a chart upgrade, so you have to install the CRDs manually before updating: + +```console +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.52.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml +``` + +### From 18.x to 19.x + +`kubeStateMetrics.serviceMonitor.namespaceOverride` was removed. +Please use `kube-state-metrics.namespaceOverride` instead. + +### From 17.x to 18.x + +Version 18 upgrades prometheus-operator from 0.49.x to 0.50.x. Helm does not automatically upgrade or install new CRDs on a chart upgrade, so you have to install the CRDs manually before updating: + +```console +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.50.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.50.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.50.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.50.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.50.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.50.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.50.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.50.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml +``` + +### From 16.x to 17.x + +Version 17 upgrades prometheus-operator from 0.48.x to 0.49.x. Helm does not automatically upgrade or install new CRDs on a chart upgrade, so you have to install the CRDs manually before updating: + +```console +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.49.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.49.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.49.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.49.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.49.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.49.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.49.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.49.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml +``` + +### From 15.x to 16.x + +Version 16 upgrades kube-state-metrics to v2.0.0. This includes changed command-line arguments and removed metrics, see this [blog post](https://kubernetes.io/blog/2021/04/13/kube-state-metrics-v-2-0/). This version also removes Grafana dashboards that supported Kubernetes 1.14 or earlier. + +### From 14.x to 15.x + +Version 15 upgrades prometheus-operator from 0.46.x to 0.47.x. Helm does not automatically upgrade or install new CRDs on a chart upgrade, so you have to install the CRDs manually before updating: + +```console +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.47.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.47.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.47.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.47.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.47.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.47.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.47.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml +``` + +### From 13.x to 14.x + +Version 14 upgrades prometheus-operator from 0.45.x to 0.46.x. Helm does not automatically upgrade or install new CRDs on a chart upgrade, so you have to install the CRDs manually before updating: + +```console +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.46.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.46.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.46.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.46.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.46.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.46.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.46.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml +``` + +### From 12.x to 13.x + +Version 13 upgrades prometheus-operator from 0.44.x to 0.45.x. Helm does not automatically upgrade or install new CRDs on a chart upgrade, so you have to install the CRD manually before updating: + +```console +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.45.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.45.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.45.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +``` + +### From 11.x to 12.x + +Version 12 upgrades prometheus-operator from 0.43.x to 0.44.x. Helm does not automatically upgrade or install new CRDs on a chart upgrade, so you have to install the CRD manually before updating: + +```console +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.44/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +``` + +The chart was migrated to support only helm v3 and later. + +### From 10.x to 11.x + +Version 11 upgrades prometheus-operator from 0.42.x to 0.43.x. Starting with 0.43.x an additional `AlertmanagerConfigs` CRD is introduced. Helm does not automatically upgrade or install new CRDs on a chart upgrade, so you have to install the CRD manually before updating: + +```console +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.43/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +``` + +Version 11 removes the deprecated tlsProxy via ghostunnel in favor of native TLS support the prometheus-operator gained with v0.39.0. + +### From 9.x to 10.x + +Version 10 upgrades prometheus-operator from 0.38.x to 0.42.x. Starting with 0.40.x an additional `Probes` CRD is introduced. Helm does not automatically upgrade or install new CRDs on a chart upgrade, so you have to install the CRD manually before updating: + +```console +kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.42/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +``` + +### From 8.x to 9.x + +Version 9 of the helm chart removes the existing `additionalScrapeConfigsExternal` in favour of `additionalScrapeConfigsSecret`. This change lets users specify the secret name and secret key to use for the additional scrape configuration of prometheus. This is useful for users that have prometheus-operator as a subchart and also have a template that creates the additional scrape configuration. + +### From 7.x to 8.x + +Due to new template functions being used in the rules in version 8.x.x of the chart, an upgrade to Prometheus Operator and Prometheus is necessary in order to support them. First, upgrade to the latest version of 7.x.x + +```console +helm upgrade [RELEASE_NAME] prometheus-community/kube-prometheus-stack --version 7.5.0 +``` + +Then upgrade to 8.x.x + +```console +helm upgrade [RELEASE_NAME] prometheus-community/kube-prometheus-stack --version [8.x.x] +``` + +Minimal recommended Prometheus version for this chart release is `2.12.x` + +### From 6.x to 7.x + +Due to a change in grafana subchart, version 7.x.x now requires Helm >= 2.12.0. + +### From 5.x to 6.x + +Due to a change in deployment labels of kube-state-metrics, the upgrade requires `helm upgrade --force` in order to re-create the deployment. If this is not done an error will occur indicating that the deployment cannot be modified: + +```console +invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/name":"kube-state-metrics"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable +``` + +If this error has already been encountered, a `helm history` command can be used to determine which release has worked, then `helm rollback` to the release, then `helm upgrade --force` to this new one + +## Configuration + +See [Customizing the Chart Before Installing](https://helm.sh/docs/intro/using_helm/#customizing-the-chart-before-installing). To see all configurable options with detailed comments: + +```console +helm show values prometheus-community/kube-prometheus-stack +``` + +You may also `helm show values` on this chart's [dependencies](#dependencies) for additional options. + +### Multiple releases + +The same chart can be used to run multiple Prometheus instances in the same cluster if required. To achieve this, it is necessary to run only one instance of prometheus-operator and a pair of alertmanager pods for an HA configuration, while all other components need to be disabled. To disable a dependency during installation, set `kubeStateMetrics.enabled`, `nodeExporter.enabled` and `grafana.enabled` to `false`. + +## Work-Arounds for Known Issues + +### Running on private GKE clusters + +When Google configure the control plane for private clusters, they automatically configure VPC peering between your Kubernetes cluster’s network and a separate Google managed project. In order to restrict what Google are able to access within your cluster, the firewall rules configured restrict access to your Kubernetes pods. This means that in order to use the webhook component with a GKE private cluster, you must configure an additional firewall rule to allow the GKE control plane access to your webhook pod. + +You can read more information on how to add firewall rules for the GKE control plane nodes in the [GKE docs](https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#add_firewall_rules) + +Alternatively, you can disable the hooks by setting `prometheusOperator.admissionWebhooks.enabled=false`. + +## PrometheusRules Admission Webhooks + +With Prometheus Operator version 0.30+, the core Prometheus Operator pod exposes an endpoint that will integrate with the `validatingwebhookconfiguration` Kubernetes feature to prevent malformed rules from being added to the cluster. + +### How the Chart Configures the Hooks + +A validating and mutating webhook configuration requires the endpoint to which the request is sent to use TLS. It is possible to set up custom certificates to do this, but in most cases, a self-signed certificate is enough. The setup of this component requires some more complex orchestration when using helm. The steps are created to be idempotent and to allow turning the feature on and off without running into helm quirks. + +1. A pre-install hook provisions a certificate into the same namespace using a format compatible with provisioning using end user certificates. If the certificate already exists, the hook exits. +2. The prometheus operator pod is configured to use a TLS proxy container, which will load that certificate. +3. Validating and Mutating webhook configurations are created in the cluster, with their failure mode set to Ignore. This allows rules to be created by the same chart at the same time, even though the webhook has not yet been fully set up - it does not have the correct CA field set. +4. A post-install hook reads the CA from the secret created by step 1 and patches the Validating and Mutating webhook configurations. This process will allow a custom CA provisioned by some other process to also be patched into the webhook configurations. The chosen failure policy is also patched into the webhook configurations + +### Alternatives + +It should be possible to use [jetstack/cert-manager](https://github.com/jetstack/cert-manager) if a more complete solution is required, but it has not been tested. + +You can enable automatic self-signed TLS certificate provisioning via cert-manager by setting the `prometheusOperator.admissionWebhooks.certManager.enabled` value to true. + +### Limitations + +Because the operator can only run as a single pod, there is potential for this component failure to cause rule deployment failure. Because this risk is outweighed by the benefit of having validation, the feature is enabled by default. + +## Developing Prometheus Rules and Grafana Dashboards + +This chart Grafana Dashboards and Prometheus Rules are just a copy from [prometheus-operator/prometheus-operator](https://github.com/prometheus-operator/prometheus-operator) and other sources, synced (with alterations) by scripts in [hack](hack) folder. In order to introduce any changes you need to first [add them to the original repository](https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/developing-prometheus-rules-and-grafana-dashboards.md) and then sync there by scripts. + +## Further Information + +For more in-depth documentation of configuration options meanings, please see + +- [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) +- [Prometheus](https://prometheus.io/docs/introduction/overview/) +- [Grafana](https://github.com/grafana/helm-charts/tree/main/charts/grafana#grafana-helm-chart) + +## prometheus.io/scrape + +The prometheus operator does not support annotation-based discovery of services, using the `PodMonitor` or `ServiceMonitor` CRD in its place as they provide far more configuration options. +For information on how to use PodMonitors/ServiceMonitors, please see the documentation on the `prometheus-operator/prometheus-operator` documentation here: + +- [ServiceMonitors](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/getting-started.md#include-servicemonitors) +- [PodMonitors](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/getting-started.md#include-podmonitors) +- [Running Exporters](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/running-exporters.md) + +By default, Prometheus discovers PodMonitors and ServiceMonitors within its namespace, that are labeled with the same release tag as the prometheus-operator release. +Sometimes, you may need to discover custom PodMonitors/ServiceMonitors, for example used to scrape data from third-party applications. +An easy way of doing this, without compromising the default PodMonitors/ServiceMonitors discovery, is allowing Prometheus to discover all PodMonitors/ServiceMonitors within its namespace, without applying label filtering. +To do so, you can set `prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues` and `prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues` to `false`. + +## Migrating from stable/prometheus-operator chart + +## Zero downtime + +Since `kube-prometheus-stack` is fully compatible with the `stable/prometheus-operator` chart, a migration without downtime can be achieved. +However, the old name prefix needs to be kept. If you want the new name please follow the step by step guide below (with downtime). + +You can override the name to achieve this: + +```console +helm upgrade prometheus-operator prometheus-community/kube-prometheus-stack -n monitoring --reuse-values --set nameOverride=prometheus-operator +``` + +**Note**: It is recommended to run this first with `--dry-run --debug`. + +## Redeploy with new name (downtime) + +If the **prometheus-operator** values are compatible with the new **kube-prometheus-stack** chart, please follow the below steps for migration: + +> The guide presumes that chart is deployed in `monitoring` namespace and the deployments are running there. If in other namespace, please replace the `monitoring` to the deployed namespace. + +1. Patch the PersistenceVolume created/used by the prometheus-operator chart to `Retain` claim policy: + + ```console + kubectl patch pv/ -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' + ``` + + **Note:** To execute the above command, the user must have a cluster wide permission. Please refer [Kubernetes RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) + +2. Uninstall the **prometheus-operator** release and delete the existing PersistentVolumeClaim, and verify PV become Released. + + ```console + helm uninstall prometheus-operator -n monitoring + kubectl delete pvc/ -n monitoring + ``` + + Additionally, you have to manually remove the remaining `prometheus-operator-kubelet` service. + + ```console + kubectl delete service/prometheus-operator-kubelet -n kube-system + ``` + + You can choose to remove all your existing CRDs (ServiceMonitors, Podmonitors, etc.) if you want to. + +3. Remove current `spec.claimRef` values to change the PV's status from Released to Available. + + ```console + kubectl patch pv/ --type json -p='[{"op": "remove", "path": "/spec/claimRef"}]' -n monitoring + ``` + +**Note:** To execute the above command, the user must have a cluster wide permission. Please refer to [Kubernetes RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) + +After these steps, proceed to a fresh **kube-prometheus-stack** installation and make sure the current release of **kube-prometheus-stack** matching the `volumeClaimTemplate` values in the `values.yaml`. + +The binding is done via matching a specific amount of storage requested and with certain access modes. + +For example, if you had storage specified as this with **prometheus-operator**: + +```yaml +volumeClaimTemplate: + spec: + storageClassName: gp2 + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 50Gi +``` + +You have to specify matching `volumeClaimTemplate` with 50Gi storage and `ReadWriteOnce` access mode. + +Additionally, you should check the current AZ of your legacy installation's PV, and configure the fresh release to use the same AZ as the old one. If the pods are in a different AZ than the PV, the release will fail to bind the existing one, hence creating a new PV. + +This can be achieved either by specifying the labels through `values.yaml`, e.g. setting `prometheus.prometheusSpec.nodeSelector` to: + +```yaml +nodeSelector: + failure-domain.beta.kubernetes.io/zone: east-west-1a +``` + +or passing these values as `--set` overrides during installation. + +The new release should now re-attach your previously released PV with its content. + +## Migrating from coreos/prometheus-operator chart + +The multiple charts have been combined into a single chart that installs prometheus operator, prometheus, alertmanager, grafana as well as the multitude of exporters necessary to monitor a cluster. + +There is no simple and direct migration path between the charts as the changes are extensive and intended to make the chart easier to support. + +The capabilities of the old chart are all available in the new chart, including the ability to run multiple prometheus instances on a single cluster - you will need to disable the parts of the chart you do not wish to deploy. + +You can check out the tickets for this change [here](https://github.com/prometheus-operator/prometheus-operator/issues/592) and [here](https://github.com/helm/charts/pull/6765). + +### High-level overview of Changes + +#### Added dependencies + +The chart has added 3 [dependencies](#dependencies). + +- Node-Exporter, Kube-State-Metrics: These components are loaded as dependencies into the chart, and are relatively simple components +- Grafana: The Grafana chart is more feature-rich than this chart - it contains a sidecar that is able to load data sources and dashboards from configmaps deployed into the same cluster. For more information check out the [documentation for the chart](https://github.com/grafana/helm-charts/blob/main/charts/grafana/README.md) + +#### Kubelet Service + +Because the kubelet service has a new name in the chart, make sure to clean up the old kubelet service in the `kube-system` namespace to prevent counting container metrics twice. + +#### Persistent Volumes + +If you would like to keep the data of the current persistent volumes, it should be possible to attach existing volumes to new PVCs and PVs that are created using the conventions in the new chart. For example, in order to use an existing Azure disk for a helm release called `prometheus-migration` the following resources can be created: + +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: pvc-prometheus-migration-prometheus-0 +spec: + accessModes: + - ReadWriteOnce + azureDisk: + cachingMode: None + diskName: pvc-prometheus-migration-prometheus-0 + diskURI: /subscriptions/f5125d82-2622-4c50-8d25-3f7ba3e9ac4b/resourceGroups/sample-migration-resource-group/providers/Microsoft.Compute/disks/pvc-prometheus-migration-prometheus-0 + fsType: "" + kind: Managed + readOnly: false + capacity: + storage: 1Gi + persistentVolumeReclaimPolicy: Delete + storageClassName: prometheus + volumeMode: Filesystem +``` + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + labels: + app.kubernetes.io/name: prometheus + prometheus: prometheus-migration-prometheus + name: prometheus-prometheus-migration-prometheus-db-prometheus-prometheus-migration-prometheus-0 + namespace: monitoring +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: prometheus + volumeMode: Filesystem + volumeName: pvc-prometheus-migration-prometheus-0 +``` + +The PVC will take ownership of the PV and when you create a release using a persistent volume claim template it will use the existing PVCs as they match the naming convention used by the chart. For other cloud providers similar approaches can be used. + +#### KubeProxy + +The metrics bind address of kube-proxy is default to `127.0.0.1:10249` that prometheus instances **cannot** access to. You should expose metrics by changing `metricsBindAddress` field value to `0.0.0.0:10249` if you want to collect them. + +Depending on the cluster, the relevant part `config.conf` will be in ConfigMap `kube-system/kube-proxy` or `kube-system/kube-proxy-config`. For example: + +```console +kubectl -n kube-system edit cm kube-proxy +``` + +```yaml +apiVersion: v1 +data: + config.conf: |- + apiVersion: kubeproxy.config.k8s.io/v1alpha1 + kind: KubeProxyConfiguration + # ... + # metricsBindAddress: 127.0.0.1:10249 + metricsBindAddress: 0.0.0.0:10249 + # ... + kubeconfig.conf: |- + # ... +kind: ConfigMap +metadata: + labels: + app: kube-proxy + name: kube-proxy + namespace: kube-system +``` diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/.helmignore b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/.helmignore new file mode 100644 index 00000000..8cade131 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/.helmignore @@ -0,0 +1,23 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*~ +# Various IDEs +.vscode +.project +.idea/ +*.tmproj +OWNERS diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/Chart.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/Chart.yaml new file mode 100644 index 00000000..6d313a9c --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/Chart.yaml @@ -0,0 +1,22 @@ +apiVersion: v2 +appVersion: 9.3.1 +description: The leading tool for querying and visualizing time series and metrics. +home: https://grafana.net +icon: https://raw.githubusercontent.com/grafana/grafana/master/public/img/logo_transparent_400x.png +kubeVersion: ^1.8.0-0 +maintainers: +- email: zanhsieh@gmail.com + name: zanhsieh +- email: rluckie@cisco.com + name: rtluckie +- email: maor.friedman@redhat.com + name: maorfr +- email: miroslav.hadzhiev@gmail.com + name: Xtigyro +- email: mail@torstenwalter.de + name: torstenwalter +name: grafana +sources: +- https://github.com/grafana/grafana +type: application +version: 6.48.2 diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/README.md b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/README.md new file mode 100644 index 00000000..d5210caa --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/README.md @@ -0,0 +1,603 @@ +# Grafana Helm Chart + +* Installs the web dashboarding system [Grafana](http://grafana.org/) + +## Get Repo Info + +```console +helm repo add grafana https://grafana.github.io/helm-charts +helm repo update +``` + +_See [helm repo](https://helm.sh/docs/helm/helm_repo/) for command documentation._ + +## Installing the Chart + +To install the chart with the release name `my-release`: + +```console +helm install my-release grafana/grafana +``` + +## Uninstalling the Chart + +To uninstall/delete the my-release deployment: + +```console +helm delete my-release +``` + +The command removes all the Kubernetes components associated with the chart and deletes the release. + +## Upgrading an existing Release to a new major version + +A major chart version change (like v1.2.3 -> v2.0.0) indicates that there is an +incompatible breaking change needing manual actions. + +### To 4.0.0 (And 3.12.1) + +This version requires Helm >= 2.12.0. + +### To 5.0.0 + +You have to add --force to your helm upgrade command as the labels of the chart have changed. + +### To 6.0.0 + +This version requires Helm >= 3.1.0. + +## Configuration + +| Parameter | Description | Default | +|-------------------------------------------|-----------------------------------------------|---------------------------------------------------------| +| `replicas` | Number of nodes | `1` | +| `podDisruptionBudget.minAvailable` | Pod disruption minimum available | `nil` | +| `podDisruptionBudget.maxUnavailable` | Pod disruption maximum unavailable | `nil` | +| `deploymentStrategy` | Deployment strategy | `{ "type": "RollingUpdate" }` | +| `livenessProbe` | Liveness Probe settings | `{ "httpGet": { "path": "/api/health", "port": 3000 } "initialDelaySeconds": 60, "timeoutSeconds": 30, "failureThreshold": 10 }` | +| `readinessProbe` | Readiness Probe settings | `{ "httpGet": { "path": "/api/health", "port": 3000 } }`| +| `securityContext` | Deployment securityContext | `{"runAsUser": 472, "runAsGroup": 472, "fsGroup": 472}` | +| `priorityClassName` | Name of Priority Class to assign pods | `nil` | +| `image.repository` | Image repository | `grafana/grafana` | +| `image.tag` | Overrides the Grafana image tag whose default is the chart appVersion (`Must be >= 5.0.0`) | `` | +| `image.sha` | Image sha (optional) | `` | +| `image.pullPolicy` | Image pull policy | `IfNotPresent` | +| `image.pullSecrets` | Image pull secrets (can be templated) | `[]` | +| `service.enabled` | Enable grafana service | `true` | +| `service.type` | Kubernetes service type | `ClusterIP` | +| `service.port` | Kubernetes port where service is exposed | `80` | +| `service.portName` | Name of the port on the service | `service` | +| `service.appProtocol` | Adds the appProtocol field to the service | `` | +| `service.targetPort` | Internal service is port | `3000` | +| `service.nodePort` | Kubernetes service nodePort | `nil` | +| `service.annotations` | Service annotations (can be templated) | `{}` | +| `service.labels` | Custom labels | `{}` | +| `service.clusterIP` | internal cluster service IP | `nil` | +| `service.loadBalancerIP` | IP address to assign to load balancer (if supported) | `nil` | +| `service.loadBalancerSourceRanges` | list of IP CIDRs allowed access to lb (if supported) | `[]` | +| `service.externalIPs` | service external IP addresses | `[]` | +| `headlessService` | Create a headless service | `false` | +| `extraExposePorts` | Additional service ports for sidecar containers| `[]` | +| `hostAliases` | adds rules to the pod's /etc/hosts | `[]` | +| `ingress.enabled` | Enables Ingress | `false` | +| `ingress.annotations` | Ingress annotations (values are templated) | `{}` | +| `ingress.labels` | Custom labels | `{}` | +| `ingress.path` | Ingress accepted path | `/` | +| `ingress.pathType` | Ingress type of path | `Prefix` | +| `ingress.hosts` | Ingress accepted hostnames | `["chart-example.local"]` | +| `ingress.extraPaths` | Ingress extra paths to prepend to every host configuration. Useful when configuring [custom actions with AWS ALB Ingress Controller](https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/ingress/annotation/#actions). Requires `ingress.hosts` to have one or more host entries. | `[]` | +| `ingress.tls` | Ingress TLS configuration | `[]` | +| `resources` | CPU/Memory resource requests/limits | `{}` | +| `nodeSelector` | Node labels for pod assignment | `{}` | +| `tolerations` | Toleration labels for pod assignment | `[]` | +| `affinity` | Affinity settings for pod assignment | `{}` | +| `extraInitContainers` | Init containers to add to the grafana pod | `{}` | +| `extraContainers` | Sidecar containers to add to the grafana pod | `""` | +| `extraContainerVolumes` | Volumes that can be mounted in sidecar containers | `[]` | +| `extraLabels` | Custom labels for all manifests | `{}` | +| `schedulerName` | Name of the k8s scheduler (other than default) | `nil` | +| `persistence.enabled` | Use persistent volume to store data | `false` | +| `persistence.type` | Type of persistence (`pvc` or `statefulset`) | `pvc` | +| `persistence.size` | Size of persistent volume claim | `10Gi` | +| `persistence.existingClaim` | Use an existing PVC to persist data (can be templated) | `nil` | +| `persistence.storageClassName` | Type of persistent volume claim | `nil` | +| `persistence.accessModes` | Persistence access modes | `[ReadWriteOnce]` | +| `persistence.annotations` | PersistentVolumeClaim annotations | `{}` | +| `persistence.finalizers` | PersistentVolumeClaim finalizers | `[ "kubernetes.io/pvc-protection" ]` | +| `persistence.extraPvcLabels` | Extra labels to apply to a PVC. | `{}` | +| `persistence.subPath` | Mount a sub dir of the persistent volume (can be templated) | `nil` | +| `persistence.inMemory.enabled` | If persistence is not enabled, whether to mount the local storage in-memory to improve performance | `false` | +| `persistence.inMemory.sizeLimit` | SizeLimit for the in-memory local storage | `nil` | +| `initChownData.enabled` | If false, don't reset data ownership at startup | true | +| `initChownData.image.repository` | init-chown-data container image repository | `busybox` | +| `initChownData.image.tag` | init-chown-data container image tag | `1.31.1` | +| `initChownData.image.sha` | init-chown-data container image sha (optional)| `""` | +| `initChownData.image.pullPolicy` | init-chown-data container image pull policy | `IfNotPresent` | +| `initChownData.resources` | init-chown-data pod resource requests & limits | `{}` | +| `schedulerName` | Alternate scheduler name | `nil` | +| `env` | Extra environment variables passed to pods | `{}` | +| `envValueFrom` | Environment variables from alternate sources. See the API docs on [EnvVarSource](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#envvarsource-v1-core) for format details. Can be templated | `{}` | +| `envFromSecret` | Name of a Kubernetes secret (must be manually created in the same namespace) containing values to be added to the environment. Can be templated | `""` | +| `envFromSecrets` | List of Kubernetes secrets (must be manually created in the same namespace) containing values to be added to the environment. Can be templated | `[]` | +| `envFromConfigMaps` | List of Kubernetes ConfigMaps (must be manually created in the same namespace) containing values to be added to the environment. Can be templated | `[]` | +| `envRenderSecret` | Sensible environment variables passed to pods and stored as secret | `{}` | +| `enableServiceLinks` | Inject Kubernetes services as environment variables. | `true` | +| `extraSecretMounts` | Additional grafana server secret mounts | `[]` | +| `extraVolumeMounts` | Additional grafana server volume mounts | `[]` | +| `createConfigmap` | Enable creating the grafana configmap | `true` | +| `extraConfigmapMounts` | Additional grafana server configMap volume mounts (values are templated) | `[]` | +| `extraEmptyDirMounts` | Additional grafana server emptyDir volume mounts | `[]` | +| `plugins` | Plugins to be loaded along with Grafana | `[]` | +| `datasources` | Configure grafana datasources (passed through tpl) | `{}` | +| `alerting` | Configure grafana alerting (passed through tpl) | `{}` | +| `notifiers` | Configure grafana notifiers | `{}` | +| `dashboardProviders` | Configure grafana dashboard providers | `{}` | +| `dashboards` | Dashboards to import | `{}` | +| `dashboardsConfigMaps` | ConfigMaps reference that contains dashboards | `{}` | +| `grafana.ini` | Grafana's primary configuration | `{}` | +| `global.imagePullSecrets` | Global image pull secrets (can be templated). Allows either an array of {name: pullSecret} maps (k8s-style), or an array of strings (more common helm-style). | `[]` | +| `ldap.enabled` | Enable LDAP authentication | `false` | +| `ldap.existingSecret` | The name of an existing secret containing the `ldap.toml` file, this must have the key `ldap-toml`. | `""` | +| `ldap.config` | Grafana's LDAP configuration | `""` | +| `annotations` | Deployment annotations | `{}` | +| `labels` | Deployment labels | `{}` | +| `podAnnotations` | Pod annotations | `{}` | +| `podLabels` | Pod labels | `{}` | +| `podPortName` | Name of the grafana port on the pod | `grafana` | +| `lifecycleHooks` | Lifecycle hooks for podStart and preStop [Example](https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/#define-poststart-and-prestop-handlers) | `{}` | +| `sidecar.image.repository` | Sidecar image repository | `quay.io/kiwigrid/k8s-sidecar` | +| `sidecar.image.tag` | Sidecar image tag | `1.19.2` | +| `sidecar.image.sha` | Sidecar image sha (optional) | `""` | +| `sidecar.imagePullPolicy` | Sidecar image pull policy | `IfNotPresent` | +| `sidecar.resources` | Sidecar resources | `{}` | +| `sidecar.securityContext` | Sidecar securityContext | `{}` | +| `sidecar.enableUniqueFilenames` | Sets the kiwigrid/k8s-sidecar UNIQUE_FILENAMES environment variable. If set to `true` the sidecar will create unique filenames where duplicate data keys exist between ConfigMaps and/or Secrets within the same or multiple Namespaces. | `false` | +| `sidecar.alerts.enabled` | Enables the cluster wide search for alerts and adds/updates/deletes them in grafana |`false` | +| `sidecar.alerts.label` | Label that config maps with alerts should have to be added | `grafana_alert` | +| `sidecar.alerts.labelValue` | Label value that config maps with alerts should have to be added | `""` | +| `sidecar.alerts.searchNamespace` | Namespaces list. If specified, the sidecar will search for alerts config-maps inside these namespaces. Otherwise the namespace in which the sidecar is running will be used. It's also possible to specify ALL to search in all namespaces. | `nil` | +| `sidecar.alerts.watchMethod` | Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH requests, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds. | `WATCH` | +| `sidecar.alerts.resource` | Should the sidecar looks into secrets, configmaps or both. | `both` | +| `sidecar.alerts.reloadURL` | Full url of datasource configuration reload API endpoint, to invoke after a config-map change | `"http://localhost:3000/api/admin/provisioning/alerting/reload"` | +| `sidecar.alerts.skipReload` | Enabling this omits defining the REQ_URL and REQ_METHOD environment variables | `false` | +| `sidecar.alerts.initDatasources` | Set to true to deploy the datasource sidecar as an initContainer in addition to a container. This is needed if skipReload is true, to load any alerts defined at startup time. | `false` | +| `sidecar.dashboards.enabled` | Enables the cluster wide search for dashboards and adds/updates/deletes them in grafana | `false` | +| `sidecar.dashboards.SCProvider` | Enables creation of sidecar provider | `true` | +| `sidecar.dashboards.provider.name` | Unique name of the grafana provider | `sidecarProvider` | +| `sidecar.dashboards.provider.orgid` | Id of the organisation, to which the dashboards should be added | `1` | +| `sidecar.dashboards.provider.folder` | Logical folder in which grafana groups dashboards | `""` | +| `sidecar.dashboards.provider.disableDelete` | Activate to avoid the deletion of imported dashboards | `false` | +| `sidecar.dashboards.provider.allowUiUpdates` | Allow updating provisioned dashboards from the UI | `false` | +| `sidecar.dashboards.provider.type` | Provider type | `file` | +| `sidecar.dashboards.provider.foldersFromFilesStructure` | Allow Grafana to replicate dashboard structure from filesystem. | `false` | +| `sidecar.dashboards.watchMethod` | Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH requests, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds. | `WATCH` | +| `sidecar.skipTlsVerify` | Set to true to skip tls verification for kube api calls | `nil` | +| `sidecar.dashboards.label` | Label that config maps with dashboards should have to be added | `grafana_dashboard` | +| `sidecar.dashboards.labelValue` | Label value that config maps with dashboards should have to be added | `""` | +| `sidecar.dashboards.folder` | Folder in the pod that should hold the collected dashboards (unless `sidecar.dashboards.defaultFolderName` is set). This path will be mounted. | `/tmp/dashboards` | +| `sidecar.dashboards.folderAnnotation` | The annotation the sidecar will look for in configmaps to override the destination folder for files | `nil` | +| `sidecar.dashboards.defaultFolderName` | The default folder name, it will create a subfolder under the `sidecar.dashboards.folder` and put dashboards in there instead | `nil` | +| `sidecar.dashboards.searchNamespace` | Namespaces list. If specified, the sidecar will search for dashboards config-maps inside these namespaces. Otherwise the namespace in which the sidecar is running will be used. It's also possible to specify ALL to search in all namespaces. | `nil` | +| `sidecar.dashboards.script` | Absolute path to shell script to execute after a configmap got reloaded. | `nil` | +| `sidecar.dashboards.reloadURL` | Full url of dashboards configuration reload API endpoint, to invoke after a config-map change | `"http://localhost:3000/api/admin/provisioning/dashboards/reload"` | +| `sidecar.dashboards.skipReload` | Enabling this omits defining the REQ_URL and REQ_METHOD environment variables | `false` | +| `sidecar.dashboards.resource` | Should the sidecar looks into secrets, configmaps or both. | `both` | +| `sidecar.dashboards.extraMounts` | Additional dashboard sidecar volume mounts. | `[]` | +| `sidecar.datasources.enabled` | Enables the cluster wide search for datasources and adds/updates/deletes them in grafana |`false` | +| `sidecar.datasources.label` | Label that config maps with datasources should have to be added | `grafana_datasource` | +| `sidecar.datasources.labelValue` | Label value that config maps with datasources should have to be added | `""` | +| `sidecar.datasources.searchNamespace` | Namespaces list. If specified, the sidecar will search for datasources config-maps inside these namespaces. Otherwise the namespace in which the sidecar is running will be used. It's also possible to specify ALL to search in all namespaces. | `nil` | +| `sidecar.datasources.watchMethod` | Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH requests, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds. | `WATCH` | +| `sidecar.datasources.resource` | Should the sidecar looks into secrets, configmaps or both. | `both` | +| `sidecar.datasources.reloadURL` | Full url of datasource configuration reload API endpoint, to invoke after a config-map change | `"http://localhost:3000/api/admin/provisioning/datasources/reload"` | +| `sidecar.datasources.skipReload` | Enabling this omits defining the REQ_URL and REQ_METHOD environment variables | `false` | +| `sidecar.datasources.initDatasources` | Set to true to deploy the datasource sidecar as an initContainer in addition to a container. This is needed if skipReload is true, to load any datasources defined at startup time. | `false` | +| `sidecar.notifiers.enabled` | Enables the cluster wide search for notifiers and adds/updates/deletes them in grafana | `false` | +| `sidecar.notifiers.label` | Label that config maps with notifiers should have to be added | `grafana_notifier` | +| `sidecar.notifiers.labelValue` | Label value that config maps with notifiers should have to be added | `""` | +| `sidecar.notifiers.searchNamespace` | Namespaces list. If specified, the sidecar will search for notifiers config-maps (or secrets) inside these namespaces. Otherwise the namespace in which the sidecar is running will be used. It's also possible to specify ALL to search in all namespaces. | `nil` | +| `sidecar.notifiers.watchMethod` | Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH requests, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds. | `WATCH` | +| `sidecar.notifiers.resource` | Should the sidecar looks into secrets, configmaps or both. | `both` | +| `sidecar.notifiers.reloadURL` | Full url of notifier configuration reload API endpoint, to invoke after a config-map change | `"http://localhost:3000/api/admin/provisioning/notifications/reload"` | +| `sidecar.notifiers.skipReload` | Enabling this omits defining the REQ_URL and REQ_METHOD environment variables | `false` | +| `sidecar.notifiers.initNotifiers` | Set to true to deploy the notifier sidecar as an initContainer in addition to a container. This is needed if skipReload is true, to load any notifiers defined at startup time. | `false` | +| `smtp.existingSecret` | The name of an existing secret containing the SMTP credentials. | `""` | +| `smtp.userKey` | The key in the existing SMTP secret containing the username. | `"user"` | +| `smtp.passwordKey` | The key in the existing SMTP secret containing the password. | `"password"` | +| `admin.existingSecret` | The name of an existing secret containing the admin credentials (can be templated). | `""` | +| `admin.userKey` | The key in the existing admin secret containing the username. | `"admin-user"` | +| `admin.passwordKey` | The key in the existing admin secret containing the password. | `"admin-password"` | +| `serviceAccount.autoMount` | Automount the service account token in the pod| `true` | +| `serviceAccount.annotations` | ServiceAccount annotations | | +| `serviceAccount.create` | Create service account | `true` | +| `serviceAccount.labels` | ServiceAccount labels | `{}` | +| `serviceAccount.name` | Service account name to use, when empty will be set to created account if `serviceAccount.create` is set else to `default` | `` | +| `serviceAccount.nameTest` | Service account name to use for test, when empty will be set to created account if `serviceAccount.create` is set else to `default` | `nil` | +| `rbac.create` | Create and use RBAC resources | `true` | +| `rbac.namespaced` | Creates Role and Rolebinding instead of the default ClusterRole and ClusteRoleBindings for the grafana instance | `false` | +| `rbac.useExistingRole` | Set to a rolename to use existing role - skipping role creating - but still doing serviceaccount and rolebinding to the rolename set here. | `nil` | +| `rbac.pspEnabled` | Create PodSecurityPolicy (with `rbac.create`, grant roles permissions as well) | `true` | +| `rbac.pspUseAppArmor` | Enforce AppArmor in created PodSecurityPolicy (requires `rbac.pspEnabled`) | `true` | +| `rbac.extraRoleRules` | Additional rules to add to the Role | [] | +| `rbac.extraClusterRoleRules` | Additional rules to add to the ClusterRole | [] | +| `command` | Define command to be executed by grafana container at startup | `nil` | +| `testFramework.enabled` | Whether to create test-related resources | `true` | +| `testFramework.image` | `test-framework` image repository. | `bats/bats` | +| `testFramework.tag` | `test-framework` image tag. | `v1.4.1` | +| `testFramework.imagePullPolicy` | `test-framework` image pull policy. | `IfNotPresent` | +| `testFramework.securityContext` | `test-framework` securityContext | `{}` | +| `downloadDashboards.env` | Environment variables to be passed to the `download-dashboards` container | `{}` | +| `downloadDashboards.envFromSecret` | Name of a Kubernetes secret (must be manually created in the same namespace) containing values to be added to the environment. Can be templated | `""` | +| `downloadDashboards.resources` | Resources of `download-dashboards` container | `{}` | +| `downloadDashboardsImage.repository` | Curl docker image repo | `curlimages/curl` | +| `downloadDashboardsImage.tag` | Curl docker image tag | `7.73.0` | +| `downloadDashboardsImage.sha` | Curl docker image sha (optional) | `""` | +| `downloadDashboardsImage.pullPolicy` | Curl docker image pull policy | `IfNotPresent` | +| `namespaceOverride` | Override the deployment namespace | `""` (`Release.Namespace`) | +| `serviceMonitor.enabled` | Use servicemonitor from prometheus operator | `false` | +| `serviceMonitor.namespace` | Namespace this servicemonitor is installed in | | +| `serviceMonitor.interval` | How frequently Prometheus should scrape | `1m` | +| `serviceMonitor.path` | Path to scrape | `/metrics` | +| `serviceMonitor.scheme` | Scheme to use for metrics scraping | `http` | +| `serviceMonitor.tlsConfig` | TLS configuration block for the endpoint | `{}` | +| `serviceMonitor.labels` | Labels for the servicemonitor passed to Prometheus Operator | `{}` | +| `serviceMonitor.scrapeTimeout` | Timeout after which the scrape is ended | `30s` | +| `serviceMonitor.relabelings` | MetricRelabelConfigs to apply to samples before ingestion. | `[]` | +| `revisionHistoryLimit` | Number of old ReplicaSets to retain | `10` | +| `imageRenderer.enabled` | Enable the image-renderer deployment & service | `false` | +| `imageRenderer.image.repository` | image-renderer Image repository | `grafana/grafana-image-renderer` | +| `imageRenderer.image.tag` | image-renderer Image tag | `latest` | +| `imageRenderer.image.sha` | image-renderer Image sha (optional) | `""` | +| `imageRenderer.image.pullPolicy` | image-renderer ImagePullPolicy | `Always` | +| `imageRenderer.env` | extra env-vars for image-renderer | `{}` | +| `imageRenderer.serviceAccountName` | image-renderer deployment serviceAccountName | `""` | +| `imageRenderer.securityContext` | image-renderer deployment securityContext | `{}` | +| `imageRenderer.hostAliases` | image-renderer deployment Host Aliases | `[]` | +| `imageRenderer.priorityClassName` | image-renderer deployment priority class | `''` | +| `imageRenderer.service.enabled` | Enable the image-renderer service | `true` | +| `imageRenderer.service.portName` | image-renderer service port name | `http` | +| `imageRenderer.service.port` | image-renderer port used by deployment | `8081` | +| `imageRenderer.service.targetPort` | image-renderer service port used by service | `8081` | +| `imageRenderer.appProtocol` | Adds the appProtocol field to the service | `` | +| `imageRenderer.grafanaSubPath` | Grafana sub path to use for image renderer callback url | `''` | +| `imageRenderer.podPortName` | name of the image-renderer port on the pod | `http` | +| `imageRenderer.revisionHistoryLimit` | number of image-renderer replica sets to keep | `10` | +| `imageRenderer.networkPolicy.limitIngress` | Enable a NetworkPolicy to limit inbound traffic from only the created grafana pods | `true` | +| `imageRenderer.networkPolicy.limitEgress` | Enable a NetworkPolicy to limit outbound traffic to only the created grafana pods | `false` | +| `imageRenderer.resources` | Set resource limits for image-renderer pdos | `{}` | +| `imageRenderer.nodeSelector` | Node labels for pod assignment | `{}` | +| `imageRenderer.tolerations` | Toleration labels for pod assignment | `[]` | +| `imageRenderer.affinity` | Affinity settings for pod assignment | `{}` | +| `networkPolicy.enabled` | Enable creation of NetworkPolicy resources. | `false` | +| `networkPolicy.allowExternal` | Don't require client label for connections | `true` | +| `networkPolicy.explicitNamespacesSelector` | A Kubernetes LabelSelector to explicitly select namespaces from which traffic could be allowed | `{}` | +| `networkPolicy.ingress` | Enable the creation of an ingress network policy | `true` | +| `networkPolicy.egress.enabled` | Enable the creation of an egress network policy | `false` | +| `networkPolicy.egress.ports` | An array of ports to allow for the egress | `[]` | +| `enableKubeBackwardCompatibility` | Enable backward compatibility of kubernetes where pod's defintion version below 1.13 doesn't have the enableServiceLinks option | `false` | + + + +### Example ingress with path + +With grafana 6.3 and above +```yaml +grafana.ini: + server: + domain: monitoring.example.com + root_url: "%(protocol)s://%(domain)s/grafana" + serve_from_sub_path: true +ingress: + enabled: true + hosts: + - "monitoring.example.com" + path: "/grafana" +``` + +### Example of extraVolumeMounts + +Volume can be type persistentVolumeClaim or hostPath but not both at same time. +If neither existingClaim or hostPath argument is given then type is emptyDir. + +```yaml +- extraVolumeMounts: + - name: plugins + mountPath: /var/lib/grafana/plugins + subPath: configs/grafana/plugins + existingClaim: existing-grafana-claim + readOnly: false + - name: dashboards + mountPath: /var/lib/grafana/dashboards + hostPath: /usr/shared/grafana/dashboards + readOnly: false +``` + +## Import dashboards + +There are a few methods to import dashboards to Grafana. Below are some examples and explanations as to how to use each method: + +```yaml +dashboards: + default: + some-dashboard: + json: | + { + "annotations": + + ... + # Complete json file here + ... + + "title": "Some Dashboard", + "uid": "abcd1234", + "version": 1 + } + custom-dashboard: + # This is a path to a file inside the dashboards directory inside the chart directory + file: dashboards/custom-dashboard.json + prometheus-stats: + # Ref: https://grafana.com/dashboards/2 + gnetId: 2 + revision: 2 + datasource: Prometheus + loki-dashboard-quick-search: + gnetId: 12019 + revision: 2 + datasource: + - name: DS_PROMETHEUS + value: Prometheus + - name: DS_LOKI + value: Loki + local-dashboard: + url: https://raw.githubusercontent.com/user/repository/master/dashboards/dashboard.json +``` + +## BASE64 dashboards + +Dashboards could be stored on a server that does not return JSON directly and instead of it returns a Base64 encoded file (e.g. Gerrit) +A new parameter has been added to the url use case so if you specify a b64content value equals to true after the url entry a Base64 decoding is applied before save the file to disk. +If this entry is not set or is equals to false not decoding is applied to the file before saving it to disk. + +### Gerrit use case + +Gerrit API for download files has the following schema: where {project-name} and +{file-id} usually has '/' in their values and so they MUST be replaced by %2F so if project-name is user/repo, branch-id is master and file-id is equals to dir1/dir2/dashboard +the url value is + +## Sidecar for dashboards + +If the parameter `sidecar.dashboards.enabled` is set, a sidecar container is deployed in the grafana +pod. This container watches all configmaps (or secrets) in the cluster and filters out the ones with +a label as defined in `sidecar.dashboards.label`. The files defined in those configmaps are written +to a folder and accessed by grafana. Changes to the configmaps are monitored and the imported +dashboards are deleted/updated. + +A recommendation is to use one configmap per dashboard, as a reduction of multiple dashboards inside +one configmap is currently not properly mirrored in grafana. + +Example dashboard config: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: sample-grafana-dashboard + labels: + grafana_dashboard: "1" +data: + k8s-dashboard.json: |- + [...] +``` + +## Sidecar for datasources + +If the parameter `sidecar.datasources.enabled` is set, an init container is deployed in the grafana +pod. This container lists all secrets (or configmaps, though not recommended) in the cluster and +filters out the ones with a label as defined in `sidecar.datasources.label`. The files defined in +those secrets are written to a folder and accessed by grafana on startup. Using these yaml files, +the data sources in grafana can be imported. + +Secrets are recommended over configmaps for this usecase because datasources usually contain private +data like usernames and passwords. Secrets are the more appropriate cluster resource to manage those. + +Example values to add a datasource adapted from [Grafana](http://docs.grafana.org/administration/provisioning/#example-datasource-config-file): + +```yaml +datasources: + datasources.yaml: + apiVersion: 1 + datasources: + # name of the datasource. Required + - name: Graphite + # datasource type. Required + type: graphite + # access mode. proxy or direct (Server or Browser in the UI). Required + access: proxy + # org id. will default to orgId 1 if not specified + orgId: 1 + # url + url: http://localhost:8080 + # database password, if used + password: + # database user, if used + user: + # database name, if used + database: + # enable/disable basic auth + basicAuth: + # basic auth username + basicAuthUser: + # basic auth password + basicAuthPassword: + # enable/disable with credentials headers + withCredentials: + # mark as default datasource. Max one per org + isDefault: + # fields that will be converted to json and stored in json_data + jsonData: + graphiteVersion: "1.1" + tlsAuth: true + tlsAuthWithCACert: true + # json object of data that will be encrypted. + secureJsonData: + tlsCACert: "..." + tlsClientCert: "..." + tlsClientKey: "..." + version: 1 + # allow users to edit datasources from the UI. + editable: false +``` + +## Sidecar for notifiers + +If the parameter `sidecar.notifiers.enabled` is set, an init container is deployed in the grafana +pod. This container lists all secrets (or configmaps, though not recommended) in the cluster and +filters out the ones with a label as defined in `sidecar.notifiers.label`. The files defined in +those secrets are written to a folder and accessed by grafana on startup. Using these yaml files, +the notification channels in grafana can be imported. The secrets must be created before +`helm install` so that the notifiers init container can list the secrets. + +Secrets are recommended over configmaps for this usecase because alert notification channels usually contain +private data like SMTP usernames and passwords. Secrets are the more appropriate cluster resource to manage those. + +Example datasource config adapted from [Grafana](https://grafana.com/docs/grafana/latest/administration/provisioning/#alert-notification-channels): + +```yaml +notifiers: + - name: notification-channel-1 + type: slack + uid: notifier1 + # either + org_id: 2 + # or + org_name: Main Org. + is_default: true + send_reminder: true + frequency: 1h + disable_resolve_message: false + # See `Supported Settings` section for settings supporter for each + # alert notification type. + settings: + recipient: 'XXX' + token: 'xoxb' + uploadImage: true + url: https://slack.com + +delete_notifiers: + - name: notification-channel-1 + uid: notifier1 + org_id: 2 + - name: notification-channel-2 + # default org_id: 1 +``` + +## How to serve Grafana with a path prefix (/grafana) + +In order to serve Grafana with a prefix (e.g., ), add the following to your values.yaml. + +```yaml +ingress: + enabled: true + annotations: + kubernetes.io/ingress.class: "nginx" + nginx.ingress.kubernetes.io/rewrite-target: /$1 + nginx.ingress.kubernetes.io/use-regex: "true" + + path: /grafana/?(.*) + hosts: + - k8s.example.dev + +grafana.ini: + server: + root_url: http://localhost:3000/grafana # this host can be localhost +``` + +## How to securely reference secrets in grafana.ini + +This example uses Grafana [file providers](https://grafana.com/docs/grafana/latest/administration/configuration/#file-provider) for secret values and the `extraSecretMounts` configuration flag (Additional grafana server secret mounts) to mount the secrets. + +In grafana.ini: + +```yaml +grafana.ini: + [auth.generic_oauth] + enabled = true + client_id = $__file{/etc/secrets/auth_generic_oauth/client_id} + client_secret = $__file{/etc/secrets/auth_generic_oauth/client_secret} +``` + +Existing secret, or created along with helm: + +```yaml +--- +apiVersion: v1 +kind: Secret +metadata: + name: auth-generic-oauth-secret +type: Opaque +stringData: + client_id: + client_secret: +``` + +Include in the `extraSecretMounts` configuration flag: + +```yaml +- extraSecretMounts: + - name: auth-generic-oauth-secret-mount + secretName: auth-generic-oauth-secret + defaultMode: 0440 + mountPath: /etc/secrets/auth_generic_oauth + readOnly: true +``` + +### extraSecretMounts using a Container Storage Interface (CSI) provider + +This example uses a CSI driver e.g. retrieving secrets using [Azure Key Vault Provider](https://github.com/Azure/secrets-store-csi-driver-provider-azure) + +```yaml +- extraSecretMounts: + - name: secrets-store-inline + mountPath: /run/secrets + readOnly: true + csi: + driver: secrets-store.csi.k8s.io + readOnly: true + volumeAttributes: + secretProviderClass: "my-provider" + nodePublishSecretRef: + name: akv-creds +``` + +## Image Renderer Plug-In + +This chart supports enabling [remote image rendering](https://github.com/grafana/grafana-image-renderer/blob/master/README.md#run-in-docker) + +```yaml +imageRenderer: + enabled: true +``` + +### Image Renderer NetworkPolicy + +By default the image-renderer pods will have a network policy which only allows ingress traffic from the created grafana instance + +### High Availability for unified alerting + +If you want to run Grafana in a high availability cluster you need to enable +the headless service by setting `headlessService: true` in your `values.yaml` +file. + +As next step you have to setup the `grafana.ini` in your `values.yaml` in a way +that it will make use of the headless service to obtain all the IPs of the +cluster. You should replace ``{{ Name }}`` with the name of your helm deployment. + +```yaml +grafana.ini: + ... + unified_alerting: + enabled: true + ha_peers: {{ Name }}-headless:9094 + alerting: + enabled: false +``` diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/default-values.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/default-values.yaml new file mode 100644 index 00000000..fc2ba605 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/default-values.yaml @@ -0,0 +1 @@ +# Leave this file empty to ensure that CI runs builds against the default configuration in values.yaml. diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-affinity-values.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-affinity-values.yaml new file mode 100644 index 00000000..f5b9b53e --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-affinity-values.yaml @@ -0,0 +1,16 @@ +affinity: + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - podAffinityTerm: + labelSelector: + matchLabels: + app.kubernetes.io/instance: grafana-test + app.kubernetes.io/name: grafana + topologyKey: failure-domain.beta.kubernetes.io/zone + weight: 100 + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: + app.kubernetes.io/instance: grafana-test + app.kubernetes.io/name: grafana + topologyKey: kubernetes.io/hostname diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-dashboard-json-values.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-dashboard-json-values.yaml new file mode 100644 index 00000000..e0c4e416 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-dashboard-json-values.yaml @@ -0,0 +1,53 @@ +dashboards: + my-provider: + my-awesome-dashboard: + # An empty but valid dashboard + json: | + { + "__inputs": [], + "__requires": [ + { + "type": "grafana", + "id": "grafana", + "name": "Grafana", + "version": "6.3.5" + } + ], + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": "-- Grafana --", + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "gnetId": null, + "graphTooltip": 0, + "id": null, + "links": [], + "panels": [], + "schemaVersion": 19, + "style": "dark", + "tags": [], + "templating": { + "list": [] + }, + "time": { + "from": "now-6h", + "to": "now" + }, + "timepicker": { + "refresh_intervals": ["5s"] + }, + "timezone": "", + "title": "Dummy Dashboard", + "uid": "IdcYQooWk", + "version": 1 + } + datasource: Prometheus diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-dashboard-values.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-dashboard-values.yaml new file mode 100644 index 00000000..7b662c5f --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-dashboard-values.yaml @@ -0,0 +1,19 @@ +dashboards: + my-provider: + my-awesome-dashboard: + gnetId: 10000 + revision: 1 + datasource: Prometheus +dashboardProviders: + dashboardproviders.yaml: + apiVersion: 1 + providers: + - name: 'my-provider' + orgId: 1 + folder: '' + type: file + updateIntervalSeconds: 10 + disableDeletion: true + editable: true + options: + path: /var/lib/grafana/dashboards/my-provider diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-extraconfigmapmounts-values.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-extraconfigmapmounts-values.yaml new file mode 100644 index 00000000..5cc44a05 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-extraconfigmapmounts-values.yaml @@ -0,0 +1,7 @@ +extraConfigmapMounts: + - name: '{{ include "grafana.fullname" . }}' + configMap: '{{ include "grafana.fullname" . }}' + mountPath: /var/lib/grafana/dashboards/test-dashboard.json + # This is not a realistic test, but for this we only care about extraConfigmapMounts not being empty and pointing to an existing ConfigMap + subPath: grafana.ini + readOnly: true diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-image-renderer-values.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-image-renderer-values.yaml new file mode 100644 index 00000000..32f30743 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-image-renderer-values.yaml @@ -0,0 +1,19 @@ +podLabels: + customLableA: Aaaaa +imageRenderer: + enabled: true + env: + RENDERING_ARGS: --disable-gpu,--window-size=1280x758 + RENDERING_MODE: clustered + podLabels: + customLableB: Bbbbb + networkPolicy: + limitIngress: true + limitEgress: true + resources: + limits: + cpu: 1000m + memory: 1000Mi + requests: + cpu: 500m + memory: 50Mi diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-persistence.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-persistence.yaml new file mode 100644 index 00000000..b92ca02c --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/ci/with-persistence.yaml @@ -0,0 +1,3 @@ +persistence: + type: pvc + enabled: true diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/dashboards/custom-dashboard.json b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/dashboards/custom-dashboard.json new file mode 100644 index 00000000..0967ef42 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/dashboards/custom-dashboard.json @@ -0,0 +1 @@ +{} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/NOTES.txt b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/NOTES.txt new file mode 100644 index 00000000..f399f43f --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/NOTES.txt @@ -0,0 +1,54 @@ +1. Get your '{{ .Values.adminUser }}' user password by running: + + kubectl get secret --namespace {{ include "grafana.namespace" . }} {{ include "grafana.fullname" . }} -o jsonpath="{.data.admin-password}" | base64 --decode ; echo + +2. The Grafana server can be accessed via port {{ .Values.service.port }} on the following DNS name from within your cluster: + + {{ include "grafana.fullname" . }}.{{ include "grafana.namespace" . }}.svc.cluster.local +{{ if .Values.ingress.enabled }} + If you bind grafana to 80, please update values in values.yaml and reinstall: + ``` + securityContext: + runAsUser: 0 + runAsGroup: 0 + fsGroup: 0 + + command: + - "setcap" + - "'cap_net_bind_service=+ep'" + - "/usr/sbin/grafana-server &&" + - "sh" + - "/run.sh" + ``` + Details refer to https://grafana.com/docs/installation/configuration/#http-port. + Or grafana would always crash. + + From outside the cluster, the server URL(s) are: + {{- range .Values.ingress.hosts }} + http://{{ . }} + {{- end }} +{{- else }} + Get the Grafana URL to visit by running these commands in the same shell: + {{- if contains "NodePort" .Values.service.type }} + export NODE_PORT=$(kubectl get --namespace {{ include "grafana.namespace" . }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "grafana.fullname" . }}) + export NODE_IP=$(kubectl get nodes --namespace {{ include "grafana.namespace" . }} -o jsonpath="{.items[0].status.addresses[0].address}") + echo http://$NODE_IP:$NODE_PORT + {{- else if contains "LoadBalancer" .Values.service.type }} + NOTE: It may take a few minutes for the LoadBalancer IP to be available. + You can watch the status of by running 'kubectl get svc --namespace {{ include "grafana.namespace" . }} -w {{ include "grafana.fullname" . }}' + export SERVICE_IP=$(kubectl get svc --namespace {{ include "grafana.namespace" . }} {{ include "grafana.fullname" . }} -o jsonpath='{.status.loadBalancer.ingress[0].ip}') + http://$SERVICE_IP:{{ .Values.service.port -}} + {{- else if contains "ClusterIP" .Values.service.type }} + export POD_NAME=$(kubectl get pods --namespace {{ include "grafana.namespace" . }} -l "app.kubernetes.io/name={{ include "grafana.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}") + kubectl --namespace {{ include "grafana.namespace" . }} port-forward $POD_NAME 3000 + {{- end }} +{{- end }} + +3. Login with the password from step 1 and the username: {{ .Values.adminUser }} + +{{- if not .Values.persistence.enabled }} +################################################################################# +###### WARNING: Persistence is disabled!!! You will lose your data when ##### +###### the Grafana pod is terminated. ##### +################################################################################# +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/_helpers.tpl b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/_helpers.tpl new file mode 100644 index 00000000..ef8dc392 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/_helpers.tpl @@ -0,0 +1,199 @@ +{{/* vim: set filetype=mustache: */}} +{{/* +Expand the name of the chart. +*/}} +{{- define "grafana.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +If release name contains chart name it will be used as a full name. +*/}} +{{- define "grafana.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "grafana.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create the name of the service account +*/}} +{{- define "grafana.serviceAccountName" -}} +{{- if .Values.serviceAccount.create }} +{{- default (include "grafana.fullname" .) .Values.serviceAccount.name }} +{{- else }} +{{- default "default" .Values.serviceAccount.name }} +{{- end }} +{{- end }} + +{{- define "grafana.serviceAccountNameTest" -}} +{{- if .Values.serviceAccount.create }} +{{- default (print (include "grafana.fullname" .) "-test") .Values.serviceAccount.nameTest }} +{{- else }} +{{- default "default" .Values.serviceAccount.nameTest }} +{{- end }} +{{- end }} + +{{/* +Allow the release namespace to be overridden for multi-namespace deployments in combined charts +*/}} +{{- define "grafana.namespace" -}} +{{- if .Values.namespaceOverride }} +{{- .Values.namespaceOverride }} +{{- else }} +{{- .Release.Namespace }} +{{- end }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "grafana.labels" -}} +helm.sh/chart: {{ include "grafana.chart" . }} +{{ include "grafana.selectorLabels" . }} +{{- if or .Chart.AppVersion .Values.image.tag }} +app.kubernetes.io/version: {{ mustRegexReplaceAllLiteral "@sha.*" .Values.image.tag "" | default .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- with .Values.extraLabels }} +{{ toYaml . }} +{{- end }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "grafana.selectorLabels" -}} +app.kubernetes.io/name: {{ include "grafana.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "grafana.imageRenderer.labels" -}} +helm.sh/chart: {{ include "grafana.chart" . }} +{{ include "grafana.imageRenderer.selectorLabels" . }} +{{- if or .Chart.AppVersion .Values.image.tag }} +app.kubernetes.io/version: {{ mustRegexReplaceAllLiteral "@sha.*" .Values.image.tag "" | default .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} + +{{/* +Selector labels ImageRenderer +*/}} +{{- define "grafana.imageRenderer.selectorLabels" -}} +app.kubernetes.io/name: {{ include "grafana.name" . }}-image-renderer +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Looks if there's an existing secret and reuse its password. If not it generates +new password and use it. +*/}} +{{- define "grafana.password" -}} +{{- $secret := (lookup "v1" "Secret" (include "grafana.namespace" .) (include "grafana.fullname" .) ) }} +{{- if $secret }} +{{- index $secret "data" "admin-password" }} +{{- else }} +{{- (randAlphaNum 40) | b64enc | quote }} +{{- end }} +{{- end }} + +{{/* +Return the appropriate apiVersion for rbac. +*/}} +{{- define "grafana.rbac.apiVersion" -}} +{{- if $.Capabilities.APIVersions.Has "rbac.authorization.k8s.io/v1" }} +{{- print "rbac.authorization.k8s.io/v1" }} +{{- else }} +{{- print "rbac.authorization.k8s.io/v1beta1" }} +{{- end }} +{{- end }} + +{{/* +Return the appropriate apiVersion for ingress. +*/}} +{{- define "grafana.ingress.apiVersion" -}} +{{- if and ($.Capabilities.APIVersions.Has "networking.k8s.io/v1") (semverCompare ">= 1.19-0" .Capabilities.KubeVersion.Version) }} +{{- print "networking.k8s.io/v1" }} +{{- else if $.Capabilities.APIVersions.Has "networking.k8s.io/v1beta1" }} +{{- print "networking.k8s.io/v1beta1" }} +{{- else }} +{{- print "extensions/v1beta1" }} +{{- end }} +{{- end }} + +{{/* +Return the appropriate apiVersion for Horizontal Pod Autoscaler. +*/}} +{{- define "grafana.hpa.apiVersion" -}} +{{- if semverCompare "<1.23-0" .Capabilities.KubeVersion.Version }} +{{- print "autoscaling/v2beta1" }} +{{- else }} +{{- print "autoscaling/v2" }} +{{- end }} +{{- end }} + +{{/* +Return the appropriate apiVersion for podDisruptionBudget. +*/}} +{{- define "grafana.podDisruptionBudget.apiVersion" -}} +{{- if $.Capabilities.APIVersions.Has "policy/v1/PodDisruptionBudget" }} +{{- print "policy/v1" }} +{{- else }} +{{- print "policy/v1beta1" }} +{{- end }} +{{- end }} + +{{/* +Return if ingress is stable. +*/}} +{{- define "grafana.ingress.isStable" -}} +{{- eq (include "grafana.ingress.apiVersion" .) "networking.k8s.io/v1" }} +{{- end }} + +{{/* +Return if ingress supports ingressClassName. +*/}} +{{- define "grafana.ingress.supportsIngressClassName" -}} +{{- or (eq (include "grafana.ingress.isStable" .) "true") (and (eq (include "grafana.ingress.apiVersion" .) "networking.k8s.io/v1beta1") (semverCompare ">= 1.18-0" .Capabilities.KubeVersion.Version)) }} +{{- end }} + +{{/* +Return if ingress supports pathType. +*/}} +{{- define "grafana.ingress.supportsPathType" -}} +{{- or (eq (include "grafana.ingress.isStable" .) "true") (and (eq (include "grafana.ingress.apiVersion" .) "networking.k8s.io/v1beta1") (semverCompare ">= 1.18-0" .Capabilities.KubeVersion.Version)) }} +{{- end }} + +{{/* +Formats imagePullSecrets. Input is (dict "root" . "imagePullSecrets" .{specific imagePullSecrets}) +*/}} +{{- define "grafana.imagePullSecrets" -}} +{{- $root := .root }} +{{- range (concat .root.Values.global.imagePullSecrets .imagePullSecrets) }} +{{- if eq (typeOf .) "map[string]interface {}" }} +- {{ toYaml (dict "name" (tpl .name $root)) | trim }} +{{- else }} +- name: {{ tpl . $root }} +{{- end }} +{{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/_pod.tpl b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/_pod.tpl new file mode 100644 index 00000000..0bf6b4d4 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/_pod.tpl @@ -0,0 +1,1143 @@ +{{- define "grafana.pod" -}} +{{- $sts := list "sts" "StatefulSet" "statefulset" -}} +{{- $root := . -}} +{{- with .Values.schedulerName }} +schedulerName: "{{ . }}" +{{- end }} +serviceAccountName: {{ include "grafana.serviceAccountName" . }} +automountServiceAccountToken: {{ .Values.serviceAccount.autoMount }} +{{- with .Values.securityContext }} +securityContext: + {{- toYaml . | nindent 2 }} +{{- end }} +{{- with .Values.hostAliases }} +hostAliases: + {{- toYaml . | nindent 2 }} +{{- end }} +{{- with .Values.priorityClassName }} +priorityClassName: {{ . }} +{{- end }} +{{- if ( or .Values.persistence.enabled .Values.dashboards .Values.extraInitContainers (and .Values.sidecar.datasources.enabled .Values.sidecar.datasources.initDatasources) (and .Values.sidecar.notifiers.enabled .Values.sidecar.notifiers.initNotifiers)) }} +initContainers: +{{- end }} +{{- if ( and .Values.persistence.enabled .Values.initChownData.enabled ) }} + - name: init-chown-data + {{- if .Values.initChownData.image.sha }} + image: "{{ .Values.initChownData.image.repository }}:{{ .Values.initChownData.image.tag }}@sha256:{{ .Values.initChownData.image.sha }}" + {{- else }} + image: "{{ .Values.initChownData.image.repository }}:{{ .Values.initChownData.image.tag }}" + {{- end }} + imagePullPolicy: {{ .Values.initChownData.image.pullPolicy }} + {{- with .Values.initChownData.securityContext }} + securityContext: + {{- toYaml . | nindent 6 }} + {{- end }} + command: + - chown + - -R + - {{ .Values.securityContext.runAsUser }}:{{ .Values.securityContext.runAsGroup }} + - /var/lib/grafana + {{- with .Values.initChownData.resources }} + resources: + {{- toYaml . | nindent 6 }} + {{- end }} + volumeMounts: + - name: storage + mountPath: "/var/lib/grafana" + {{- with .Values.persistence.subPath }} + subPath: {{ tpl . $root }} + {{- end }} +{{- end }} +{{- if .Values.dashboards }} + - name: download-dashboards + {{- if .Values.downloadDashboardsImage.sha }} + image: "{{ .Values.downloadDashboardsImage.repository }}:{{ .Values.downloadDashboardsImage.tag }}@sha256:{{ .Values.downloadDashboardsImage.sha }}" + {{- else }} + image: "{{ .Values.downloadDashboardsImage.repository }}:{{ .Values.downloadDashboardsImage.tag }}" + {{- end }} + imagePullPolicy: {{ .Values.downloadDashboardsImage.pullPolicy }} + command: ["/bin/sh"] + args: [ "-c", "mkdir -p /var/lib/grafana/dashboards/default && /bin/sh -x /etc/grafana/download_dashboards.sh" ] + {{- with .Values.downloadDashboards.resources }} + resources: + {{- toYaml . | nindent 6 }} + {{- end }} + env: + {{- range $key, $value := .Values.downloadDashboards.env }} + - name: "{{ $key }}" + value: "{{ $value }}" + {{- end }} + {{- range $key, $value := .Values.downloadDashboards.envValueFrom }} + - name: {{ $key | quote }} + valueFrom: + {{- tpl (toYaml $value) $ | nindent 10 }} + {{- end }} + {{- with .Values.downloadDashboards.securityContext }} + securityContext: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.downloadDashboards.envFromSecret }} + envFrom: + - secretRef: + name: {{ tpl . $root }} + {{- end }} + volumeMounts: + - name: config + mountPath: "/etc/grafana/download_dashboards.sh" + subPath: download_dashboards.sh + - name: storage + mountPath: "/var/lib/grafana" + {{- with .Values.persistence.subPath }} + subPath: {{ tpl . $root }} + {{- end }} + {{- range .Values.extraSecretMounts }} + - name: {{ .name }} + mountPath: {{ .mountPath }} + readOnly: {{ .readOnly }} + {{- end }} +{{- end }} +{{- if and .Values.sidecar.datasources.enabled .Values.sidecar.datasources.initDatasources }} + - name: {{ include "grafana.name" . }}-init-sc-datasources + {{- if .Values.sidecar.image.sha }} + image: "{{ .Values.sidecar.image.repository }}:{{ .Values.sidecar.image.tag }}@sha256:{{ .Values.sidecar.image.sha }}" + {{- else }} + image: "{{ .Values.sidecar.image.repository }}:{{ .Values.sidecar.image.tag }}" + {{- end }} + imagePullPolicy: {{ .Values.sidecar.imagePullPolicy }} + env: + {{- range $key, $value := .Values.sidecar.datasources.env }} + - name: "{{ $key }}" + value: "{{ $value }}" + {{- end }} + {{- if .Values.sidecar.datasources.ignoreAlreadyProcessed }} + - name: IGNORE_ALREADY_PROCESSED + value: "true" + {{- end }} + - name: METHOD + value: "LIST" + - name: LABEL + value: "{{ .Values.sidecar.datasources.label }}" + {{- with .Values.sidecar.datasources.labelValue }} + - name: LABEL_VALUE + value: {{ quote . }} + {{- end }} + {{- if or .Values.sidecar.logLevel .Values.sidecar.datasources.logLevel }} + - name: LOG_LEVEL + value: {{ default .Values.sidecar.logLevel .Values.sidecar.datasources.logLevel }} + {{- end }} + - name: FOLDER + value: "/etc/grafana/provisioning/datasources" + - name: RESOURCE + value: {{ quote .Values.sidecar.datasources.resource }} + {{- with .Values.sidecar.enableUniqueFilenames }} + - name: UNIQUE_FILENAMES + value: "{{ . }}" + {{- end }} + {{- if .Values.sidecar.datasources.searchNamespace }} + - name: NAMESPACE + value: "{{ tpl (.Values.sidecar.datasources.searchNamespace | join ",") . }}" + {{- end }} + {{- with .Values.sidecar.skipTlsVerify }} + - name: SKIP_TLS_VERIFY + value: "{{ . }}" + {{- end }} + {{- with .Values.sidecar.resources }} + resources: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.securityContext }} + securityContext: + {{- toYaml . | nindent 6 }} + {{- end }} + volumeMounts: + - name: sc-datasources-volume + mountPath: "/etc/grafana/provisioning/datasources" +{{- end }} +{{- if and .Values.sidecar.notifiers.enabled .Values.sidecar.notifiers.initNotifiers }} + - name: {{ include "grafana.name" . }}-init-sc-notifiers + {{- if .Values.sidecar.image.sha }} + image: "{{ .Values.sidecar.image.repository }}:{{ .Values.sidecar.image.tag }}@sha256:{{ .Values.sidecar.image.sha }}" + {{- else }} + image: "{{ .Values.sidecar.image.repository }}:{{ .Values.sidecar.image.tag }}" + {{- end }} + imagePullPolicy: {{ .Values.sidecar.imagePullPolicy }} + env: + {{- range $key, $value := .Values.sidecar.notifiers.env }} + - name: "{{ $key }}" + value: "{{ $value }}" + {{- end }} + {{- if .Values.sidecar.notifiers.ignoreAlreadyProcessed }} + - name: IGNORE_ALREADY_PROCESSED + value: "true" + {{- end }} + - name: METHOD + value: LIST + - name: LABEL + value: "{{ .Values.sidecar.notifiers.label }}" + {{- with .Values.sidecar.notifiers.labelValue }} + - name: LABEL_VALUE + value: {{ quote . }} + {{- end }} + {{- if or .Values.sidecar.logLevel .Values.sidecar.notifiers.logLevel }} + - name: LOG_LEVEL + value: {{ default .Values.sidecar.logLevel .Values.sidecar.notifiers.logLevel }} + {{- end }} + - name: FOLDER + value: "/etc/grafana/provisioning/notifiers" + - name: RESOURCE + value: {{ quote .Values.sidecar.notifiers.resource }} + {{- with .Values.sidecar.enableUniqueFilenames }} + - name: UNIQUE_FILENAMES + value: "{{ . }}" + {{- end }} + {{- with .Values.sidecar.notifiers.searchNamespace }} + - name: NAMESPACE + value: "{{ tpl (. | join ",") $root }}" + {{- end }} + {{- with .Values.sidecar.skipTlsVerify }} + - name: SKIP_TLS_VERIFY + value: "{{ . }}" + {{- end }} + {{- with .Values.sidecar.livenessProbe }} + livenessProbe: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.readinessProbe }} + readinessProbe: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.resources }} + resources: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.securityContext }} + securityContext: + {{- toYaml . | nindent 6 }} + {{- end }} + volumeMounts: + - name: sc-notifiers-volume + mountPath: "/etc/grafana/provisioning/notifiers" +{{- end}} +{{- with .Values.extraInitContainers }} + {{- tpl (toYaml .) $root | nindent 2 }} +{{- end }} +{{- if or .Values.image.pullSecrets .Values.global.imagePullSecrets }} +imagePullSecrets: + {{- include "grafana.imagePullSecrets" (dict "root" $root "imagePullSecrets" .Values.image.pullSecrets) | nindent 2 }} +{{- end }} +{{- if not .Values.enableKubeBackwardCompatibility }} +enableServiceLinks: {{ .Values.enableServiceLinks }} +{{- end }} +containers: +{{- if .Values.sidecar.alerts.enabled }} + - name: {{ include "grafana.name" . }}-sc-alerts + {{- if .Values.sidecar.image.sha }} + image: "{{ .Values.sidecar.image.repository }}:{{ .Values.sidecar.image.tag }}@sha256:{{ .Values.sidecar.image.sha }}" + {{- else }} + image: "{{ .Values.sidecar.image.repository }}:{{ .Values.sidecar.image.tag }}" + {{- end }} + imagePullPolicy: {{ .Values.sidecar.imagePullPolicy }} + env: + {{- range $key, $value := .Values.sidecar.alerts.env }} + - name: "{{ $key }}" + value: "{{ $value }}" + {{- end }} + {{- if .Values.sidecar.alerts.ignoreAlreadyProcessed }} + - name: IGNORE_ALREADY_PROCESSED + value: "true" + {{- end }} + - name: METHOD + value: {{ .Values.sidecar.alerts.watchMethod }} + - name: LABEL + value: "{{ .Values.sidecar.alerts.label }}" + {{- with .Values.sidecar.alerts.labelValue }} + - name: LABEL_VALUE + value: {{ quote . }} + {{- end }} + {{- if or .Values.sidecar.logLevel .Values.sidecar.alerts.logLevel }} + - name: LOG_LEVEL + value: {{ default .Values.sidecar.logLevel .Values.sidecar.alerts.logLevel }} + {{- end }} + - name: FOLDER + value: "/etc/grafana/provisioning/alerting" + - name: RESOURCE + value: {{ quote .Values.sidecar.alerts.resource }} + {{- with .Values.sidecar.enableUniqueFilenames }} + - name: UNIQUE_FILENAMES + value: "{{ . }}" + {{- end }} + {{- with .Values.sidecar.alerts.searchNamespace }} + - name: NAMESPACE + value: {{ . | join "," | quote }} + {{- end }} + {{- with .Values.sidecar.alerts.skipTlsVerify }} + - name: SKIP_TLS_VERIFY + value: {{ quote . }} + {{- end }} + {{- with .Values.sidecar.alerts.script }} + - name: SCRIPT + value: {{ quote . }} + {{- end }} + {{- if and (not .Values.env.GF_SECURITY_ADMIN_USER) (not .Values.env.GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION) }} + - name: REQ_USERNAME + valueFrom: + secretKeyRef: + name: {{ (tpl .Values.admin.existingSecret .) | default (include "grafana.fullname" .) }} + key: {{ .Values.admin.userKey | default "admin-user" }} + {{- end }} + {{- if and (not .Values.env.GF_SECURITY_ADMIN_PASSWORD) (not .Values.env.GF_SECURITY_ADMIN_PASSWORD__FILE) (not .Values.env.GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION) }} + - name: REQ_PASSWORD + valueFrom: + secretKeyRef: + name: {{ (tpl .Values.admin.existingSecret .) | default (include "grafana.fullname" .) }} + key: {{ .Values.admin.passwordKey | default "admin-password" }} + {{- end }} + {{- if not .Values.sidecar.alerts.skipReload }} + - name: REQ_URL + value: {{ .Values.sidecar.alerts.reloadURL }} + - name: REQ_METHOD + value: POST + {{- end }} + {{- if .Values.sidecar.alerts.watchServerTimeout }} + {{- if ne .Values.sidecar.alerts.watchMethod "WATCH" }} + {{- fail (printf "Cannot use .Values.sidecar.alerts.watchServerTimeout with .Values.sidecar.alerts.watchMethod %s" .Values.sidecar.alerts.watchMethod) }} + {{- end }} + - name: WATCH_SERVER_TIMEOUT + value: "{{ .Values.sidecar.alerts.watchServerTimeout }}" + {{- end }} + {{- if .Values.sidecar.alerts.watchClientTimeout }} + {{- if ne .Values.sidecar.alerts.watchMethod "WATCH" }} + {{- fail (printf "Cannot use .Values.sidecar.alerts.watchClientTimeout with .Values.sidecar.alerts.watchMethod %s" .Values.sidecar.alerts.watchMethod) }} + {{- end }} + - name: WATCH_CLIENT_TIMEOUT + value: "{{ .Values.sidecar.alerts.watchClientTimeout }}" + {{- end }} + {{- with .Values.sidecar.livenessProbe }} + livenessProbe: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.readinessProbe }} + readinessProbe: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.resources }} + resources: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.securityContext }} + securityContext: + {{- toYaml . | nindent 6 }} + {{- end }} + volumeMounts: + - name: sc-alerts-volume + mountPath: "/etc/grafana/provisioning/alerting" +{{- end}} +{{- if .Values.sidecar.dashboards.enabled }} + - name: {{ include "grafana.name" . }}-sc-dashboard + {{- if .Values.sidecar.image.sha }} + image: "{{ .Values.sidecar.image.repository }}:{{ .Values.sidecar.image.tag }}@sha256:{{ .Values.sidecar.image.sha }}" + {{- else }} + image: "{{ .Values.sidecar.image.repository }}:{{ .Values.sidecar.image.tag }}" + {{- end }} + imagePullPolicy: {{ .Values.sidecar.imagePullPolicy }} + env: + {{- range $key, $value := .Values.sidecar.dashboards.env }} + - name: "{{ $key }}" + value: "{{ $value }}" + {{- end }} + {{- if .Values.sidecar.dashboards.ignoreAlreadyProcessed }} + - name: IGNORE_ALREADY_PROCESSED + value: "true" + {{- end }} + - name: METHOD + value: {{ .Values.sidecar.dashboards.watchMethod }} + - name: LABEL + value: "{{ .Values.sidecar.dashboards.label }}" + {{- with .Values.sidecar.dashboards.labelValue }} + - name: LABEL_VALUE + value: {{ quote . }} + {{- end }} + {{- if or .Values.sidecar.logLevel .Values.sidecar.dashboards.logLevel }} + - name: LOG_LEVEL + value: {{ default .Values.sidecar.logLevel .Values.sidecar.dashboards.logLevel }} + {{- end }} + - name: FOLDER + value: "{{ .Values.sidecar.dashboards.folder }}{{- with .Values.sidecar.dashboards.defaultFolderName }}/{{ . }}{{- end }}" + - name: RESOURCE + value: {{ quote .Values.sidecar.dashboards.resource }} + {{- with .Values.sidecar.enableUniqueFilenames }} + - name: UNIQUE_FILENAMES + value: "{{ . }}" + {{- end }} + {{- with .Values.sidecar.dashboards.searchNamespace }} + - name: NAMESPACE + value: "{{ tpl (. | join ",") $root }}" + {{- end }} + {{- with .Values.sidecar.skipTlsVerify }} + - name: SKIP_TLS_VERIFY + value: "{{ . }}" + {{- end }} + {{- with .Values.sidecar.dashboards.folderAnnotation }} + - name: FOLDER_ANNOTATION + value: "{{ . }}" + {{- end }} + {{- with .Values.sidecar.dashboards.script }} + - name: SCRIPT + value: "{{ . }}" + {{- end }} + {{- if and (not .Values.env.GF_SECURITY_ADMIN_USER) (not .Values.env.GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION) }} + - name: REQ_USERNAME + valueFrom: + secretKeyRef: + name: {{ (tpl .Values.admin.existingSecret .) | default (include "grafana.fullname" .) }} + key: {{ .Values.admin.userKey | default "admin-user" }} + {{- end }} + {{- if and (not .Values.env.GF_SECURITY_ADMIN_PASSWORD) (not .Values.env.GF_SECURITY_ADMIN_PASSWORD__FILE) (not .Values.env.GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION) }} + - name: REQ_PASSWORD + valueFrom: + secretKeyRef: + name: {{ (tpl .Values.admin.existingSecret .) | default (include "grafana.fullname" .) }} + key: {{ .Values.admin.passwordKey | default "admin-password" }} + {{- end }} + {{- if not .Values.sidecar.dashboards.skipReload }} + - name: REQ_URL + value: {{ .Values.sidecar.dashboards.reloadURL }} + - name: REQ_METHOD + value: POST + {{- end }} + {{- if .Values.sidecar.dashboards.watchServerTimeout }} + {{- if ne .Values.sidecar.dashboards.watchMethod "WATCH" }} + {{- fail (printf "Cannot use .Values.sidecar.dashboards.watchServerTimeout with .Values.sidecar.dashboards.watchMethod %s" .Values.sidecar.dashboards.watchMethod) }} + {{- end }} + - name: WATCH_SERVER_TIMEOUT + value: "{{ .Values.sidecar.dashboards.watchServerTimeout }}" + {{- end }} + {{- if .Values.sidecar.dashboards.watchClientTimeout }} + {{- if ne .Values.sidecar.dashboards.watchMethod "WATCH" }} + {{- fail (printf "Cannot use .Values.sidecar.dashboards.watchClientTimeout with .Values.sidecar.dashboards.watchMethod %s" .Values.sidecar.dashboards.watchMethod) }} + {{- end }} + - name: WATCH_CLIENT_TIMEOUT + value: {{ .Values.sidecar.dashboards.watchClientTimeout | quote }} + {{- end }} + {{- with .Values.sidecar.livenessProbe }} + livenessProbe: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.readinessProbe }} + readinessProbe: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.resources }} + resources: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.securityContext }} + securityContext: + {{- toYaml . | nindent 6 }} + {{- end }} + volumeMounts: + - name: sc-dashboard-volume + mountPath: {{ .Values.sidecar.dashboards.folder | quote }} + {{- with .Values.sidecar.dashboards.extraMounts }} + {{- toYaml . | trim | nindent 6 }} + {{- end }} +{{- end}} +{{- if .Values.sidecar.datasources.enabled }} + - name: {{ include "grafana.name" . }}-sc-datasources + {{- if .Values.sidecar.image.sha }} + image: "{{ .Values.sidecar.image.repository }}:{{ .Values.sidecar.image.tag }}@sha256:{{ .Values.sidecar.image.sha }}" + {{- else }} + image: "{{ .Values.sidecar.image.repository }}:{{ .Values.sidecar.image.tag }}" + {{- end }} + imagePullPolicy: {{ .Values.sidecar.imagePullPolicy }} + env: + {{- range $key, $value := .Values.sidecar.datasources.env }} + - name: "{{ $key }}" + value: "{{ $value }}" + {{- end }} + {{- if .Values.sidecar.datasources.ignoreAlreadyProcessed }} + - name: IGNORE_ALREADY_PROCESSED + value: "true" + {{- end }} + - name: METHOD + value: {{ .Values.sidecar.datasources.watchMethod }} + - name: LABEL + value: "{{ .Values.sidecar.datasources.label }}" + {{- with .Values.sidecar.datasources.labelValue }} + - name: LABEL_VALUE + value: {{ quote . }} + {{- end }} + {{- if or .Values.sidecar.logLevel .Values.sidecar.datasources.logLevel }} + - name: LOG_LEVEL + value: {{ default .Values.sidecar.logLevel .Values.sidecar.datasources.logLevel }} + {{- end }} + - name: FOLDER + value: "/etc/grafana/provisioning/datasources" + - name: RESOURCE + value: {{ quote .Values.sidecar.datasources.resource }} + {{- with .Values.sidecar.enableUniqueFilenames }} + - name: UNIQUE_FILENAMES + value: "{{ . }}" + {{- end }} + {{- with .Values.sidecar.datasources.searchNamespace }} + - name: NAMESPACE + value: "{{ tpl (. | join ",") $root }}" + {{- end }} + {{- if .Values.sidecar.skipTlsVerify }} + - name: SKIP_TLS_VERIFY + value: "{{ .Values.sidecar.skipTlsVerify }}" + {{- end }} + {{- if .Values.sidecar.datasources.script }} + - name: SCRIPT + value: "{{ .Values.sidecar.datasources.script }}" + {{- end }} + {{- if and (not .Values.env.GF_SECURITY_ADMIN_USER) (not .Values.env.GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION) }} + - name: REQ_USERNAME + valueFrom: + secretKeyRef: + name: {{ (tpl .Values.admin.existingSecret .) | default (include "grafana.fullname" .) }} + key: {{ .Values.admin.userKey | default "admin-user" }} + {{- end }} + {{- if and (not .Values.env.GF_SECURITY_ADMIN_PASSWORD) (not .Values.env.GF_SECURITY_ADMIN_PASSWORD__FILE) (not .Values.env.GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION) }} + - name: REQ_PASSWORD + valueFrom: + secretKeyRef: + name: {{ (tpl .Values.admin.existingSecret .) | default (include "grafana.fullname" .) }} + key: {{ .Values.admin.passwordKey | default "admin-password" }} + {{- end }} + {{- if not .Values.sidecar.datasources.skipReload }} + - name: REQ_URL + value: {{ .Values.sidecar.datasources.reloadURL }} + - name: REQ_METHOD + value: POST + {{- end }} + {{- if .Values.sidecar.datasources.watchServerTimeout }} + {{- if ne .Values.sidecar.datasources.watchMethod "WATCH" }} + {{- fail (printf "Cannot use .Values.sidecar.datasources.watchServerTimeout with .Values.sidecar.datasources.watchMethod %s" .Values.sidecar.datasources.watchMethod) }} + {{- end }} + - name: WATCH_SERVER_TIMEOUT + value: "{{ .Values.sidecar.datasources.watchServerTimeout }}" + {{- end }} + {{- if .Values.sidecar.datasources.watchClientTimeout }} + {{- if ne .Values.sidecar.datasources.watchMethod "WATCH" }} + {{- fail (printf "Cannot use .Values.sidecar.datasources.watchClientTimeout with .Values.sidecar.datasources.watchMethod %s" .Values.sidecar.datasources.watchMethod) }} + {{- end }} + - name: WATCH_CLIENT_TIMEOUT + value: "{{ .Values.sidecar.datasources.watchClientTimeout }}" + {{- end }} + {{- with .Values.sidecar.livenessProbe }} + livenessProbe: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.readinessProbe }} + readinessProbe: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.resources }} + resources: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.securityContext }} + securityContext: + {{- toYaml . | nindent 6 }} + {{- end }} + volumeMounts: + - name: sc-datasources-volume + mountPath: "/etc/grafana/provisioning/datasources" +{{- end}} +{{- if .Values.sidecar.notifiers.enabled }} + - name: {{ include "grafana.name" . }}-sc-notifiers + {{- if .Values.sidecar.image.sha }} + image: "{{ .Values.sidecar.image.repository }}:{{ .Values.sidecar.image.tag }}@sha256:{{ .Values.sidecar.image.sha }}" + {{- else }} + image: "{{ .Values.sidecar.image.repository }}:{{ .Values.sidecar.image.tag }}" + {{- end }} + imagePullPolicy: {{ .Values.sidecar.imagePullPolicy }} + env: + {{- range $key, $value := .Values.sidecar.notifiers.env }} + - name: "{{ $key }}" + value: "{{ $value }}" + {{- end }} + {{- if .Values.sidecar.notifiers.ignoreAlreadyProcessed }} + - name: IGNORE_ALREADY_PROCESSED + value: "true" + {{- end }} + - name: METHOD + value: {{ .Values.sidecar.notifiers.watchMethod }} + - name: LABEL + value: "{{ .Values.sidecar.notifiers.label }}" + {{- with .Values.sidecar.notifiers.labelValue }} + - name: LABEL_VALUE + value: {{ quote . }} + {{- end }} + {{- if or .Values.sidecar.logLevel .Values.sidecar.notifiers.logLevel }} + - name: LOG_LEVEL + value: {{ default .Values.sidecar.logLevel .Values.sidecar.notifiers.logLevel }} + {{- end }} + - name: FOLDER + value: "/etc/grafana/provisioning/notifiers" + - name: RESOURCE + value: {{ quote .Values.sidecar.notifiers.resource }} + {{- if .Values.sidecar.enableUniqueFilenames }} + - name: UNIQUE_FILENAMES + value: "{{ .Values.sidecar.enableUniqueFilenames }}" + {{- end }} + {{- with .Values.sidecar.notifiers.searchNamespace }} + - name: NAMESPACE + value: "{{ tpl (. | join ",") $root }}" + {{- end }} + {{- with .Values.sidecar.skipTlsVerify }} + - name: SKIP_TLS_VERIFY + value: "{{ . }}" + {{- end }} + {{- if .Values.sidecar.notifiers.script }} + - name: SCRIPT + value: "{{ .Values.sidecar.notifiers.script }}" + {{- end }} + {{- if and (not .Values.env.GF_SECURITY_ADMIN_USER) (not .Values.env.GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION) }} + - name: REQ_USERNAME + valueFrom: + secretKeyRef: + name: {{ (tpl .Values.admin.existingSecret .) | default (include "grafana.fullname" .) }} + key: {{ .Values.admin.userKey | default "admin-user" }} + {{- end }} + {{- if and (not .Values.env.GF_SECURITY_ADMIN_PASSWORD) (not .Values.env.GF_SECURITY_ADMIN_PASSWORD__FILE) (not .Values.env.GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION) }} + - name: REQ_PASSWORD + valueFrom: + secretKeyRef: + name: {{ (tpl .Values.admin.existingSecret .) | default (include "grafana.fullname" .) }} + key: {{ .Values.admin.passwordKey | default "admin-password" }} + {{- end }} + {{- if not .Values.sidecar.notifiers.skipReload }} + - name: REQ_URL + value: {{ .Values.sidecar.notifiers.reloadURL }} + - name: REQ_METHOD + value: POST + {{- end }} + {{- if .Values.sidecar.notifiers.watchServerTimeout }} + {{- if ne .Values.sidecar.notifiers.watchMethod "WATCH" }} + {{- fail (printf "Cannot use .Values.sidecar.notifiers.watchServerTimeout with .Values.sidecar.notifiers.watchMethod %s" .Values.sidecar.notifiers.watchMethod) }} + {{- end }} + - name: WATCH_SERVER_TIMEOUT + value: "{{ .Values.sidecar.notifiers.watchServerTimeout }}" + {{- end }} + {{- if .Values.sidecar.notifiers.watchClientTimeout }} + {{- if ne .Values.sidecar.notifiers.watchMethod "WATCH" }} + {{- fail (printf "Cannot use .Values.sidecar.notifiers.watchClientTimeout with .Values.sidecar.notifiers.watchMethod %s" .Values.sidecar.notifiers.watchMethod) }} + {{- end }} + - name: WATCH_CLIENT_TIMEOUT + value: "{{ .Values.sidecar.notifiers.watchClientTimeout }}" + {{- end }} + {{- with .Values.sidecar.livenessProbe }} + livenessProbe: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.readinessProbe }} + readinessProbe: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.resources }} + resources: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.securityContext }} + securityContext: + {{- toYaml . | nindent 6 }} + {{- end }} + volumeMounts: + - name: sc-notifiers-volume + mountPath: "/etc/grafana/provisioning/notifiers" +{{- end}} +{{- if .Values.sidecar.plugins.enabled }} + - name: {{ include "grafana.name" . }}-sc-plugins + {{- if .Values.sidecar.image.sha }} + image: "{{ .Values.sidecar.image.repository }}:{{ .Values.sidecar.image.tag }}@sha256:{{ .Values.sidecar.image.sha }}" + {{- else }} + image: "{{ .Values.sidecar.image.repository }}:{{ .Values.sidecar.image.tag }}" + {{- end }} + imagePullPolicy: {{ .Values.sidecar.imagePullPolicy }} + env: + {{- range $key, $value := .Values.sidecar.plugins.env }} + - name: "{{ $key }}" + value: "{{ $value }}" + {{- end }} + {{- if .Values.sidecar.plugins.ignoreAlreadyProcessed }} + - name: IGNORE_ALREADY_PROCESSED + value: "true" + {{- end }} + - name: METHOD + value: {{ .Values.sidecar.plugins.watchMethod }} + - name: LABEL + value: "{{ .Values.sidecar.plugins.label }}" + {{- if .Values.sidecar.plugins.labelValue }} + - name: LABEL_VALUE + value: {{ quote .Values.sidecar.plugins.labelValue }} + {{- end }} + {{- if or .Values.sidecar.logLevel .Values.sidecar.plugins.logLevel }} + - name: LOG_LEVEL + value: {{ default .Values.sidecar.logLevel .Values.sidecar.plugins.logLevel }} + {{- end }} + - name: FOLDER + value: "/etc/grafana/provisioning/plugins" + - name: RESOURCE + value: {{ quote .Values.sidecar.plugins.resource }} + {{- with .Values.sidecar.enableUniqueFilenames }} + - name: UNIQUE_FILENAMES + value: "{{ . }}" + {{- end }} + {{- with .Values.sidecar.plugins.searchNamespace }} + - name: NAMESPACE + value: "{{ tpl (. | join ",") $root }}" + {{- end }} + {{- with .Values.sidecar.plugins.script }} + - name: SCRIPT + value: "{{ . }}" + {{- end }} + {{- with .Values.sidecar.skipTlsVerify }} + - name: SKIP_TLS_VERIFY + value: "{{ . }}" + {{- end }} + {{- if and (not .Values.env.GF_SECURITY_ADMIN_USER) (not .Values.env.GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION) }} + - name: REQ_USERNAME + valueFrom: + secretKeyRef: + name: {{ (tpl .Values.admin.existingSecret .) | default (include "grafana.fullname" .) }} + key: {{ .Values.admin.userKey | default "admin-user" }} + {{- end }} + {{- if and (not .Values.env.GF_SECURITY_ADMIN_PASSWORD) (not .Values.env.GF_SECURITY_ADMIN_PASSWORD__FILE) (not .Values.env.GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION) }} + - name: REQ_PASSWORD + valueFrom: + secretKeyRef: + name: {{ (tpl .Values.admin.existingSecret .) | default (include "grafana.fullname" .) }} + key: {{ .Values.admin.passwordKey | default "admin-password" }} + {{- end }} + {{- if not .Values.sidecar.plugins.skipReload }} + - name: REQ_URL + value: {{ .Values.sidecar.plugins.reloadURL }} + - name: REQ_METHOD + value: POST + {{- end }} + {{- if .Values.sidecar.plugins.watchServerTimeout }} + {{- if ne .Values.sidecar.plugins.watchMethod "WATCH" }} + {{- fail (printf "Cannot use .Values.sidecar.plugins.watchServerTimeout with .Values.sidecar.plugins.watchMethod %s" .Values.sidecar.plugins.watchMethod) }} + {{- end }} + - name: WATCH_SERVER_TIMEOUT + value: "{{ .Values.sidecar.plugins.watchServerTimeout }}" + {{- end }} + {{- if .Values.sidecar.plugins.watchClientTimeout }} + {{- if ne .Values.sidecar.plugins.watchMethod "WATCH" }} + {{- fail (printf "Cannot use .Values.sidecar.plugins.watchClientTimeout with .Values.sidecar.plugins.watchMethod %s" .Values.sidecar.plugins.watchMethod) }} + {{- end }} + - name: WATCH_CLIENT_TIMEOUT + value: "{{ .Values.sidecar.plugins.watchClientTimeout }}" + {{- end }} + {{- with .Values.sidecar.livenessProbe }} + livenessProbe: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.readinessProbe }} + readinessProbe: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.resources }} + resources: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.sidecar.securityContext }} + securityContext: + {{- toYaml . | nindent 6 }} + {{- end }} + volumeMounts: + - name: sc-plugins-volume + mountPath: "/etc/grafana/provisioning/plugins" +{{- end}} + - name: {{ .Chart.Name }} + {{- if .Values.image.sha }} + image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}@sha256:{{ .Values.image.sha }}" + {{- else }} + image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" + {{- end }} + imagePullPolicy: {{ .Values.image.pullPolicy }} + {{- if .Values.command }} + command: + {{- range .Values.command }} + - {{ . | quote }} + {{- end }} + {{- end}} + {{- with .Values.containerSecurityContext }} + securityContext: + {{- toYaml . | nindent 6 }} + {{- end }} + volumeMounts: + - name: config + mountPath: "/etc/grafana/grafana.ini" + subPath: grafana.ini + {{- if .Values.ldap.enabled }} + - name: ldap + mountPath: "/etc/grafana/ldap.toml" + subPath: ldap.toml + {{- end }} + {{- range .Values.extraConfigmapMounts }} + - name: {{ tpl .name $root }} + mountPath: {{ tpl .mountPath $root }} + subPath: {{ (tpl .subPath $root) | default "" }} + readOnly: {{ .readOnly }} + {{- end }} + - name: storage + mountPath: "/var/lib/grafana" + {{- with .Values.persistence.subPath }} + subPath: {{ tpl . $root }} + {{- end }} + {{- with .Values.dashboards }} + {{- range $provider, $dashboards := . }} + {{- range $key, $value := $dashboards }} + {{- if (or (hasKey $value "json") (hasKey $value "file")) }} + - name: dashboards-{{ $provider }} + mountPath: "/var/lib/grafana/dashboards/{{ $provider }}/{{ $key }}.json" + subPath: "{{ $key }}.json" + {{- end }} + {{- end }} + {{- end }} + {{- end }} + {{- with .Values.dashboardsConfigMaps }} + {{- range (keys . | sortAlpha) }} + - name: dashboards-{{ . }} + mountPath: "/var/lib/grafana/dashboards/{{ . }}" + {{- end }} + {{- end }} + {{- with .Values.datasources }} + {{- range (keys . | sortAlpha) }} + - name: config + mountPath: "/etc/grafana/provisioning/datasources/{{ . }}" + subPath: {{ . | quote }} + {{- end }} + {{- end }} + {{- with .Values.notifiers }} + {{- range (keys . | sortAlpha) }} + - name: config + mountPath: "/etc/grafana/provisioning/notifiers/{{ . }}" + subPath: {{ . | quote }} + {{- end }} + {{- end }} + {{- with .Values.alerting }} + {{- range (keys . | sortAlpha) }} + - name: config + mountPath: "/etc/grafana/provisioning/alerting/{{ . }}" + subPath: {{ . | quote }} + {{- end }} + {{- end }} + {{- with .Values.dashboardProviders }} + {{- range (keys . | sortAlpha) }} + - name: config + mountPath: "/etc/grafana/provisioning/dashboards/{{ . }}" + subPath: {{ . | quote }} + {{- end }} + {{- end }} + {{- with .Values.sidecar.alerts.enabled }} + - name: sc-alerts-volume + mountPath: "/etc/grafana/provisioning/alerting" + {{- end}} + {{- if .Values.sidecar.dashboards.enabled }} + - name: sc-dashboard-volume + mountPath: {{ .Values.sidecar.dashboards.folder | quote }} + {{- if .Values.sidecar.dashboards.SCProvider }} + - name: sc-dashboard-provider + mountPath: "/etc/grafana/provisioning/dashboards/sc-dashboardproviders.yaml" + subPath: provider.yaml + {{- end}} + {{- end}} + {{- if .Values.sidecar.datasources.enabled }} + - name: sc-datasources-volume + mountPath: "/etc/grafana/provisioning/datasources" + {{- end}} + {{- if .Values.sidecar.plugins.enabled }} + - name: sc-plugins-volume + mountPath: "/etc/grafana/provisioning/plugins" + {{- end}} + {{- if .Values.sidecar.notifiers.enabled }} + - name: sc-notifiers-volume + mountPath: "/etc/grafana/provisioning/notifiers" + {{- end}} + {{- range .Values.extraSecretMounts }} + - name: {{ .name }} + mountPath: {{ .mountPath }} + readOnly: {{ .readOnly }} + subPath: {{ .subPath | default "" }} + {{- end }} + {{- range .Values.extraVolumeMounts }} + - name: {{ .name }} + mountPath: {{ .mountPath }} + subPath: {{ .subPath | default "" }} + readOnly: {{ .readOnly }} + {{- end }} + {{- range .Values.extraEmptyDirMounts }} + - name: {{ .name }} + mountPath: {{ .mountPath }} + {{- end }} + ports: + - name: {{ .Values.podPortName }} + containerPort: {{ .Values.service.targetPort }} + protocol: TCP + env: + {{- if and (not .Values.env.GF_SECURITY_ADMIN_USER) (not .Values.env.GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION) }} + - name: GF_SECURITY_ADMIN_USER + valueFrom: + secretKeyRef: + name: {{ (tpl .Values.admin.existingSecret .) | default (include "grafana.fullname" .) }} + key: {{ .Values.admin.userKey | default "admin-user" }} + {{- end }} + {{- if and (not .Values.env.GF_SECURITY_ADMIN_PASSWORD) (not .Values.env.GF_SECURITY_ADMIN_PASSWORD__FILE) (not .Values.env.GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION) }} + - name: GF_SECURITY_ADMIN_PASSWORD + valueFrom: + secretKeyRef: + name: {{ (tpl .Values.admin.existingSecret .) | default (include "grafana.fullname" .) }} + key: {{ .Values.admin.passwordKey | default "admin-password" }} + {{- end }} + {{- if .Values.plugins }} + - name: GF_INSTALL_PLUGINS + valueFrom: + configMapKeyRef: + name: {{ include "grafana.fullname" . }} + key: plugins + {{- end }} + {{- if .Values.smtp.existingSecret }} + - name: GF_SMTP_USER + valueFrom: + secretKeyRef: + name: {{ .Values.smtp.existingSecret }} + key: {{ .Values.smtp.userKey | default "user" }} + - name: GF_SMTP_PASSWORD + valueFrom: + secretKeyRef: + name: {{ .Values.smtp.existingSecret }} + key: {{ .Values.smtp.passwordKey | default "password" }} + {{- end }} + {{- if .Values.imageRenderer.enabled }} + - name: GF_RENDERING_SERVER_URL + value: http://{{ include "grafana.fullname" . }}-image-renderer.{{ include "grafana.namespace" . }}:{{ .Values.imageRenderer.service.port }}/render + - name: GF_RENDERING_CALLBACK_URL + value: {{ .Values.imageRenderer.grafanaProtocol }}://{{ include "grafana.fullname" . }}.{{ include "grafana.namespace" . }}:{{ .Values.service.port }}/{{ .Values.imageRenderer.grafanaSubPath }} + {{- end }} + - name: GF_PATHS_DATA + value: {{ (get .Values "grafana.ini").paths.data }} + - name: GF_PATHS_LOGS + value: {{ (get .Values "grafana.ini").paths.logs }} + - name: GF_PATHS_PLUGINS + value: {{ (get .Values "grafana.ini").paths.plugins }} + - name: GF_PATHS_PROVISIONING + value: {{ (get .Values "grafana.ini").paths.provisioning }} + {{- range $key, $value := .Values.envValueFrom }} + - name: {{ $key | quote }} + valueFrom: + {{- tpl (toYaml $value) $ | nindent 10 }} + {{- end }} + {{- range $key, $value := .Values.env }} + - name: "{{ tpl $key $ }}" + value: "{{ tpl (print $value) $ }}" + {{- end }} + {{- if or .Values.envFromSecret (or .Values.envRenderSecret .Values.envFromSecrets) .Values.envFromConfigMaps }} + envFrom: + {{- if .Values.envFromSecret }} + - secretRef: + name: {{ tpl .Values.envFromSecret . }} + {{- end }} + {{- if .Values.envRenderSecret }} + - secretRef: + name: {{ include "grafana.fullname" . }}-env + {{- end }} + {{- range .Values.envFromSecrets }} + - secretRef: + name: {{ tpl .name $ }} + optional: {{ .optional | default false }} + {{- end }} + {{- range .Values.envFromConfigMaps }} + - configMapRef: + name: {{ tpl .name $ }} + optional: {{ .optional | default false }} + {{- end }} + {{- end }} + {{- with .Values.livenessProbe }} + livenessProbe: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.readinessProbe }} + readinessProbe: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.lifecycleHooks }} + lifecycle: + {{- tpl (toYaml .) $root | nindent 6 }} + {{- end }} + {{- with .Values.resources }} + resources: + {{- toYaml . | nindent 6 }} + {{- end }} +{{- with .Values.extraContainers }} + {{- tpl . $ | nindent 2 }} +{{- end }} +{{- with .Values.nodeSelector }} +nodeSelector: + {{- toYaml . | nindent 2 }} +{{- end }} +{{- with .Values.affinity }} +affinity: + {{- tpl (toYaml .) $root | nindent 2 }} +{{- end }} +{{- with .Values.topologySpreadConstraints }} +topologySpreadConstraints: + {{- toYaml . | nindent 2 }} +{{- end }} +{{- with .Values.tolerations }} +tolerations: + {{- toYaml . | nindent 2 }} +{{- end }} +volumes: + - name: config + configMap: + name: {{ include "grafana.fullname" . }} + {{- range .Values.extraConfigmapMounts }} + - name: {{ tpl .name $root }} + configMap: + name: {{ tpl .configMap $root }} + {{- with .items }} + items: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- end }} + {{- if .Values.dashboards }} + {{- range (keys .Values.dashboards | sortAlpha) }} + - name: dashboards-{{ . }} + configMap: + name: {{ include "grafana.fullname" $ }}-dashboards-{{ . }} + {{- end }} + {{- end }} + {{- if .Values.dashboardsConfigMaps }} + {{- range $provider, $name := .Values.dashboardsConfigMaps }} + - name: dashboards-{{ $provider }} + configMap: + name: {{ tpl $name $root }} + {{- end }} + {{- end }} + {{- if .Values.ldap.enabled }} + - name: ldap + secret: + {{- if .Values.ldap.existingSecret }} + secretName: {{ .Values.ldap.existingSecret }} + {{- else }} + secretName: {{ include "grafana.fullname" . }} + {{- end }} + items: + - key: ldap-toml + path: ldap.toml + {{- end }} + {{- if and .Values.persistence.enabled (eq .Values.persistence.type "pvc") }} + - name: storage + persistentVolumeClaim: + claimName: {{ tpl (.Values.persistence.existingClaim | default (include "grafana.fullname" .)) . }} + {{- else if and .Values.persistence.enabled (has .Values.persistence.type $sts) }} + {{/* nothing */}} + {{- else }} + - name: storage + {{- if .Values.persistence.inMemory.enabled }} + emptyDir: + medium: Memory + {{- with .Values.persistence.inMemory.sizeLimit }} + sizeLimit: {{ . }} + {{- end }} + {{- else }} + emptyDir: {} + {{- end }} + {{- end }} + {{- if .Values.sidecar.alerts.enabled }} + - name: sc-alerts-volume + emptyDir: + {{- with .Values.sidecar.alerts.sizeLimit }} + sizeLimit: {{ . }} + {{- else }} + {} + {{- end }} + {{- end }} + {{- if .Values.sidecar.dashboards.enabled }} + - name: sc-dashboard-volume + emptyDir: + {{- with .Values.sidecar.dashboards.sizeLimit }} + sizeLimit: {{ . }} + {{- else }} + {} + {{- end }} + {{- if .Values.sidecar.dashboards.SCProvider }} + - name: sc-dashboard-provider + configMap: + name: {{ include "grafana.fullname" . }}-config-dashboards + {{- end }} + {{- end }} + {{- if .Values.sidecar.datasources.enabled }} + - name: sc-datasources-volume + emptyDir: + {{- with .Values.sidecar.datasources.sizeLimit }} + sizeLimit: {{ . }} + {{- else }} + {} + {{- end }} + {{- end }} + {{- if .Values.sidecar.plugins.enabled }} + - name: sc-plugins-volume + emptyDir: + {{- with .Values.sidecar.plugins.sizeLimit }} + sizeLimit: {{ . }} + {{- else }} + {} + {{- end }} + {{- end }} + {{- if .Values.sidecar.notifiers.enabled }} + - name: sc-notifiers-volume + emptyDir: + {{- with .Values.sidecar.notifiers.sizeLimit }} + sizeLimit: {{ . }} + {{- else }} + {} + {{- end }} + {{- end }} + {{- range .Values.extraSecretMounts }} + {{- if .secretName }} + - name: {{ .name }} + secret: + secretName: {{ .secretName }} + defaultMode: {{ .defaultMode }} + {{- with .items }} + items: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- else if .projected }} + - name: {{ .name }} + projected: + {{- toYaml .projected | nindent 6 }} + {{- else if .csi }} + - name: {{ .name }} + csi: + {{- toYaml .csi | nindent 6 }} + {{- end }} + {{- end }} + {{- range .Values.extraVolumeMounts }} + - name: {{ .name }} + {{- if .existingClaim }} + persistentVolumeClaim: + claimName: {{ .existingClaim }} + {{- else if .hostPath }} + hostPath: + path: {{ .hostPath }} + {{- else if .csi }} + csi: + {{- toYaml .data | nindent 6 }} + {{- else }} + emptyDir: {} + {{- end }} + {{- end }} + {{- range .Values.extraEmptyDirMounts }} + - name: {{ .name }} + emptyDir: {} + {{- end }} + {{- with .Values.extraContainerVolumes }} + {{- tpl (toYaml .) $root | nindent 2 }} + {{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/clusterrole.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/clusterrole.yaml new file mode 100644 index 00000000..3396713a --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/clusterrole.yaml @@ -0,0 +1,25 @@ +{{- if and .Values.rbac.create (not .Values.rbac.namespaced) (not .Values.rbac.useExistingRole) }} +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} + name: {{ include "grafana.fullname" . }}-clusterrole +{{- if or .Values.sidecar.dashboards.enabled (or .Values.rbac.extraClusterRoleRules (or .Values.sidecar.datasources.enabled .Values.sidecar.plugins.enabled)) }} +rules: + {{- if or .Values.sidecar.dashboards.enabled (or .Values.sidecar.datasources.enabled .Values.sidecar.plugins.enabled) }} + - apiGroups: [""] # "" indicates the core API group + resources: ["configmaps", "secrets"] + verbs: ["get", "watch", "list"] + {{- end}} + {{- with .Values.rbac.extraClusterRoleRules }} + {{- toYaml . | nindent 2 }} + {{- end}} +{{- else }} +rules: [] +{{- end}} +{{- end}} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/clusterrolebinding.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/clusterrolebinding.yaml new file mode 100644 index 00000000..48411fe8 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/clusterrolebinding.yaml @@ -0,0 +1,24 @@ +{{- if and .Values.rbac.create (not .Values.rbac.namespaced) }} +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: {{ include "grafana.fullname" . }}-clusterrolebinding + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +subjects: + - kind: ServiceAccount + name: {{ include "grafana.serviceAccountName" . }} + namespace: {{ include "grafana.namespace" . }} +roleRef: + kind: ClusterRole + {{- if .Values.rbac.useExistingRole }} + name: {{ .Values.rbac.useExistingRole }} + {{- else }} + name: {{ include "grafana.fullname" . }}-clusterrole + {{- end }} + apiGroup: rbac.authorization.k8s.io +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/configmap-dashboard-provider.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/configmap-dashboard-provider.yaml new file mode 100644 index 00000000..1f706a8b --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/configmap-dashboard-provider.yaml @@ -0,0 +1,29 @@ +{{- if and .Values.sidecar.dashboards.enabled .Values.sidecar.dashboards.SCProvider }} +apiVersion: v1 +kind: ConfigMap +metadata: + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} + name: {{ include "grafana.fullname" . }}-config-dashboards + namespace: {{ include "grafana.namespace" . }} +data: + provider.yaml: |- + apiVersion: 1 + providers: + - name: '{{ .Values.sidecar.dashboards.provider.name }}' + orgId: {{ .Values.sidecar.dashboards.provider.orgid }} + {{- if not .Values.sidecar.dashboards.provider.foldersFromFilesStructure }} + folder: '{{ .Values.sidecar.dashboards.provider.folder }}' + {{- end }} + type: {{ .Values.sidecar.dashboards.provider.type }} + disableDeletion: {{ .Values.sidecar.dashboards.provider.disableDelete }} + allowUiUpdates: {{ .Values.sidecar.dashboards.provider.allowUiUpdates }} + updateIntervalSeconds: {{ .Values.sidecar.dashboards.provider.updateIntervalSeconds | default 30 }} + options: + foldersFromFilesStructure: {{ .Values.sidecar.dashboards.provider.foldersFromFilesStructure }} + path: {{ .Values.sidecar.dashboards.folder }}{{- with .Values.sidecar.dashboards.defaultFolderName }}/{{ . }}{{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/configmap.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/configmap.yaml new file mode 100644 index 00000000..b0735a2b --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/configmap.yaml @@ -0,0 +1,125 @@ +{{- if .Values.createConfigmap }} +{{- $root := . -}} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "grafana.fullname" . }} + namespace: {{ include "grafana.namespace" . }} + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +data: + {{- with .Values.plugins }} + plugins: {{ join "," . }} + {{- end }} + grafana.ini: | + {{- range $elem, $elemVal := index .Values "grafana.ini" }} + {{- if not (kindIs "map" $elemVal) }} + {{- if kindIs "invalid" $elemVal }} + {{ $elem }} = + {{- else if kindIs "string" $elemVal }} + {{ $elem }} = {{ tpl $elemVal $ }} + {{- else }} + {{ $elem }} = {{ $elemVal }} + {{- end }} + {{- end }} + {{- end }} + {{- range $key, $value := index .Values "grafana.ini" }} + {{- if kindIs "map" $value }} + [{{ $key }}] + {{- range $elem, $elemVal := $value }} + {{- if kindIs "invalid" $elemVal }} + {{ $elem }} = + {{- else if kindIs "string" $elemVal }} + {{ $elem }} = {{ tpl $elemVal $ }} + {{- else }} + {{ $elem }} = {{ $elemVal }} + {{- end }} + {{- end }} + {{- end }} + {{- end }} + + {{- range $key, $value := .Values.datasources }} + {{- $key | nindent 2 }}: | + {{- tpl (toYaml $value | nindent 4) $root }} + {{- end }} + + {{- range $key, $value := .Values.notifiers }} + {{- $key | nindent 2 }}: | + {{- toYaml $value | nindent 4 }} + {{- end }} + + {{- range $key, $value := .Values.alerting }} + {{- $key | nindent 2 }}: | + {{- tpl (toYaml $value | nindent 4) $root }} + {{- end }} + + {{- range $key, $value := .Values.dashboardProviders }} + {{- $key | nindent 2 }}: | + {{- toYaml $value | nindent 4 }} + {{- end }} + +{{- if .Values.dashboards }} + download_dashboards.sh: | + #!/usr/bin/env sh + set -euf + {{- if .Values.dashboardProviders }} + {{- range $key, $value := .Values.dashboardProviders }} + {{- range $value.providers }} + mkdir -p {{ .options.path }} + {{- end }} + {{- end }} + {{- end }} + {{ $dashboardProviders := .Values.dashboardProviders }} + {{- range $provider, $dashboards := .Values.dashboards }} + {{- range $key, $value := $dashboards }} + {{- if (or (hasKey $value "gnetId") (hasKey $value "url")) }} + curl -skf \ + --connect-timeout 60 \ + --max-time 60 \ + {{- if not $value.b64content }} + -H "Accept: application/json" \ + {{- if $value.token }} + -H "Authorization: token {{ $value.token }}" \ + {{- end }} + {{- if $value.bearerToken }} + -H "Authorization: Bearer {{ $value.bearerToken }}" \ + {{- end }} + {{- if $value.gitlabToken }} + -H "PRIVATE-TOKEN: {{ $value.gitlabToken }}" \ + {{- end }} + -H "Content-Type: application/json;charset=UTF-8" \ + {{- end }} + {{- $dpPath := "" -}} + {{- range $kd := (index $dashboardProviders "dashboardproviders.yaml").providers }} + {{- if eq $kd.name $provider }} + {{- $dpPath = $kd.options.path }} + {{- end }} + {{- end }} + {{- if $value.url }} + "{{ $value.url }}" \ + {{- else }} + "https://grafana.com/api/dashboards/{{ $value.gnetId }}/revisions/{{- if $value.revision -}}{{ $value.revision }}{{- else -}}1{{- end -}}/download" \ + {{- end }} + {{- if $value.datasource }} + {{- if kindIs "string" $value.datasource }} + | sed '/-- .* --/! s/"datasource":.*,/"datasource": "{{ $value.datasource }}",/g' \ + {{- end }} + {{- if kindIs "slice" $value.datasource }} + {{- range $value.datasource }} + | sed '/-- .* --/! s/${{"{"}}{{ .name }}}/{{ .value }}/g' \ + {{- end }} + {{- end }} + {{- end }} + {{- if $value.b64content }} + | base64 -d \ + {{- end }} + > "{{- if $dpPath -}}{{ $dpPath }}{{- else -}}/var/lib/grafana/dashboards/{{ $provider }}{{- end -}}/{{ $key }}.json" + {{ end }} + {{- end }} + {{- end }} +{{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/dashboards-json-configmap.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/dashboards-json-configmap.yaml new file mode 100644 index 00000000..df0ed0d8 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/dashboards-json-configmap.yaml @@ -0,0 +1,35 @@ +{{- if .Values.dashboards }} +{{ $files := .Files }} +{{- range $provider, $dashboards := .Values.dashboards }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "grafana.fullname" $ }}-dashboards-{{ $provider }} + namespace: {{ include "grafana.namespace" $ }} + labels: + {{- include "grafana.labels" $ | nindent 4 }} + dashboard-provider: {{ $provider }} +{{- if $dashboards }} +data: +{{- $dashboardFound := false }} +{{- range $key, $value := $dashboards }} +{{- if (or (hasKey $value "json") (hasKey $value "file")) }} +{{- $dashboardFound = true }} + {{- print $key | nindent 2 }}.json: + {{- if hasKey $value "json" }} + |- + {{- $value.json | nindent 6 }} + {{- end }} + {{- if hasKey $value "file" }} + {{- toYaml ( $files.Get $value.file ) | nindent 4}} + {{- end }} +{{- end }} +{{- end }} +{{- if not $dashboardFound }} + {} +{{- end }} +{{- end }} +--- +{{- end }} + +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/deployment.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/deployment.yaml new file mode 100644 index 00000000..96eac4d2 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/deployment.yaml @@ -0,0 +1,50 @@ +{{- if (and (not .Values.useStatefulSet) (or (not .Values.persistence.enabled) (eq .Values.persistence.type "pvc"))) }} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "grafana.fullname" . }} + namespace: {{ include "grafana.namespace" . }} + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.labels }} + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + {{- if and (not .Values.autoscaling.enabled) (.Values.replicas) }} + replicas: {{ .Values.replicas }} + {{- end }} + revisionHistoryLimit: {{ .Values.revisionHistoryLimit }} + selector: + matchLabels: + {{- include "grafana.selectorLabels" . | nindent 6 }} + {{- with .Values.deploymentStrategy }} + strategy: + {{- toYaml . | trim | nindent 4 }} + {{- end }} + template: + metadata: + labels: + {{- include "grafana.selectorLabels" . | nindent 8 }} + {{- with .Values.podLabels }} + {{- toYaml . | nindent 8 }} + {{- end }} + annotations: + checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} + checksum/dashboards-json-config: {{ include (print $.Template.BasePath "/dashboards-json-configmap.yaml") . | sha256sum }} + checksum/sc-dashboard-provider-config: {{ include (print $.Template.BasePath "/configmap-dashboard-provider.yaml") . | sha256sum }} + {{- if and (or (and (not .Values.admin.existingSecret) (not .Values.env.GF_SECURITY_ADMIN_PASSWORD__FILE) (not .Values.env.GF_SECURITY_ADMIN_PASSWORD)) (and .Values.ldap.enabled (not .Values.ldap.existingSecret))) (not .Values.env.GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION) }} + checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }} + {{- end }} + {{- if .Values.envRenderSecret }} + checksum/secret-env: {{ include (print $.Template.BasePath "/secret-env.yaml") . | sha256sum }} + {{- end }} + {{- with .Values.podAnnotations }} + {{- toYaml . | nindent 8 }} + {{- end }} + spec: + {{- include "grafana.pod" . | nindent 6 }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/extra-manifests.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/extra-manifests.yaml new file mode 100644 index 00000000..a9bb3b6b --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/extra-manifests.yaml @@ -0,0 +1,4 @@ +{{ range .Values.extraObjects }} +--- +{{ tpl (toYaml .) $ }} +{{ end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/headless-service.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/headless-service.yaml new file mode 100644 index 00000000..caaed5d5 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/headless-service.yaml @@ -0,0 +1,23 @@ +{{- $sts := list "sts" "StatefulSet" "statefulset" -}} +{{- if or .Values.headlessService (and .Values.persistence.enabled (not .Values.persistence.existingClaim) (has .Values.persistence.type $sts)) }} +apiVersion: v1 +kind: Service +metadata: + name: {{ include "grafana.fullname" . }}-headless + namespace: {{ include "grafana.namespace" . }} + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + clusterIP: None + selector: + {{- include "grafana.selectorLabels" . | nindent 4 }} + type: ClusterIP + ports: + - protocol: TCP + port: 3000 + targetPort: {{ .Values.service.targetPort }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/hpa.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/hpa.yaml new file mode 100644 index 00000000..f53dfc83 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/hpa.yaml @@ -0,0 +1,49 @@ +{{- $sts := list "sts" "StatefulSet" "statefulset" -}} +{{- if .Values.autoscaling.enabled }} +apiVersion: {{ include "grafana.hpa.apiVersion" . }} +kind: HorizontalPodAutoscaler +metadata: + name: {{ include "grafana.fullname" . }} + namespace: {{ include "grafana.namespace" . }} + labels: + app.kubernetes.io/name: {{ include "grafana.name" . }} + helm.sh/chart: {{ include "grafana.chart" . }} + app.kubernetes.io/managed-by: {{ .Release.Service }} + app.kubernetes.io/instance: {{ .Release.Name }} +spec: + scaleTargetRef: + apiVersion: apps/v1 + {{- if has .Values.persistence.type $sts }} + kind: StatefulSet + {{- else }} + kind: Deployment + {{- end }} + name: {{ include "grafana.fullname" . }} + minReplicas: {{ .Values.autoscaling.minReplicas }} + maxReplicas: {{ .Values.autoscaling.maxReplicas }} + metrics: + {{- if .Values.autoscaling.targetMemory }} + - type: Resource + resource: + name: memory + {{- if semverCompare "<1.23-0" .Capabilities.KubeVersion.Version }} + targetAverageUtilization: {{ .Values.autoscaling.targetMemory }} + {{- else }} + target: + type: Utilization + averageUtilization: {{ .Values.autoscaling.targetMemory }} + {{- end }} + {{- end }} + {{- if .Values.autoscaling.targetCPU }} + - type: Resource + resource: + name: cpu + {{- if semverCompare "<1.23-0" .Capabilities.KubeVersion.Version }} + targetAverageUtilization: {{ .Values.autoscaling.targetCPU }} + {{- else }} + target: + type: Utilization + averageUtilization: {{ .Values.autoscaling.targetCPU }} + {{- end }} + {{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/image-renderer-deployment.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/image-renderer-deployment.yaml new file mode 100644 index 00000000..b087179f --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/image-renderer-deployment.yaml @@ -0,0 +1,119 @@ +{{ if .Values.imageRenderer.enabled }} +{{- $root := . -}} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "grafana.fullname" . }}-image-renderer + namespace: {{ include "grafana.namespace" . }} + labels: + {{- include "grafana.imageRenderer.labels" . | nindent 4 }} + {{- with .Values.imageRenderer.labels }} + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.imageRenderer.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + replicas: {{ .Values.imageRenderer.replicas }} + revisionHistoryLimit: {{ .Values.imageRenderer.revisionHistoryLimit }} + selector: + matchLabels: + {{- include "grafana.imageRenderer.selectorLabels" . | nindent 6 }} + + {{- with .Values.imageRenderer.deploymentStrategy }} + strategy: + {{- toYaml . | trim | nindent 4 }} + {{- end }} + template: + metadata: + labels: + {{- include "grafana.imageRenderer.selectorLabels" . | nindent 8 }} + {{- with .Values.imageRenderer.podLabels }} + {{- toYaml . | nindent 8 }} + {{- end }} + annotations: + checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} + {{- with .Values.imageRenderer.podAnnotations }} + {{- toYaml . | nindent 8 }} + {{- end }} + spec: + {{- with .Values.imageRenderer.schedulerName }} + schedulerName: "{{ . }}" + {{- end }} + {{- with .Values.imageRenderer.serviceAccountName }} + serviceAccountName: "{{ . }}" + {{- end }} + {{- with .Values.imageRenderer.securityContext }} + securityContext: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.imageRenderer.hostAliases }} + hostAliases: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.imageRenderer.priorityClassName }} + priorityClassName: {{ . }} + {{- end }} + {{- with .Values.imageRenderer.image.pullSecrets }} + imagePullSecrets: + {{- range . }} + - name: {{ tpl . $root }} + {{- end}} + {{- end }} + containers: + - name: {{ .Chart.Name }}-image-renderer + {{- if .Values.imageRenderer.image.sha }} + image: "{{ .Values.imageRenderer.image.repository }}:{{ .Values.imageRenderer.image.tag }}@sha256:{{ .Values.imageRenderer.image.sha }}" + {{- else }} + image: "{{ .Values.imageRenderer.image.repository }}:{{ .Values.imageRenderer.image.tag }}" + {{- end }} + imagePullPolicy: {{ .Values.imageRenderer.image.pullPolicy }} + {{- if .Values.imageRenderer.command }} + command: + {{- range .Values.imageRenderer.command }} + - {{ . }} + {{- end }} + {{- end}} + ports: + - name: {{ .Values.imageRenderer.service.portName }} + containerPort: {{ .Values.imageRenderer.service.targetPort }} + protocol: TCP + livenessProbe: + httpGet: + path: / + port: {{ .Values.imageRenderer.service.portName }} + env: + - name: HTTP_PORT + value: {{ .Values.imageRenderer.service.targetPort | quote }} + {{- range $key, $value := .Values.imageRenderer.env }} + - name: {{ $key | quote }} + value: {{ $value | quote }} + {{- end }} + {{- with .Values.imageRenderer.containerSecurityContext }} + securityContext: + {{- toYaml . | nindent 12 }} + {{- end }} + volumeMounts: + - mountPath: /tmp + name: image-renderer-tmpfs + {{- with .Values.imageRenderer.resources }} + resources: + {{- toYaml . | nindent 12 }} + {{- end }} + {{- with .Values.imageRenderer.nodeSelector }} + nodeSelector: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.imageRenderer.affinity }} + affinity: + {{- tpl (toYaml .) $root | nindent 8 }} + {{- end }} + {{- with .Values.imageRenderer.tolerations }} + tolerations: + {{- toYaml . | nindent 8 }} + {{- end }} + volumes: + - name: image-renderer-tmpfs + emptyDir: {} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/image-renderer-network-policy.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/image-renderer-network-policy.yaml new file mode 100644 index 00000000..fb694451 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/image-renderer-network-policy.yaml @@ -0,0 +1,73 @@ +{{- if and .Values.imageRenderer.enabled .Values.imageRenderer.networkPolicy.limitIngress }} +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ include "grafana.fullname" . }}-image-renderer-ingress + namespace: {{ include "grafana.namespace" . }} + annotations: + comment: Limit image-renderer ingress traffic from grafana +spec: + podSelector: + matchLabels: + {{- include "grafana.imageRenderer.selectorLabels" . | nindent 6 }} + {{- with .Values.imageRenderer.podLabels }} + {{- toYaml . | nindent 6 }} + {{- end }} + + policyTypes: + - Ingress + ingress: + - ports: + - port: {{ .Values.imageRenderer.service.targetPort }} + protocol: TCP + from: + - namespaceSelector: + matchLabels: + name: {{ include "grafana.namespace" . }} + - podSelector: + matchLabels: + {{- include "grafana.selectorLabels" . | nindent 14 }} + {{- with .Values.podLabels }} + {{- toYaml . | nindent 14 }} + {{- end }} +{{- end }} + +{{- if and .Values.imageRenderer.enabled .Values.imageRenderer.networkPolicy.limitEgress }} +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ include "grafana.fullname" . }}-image-renderer-egress + namespace: {{ include "grafana.namespace" . }} + annotations: + comment: Limit image-renderer egress traffic to grafana +spec: + podSelector: + matchLabels: + {{- include "grafana.imageRenderer.selectorLabels" . | nindent 6 }} + {{- with .Values.imageRenderer.podLabels }} + {{- toYaml . | nindent 6 }} + {{- end }} + + policyTypes: + - Egress + egress: + # allow dns resolution + - ports: + - port: 53 + protocol: UDP + - port: 53 + protocol: TCP + # talk only to grafana + - ports: + - port: {{ .Values.service.port }} + protocol: TCP + to: + - podSelector: + matchLabels: + {{- include "grafana.selectorLabels" . | nindent 14 }} + {{- with .Values.podLabels }} + {{- toYaml . | nindent 14 }} + {{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/image-renderer-service.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/image-renderer-service.yaml new file mode 100644 index 00000000..f8da127c --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/image-renderer-service.yaml @@ -0,0 +1,31 @@ +{{- if and .Values.imageRenderer.enabled .Values.imageRenderer.service.enabled }} +apiVersion: v1 +kind: Service +metadata: + name: {{ include "grafana.fullname" . }}-image-renderer + namespace: {{ include "grafana.namespace" . }} + labels: + {{- include "grafana.imageRenderer.labels" . | nindent 4 }} + {{- with .Values.imageRenderer.service.labels }} + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.imageRenderer.service.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + type: ClusterIP + {{- with .Values.imageRenderer.service.clusterIP }} + clusterIP: {{ . }} + {{- end }} + ports: + - name: {{ .Values.imageRenderer.service.portName }} + port: {{ .Values.imageRenderer.service.port }} + protocol: TCP + targetPort: {{ .Values.imageRenderer.service.targetPort }} + {{- with .Values.imageRenderer.appProtocol }} + appProtocol: {{ . }} + {{- end }} + selector: + {{- include "grafana.imageRenderer.selectorLabels" . | nindent 4 }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/ingress.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/ingress.yaml new file mode 100644 index 00000000..063cdfaa --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/ingress.yaml @@ -0,0 +1,78 @@ +{{- if .Values.ingress.enabled -}} +{{- $ingressApiIsStable := eq (include "grafana.ingress.isStable" .) "true" -}} +{{- $ingressSupportsIngressClassName := eq (include "grafana.ingress.supportsIngressClassName" .) "true" -}} +{{- $ingressSupportsPathType := eq (include "grafana.ingress.supportsPathType" .) "true" -}} +{{- $fullName := include "grafana.fullname" . -}} +{{- $servicePort := .Values.service.port -}} +{{- $ingressPath := .Values.ingress.path -}} +{{- $ingressPathType := .Values.ingress.pathType -}} +{{- $extraPaths := .Values.ingress.extraPaths -}} +apiVersion: {{ include "grafana.ingress.apiVersion" . }} +kind: Ingress +metadata: + name: {{ $fullName }} + namespace: {{ include "grafana.namespace" . }} + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.ingress.labels }} + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.ingress.annotations }} + annotations: + {{- range $key, $value := . }} + {{ $key }}: {{ tpl $value $ | quote }} + {{- end }} + {{- end }} +spec: + {{- if and $ingressSupportsIngressClassName .Values.ingress.ingressClassName }} + ingressClassName: {{ .Values.ingress.ingressClassName }} + {{- end -}} + {{- with .Values.ingress.tls }} + tls: + {{- tpl (toYaml .) $ | nindent 4 }} + {{- end }} + rules: + {{- if .Values.ingress.hosts }} + {{- range .Values.ingress.hosts }} + - host: {{ tpl . $ }} + http: + paths: + {{- with $extraPaths }} + {{- toYaml . | nindent 10 }} + {{- end }} + - path: {{ $ingressPath }} + {{- if $ingressSupportsPathType }} + pathType: {{ $ingressPathType }} + {{- end }} + backend: + {{- if $ingressApiIsStable }} + service: + name: {{ $fullName }} + port: + number: {{ $servicePort }} + {{- else }} + serviceName: {{ $fullName }} + servicePort: {{ $servicePort }} + {{- end }} + {{- end }} + {{- else }} + - http: + paths: + - backend: + {{- if $ingressApiIsStable }} + service: + name: {{ $fullName }} + port: + number: {{ $servicePort }} + {{- else }} + serviceName: {{ $fullName }} + servicePort: {{ $servicePort }} + {{- end }} + {{- with $ingressPath }} + path: {{ . }} + {{- end }} + {{- if $ingressSupportsPathType }} + pathType: {{ $ingressPathType }} + {{- end }} + {{- end -}} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/networkpolicy.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/networkpolicy.yaml new file mode 100644 index 00000000..ea4578be --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/networkpolicy.yaml @@ -0,0 +1,52 @@ +{{- if .Values.networkPolicy.enabled }} +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ include "grafana.fullname" . }} + namespace: {{ include "grafana.namespace" . }} + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.labels }} + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + policyTypes: + {{- if .Values.networkPolicy.ingress }} + - Ingress + {{- end }} + {{- if .Values.networkPolicy.egress.enabled }} + - Egress + {{- end }} + podSelector: + matchLabels: + {{- include "grafana.selectorLabels" . | nindent 6 }} + + {{- if .Values.networkPolicy.egress.enabled }} + egress: + - ports: + {{ .Values.networkPolicy.egress.ports | toJson }} + {{- end }} + {{- if .Values.networkPolicy.ingress }} + ingress: + - ports: + - port: {{ .Values.service.targetPort }} + {{- if not .Values.networkPolicy.allowExternal }} + from: + - podSelector: + matchLabels: + {{ include "grafana.fullname" . }}-client: "true" + {{- with .Values.networkPolicy.explicitNamespacesSelector }} + - namespaceSelector: + {{- toYaml . | nindent 12 }} + {{- end }} + - podSelector: + matchLabels: + {{- include "grafana.labels" . | nindent 14 }} + role: read + {{- end }} + {{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/poddisruptionbudget.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/poddisruptionbudget.yaml new file mode 100644 index 00000000..05251214 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/poddisruptionbudget.yaml @@ -0,0 +1,22 @@ +{{- if .Values.podDisruptionBudget }} +apiVersion: {{ include "grafana.podDisruptionBudget.apiVersion" . }} +kind: PodDisruptionBudget +metadata: + name: {{ include "grafana.fullname" . }} + namespace: {{ include "grafana.namespace" . }} + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.labels }} + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + {{- with .Values.podDisruptionBudget.minAvailable }} + minAvailable: {{ . }} + {{- end }} + {{- with .Values.podDisruptionBudget.maxUnavailable }} + maxUnavailable: {{ . }} + {{- end }} + selector: + matchLabels: + {{- include "grafana.selectorLabels" . | nindent 6 }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/podsecuritypolicy.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/podsecuritypolicy.yaml new file mode 100644 index 00000000..eed7af95 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/podsecuritypolicy.yaml @@ -0,0 +1,49 @@ +{{- if and .Values.rbac.pspEnabled (.Capabilities.APIVersions.Has "policy/v1beta1/PodSecurityPolicy") }} +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: {{ include "grafana.fullname" . }} + labels: + {{- include "grafana.labels" . | nindent 4 }} + annotations: + seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default,runtime/default' + seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default' + {{- if .Values.rbac.pspUseAppArmor }} + apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default' + apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default' + {{- end }} +spec: + privileged: false + allowPrivilegeEscalation: false + requiredDropCapabilities: + # Default set from Docker, with DAC_OVERRIDE and CHOWN + - ALL + volumes: + - 'configMap' + - 'emptyDir' + - 'projected' + - 'csi' + - 'secret' + - 'downwardAPI' + - 'persistentVolumeClaim' + hostNetwork: false + hostIPC: false + hostPID: false + runAsUser: + rule: 'RunAsAny' + seLinux: + rule: 'RunAsAny' + supplementalGroups: + rule: 'MustRunAs' + ranges: + # Forbid adding the root group. + - min: 1 + max: 65535 + fsGroup: + rule: 'MustRunAs' + ranges: + # Forbid adding the root group. + - min: 1 + max: 65535 + readOnlyRootFilesystem: false +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/pvc.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/pvc.yaml new file mode 100644 index 00000000..eb8f87f0 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/pvc.yaml @@ -0,0 +1,36 @@ +{{- if and .Values.persistence.enabled (not .Values.persistence.existingClaim) (eq .Values.persistence.type "pvc")}} +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: {{ include "grafana.fullname" . }} + namespace: {{ include "grafana.namespace" . }} + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.persistence.extraPvcLabels }} + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.persistence.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.persistence.finalizers }} + finalizers: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + accessModes: + {{- range .Values.persistence.accessModes }} + - {{ . | quote }} + {{- end }} + resources: + requests: + storage: {{ .Values.persistence.size | quote }} + {{- with .Values.persistence.storageClassName }} + storageClassName: {{ . }} + {{- end }} + {{- with .Values.persistence.selectorLabels }} + selector: + matchLabels: + {{- toYaml . | nindent 6 }} + {{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/role.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/role.yaml new file mode 100644 index 00000000..ffdb16f6 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/role.yaml @@ -0,0 +1,32 @@ +{{- if and .Values.rbac.create (not .Values.rbac.useExistingRole) -}} +apiVersion: {{ include "grafana.rbac.apiVersion" . }} +kind: Role +metadata: + name: {{ include "grafana.fullname" . }} + namespace: {{ include "grafana.namespace" . }} + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- if or .Values.rbac.pspEnabled (and .Values.rbac.namespaced (or .Values.sidecar.dashboards.enabled .Values.sidecar.datasources.enabled .Values.sidecar.plugins.enabled .Values.rbac.extraRoleRules)) }} +rules: + {{- if .Values.rbac.pspEnabled }} + - apiGroups: ['extensions'] + resources: ['podsecuritypolicies'] + verbs: ['use'] + resourceNames: [{{ include "grafana.fullname" . }}] + {{- end }} + {{- if and .Values.rbac.namespaced (or .Values.sidecar.dashboards.enabled .Values.sidecar.datasources.enabled .Values.sidecar.plugins.enabled) }} + - apiGroups: [""] # "" indicates the core API group + resources: ["configmaps", "secrets"] + verbs: ["get", "watch", "list"] + {{- end }} + {{- with .Values.rbac.extraRoleRules }} + {{- toYaml . | nindent 2 }} + {{- end}} +{{- else }} +rules: [] +{{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/rolebinding.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/rolebinding.yaml new file mode 100644 index 00000000..cc07bd9a --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/rolebinding.yaml @@ -0,0 +1,25 @@ +{{- if .Values.rbac.create }} +apiVersion: {{ include "grafana.rbac.apiVersion" . }} +kind: RoleBinding +metadata: + name: {{ include "grafana.fullname" . }} + namespace: {{ include "grafana.namespace" . }} + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + {{- if .Values.rbac.useExistingRole }} + name: {{ .Values.rbac.useExistingRole }} + {{- else }} + name: {{ include "grafana.fullname" . }} + {{- end }} +subjects: +- kind: ServiceAccount + name: {{ include "grafana.serviceAccountName" . }} + namespace: {{ include "grafana.namespace" . }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/secret-env.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/secret-env.yaml new file mode 100644 index 00000000..c7655671 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/secret-env.yaml @@ -0,0 +1,14 @@ +{{- if .Values.envRenderSecret }} +apiVersion: v1 +kind: Secret +metadata: + name: {{ include "grafana.fullname" . }}-env + namespace: {{ include "grafana.namespace" . }} + labels: + {{- include "grafana.labels" . | nindent 4 }} +type: Opaque +data: +{{- range $key, $val := .Values.envRenderSecret }} + {{ $key }}: {{ $val | b64enc | quote }} +{{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/secret.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/secret.yaml new file mode 100644 index 00000000..5cbd5274 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/secret.yaml @@ -0,0 +1,26 @@ +{{- if or (and (not .Values.admin.existingSecret) (not .Values.env.GF_SECURITY_ADMIN_PASSWORD__FILE) (not .Values.env.GF_SECURITY_ADMIN_PASSWORD) (not .Values.env.GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION)) (and .Values.ldap.enabled (not .Values.ldap.existingSecret)) }} +apiVersion: v1 +kind: Secret +metadata: + name: {{ include "grafana.fullname" . }} + namespace: {{ include "grafana.namespace" . }} + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +type: Opaque +data: + {{- if and (not .Values.env.GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION) (not .Values.admin.existingSecret) (not .Values.env.GF_SECURITY_ADMIN_PASSWORD__FILE) (not .Values.env.GF_SECURITY_ADMIN_PASSWORD) }} + admin-user: {{ .Values.adminUser | b64enc | quote }} + {{- if .Values.adminPassword }} + admin-password: {{ .Values.adminPassword | b64enc | quote }} + {{- else }} + admin-password: {{ include "grafana.password" . }} + {{- end }} + {{- end }} + {{- if not .Values.ldap.existingSecret }} + ldap-toml: {{ tpl .Values.ldap.config $ | b64enc | quote }} + {{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/service.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/service.yaml new file mode 100644 index 00000000..43d360b5 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/service.yaml @@ -0,0 +1,55 @@ +{{- if .Values.service.enabled }} +{{- $root := . }} +apiVersion: v1 +kind: Service +metadata: + name: {{ include "grafana.fullname" . }} + namespace: {{ include "grafana.namespace" . }} + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.service.labels }} + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.service.annotations }} + annotations: + {{- tpl (toYaml . | nindent 4) $root }} + {{- end }} +spec: + {{- if (or (eq .Values.service.type "ClusterIP") (empty .Values.service.type)) }} + type: ClusterIP + {{- with .Values.service.clusterIP }} + clusterIP: {{ . }} + {{- end }} + {{- else if eq .Values.service.type "LoadBalancer" }} + type: {{ .Values.service.type }} + {{- with .Values.service.loadBalancerIP }} + loadBalancerIP: {{ . }} + {{- end }} + {{- with .Values.service.loadBalancerSourceRanges }} + loadBalancerSourceRanges: + {{- toYaml . | nindent 4 }} + {{- end }} + {{- else }} + type: {{ .Values.service.type }} + {{- end }} + {{- with .Values.service.externalIPs }} + externalIPs: + {{- toYaml . | nindent 4 }} + {{- end }} + ports: + - name: {{ .Values.service.portName }} + port: {{ .Values.service.port }} + protocol: TCP + targetPort: {{ .Values.service.targetPort }} + {{- with .Values.service.appProtocol }} + appProtocol: {{ . }} + {{- end }} + {{- if (and (eq .Values.service.type "NodePort") (not (empty .Values.service.nodePort))) }} + nodePort: {{ .Values.service.nodePort }} + {{- end }} + {{- with .Values.extraExposePorts }} + {{- tpl (toYaml . | nindent 4) $root }} + {{- end }} + selector: + {{- include "grafana.selectorLabels" . | nindent 4 }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/serviceaccount.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/serviceaccount.yaml new file mode 100644 index 00000000..784e71ba --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/serviceaccount.yaml @@ -0,0 +1,17 @@ +{{- if .Values.serviceAccount.create }} +{{- $root := . -}} +apiVersion: v1 +kind: ServiceAccount +metadata: + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.labels }} + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- tpl (toYaml . | nindent 4) $root }} + {{- end }} + name: {{ include "grafana.serviceAccountName" . }} + namespace: {{ include "grafana.namespace" . }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/servicemonitor.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/servicemonitor.yaml new file mode 100644 index 00000000..6575fb9a --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/servicemonitor.yaml @@ -0,0 +1,44 @@ +{{- if .Values.serviceMonitor.enabled }} +--- +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: {{ include "grafana.fullname" . }} + {{- if .Values.serviceMonitor.namespace }} + namespace: {{ tpl .Values.serviceMonitor.namespace . }} + {{- else }} + namespace: {{ include "grafana.namespace" . }} + {{- end }} + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.serviceMonitor.labels }} + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + endpoints: + - port: {{ .Values.service.portName }} + {{- with .Values.serviceMonitor.interval }} + interval: {{ . }} + {{- end }} + {{- with .Values.serviceMonitor.scrapeTimeout }} + scrapeTimeout: {{ . }} + {{- end }} + honorLabels: true + path: {{ .Values.serviceMonitor.path }} + scheme: {{ .Values.serviceMonitor.scheme }} + {{- with .Values.serviceMonitor.tlsConfig }} + tlsConfig: + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.serviceMonitor.relabelings }} + relabelings: + {{- toYaml . | nindent 6 }} + {{- end }} + jobLabel: "{{ .Release.Name }}" + selector: + matchLabels: + {{- include "grafana.selectorLabels" . | nindent 6 }} + namespaceSelector: + matchNames: + - {{ include "grafana.namespace" . }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/statefulset.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/statefulset.yaml new file mode 100644 index 00000000..acfab4dc --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/statefulset.yaml @@ -0,0 +1,55 @@ +{{- $sts := list "sts" "StatefulSet" "statefulset" -}} +{{- if (or (.Values.useStatefulSet) (and .Values.persistence.enabled (not .Values.persistence.existingClaim) (has .Values.persistence.type $sts)))}} +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: {{ include "grafana.fullname" . }} + namespace: {{ include "grafana.namespace" . }} + labels: + {{- include "grafana.labels" . | nindent 4 }} + {{- with .Values.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + replicas: {{ .Values.replicas }} + selector: + matchLabels: + {{- include "grafana.selectorLabels" . | nindent 6 }} + serviceName: {{ include "grafana.fullname" . }}-headless + template: + metadata: + labels: + {{- include "grafana.selectorLabels" . | nindent 8 }} + {{- with .Values.podLabels }} + {{- toYaml . | nindent 8 }} + {{- end }} + annotations: + checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} + checksum/dashboards-json-config: {{ include (print $.Template.BasePath "/dashboards-json-configmap.yaml") . | sha256sum }} + checksum/sc-dashboard-provider-config: {{ include (print $.Template.BasePath "/configmap-dashboard-provider.yaml") . | sha256sum }} + {{- if and (or (and (not .Values.admin.existingSecret) (not .Values.env.GF_SECURITY_ADMIN_PASSWORD__FILE) (not .Values.env.GF_SECURITY_ADMIN_PASSWORD)) (and .Values.ldap.enabled (not .Values.ldap.existingSecret))) (not .Values.env.GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION) }} + checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }} + {{- end }} + {{- with .Values.podAnnotations }} + {{- toYaml . | nindent 8 }} + {{- end }} + spec: + {{- include "grafana.pod" . | nindent 6 }} + {{- if .Values.persistence.enabled}} + volumeClaimTemplates: + - metadata: + name: storage + spec: + accessModes: {{ .Values.persistence.accessModes }} + storageClassName: {{ .Values.persistence.storageClassName }} + resources: + requests: + storage: {{ .Values.persistence.size }} + {{- with .Values.persistence.selectorLabels }} + selector: + matchLabels: + {{- toYaml . | nindent 10 }} + {{- end }} + {{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-configmap.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-configmap.yaml new file mode 100644 index 00000000..01c96c92 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-configmap.yaml @@ -0,0 +1,20 @@ +{{- if .Values.testFramework.enabled }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "grafana.fullname" . }}-test + namespace: {{ include "grafana.namespace" . }} + annotations: + "helm.sh/hook": test-success + "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded" + labels: + {{- include "grafana.labels" . | nindent 4 }} +data: + run.sh: |- + @test "Test Health" { + url="http://{{ include "grafana.fullname" . }}/api/health" + + code=$(wget --server-response --spider --timeout 90 --tries 10 ${url} 2>&1 | awk '/^ HTTP/{print $2}') + [ "$code" == "200" ] + } +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-podsecuritypolicy.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-podsecuritypolicy.yaml new file mode 100644 index 00000000..1821772a --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-podsecuritypolicy.yaml @@ -0,0 +1,32 @@ +{{- if and (.Capabilities.APIVersions.Has "policy/v1beta1/PodSecurityPolicy") .Values.testFramework.enabled .Values.rbac.pspEnabled }} +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: {{ include "grafana.fullname" . }}-test + annotations: + "helm.sh/hook": test-success + "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded" + labels: + {{- include "grafana.labels" . | nindent 4 }} +spec: + allowPrivilegeEscalation: true + privileged: false + hostNetwork: false + hostIPC: false + hostPID: false + fsGroup: + rule: RunAsAny + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + runAsUser: + rule: RunAsAny + volumes: + - configMap + - downwardAPI + - emptyDir + - projected + - csi + - secret +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-role.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-role.yaml new file mode 100644 index 00000000..cb4c7820 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-role.yaml @@ -0,0 +1,17 @@ +{{- if and (.Capabilities.APIVersions.Has "policy/v1beta1/PodSecurityPolicy") .Values.testFramework.enabled .Values.rbac.pspEnabled }} +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ include "grafana.fullname" . }}-test + namespace: {{ include "grafana.namespace" . }} + annotations: + "helm.sh/hook": test-success + "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded" + labels: + {{- include "grafana.labels" . | nindent 4 }} +rules: + - apiGroups: ['policy'] + resources: ['podsecuritypolicies'] + verbs: ['use'] + resourceNames: [{{ include "grafana.fullname" . }}-test] +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-rolebinding.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-rolebinding.yaml new file mode 100644 index 00000000..f40d791f --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-rolebinding.yaml @@ -0,0 +1,20 @@ +{{- if and (.Capabilities.APIVersions.Has "policy/v1beta1/PodSecurityPolicy") .Values.testFramework.enabled .Values.rbac.pspEnabled }} +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ include "grafana.fullname" . }}-test + namespace: {{ include "grafana.namespace" . }} + annotations: + "helm.sh/hook": test-success + "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded" + labels: + {{- include "grafana.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ include "grafana.fullname" . }}-test +subjects: + - kind: ServiceAccount + name: {{ include "grafana.serviceAccountNameTest" . }} + namespace: {{ include "grafana.namespace" . }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-serviceaccount.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-serviceaccount.yaml new file mode 100644 index 00000000..38fba359 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-serviceaccount.yaml @@ -0,0 +1,12 @@ +{{- if and .Values.testFramework.enabled .Values.serviceAccount.create }} +apiVersion: v1 +kind: ServiceAccount +metadata: + labels: + {{- include "grafana.labels" . | nindent 4 }} + name: {{ include "grafana.serviceAccountNameTest" . }} + namespace: {{ include "grafana.namespace" . }} + annotations: + "helm.sh/hook": test-success + "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded" +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test.yaml new file mode 100644 index 00000000..9fb88426 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/templates/tests/test.yaml @@ -0,0 +1,49 @@ +{{- if .Values.testFramework.enabled }} +{{- $root := . }} +apiVersion: v1 +kind: Pod +metadata: + name: {{ include "grafana.fullname" . }}-test + labels: + {{- include "grafana.labels" . | nindent 4 }} + annotations: + "helm.sh/hook": test-success + "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded" + namespace: {{ include "grafana.namespace" . }} +spec: + serviceAccountName: {{ include "grafana.serviceAccountNameTest" . }} + {{- with .Values.testFramework.securityContext }} + securityContext: + {{- toYaml . | nindent 4 }} + {{- end }} + {{- if or .Values.image.pullSecrets .Values.global.imagePullSecrets }} + imagePullSecrets: + {{- include "grafana.imagePullSecrets" (dict "root" $root "imagePullSecrets" .Values.image.pullSecrets) | nindent 4 }} + {{- end }} + {{- with .Values.nodeSelector }} + nodeSelector: + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.affinity }} + affinity: + {{- tpl (toYaml .) $root | nindent 4 }} + {{- end }} + {{- with .Values.tolerations }} + tolerations: + {{- toYaml . | nindent 4 }} + {{- end }} + containers: + - name: {{ .Release.Name }}-test + image: "{{ .Values.testFramework.image}}:{{ .Values.testFramework.tag }}" + imagePullPolicy: "{{ .Values.testFramework.imagePullPolicy}}" + command: ["/opt/bats/bin/bats", "-t", "/tests/run.sh"] + volumeMounts: + - mountPath: /tests + name: tests + readOnly: true + volumes: + - name: tests + configMap: + name: {{ include "grafana.fullname" . }}-test + restartPolicy: Never +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/values.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/values.yaml new file mode 100644 index 00000000..bc1834fb --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/grafana/values.yaml @@ -0,0 +1,1157 @@ +global: + # To help compatibility with other charts which use global.imagePullSecrets. + # Allow either an array of {name: pullSecret} maps (k8s-style), or an array of strings (more common helm-style). + # Can be tempalted. + # global: + # imagePullSecrets: + # - name: pullSecret1 + # - name: pullSecret2 + # or + # global: + # imagePullSecrets: + # - pullSecret1 + # - pullSecret2 + imagePullSecrets: [] + +rbac: + create: true + ## Use an existing ClusterRole/Role (depending on rbac.namespaced false/true) + # useExistingRole: name-of-some-(cluster)role + pspEnabled: true + pspUseAppArmor: true + namespaced: false + extraRoleRules: [] + # - apiGroups: [] + # resources: [] + # verbs: [] + extraClusterRoleRules: [] + # - apiGroups: [] + # resources: [] + # verbs: [] +serviceAccount: + create: true + name: + nameTest: + ## ServiceAccount labels. + labels: {} +## Service account annotations. Can be templated. +# annotations: +# eks.amazonaws.com/role-arn: arn:aws:iam::123456789000:role/iam-role-name-here + autoMount: true + +replicas: 1 + +## Create a headless service for the deployment +headlessService: false + +## Create HorizontalPodAutoscaler object for deployment type +# +autoscaling: + enabled: false + minReplicas: 1 + maxReplicas: 5 + targetCPU: "60" + targetMemory: "" + +## See `kubectl explain poddisruptionbudget.spec` for more +## ref: https://kubernetes.io/docs/tasks/run-application/configure-pdb/ +podDisruptionBudget: {} +# minAvailable: 1 +# maxUnavailable: 1 + +## See `kubectl explain deployment.spec.strategy` for more +## ref: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy +deploymentStrategy: + type: RollingUpdate + +readinessProbe: + httpGet: + path: /api/health + port: 3000 + +livenessProbe: + httpGet: + path: /api/health + port: 3000 + initialDelaySeconds: 60 + timeoutSeconds: 30 + failureThreshold: 10 + +## Use an alternate scheduler, e.g. "stork". +## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/ +## +# schedulerName: "default-scheduler" + +image: + repository: grafana/grafana + # Overrides the Grafana image tag whose default is the chart appVersion + tag: "" + sha: "" + pullPolicy: IfNotPresent + + ## Optionally specify an array of imagePullSecrets. + ## Secrets must be manually created in the namespace. + ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ + ## Can be templated. + ## + pullSecrets: [] + # - myRegistrKeySecretName + +testFramework: + enabled: true + image: "bats/bats" + tag: "v1.4.1" + imagePullPolicy: IfNotPresent + securityContext: {} + +securityContext: + runAsUser: 472 + runAsGroup: 472 + fsGroup: 472 + +containerSecurityContext: {} + +# Enable creating the grafana configmap +createConfigmap: true + +# Extra configmaps to mount in grafana pods +# Values are templated. +extraConfigmapMounts: [] + # - name: certs-configmap + # mountPath: /etc/grafana/ssl/ + # subPath: certificates.crt # (optional) + # configMap: certs-configmap + # readOnly: true + + +extraEmptyDirMounts: [] + # - name: provisioning-notifiers + # mountPath: /etc/grafana/provisioning/notifiers + + +# Apply extra labels to common labels. +extraLabels: {} + +## Assign a PriorityClassName to pods if set +# priorityClassName: + +downloadDashboardsImage: + repository: curlimages/curl + tag: 7.85.0 + sha: "" + pullPolicy: IfNotPresent + +downloadDashboards: + env: {} + envFromSecret: "" + resources: {} + securityContext: {} + envValueFrom: {} + # ENV_NAME: + # configMapKeyRef: + # name: configmap-name + # key: value_key + +## Pod Annotations +# podAnnotations: {} + +## Pod Labels +# podLabels: {} + +podPortName: grafana + +## Deployment annotations +# annotations: {} + +## Expose the grafana service to be accessed from outside the cluster (LoadBalancer service). +## or access it from within the cluster (ClusterIP service). Set the service type and the port to serve it. +## ref: http://kubernetes.io/docs/user-guide/services/ +## +service: + enabled: true + type: ClusterIP + port: 80 + targetPort: 3000 + # targetPort: 4181 To be used with a proxy extraContainer + ## Service annotations. Can be templated. + annotations: {} + labels: {} + portName: service + # Adds the appProtocol field to the service. This allows to work with istio protocol selection. Ex: "http" or "tcp" + appProtocol: "" + +serviceMonitor: + ## If true, a ServiceMonitor CRD is created for a prometheus operator + ## https://github.com/coreos/prometheus-operator + ## + enabled: false + path: /metrics + # namespace: monitoring (defaults to use the namespace this chart is deployed to) + labels: {} + interval: 1m + scheme: http + tlsConfig: {} + scrapeTimeout: 30s + relabelings: [] + +extraExposePorts: [] + # - name: keycloak + # port: 8080 + # targetPort: 8080 + # type: ClusterIP + +# overrides pod.spec.hostAliases in the grafana deployment's pods +hostAliases: [] + # - ip: "1.2.3.4" + # hostnames: + # - "my.host.com" + +ingress: + enabled: false + # For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName + # See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress + # ingressClassName: nginx + # Values can be templated + annotations: {} + # kubernetes.io/ingress.class: nginx + # kubernetes.io/tls-acme: "true" + labels: {} + path: / + + # pathType is only for k8s >= 1.1= + pathType: Prefix + + hosts: + - chart-example.local + ## Extra paths to prepend to every host configuration. This is useful when working with annotation based services. + extraPaths: [] + # - path: /* + # backend: + # serviceName: ssl-redirect + # servicePort: use-annotation + ## Or for k8s > 1.19 + # - path: /* + # pathType: Prefix + # backend: + # service: + # name: ssl-redirect + # port: + # name: use-annotation + + + tls: [] + # - secretName: chart-example-tls + # hosts: + # - chart-example.local + +resources: {} +# limits: +# cpu: 100m +# memory: 128Mi +# requests: +# cpu: 100m +# memory: 128Mi + +## Node labels for pod assignment +## ref: https://kubernetes.io/docs/user-guide/node-selection/ +# +nodeSelector: {} + +## Tolerations for pod assignment +## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ +## +tolerations: [] + +## Affinity for pod assignment (evaluated as template) +## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity +## +affinity: {} + +## Topology Spread Constraints +## ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/ +## +topologySpreadConstraints: [] + +## Additional init containers (evaluated as template) +## ref: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ +## +extraInitContainers: [] + +## Enable an Specify container in extraContainers. This is meant to allow adding an authentication proxy to a grafana pod +extraContainers: "" +# extraContainers: | +# - name: proxy +# image: quay.io/gambol99/keycloak-proxy:latest +# args: +# - -provider=github +# - -client-id= +# - -client-secret= +# - -github-org= +# - -email-domain=* +# - -cookie-secret= +# - -http-address=http://0.0.0.0:4181 +# - -upstream-url=http://127.0.0.1:3000 +# ports: +# - name: proxy-web +# containerPort: 4181 + +## Volumes that can be used in init containers that will not be mounted to deployment pods +extraContainerVolumes: [] +# - name: volume-from-secret +# secret: +# secretName: secret-to-mount +# - name: empty-dir-volume +# emptyDir: {} + +## Enable persistence using Persistent Volume Claims +## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/ +## +persistence: + type: pvc + enabled: false + # storageClassName: default + accessModes: + - ReadWriteOnce + size: 10Gi + # annotations: {} + finalizers: + - kubernetes.io/pvc-protection + # selectorLabels: {} + ## Sub-directory of the PV to mount. Can be templated. + # subPath: "" + ## Name of an existing PVC. Can be templated. + # existingClaim: + ## Extra labels to apply to a PVC. + extraPvcLabels: {} + + ## If persistence is not enabled, this allows to mount the + ## local storage in-memory to improve performance + ## + inMemory: + enabled: false + ## The maximum usage on memory medium EmptyDir would be + ## the minimum value between the SizeLimit specified + ## here and the sum of memory limits of all containers in a pod + ## + # sizeLimit: 300Mi + +initChownData: + ## If false, data ownership will not be reset at startup + ## This allows the grafana-server to be run with an arbitrary user + ## + enabled: true + + ## initChownData container image + ## + image: + repository: busybox + tag: "1.31.1" + sha: "" + pullPolicy: IfNotPresent + + ## initChownData resource requests and limits + ## Ref: http://kubernetes.io/docs/user-guide/compute-resources/ + ## + resources: {} + # limits: + # cpu: 100m + # memory: 128Mi + # requests: + # cpu: 100m + # memory: 128Mi + securityContext: + runAsNonRoot: false + runAsUser: 0 + + +# Administrator credentials when not using an existing secret (see below) +adminUser: admin +# adminPassword: strongpassword + +# Use an existing secret for the admin user. +admin: + ## Name of the secret. Can be templated. + existingSecret: "" + userKey: admin-user + passwordKey: admin-password + +## Define command to be executed at startup by grafana container +## Needed if using `vault-env` to manage secrets (ref: https://banzaicloud.com/blog/inject-secrets-into-pods-vault/) +## Default is "run.sh" as defined in grafana's Dockerfile +# command: +# - "sh" +# - "/run.sh" + +## Extra environment variables that will be pass onto deployment pods +## +## to provide grafana with access to CloudWatch on AWS EKS: +## 1. create an iam role of type "Web identity" with provider oidc.eks.* (note the provider for later) +## 2. edit the "Trust relationships" of the role, add a line inside the StringEquals clause using the +## same oidc eks provider as noted before (same as the existing line) +## also, replace NAMESPACE and prometheus-operator-grafana with the service account namespace and name +## +## "oidc.eks.us-east-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:sub": "system:serviceaccount:NAMESPACE:prometheus-operator-grafana", +## +## 3. attach a policy to the role, you can use a built in policy called CloudWatchReadOnlyAccess +## 4. use the following env: (replace 123456789000 and iam-role-name-here with your aws account number and role name) +## +## env: +## AWS_ROLE_ARN: arn:aws:iam::123456789000:role/iam-role-name-here +## AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token +## AWS_REGION: us-east-1 +## +## 5. uncomment the EKS section in extraSecretMounts: below +## 6. uncomment the annotation section in the serviceAccount: above +## make sure to replace arn:aws:iam::123456789000:role/iam-role-name-here with your role arn + +env: {} + +## "valueFrom" environment variable references that will be added to deployment pods. Name is templated. +## ref: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#envvarsource-v1-core +## Renders in container spec as: +## env: +## ... +## - name: +## valueFrom: +## +envValueFrom: {} + # ENV_NAME: + # configMapKeyRef: + # name: configmap-name + # key: value_key + +## The name of a secret in the same kubernetes namespace which contain values to be added to the environment +## This can be useful for auth tokens, etc. Value is templated. +envFromSecret: "" + +## Sensible environment variables that will be rendered as new secret object +## This can be useful for auth tokens, etc +envRenderSecret: {} + +## The names of secrets in the same kubernetes namespace which contain values to be added to the environment +## Each entry should contain a name key, and can optionally specify whether the secret must be defined with an optional key. +## Name is templated. +envFromSecrets: [] +## - name: secret-name +## optional: true + +## The names of conifgmaps in the same kubernetes namespace which contain values to be added to the environment +## Each entry should contain a name key, and can optionally specify whether the configmap must be defined with an optional key. +## Name is templated. +## ref: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#configmapenvsource-v1-core +envFromConfigMaps: [] +## - name: configmap-name +## optional: true + +# Inject Kubernetes services as environment variables. +# See https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/#environment-variables +enableServiceLinks: true + +## Additional grafana server secret mounts +# Defines additional mounts with secrets. Secrets must be manually created in the namespace. +extraSecretMounts: [] + # - name: secret-files + # mountPath: /etc/secrets + # secretName: grafana-secret-files + # readOnly: true + # subPath: "" + # + # for AWS EKS (cloudwatch) use the following (see also instruction in env: above) + # - name: aws-iam-token + # mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount + # readOnly: true + # projected: + # defaultMode: 420 + # sources: + # - serviceAccountToken: + # audience: sts.amazonaws.com + # expirationSeconds: 86400 + # path: token + # + # for CSI e.g. Azure Key Vault use the following + # - name: secrets-store-inline + # mountPath: /run/secrets + # readOnly: true + # csi: + # driver: secrets-store.csi.k8s.io + # readOnly: true + # volumeAttributes: + # secretProviderClass: "akv-grafana-spc" + # nodePublishSecretRef: # Only required when using service principal mode + # name: grafana-akv-creds # Only required when using service principal mode + +## Additional grafana server volume mounts +# Defines additional volume mounts. +extraVolumeMounts: [] + # - name: extra-volume-0 + # mountPath: /mnt/volume0 + # readOnly: true + # existingClaim: volume-claim + # - name: extra-volume-1 + # mountPath: /mnt/volume1 + # readOnly: true + # hostPath: /usr/shared/ + # - name: grafana-secrets + # csi: true + # data: + # driver: secrets-store.csi.k8s.io + # readOnly: true + # volumeAttributes: + # secretProviderClass: "grafana-env-spc" + +## Container Lifecycle Hooks. Execute a specific bash command or make an HTTP request +lifecycleHooks: {} + # postStart: + # exec: + # command: [] + +## Pass the plugins you want installed as a list. +## +plugins: [] + # - digrich-bubblechart-panel + # - grafana-clock-panel + +## Configure grafana datasources +## ref: http://docs.grafana.org/administration/provisioning/#datasources +## +datasources: {} +# datasources.yaml: +# apiVersion: 1 +# datasources: +# - name: Prometheus +# type: prometheus +# url: http://prometheus-prometheus-server +# access: proxy +# isDefault: true +# - name: CloudWatch +# type: cloudwatch +# access: proxy +# uid: cloudwatch +# editable: false +# jsonData: +# authType: default +# defaultRegion: us-east-1 + +## Configure grafana alerting (can be templated) +## ref: http://docs.grafana.org/administration/provisioning/#alerting +## +alerting: {} + # rules.yaml: + # apiVersion: 1 + # groups: + # - orgId: 1 + # name: '{{ .Chart.Name }}_my_rule_group' + # folder: my_first_folder + # interval: 60s + # rules: + # - uid: my_id_1 + # title: my_first_rule + # condition: A + # data: + # - refId: A + # datasourceUid: '-100' + # model: + # conditions: + # - evaluator: + # params: + # - 3 + # type: gt + # operator: + # type: and + # query: + # params: + # - A + # reducer: + # type: last + # type: query + # datasource: + # type: __expr__ + # uid: '-100' + # expression: 1==0 + # intervalMs: 1000 + # maxDataPoints: 43200 + # refId: A + # type: math + # dashboardUid: my_dashboard + # panelId: 123 + # noDataState: Alerting + # for: 60s + # annotations: + # some_key: some_value + # labels: + # team: sre_team_1 + # contactpoints.yaml: + # apiVersion: 1 + # contactPoints: + # - orgId: 1 + # name: cp_1 + # receivers: + # - uid: first_uid + # type: pagerduty + # settings: + # integrationKey: XXX + # severity: critical + # class: ping failure + # component: Grafana + # group: app-stack + # summary: | + # {{ `{{ include "default.message" . }}` }} + +## Configure notifiers +## ref: http://docs.grafana.org/administration/provisioning/#alert-notification-channels +## +notifiers: {} +# notifiers.yaml: +# notifiers: +# - name: email-notifier +# type: email +# uid: email1 +# # either: +# org_id: 1 +# # or +# org_name: Main Org. +# is_default: true +# settings: +# addresses: an_email_address@example.com +# delete_notifiers: + +## Configure grafana dashboard providers +## ref: http://docs.grafana.org/administration/provisioning/#dashboards +## +## `path` must be /var/lib/grafana/dashboards/ +## +dashboardProviders: {} +# dashboardproviders.yaml: +# apiVersion: 1 +# providers: +# - name: 'default' +# orgId: 1 +# folder: '' +# type: file +# disableDeletion: false +# editable: true +# options: +# path: /var/lib/grafana/dashboards/default + +## Configure grafana dashboard to import +## NOTE: To use dashboards you must also enable/configure dashboardProviders +## ref: https://grafana.com/dashboards +## +## dashboards per provider, use provider name as key. +## +dashboards: {} + # default: + # some-dashboard: + # json: | + # $RAW_JSON + # custom-dashboard: + # file: dashboards/custom-dashboard.json + # prometheus-stats: + # gnetId: 2 + # revision: 2 + # datasource: Prometheus + # local-dashboard: + # url: https://example.com/repository/test.json + # token: '' + # local-dashboard-base64: + # url: https://example.com/repository/test-b64.json + # token: '' + # b64content: true + # local-dashboard-gitlab: + # url: https://example.com/repository/test-gitlab.json + # gitlabToken: '' + # local-dashboard-bitbucket: + # url: https://example.com/repository/test-bitbucket.json + # bearerToken: '' + +## Reference to external ConfigMap per provider. Use provider name as key and ConfigMap name as value. +## A provider dashboards must be defined either by external ConfigMaps or in values.yaml, not in both. +## ConfigMap data example: +## +## data: +## example-dashboard.json: | +## RAW_JSON +## +dashboardsConfigMaps: {} +# default: "" + +## Grafana's primary configuration +## NOTE: values in map will be converted to ini format +## ref: http://docs.grafana.org/installation/configuration/ +## +grafana.ini: + paths: + data: /var/lib/grafana/ + logs: /var/log/grafana + plugins: /var/lib/grafana/plugins + provisioning: /etc/grafana/provisioning + analytics: + check_for_updates: true + log: + mode: console + grafana_net: + url: https://grafana.net + server: + domain: "{{ if (and .Values.ingress.enabled .Values.ingress.hosts) }}{{ .Values.ingress.hosts | first }}{{ else }}''{{ end }}" +## grafana Authentication can be enabled with the following values on grafana.ini + # server: + # The full public facing url you use in browser, used for redirects and emails + # root_url: + # https://grafana.com/docs/grafana/latest/auth/github/#enable-github-in-grafana + # auth.github: + # enabled: false + # allow_sign_up: false + # scopes: user:email,read:org + # auth_url: https://github.com/login/oauth/authorize + # token_url: https://github.com/login/oauth/access_token + # api_url: https://api.github.com/user + # team_ids: + # allowed_organizations: + # client_id: + # client_secret: +## LDAP Authentication can be enabled with the following values on grafana.ini +## NOTE: Grafana will fail to start if the value for ldap.toml is invalid + # auth.ldap: + # enabled: true + # allow_sign_up: true + # config_file: /etc/grafana/ldap.toml + +## Grafana's LDAP configuration +## Templated by the template in _helpers.tpl +## NOTE: To enable the grafana.ini must be configured with auth.ldap.enabled +## ref: http://docs.grafana.org/installation/configuration/#auth-ldap +## ref: http://docs.grafana.org/installation/ldap/#configuration +ldap: + enabled: false + # `existingSecret` is a reference to an existing secret containing the ldap configuration + # for Grafana in a key `ldap-toml`. + existingSecret: "" + # `config` is the content of `ldap.toml` that will be stored in the created secret + config: "" + # config: |- + # verbose_logging = true + + # [[servers]] + # host = "my-ldap-server" + # port = 636 + # use_ssl = true + # start_tls = false + # ssl_skip_verify = false + # bind_dn = "uid=%s,ou=users,dc=myorg,dc=com" + +## Grafana's SMTP configuration +## NOTE: To enable, grafana.ini must be configured with smtp.enabled +## ref: http://docs.grafana.org/installation/configuration/#smtp +smtp: + # `existingSecret` is a reference to an existing secret containing the smtp configuration + # for Grafana. + existingSecret: "" + userKey: "user" + passwordKey: "password" + +## Sidecars that collect the configmaps with specified label and stores the included files them into the respective folders +## Requires at least Grafana 5 to work and can't be used together with parameters dashboardProviders, datasources and dashboards +sidecar: + image: + repository: quay.io/kiwigrid/k8s-sidecar + tag: 1.21.0 + sha: "" + imagePullPolicy: IfNotPresent + resources: {} +# limits: +# cpu: 100m +# memory: 100Mi +# requests: +# cpu: 50m +# memory: 50Mi + securityContext: {} + # skipTlsVerify Set to true to skip tls verification for kube api calls + # skipTlsVerify: true + enableUniqueFilenames: false + readinessProbe: {} + livenessProbe: {} + # Log level default for all sidecars. Can be one of: DEBUG, INFO, WARN, ERROR, CRITICAL. Defaults to INFO + # logLevel: INFO + alerts: + enabled: false + # Additional environment variables for the alerts sidecar + env: {} + # Do not reprocess already processed unchanged resources on k8s API reconnect. + # ignoreAlreadyProcessed: true + # label that the configmaps with alert are marked with + label: grafana_alert + # value of label that the configmaps with alert are set to + labelValue: "" + # Log level. Can be one of: DEBUG, INFO, WARN, ERROR, CRITICAL. + # logLevel: INFO + # If specified, the sidecar will search for alert config-maps inside this namespace. + # Otherwise the namespace in which the sidecar is running will be used. + # It's also possible to specify ALL to search in all namespaces + searchNamespace: null + # Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH requests, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds. + watchMethod: WATCH + # search in configmap, secret or both + resource: both + # watchServerTimeout: request to the server, asking it to cleanly close the connection after that. + # defaults to 60sec; much higher values like 3600 seconds (1h) are feasible for non-Azure K8S + # watchServerTimeout: 3600 + # + # watchClientTimeout: is a client-side timeout, configuring your local socket. + # If you have a network outage dropping all packets with no RST/FIN, + # this is how long your client waits before realizing & dropping the connection. + # defaults to 66sec (sic!) + # watchClientTimeout: 60 + # + # Endpoint to send request to reload alerts + reloadURL: "http://localhost:3000/api/admin/provisioning/alerting/reload" + # Absolute path to shell script to execute after a alert got reloaded + script: null + skipReload: false + # Deploy the alert sidecar as an initContainer in addition to a container. + # Sets the size limit of the alert sidecar emptyDir volume + sizeLimit: {} + dashboards: + enabled: false + # Additional environment variables for the dashboards sidecar + env: {} + # Do not reprocess already processed unchanged resources on k8s API reconnect. + # ignoreAlreadyProcessed: true + SCProvider: true + # label that the configmaps with dashboards are marked with + label: grafana_dashboard + # value of label that the configmaps with dashboards are set to + labelValue: "" + # Log level. Can be one of: DEBUG, INFO, WARN, ERROR, CRITICAL. + # logLevel: INFO + # folder in the pod that should hold the collected dashboards (unless `defaultFolderName` is set) + folder: /tmp/dashboards + # The default folder name, it will create a subfolder under the `folder` and put dashboards in there instead + defaultFolderName: null + # Namespaces list. If specified, the sidecar will search for config-maps/secrets inside these namespaces. + # Otherwise the namespace in which the sidecar is running will be used. + # It's also possible to specify ALL to search in all namespaces. + searchNamespace: null + # Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH requests, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds. + watchMethod: WATCH + # search in configmap, secret or both + resource: both + # If specified, the sidecar will look for annotation with this name to create folder and put graph here. + # You can use this parameter together with `provider.foldersFromFilesStructure`to annotate configmaps and create folder structure. + folderAnnotation: null + # Endpoint to send request to reload alerts + reloadURL: "http://localhost:3000/api/admin/provisioning/dashboards/reload" + # Absolute path to shell script to execute after a configmap got reloaded + script: null + skipReload: false + # watchServerTimeout: request to the server, asking it to cleanly close the connection after that. + # defaults to 60sec; much higher values like 3600 seconds (1h) are feasible for non-Azure K8S + # watchServerTimeout: 3600 + # + # watchClientTimeout: is a client-side timeout, configuring your local socket. + # If you have a network outage dropping all packets with no RST/FIN, + # this is how long your client waits before realizing & dropping the connection. + # defaults to 66sec (sic!) + # watchClientTimeout: 60 + # + # provider configuration that lets grafana manage the dashboards + provider: + # name of the provider, should be unique + name: sidecarProvider + # orgid as configured in grafana + orgid: 1 + # folder in which the dashboards should be imported in grafana + folder: '' + # type of the provider + type: file + # disableDelete to activate a import-only behaviour + disableDelete: false + # allow updating provisioned dashboards from the UI + allowUiUpdates: false + # allow Grafana to replicate dashboard structure from filesystem + foldersFromFilesStructure: false + # Additional dashboard sidecar volume mounts + extraMounts: [] + # Sets the size limit of the dashboard sidecar emptyDir volume + sizeLimit: {} + datasources: + enabled: false + # Additional environment variables for the datasourcessidecar + env: {} + # Do not reprocess already processed unchanged resources on k8s API reconnect. + # ignoreAlreadyProcessed: true + # label that the configmaps with datasources are marked with + label: grafana_datasource + # value of label that the configmaps with datasources are set to + labelValue: "" + # Log level. Can be one of: DEBUG, INFO, WARN, ERROR, CRITICAL. + # logLevel: INFO + # If specified, the sidecar will search for datasource config-maps inside this namespace. + # Otherwise the namespace in which the sidecar is running will be used. + # It's also possible to specify ALL to search in all namespaces + searchNamespace: null + # Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH requests, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds. + watchMethod: WATCH + # search in configmap, secret or both + resource: both + # watchServerTimeout: request to the server, asking it to cleanly close the connection after that. + # defaults to 60sec; much higher values like 3600 seconds (1h) are feasible for non-Azure K8S + # watchServerTimeout: 3600 + # + # watchClientTimeout: is a client-side timeout, configuring your local socket. + # If you have a network outage dropping all packets with no RST/FIN, + # this is how long your client waits before realizing & dropping the connection. + # defaults to 66sec (sic!) + # watchClientTimeout: 60 + # + # Endpoint to send request to reload datasources + reloadURL: "http://localhost:3000/api/admin/provisioning/datasources/reload" + # Absolute path to shell script to execute after a datasource got reloaded + script: null + skipReload: false + # Deploy the datasource sidecar as an initContainer in addition to a container. + # This is needed if skipReload is true, to load any datasources defined at startup time. + initDatasources: false + # Sets the size limit of the datasource sidecar emptyDir volume + sizeLimit: {} + plugins: + enabled: false + # Additional environment variables for the plugins sidecar + env: {} + # Do not reprocess already processed unchanged resources on k8s API reconnect. + # ignoreAlreadyProcessed: true + # label that the configmaps with plugins are marked with + label: grafana_plugin + # value of label that the configmaps with plugins are set to + labelValue: "" + # Log level. Can be one of: DEBUG, INFO, WARN, ERROR, CRITICAL. + # logLevel: INFO + # If specified, the sidecar will search for plugin config-maps inside this namespace. + # Otherwise the namespace in which the sidecar is running will be used. + # It's also possible to specify ALL to search in all namespaces + searchNamespace: null + # Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH requests, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds. + watchMethod: WATCH + # search in configmap, secret or both + resource: both + # watchServerTimeout: request to the server, asking it to cleanly close the connection after that. + # defaults to 60sec; much higher values like 3600 seconds (1h) are feasible for non-Azure K8S + # watchServerTimeout: 3600 + # + # watchClientTimeout: is a client-side timeout, configuring your local socket. + # If you have a network outage dropping all packets with no RST/FIN, + # this is how long your client waits before realizing & dropping the connection. + # defaults to 66sec (sic!) + # watchClientTimeout: 60 + # + # Endpoint to send request to reload plugins + reloadURL: "http://localhost:3000/api/admin/provisioning/plugins/reload" + # Absolute path to shell script to execute after a plugin got reloaded + script: null + skipReload: false + # Deploy the datasource sidecar as an initContainer in addition to a container. + # This is needed if skipReload is true, to load any plugins defined at startup time. + initPlugins: false + # Sets the size limit of the plugin sidecar emptyDir volume + sizeLimit: {} + notifiers: + enabled: false + # Additional environment variables for the notifierssidecar + env: {} + # Do not reprocess already processed unchanged resources on k8s API reconnect. + # ignoreAlreadyProcessed: true + # label that the configmaps with notifiers are marked with + label: grafana_notifier + # value of label that the configmaps with notifiers are set to + labelValue: "" + # Log level. Can be one of: DEBUG, INFO, WARN, ERROR, CRITICAL. + # logLevel: INFO + # If specified, the sidecar will search for notifier config-maps inside this namespace. + # Otherwise the namespace in which the sidecar is running will be used. + # It's also possible to specify ALL to search in all namespaces + searchNamespace: null + # Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH requests, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds. + watchMethod: WATCH + # search in configmap, secret or both + resource: both + # watchServerTimeout: request to the server, asking it to cleanly close the connection after that. + # defaults to 60sec; much higher values like 3600 seconds (1h) are feasible for non-Azure K8S + # watchServerTimeout: 3600 + # + # watchClientTimeout: is a client-side timeout, configuring your local socket. + # If you have a network outage dropping all packets with no RST/FIN, + # this is how long your client waits before realizing & dropping the connection. + # defaults to 66sec (sic!) + # watchClientTimeout: 60 + # + # Endpoint to send request to reload notifiers + reloadURL: "http://localhost:3000/api/admin/provisioning/notifications/reload" + # Absolute path to shell script to execute after a notifier got reloaded + script: null + skipReload: false + # Deploy the notifier sidecar as an initContainer in addition to a container. + # This is needed if skipReload is true, to load any notifiers defined at startup time. + initNotifiers: false + # Sets the size limit of the notifier sidecar emptyDir volume + sizeLimit: {} + +## Override the deployment namespace +## +namespaceOverride: "" + +## Number of old ReplicaSets to retain +## +revisionHistoryLimit: 10 + +## Add a seperate remote image renderer deployment/service +imageRenderer: + deploymentStrategy: {} + # Enable the image-renderer deployment & service + enabled: false + replicas: 1 + image: + # image-renderer Image repository + repository: grafana/grafana-image-renderer + # image-renderer Image tag + tag: latest + # image-renderer Image sha (optional) + sha: "" + # image-renderer ImagePullPolicy + pullPolicy: Always + # extra environment variables + env: + HTTP_HOST: "0.0.0.0" + # RENDERING_ARGS: --no-sandbox,--disable-gpu,--window-size=1280x758 + # RENDERING_MODE: clustered + # IGNORE_HTTPS_ERRORS: true + # image-renderer deployment serviceAccount + serviceAccountName: "" + # image-renderer deployment securityContext + securityContext: {} + # image-renderer deployment container securityContext + containerSecurityContext: + capabilities: + drop: ['ALL'] + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + # image-renderer deployment Host Aliases + hostAliases: [] + # image-renderer deployment priority class + priorityClassName: '' + service: + # Enable the image-renderer service + enabled: true + # image-renderer service port name + portName: 'http' + # image-renderer service port used by both service and deployment + port: 8081 + targetPort: 8081 + # Adds the appProtocol field to the image-renderer service. This allows to work with istio protocol selection. Ex: "http" or "tcp" + appProtocol: "" + # If https is enabled in Grafana, this needs to be set as 'https' to correctly configure the callback used in Grafana + grafanaProtocol: http + # In case a sub_path is used this needs to be added to the image renderer callback + grafanaSubPath: "" + # name of the image-renderer port on the pod + podPortName: http + # number of image-renderer replica sets to keep + revisionHistoryLimit: 10 + networkPolicy: + # Enable a NetworkPolicy to limit inbound traffic to only the created grafana pods + limitIngress: true + # Enable a NetworkPolicy to limit outbound traffic to only the created grafana pods + limitEgress: false + resources: {} +# limits: +# cpu: 100m +# memory: 100Mi +# requests: +# cpu: 50m +# memory: 50Mi + ## Node labels for pod assignment + ## ref: https://kubernetes.io/docs/user-guide/node-selection/ + # + nodeSelector: {} + + ## Tolerations for pod assignment + ## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ + ## + tolerations: [] + + ## Affinity for pod assignment (evaluated as template) + ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity + ## + affinity: {} + + ## Use an alternate scheduler, e.g. "stork". + ## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/ + ## + # schedulerName: "default-scheduler" + +networkPolicy: + ## @param networkPolicy.enabled Enable creation of NetworkPolicy resources. Only Ingress traffic is filtered for now. + ## + enabled: false + ## @param networkPolicy.allowExternal Don't require client label for connections + ## The Policy model to apply. When set to false, only pods with the correct + ## client label will have network access to grafana port defined. + ## When true, grafana will accept connections from any source + ## (with the correct destination port). + ## + ingress: true + ## @param networkPolicy.ingress When true enables the creation + ## an ingress network policy + ## + allowExternal: true + ## @param networkPolicy.explicitNamespacesSelector A Kubernetes LabelSelector to explicitly select namespaces from which traffic could be allowed + ## If explicitNamespacesSelector is missing or set to {}, only client Pods that are in the networkPolicy's namespace + ## and that match other criteria, the ones that have the good label, can reach the grafana. + ## But sometimes, we want the grafana to be accessible to clients from other namespaces, in this case, we can use this + ## LabelSelector to select these namespaces, note that the networkPolicy's namespace should also be explicitly added. + ## + ## Example: + ## explicitNamespacesSelector: + ## matchLabels: + ## role: frontend + ## matchExpressions: + ## - {key: role, operator: In, values: [frontend]} + ## + explicitNamespacesSelector: {} + ## + ## + ## + ## + ## + ## + egress: + ## @param networkPolicy.egress.enabled When enabled, an egress network policy will be + ## created allowing grafana to connect to external data sources from kubernetes cluster. + enabled: false + ## + ## @param networkPolicy.egress.ports Add individual ports to be allowed by the egress + ports: [] + ## Add ports to the egress by specifying - port: + ## E.X. + ## ports: + ## - port: 80 + ## - port: 443 + ## + ## + ## + ## + ## + ## + +# Enable backward compatibility of kubernetes where version below 1.13 doesn't have the enableServiceLinks option +enableKubeBackwardCompatibility: false +useStatefulSet: false +# Create a dynamic manifests via values: +extraObjects: [] + # - apiVersion: "kubernetes-client.io/v1" + # kind: ExternalSecret + # metadata: + # name: grafana-secrets + # spec: + # backendType: gcpSecretsManager + # data: + # - key: grafana-admin-password + # name: adminPassword diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/.helmignore b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/.helmignore new file mode 100644 index 00000000..f0c13194 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/.helmignore @@ -0,0 +1,21 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*~ +# Various IDEs +.project +.idea/ +*.tmproj diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/Chart.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/Chart.yaml new file mode 100644 index 00000000..29d6acc3 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/Chart.yaml @@ -0,0 +1,21 @@ +apiVersion: v2 +appVersion: 2.7.0 +description: Install kube-state-metrics to generate and expose cluster-level metrics +home: https://github.com/kubernetes/kube-state-metrics/ +keywords: +- metric +- monitoring +- prometheus +- kubernetes +maintainers: +- email: tariq.ibrahim@mulesoft.com + name: tariq1890 +- email: manuel@rueg.eu + name: mrueg +- email: david@0xdc.me + name: dotdc +name: kube-state-metrics +sources: +- https://github.com/kubernetes/kube-state-metrics/ +type: application +version: 4.24.0 diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/README.md b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/README.md new file mode 100644 index 00000000..7c2e1691 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/README.md @@ -0,0 +1,68 @@ +# kube-state-metrics Helm Chart + +Installs the [kube-state-metrics agent](https://github.com/kubernetes/kube-state-metrics). + +## Get Repo Info + +```console +helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +helm repo update +``` + +_See [helm repo](https://helm.sh/docs/helm/helm_repo/) for command documentation._ + +## Install Chart + +```console +helm install [RELEASE_NAME] prometheus-community/kube-state-metrics [flags] +``` + +_See [configuration](#configuration) below._ + +_See [helm install](https://helm.sh/docs/helm/helm_install/) for command documentation._ + +## Uninstall Chart + +```console +helm uninstall [RELEASE_NAME] +``` + +This removes all the Kubernetes components associated with the chart and deletes the release. + +_See [helm uninstall](https://helm.sh/docs/helm/helm_uninstall/) for command documentation._ + +## Upgrading Chart + +```console +helm upgrade [RELEASE_NAME] prometheus-community/kube-state-metrics [flags] +``` + +_See [helm upgrade](https://helm.sh/docs/helm/helm_upgrade/) for command documentation._ + +### Migrating from stable/kube-state-metrics and kubernetes/kube-state-metrics + +You can upgrade in-place: + +1. [get repo info](#get-repo-info) +1. [upgrade](#upgrading-chart) your existing release name using the new chart repo + + +## Upgrading to v3.0.0 + +v3.0.0 includes kube-state-metrics v2.0, see the [changelog](https://github.com/kubernetes/kube-state-metrics/blob/release-2.0/CHANGELOG.md) for major changes on the application-side. + +The upgraded chart now the following changes: +* Dropped support for helm v2 (helm v3 or later is required) +* collectors key was renamed to resources +* namespace key was renamed to namespaces + + +## Configuration + +See [Customizing the Chart Before Installing](https://helm.sh/docs/intro/using_helm/#customizing-the-chart-before-installing). To see all configurable options with detailed comments: + +```console +helm show values prometheus-community/kube-state-metrics +``` + +You may also run `helm show values` on this chart's [dependencies](#dependencies) for additional options. diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/NOTES.txt b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/NOTES.txt new file mode 100644 index 00000000..5a646e0c --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/NOTES.txt @@ -0,0 +1,10 @@ +kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. +The exposed metrics can be found here: +https://github.com/kubernetes/kube-state-metrics/blob/master/docs/README.md#exposed-metrics + +The metrics are exported on the HTTP endpoint /metrics on the listening port. +In your case, {{ template "kube-state-metrics.fullname" . }}.{{ template "kube-state-metrics.namespace" . }}.svc.cluster.local:{{ .Values.service.port }}/metrics + +They are served either as plaintext or protobuf depending on the Accept header. +They are designed to be consumed either by Prometheus itself or by a scraper that is compatible with scraping a Prometheus client endpoint. + diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/_helpers.tpl b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/_helpers.tpl new file mode 100644 index 00000000..0d193fbc --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/_helpers.tpl @@ -0,0 +1,101 @@ +{{/* vim: set filetype=mustache: */}} +{{/* +Expand the name of the chart. +*/}} +{{- define "kube-state-metrics.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +If release name contains chart name it will be used as a full name. +*/}} +{{- define "kube-state-metrics.fullname" -}} +{{- if .Values.fullnameOverride -}} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- $name := default .Chart.Name .Values.nameOverride -}} +{{- if contains $name .Release.Name -}} +{{- .Release.Name | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} +{{- end -}} +{{- end -}} +{{- end -}} + +{{/* +Create the name of the service account to use +*/}} +{{- define "kube-state-metrics.serviceAccountName" -}} +{{- if .Values.serviceAccount.create -}} + {{ default (include "kube-state-metrics.fullname" .) .Values.serviceAccount.name }} +{{- else -}} + {{ default "default" .Values.serviceAccount.name }} +{{- end -}} +{{- end -}} + +{{/* +Allow the release namespace to be overridden for multi-namespace deployments in combined charts +*/}} +{{- define "kube-state-metrics.namespace" -}} + {{- if .Values.namespaceOverride -}} + {{- .Values.namespaceOverride -}} + {{- else -}} + {{- .Release.Namespace -}} + {{- end -}} +{{- end -}} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "kube-state-metrics.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{/* +Generate basic labels +*/}} +{{- define "kube-state-metrics.labels" }} +helm.sh/chart: {{ template "kube-state-metrics.chart" . }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +app.kubernetes.io/component: metrics +app.kubernetes.io/part-of: {{ template "kube-state-metrics.name" . }} +{{- include "kube-state-metrics.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +{{- if .Values.customLabels }} +{{ toYaml .Values.customLabels }} +{{- end }} +{{- if .Values.releaseLabel }} +release: {{ .Release.Name }} +{{- end }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "kube-state-metrics.selectorLabels" }} +app.kubernetes.io/name: {{ include "kube-state-metrics.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* Sets default scrape limits for servicemonitor */}} +{{- define "servicemonitor.scrapeLimits" -}} +{{- with .sampleLimit }} +sampleLimit: {{ . }} +{{- end }} +{{- with .targetLimit }} +targetLimit: {{ . }} +{{- end }} +{{- with .labelLimit }} +labelLimit: {{ . }} +{{- end }} +{{- with .labelNameLengthLimit }} +labelNameLengthLimit: {{ . }} +{{- end }} +{{- with .labelValueLengthLimit }} +labelValueLengthLimit: {{ . }} +{{- end }} +{{- end -}} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/clusterrolebinding.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/clusterrolebinding.yaml new file mode 100644 index 00000000..cf9f628d --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/clusterrolebinding.yaml @@ -0,0 +1,20 @@ +{{- if and .Values.rbac.create .Values.rbac.useClusterRole -}} +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + labels: + {{- include "kube-state-metrics.labels" . | indent 4 }} + name: {{ template "kube-state-metrics.fullname" . }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole +{{- if .Values.rbac.useExistingRole }} + name: {{ .Values.rbac.useExistingRole }} +{{- else }} + name: {{ template "kube-state-metrics.fullname" . }} +{{- end }} +subjects: +- kind: ServiceAccount + name: {{ template "kube-state-metrics.serviceAccountName" . }} + namespace: {{ template "kube-state-metrics.namespace" . }} +{{- end -}} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/deployment.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/deployment.yaml new file mode 100644 index 00000000..e529d3fc --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/deployment.yaml @@ -0,0 +1,179 @@ +apiVersion: apps/v1 +{{- if .Values.autosharding.enabled }} +kind: StatefulSet +{{- else }} +kind: Deployment +{{- end }} +metadata: + name: {{ template "kube-state-metrics.fullname" . }} + namespace: {{ template "kube-state-metrics.namespace" . }} + labels: + {{- include "kube-state-metrics.labels" . | indent 4 }} + {{- if .Values.annotations }} + annotations: +{{ toYaml .Values.annotations | indent 4 }} + {{- end }} +spec: + selector: + matchLabels: + {{- include "kube-state-metrics.selectorLabels" . | indent 6 }} + replicas: {{ .Values.replicas }} + {{- if .Values.autosharding.enabled }} + serviceName: {{ template "kube-state-metrics.fullname" . }} + volumeClaimTemplates: [] + {{- end }} + template: + metadata: + labels: + {{- include "kube-state-metrics.labels" . | indent 8 }} + {{- if .Values.podAnnotations }} + annotations: +{{ toYaml .Values.podAnnotations | indent 8 }} + {{- end }} + spec: + hostNetwork: {{ .Values.hostNetwork }} + serviceAccountName: {{ template "kube-state-metrics.serviceAccountName" . }} + {{- if .Values.securityContext.enabled }} + securityContext: {{- omit .Values.securityContext "enabled" | toYaml | nindent 8 }} + {{- end }} + {{- if .Values.priorityClassName }} + priorityClassName: {{ .Values.priorityClassName }} + {{- end }} + containers: + - name: {{ template "kube-state-metrics.name" . }} + {{- if .Values.autosharding.enabled }} + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + {{- end }} + args: + {{- if .Values.extraArgs }} + {{- .Values.extraArgs | toYaml | nindent 8 }} + {{- end }} + {{- if .Values.service.port }} + - --port={{ .Values.service.port | default 8080}} + {{- end }} + {{- if .Values.collectors }} + - --resources={{ .Values.collectors | join "," }} + {{- end }} + {{- if .Values.metricLabelsAllowlist }} + - --metric-labels-allowlist={{ .Values.metricLabelsAllowlist | join "," }} + {{- end }} + {{- if .Values.metricAnnotationsAllowList }} + - --metric-annotations-allowlist={{ .Values.metricAnnotationsAllowList | join "," }} + {{- end }} + {{- if .Values.metricAllowlist }} + - --metric-allowlist={{ .Values.metricAllowlist | join "," }} + {{- end }} + {{- if .Values.metricDenylist }} + - --metric-denylist={{ .Values.metricDenylist | join "," }} + {{- end }} + {{- $namespaces := list }} + {{- if .Values.namespaces }} + {{- range $ns := join "," .Values.namespaces | split "," }} + {{- $namespaces = append $namespaces (tpl $ns $) }} + {{- end }} + {{- end }} + {{- if .Values.releaseNamespace }} + {{- $namespaces = append $namespaces ( include "kube-state-metrics.namespace" . ) }} + {{- end }} + {{- if $namespaces }} + - --namespaces={{ $namespaces | mustUniq | join "," }} + {{- end }} + {{- if .Values.namespacesDenylist }} + - --namespaces-denylist={{ tpl (.Values.namespacesDenylist | join ",") $ }} + {{- end }} + {{- if .Values.autosharding.enabled }} + - --pod=$(POD_NAME) + - --pod-namespace=$(POD_NAMESPACE) + {{- end }} + {{- if .Values.kubeconfig.enabled }} + - --kubeconfig=/opt/k8s/.kube/config + {{- end }} + {{- if .Values.selfMonitor.telemetryHost }} + - --telemetry-host={{ .Values.selfMonitor.telemetryHost }} + {{- end }} + {{- if .Values.selfMonitor.telemetryPort }} + - --telemetry-port={{ .Values.selfMonitor.telemetryPort | default 8081 }} + {{- end }} + {{- if or (.Values.kubeconfig.enabled) (.Values.volumeMounts) }} + volumeMounts: + {{- if .Values.kubeconfig.enabled }} + - name: kubeconfig + mountPath: /opt/k8s/.kube/ + readOnly: true + {{- end }} + {{- if .Values.volumeMounts }} +{{ toYaml .Values.volumeMounts | indent 8 }} + {{- end }} + {{- end }} + imagePullPolicy: {{ .Values.image.pullPolicy }} + {{- if .Values.image.sha }} + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}@sha256:{{ .Values.image.sha }}" + {{- else }} + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + {{- end }} + ports: + - containerPort: {{ .Values.service.port | default 8080}} + name: "http" + {{- if .Values.selfMonitor.enabled }} + - containerPort: {{ .Values.selfMonitor.telemetryPort | default 8081 }} + name: "metrics" + {{- end }} + livenessProbe: + httpGet: + path: /healthz + port: {{ .Values.service.port | default 8080}} + initialDelaySeconds: 5 + timeoutSeconds: 5 + readinessProbe: + httpGet: + path: / + port: {{ .Values.service.port | default 8080}} + initialDelaySeconds: 5 + timeoutSeconds: 5 + {{- if .Values.resources }} + resources: +{{ toYaml .Values.resources | indent 10 }} +{{- end }} +{{- if .Values.containerSecurityContext }} + securityContext: +{{ toYaml .Values.containerSecurityContext | indent 10 }} +{{- end }} +{{- if .Values.imagePullSecrets }} + imagePullSecrets: +{{ toYaml .Values.imagePullSecrets | indent 8 }} + {{- end }} + {{- if .Values.affinity }} + affinity: +{{ toYaml .Values.affinity | indent 8 }} + {{- end }} + {{- if .Values.nodeSelector }} + nodeSelector: +{{ toYaml .Values.nodeSelector | indent 8 }} + {{- end }} + {{- if .Values.tolerations }} + tolerations: +{{ toYaml .Values.tolerations | indent 8 }} + {{- end }} + {{- if .Values.topologySpreadConstraints }} + topologySpreadConstraints: +{{ toYaml .Values.topologySpreadConstraints | indent 8 }} + {{- end }} + {{- if or (.Values.kubeconfig.enabled) (.Values.volumes) }} + volumes: + {{- if .Values.kubeconfig.enabled}} + - name: kubeconfig + secret: + secretName: {{ template "kube-state-metrics.fullname" . }}-kubeconfig + {{- end }} + {{- if .Values.volumes }} +{{ toYaml .Values.volumes | indent 8 }} + {{- end }} + {{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/kubeconfig-secret.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/kubeconfig-secret.yaml new file mode 100644 index 00000000..6af00845 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/kubeconfig-secret.yaml @@ -0,0 +1,12 @@ +{{- if .Values.kubeconfig.enabled -}} +apiVersion: v1 +kind: Secret +metadata: + name: {{ template "kube-state-metrics.fullname" . }}-kubeconfig + namespace: {{ template "kube-state-metrics.namespace" . }} + labels: + {{- include "kube-state-metrics.labels" . | indent 4 }} +type: Opaque +data: + config: '{{ .Values.kubeconfig.secret }}' +{{- end -}} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/pdb.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/pdb.yaml new file mode 100644 index 00000000..3771b511 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/pdb.yaml @@ -0,0 +1,18 @@ +{{- if .Values.podDisruptionBudget -}} +{{ if $.Capabilities.APIVersions.Has "policy/v1/PodDisruptionBudget" -}} +apiVersion: policy/v1 +{{- else -}} +apiVersion: policy/v1beta1 +{{- end }} +kind: PodDisruptionBudget +metadata: + name: {{ template "kube-state-metrics.fullname" . }} + namespace: {{ template "kube-state-metrics.namespace" . }} + labels: + {{- include "kube-state-metrics.labels" . | indent 4 }} +spec: + selector: + matchLabels: + app.kubernetes.io/name: {{ template "kube-state-metrics.name" . }} +{{ toYaml .Values.podDisruptionBudget | indent 2 }} +{{- end -}} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/podsecuritypolicy.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/podsecuritypolicy.yaml new file mode 100644 index 00000000..8905e113 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/podsecuritypolicy.yaml @@ -0,0 +1,39 @@ +{{- if and .Values.podSecurityPolicy.enabled (.Capabilities.APIVersions.Has "policy/v1beta1/PodSecurityPolicy") }} +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: {{ template "kube-state-metrics.fullname" . }} + labels: + {{- include "kube-state-metrics.labels" . | indent 4 }} +{{- if .Values.podSecurityPolicy.annotations }} + annotations: +{{ toYaml .Values.podSecurityPolicy.annotations | indent 4 }} +{{- end }} +spec: + privileged: false + volumes: + - 'secret' +{{- if .Values.podSecurityPolicy.additionalVolumes }} +{{ toYaml .Values.podSecurityPolicy.additionalVolumes | indent 4 }} +{{- end }} + hostNetwork: false + hostIPC: false + hostPID: false + runAsUser: + rule: 'MustRunAsNonRoot' + seLinux: + rule: 'RunAsAny' + supplementalGroups: + rule: 'MustRunAs' + ranges: + # Forbid adding the root group. + - min: 1 + max: 65535 + fsGroup: + rule: 'MustRunAs' + ranges: + # Forbid adding the root group. + - min: 1 + max: 65535 + readOnlyRootFilesystem: false +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/psp-clusterrole.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/psp-clusterrole.yaml new file mode 100644 index 00000000..654e4a3d --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/psp-clusterrole.yaml @@ -0,0 +1,19 @@ +{{- if and .Values.podSecurityPolicy.enabled (.Capabilities.APIVersions.Has "policy/v1beta1/PodSecurityPolicy") }} +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + labels: + {{- include "kube-state-metrics.labels" . | indent 4 }} + name: psp-{{ template "kube-state-metrics.fullname" . }} +rules: +{{- $kubeTargetVersion := default .Capabilities.KubeVersion.GitVersion .Values.kubeTargetVersionOverride }} +{{- if semverCompare "> 1.15.0-0" $kubeTargetVersion }} +- apiGroups: ['policy'] +{{- else }} +- apiGroups: ['extensions'] +{{- end }} + resources: ['podsecuritypolicies'] + verbs: ['use'] + resourceNames: + - {{ template "kube-state-metrics.fullname" . }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/psp-clusterrolebinding.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/psp-clusterrolebinding.yaml new file mode 100644 index 00000000..5b62a18b --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/psp-clusterrolebinding.yaml @@ -0,0 +1,16 @@ +{{- if and .Values.podSecurityPolicy.enabled (.Capabilities.APIVersions.Has "policy/v1beta1/PodSecurityPolicy") }} +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + labels: + {{- include "kube-state-metrics.labels" . | indent 4 }} + name: psp-{{ template "kube-state-metrics.fullname" . }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp-{{ template "kube-state-metrics.fullname" . }} +subjects: + - kind: ServiceAccount + name: {{ template "kube-state-metrics.serviceAccountName" . }} + namespace: {{ template "kube-state-metrics.namespace" . }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/role.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/role.yaml new file mode 100644 index 00000000..6474914f --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/role.yaml @@ -0,0 +1,196 @@ +{{- if and (eq .Values.rbac.create true) (not .Values.rbac.useExistingRole) -}} +{{- range (ternary (join "," .Values.namespaces | split "," ) (list "") (eq $.Values.rbac.useClusterRole false)) }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +{{- if eq $.Values.rbac.useClusterRole false }} +kind: Role +{{- else }} +kind: ClusterRole +{{- end }} +metadata: + labels: + {{- include "kube-state-metrics.labels" $ | indent 4 }} + name: {{ template "kube-state-metrics.fullname" $ }} +{{- if eq $.Values.rbac.useClusterRole false }} + namespace: {{ . }} +{{- end }} +rules: +{{ if has "certificatesigningrequests" $.Values.collectors }} +- apiGroups: ["certificates.k8s.io"] + resources: + - certificatesigningrequests + verbs: ["list", "watch"] +{{ end -}} +{{ if has "configmaps" $.Values.collectors }} +- apiGroups: [""] + resources: + - configmaps + verbs: ["list", "watch"] +{{ end -}} +{{ if has "cronjobs" $.Values.collectors }} +- apiGroups: ["batch"] + resources: + - cronjobs + verbs: ["list", "watch"] +{{ end -}} +{{ if has "daemonsets" $.Values.collectors }} +- apiGroups: ["extensions", "apps"] + resources: + - daemonsets + verbs: ["list", "watch"] +{{ end -}} +{{ if has "deployments" $.Values.collectors }} +- apiGroups: ["extensions", "apps"] + resources: + - deployments + verbs: ["list", "watch"] +{{ end -}} +{{ if has "endpoints" $.Values.collectors }} +- apiGroups: [""] + resources: + - endpoints + verbs: ["list", "watch"] +{{ end -}} +{{ if has "horizontalpodautoscalers" $.Values.collectors }} +- apiGroups: ["autoscaling"] + resources: + - horizontalpodautoscalers + verbs: ["list", "watch"] +{{ end -}} +{{ if has "ingresses" $.Values.collectors }} +- apiGroups: ["extensions", "networking.k8s.io"] + resources: + - ingresses + verbs: ["list", "watch"] +{{ end -}} +{{ if has "jobs" $.Values.collectors }} +- apiGroups: ["batch"] + resources: + - jobs + verbs: ["list", "watch"] +{{ end -}} +{{ if has "leases" $.Values.collectors }} +- apiGroups: ["coordination.k8s.io"] + resources: + - leases + verbs: ["list", "watch"] +{{ end -}} +{{ if has "limitranges" $.Values.collectors }} +- apiGroups: [""] + resources: + - limitranges + verbs: ["list", "watch"] +{{ end -}} +{{ if has "mutatingwebhookconfigurations" $.Values.collectors }} +- apiGroups: ["admissionregistration.k8s.io"] + resources: + - mutatingwebhookconfigurations + verbs: ["list", "watch"] +{{ end -}} +{{ if has "namespaces" $.Values.collectors }} +- apiGroups: [""] + resources: + - namespaces + verbs: ["list", "watch"] +{{ end -}} +{{ if has "networkpolicies" $.Values.collectors }} +- apiGroups: ["networking.k8s.io"] + resources: + - networkpolicies + verbs: ["list", "watch"] +{{ end -}} +{{ if has "nodes" $.Values.collectors }} +- apiGroups: [""] + resources: + - nodes + verbs: ["list", "watch"] +{{ end -}} +{{ if has "persistentvolumeclaims" $.Values.collectors }} +- apiGroups: [""] + resources: + - persistentvolumeclaims + verbs: ["list", "watch"] +{{ end -}} +{{ if has "persistentvolumes" $.Values.collectors }} +- apiGroups: [""] + resources: + - persistentvolumes + verbs: ["list", "watch"] +{{ end -}} +{{ if has "poddisruptionbudgets" $.Values.collectors }} +- apiGroups: ["policy"] + resources: + - poddisruptionbudgets + verbs: ["list", "watch"] +{{ end -}} +{{ if has "pods" $.Values.collectors }} +- apiGroups: [""] + resources: + - pods + verbs: ["list", "watch"] +{{ end -}} +{{ if has "replicasets" $.Values.collectors }} +- apiGroups: ["extensions", "apps"] + resources: + - replicasets + verbs: ["list", "watch"] +{{ end -}} +{{ if has "replicationcontrollers" $.Values.collectors }} +- apiGroups: [""] + resources: + - replicationcontrollers + verbs: ["list", "watch"] +{{ end -}} +{{ if has "resourcequotas" $.Values.collectors }} +- apiGroups: [""] + resources: + - resourcequotas + verbs: ["list", "watch"] +{{ end -}} +{{ if has "secrets" $.Values.collectors }} +- apiGroups: [""] + resources: + - secrets + verbs: ["list", "watch"] +{{ end -}} +{{ if has "services" $.Values.collectors }} +- apiGroups: [""] + resources: + - services + verbs: ["list", "watch"] +{{ end -}} +{{ if has "statefulsets" $.Values.collectors }} +- apiGroups: ["apps"] + resources: + - statefulsets + verbs: ["list", "watch"] +{{ end -}} +{{ if has "storageclasses" $.Values.collectors }} +- apiGroups: ["storage.k8s.io"] + resources: + - storageclasses + verbs: ["list", "watch"] +{{ end -}} +{{ if has "validatingwebhookconfigurations" $.Values.collectors }} +- apiGroups: ["admissionregistration.k8s.io"] + resources: + - validatingwebhookconfigurations + verbs: ["list", "watch"] +{{ end -}} +{{ if has "volumeattachments" $.Values.collectors }} +- apiGroups: ["storage.k8s.io"] + resources: + - volumeattachments + verbs: ["list", "watch"] +{{ end -}} +{{ if has "verticalpodautoscalers" $.Values.collectors }} +- apiGroups: ["autoscaling.k8s.io"] + resources: + - verticalpodautoscalers + verbs: ["list", "watch"] +{{ end -}} +{{ if $.Values.rbac.extraRules }} +{{ toYaml $.Values.rbac.extraRules }} +{{ end }} +{{- end -}} +{{- end -}} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/rolebinding.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/rolebinding.yaml new file mode 100644 index 00000000..330651b7 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/rolebinding.yaml @@ -0,0 +1,24 @@ +{{- if and (eq .Values.rbac.create true) (eq .Values.rbac.useClusterRole false) -}} +{{- range (join "," $.Values.namespaces) | split "," }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + labels: + {{- include "kube-state-metrics.labels" $ | indent 4 }} + name: {{ template "kube-state-metrics.fullname" $ }} + namespace: {{ . }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role +{{- if (not $.Values.rbac.useExistingRole) }} + name: {{ template "kube-state-metrics.fullname" $ }} +{{- else }} + name: {{ $.Values.rbac.useExistingRole }} +{{- end }} +subjects: +- kind: ServiceAccount + name: {{ template "kube-state-metrics.serviceAccountName" $ }} + namespace: {{ template "kube-state-metrics.namespace" $ }} +{{- end -}} +{{- end -}} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/service.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/service.yaml new file mode 100644 index 00000000..6c486a66 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/service.yaml @@ -0,0 +1,49 @@ +apiVersion: v1 +kind: Service +metadata: + name: {{ template "kube-state-metrics.fullname" . }} + namespace: {{ template "kube-state-metrics.namespace" . }} + labels: + {{- include "kube-state-metrics.labels" . | indent 4 }} + annotations: + {{- if .Values.prometheusScrape }} + prometheus.io/scrape: '{{ .Values.prometheusScrape }}' + {{- end }} + {{- if .Values.service.annotations }} + {{- toYaml .Values.service.annotations | nindent 4 }} + {{- end }} +spec: + type: "{{ .Values.service.type }}" + ports: + - name: "http" + protocol: TCP + port: {{ .Values.service.port | default 8080}} + {{- if .Values.service.nodePort }} + nodePort: {{ .Values.service.nodePort }} + {{- end }} + targetPort: {{ .Values.service.port | default 8080}} + {{ if .Values.selfMonitor.enabled }} + - name: "metrics" + protocol: TCP + port: {{ .Values.selfMonitor.telemetryPort | default 8081 }} + targetPort: {{ .Values.selfMonitor.telemetryPort | default 8081 }} + {{- if .Values.selfMonitor.telemetryNodePort }} + nodePort: {{ .Values.selfMonitor.telemetryNodePort }} + {{- end }} + {{ end }} +{{- if .Values.service.loadBalancerIP }} + loadBalancerIP: "{{ .Values.service.loadBalancerIP }}" +{{- end }} +{{- if .Values.service.loadBalancerSourceRanges }} + loadBalancerSourceRanges: + {{- range $cidr := .Values.service.loadBalancerSourceRanges }} + - {{ $cidr }} + {{- end }} +{{- end }} +{{- if .Values.autosharding.enabled }} + clusterIP: None +{{- else if .Values.service.clusterIP }} + clusterIP: "{{ .Values.service.clusterIP }}" +{{- end }} + selector: + {{- include "kube-state-metrics.selectorLabels" . | indent 4 }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/serviceaccount.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/serviceaccount.yaml new file mode 100644 index 00000000..e1229eb9 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/serviceaccount.yaml @@ -0,0 +1,15 @@ +{{- if .Values.serviceAccount.create -}} +apiVersion: v1 +kind: ServiceAccount +metadata: + labels: + {{- include "kube-state-metrics.labels" . | indent 4 }} + name: {{ template "kube-state-metrics.serviceAccountName" . }} + namespace: {{ template "kube-state-metrics.namespace" . }} +{{- if .Values.serviceAccount.annotations }} + annotations: +{{ toYaml .Values.serviceAccount.annotations | indent 4 }} +{{- end }} +imagePullSecrets: +{{ toYaml .Values.serviceAccount.imagePullSecrets | indent 2 }} +{{- end -}} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/servicemonitor.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/servicemonitor.yaml new file mode 100644 index 00000000..e93df4c4 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/servicemonitor.yaml @@ -0,0 +1,81 @@ +{{- if .Values.prometheus.monitor.enabled }} +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: {{ template "kube-state-metrics.fullname" . }} + namespace: {{ template "kube-state-metrics.namespace" . }} + labels: + {{- include "kube-state-metrics.labels" . | indent 4 }} + {{- with .Values.prometheus.monitor.additionalLabels }} + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + jobLabel: {{ default "app.kubernetes.io/name" .Values.prometheus.monitor.jobLabel }} + {{- include "servicemonitor.scrapeLimits" .Values.prometheus.monitor | indent 2 }} + selector: + matchLabels: + {{- with .Values.prometheus.monitor.selectorOverride }} + {{- toYaml . | nindent 6 }} + {{- else }} + {{- include "kube-state-metrics.selectorLabels" . | indent 6 }} + {{- end }} + endpoints: + - port: http + {{- if .Values.prometheus.monitor.interval }} + interval: {{ .Values.prometheus.monitor.interval }} + {{- end }} + {{- if .Values.prometheus.monitor.scrapeTimeout }} + scrapeTimeout: {{ .Values.prometheus.monitor.scrapeTimeout }} + {{- end }} + {{- if .Values.prometheus.monitor.proxyUrl }} + proxyUrl: {{ .Values.prometheus.monitor.proxyUrl}} + {{- end }} + {{- if .Values.prometheus.monitor.honorLabels }} + honorLabels: true + {{- end }} + {{- if .Values.prometheus.monitor.metricRelabelings }} + metricRelabelings: + {{- toYaml .Values.prometheus.monitor.metricRelabelings | nindent 8 }} + {{- end }} + {{- if .Values.prometheus.monitor.relabelings }} + relabelings: + {{- toYaml .Values.prometheus.monitor.relabelings | nindent 8 }} + {{- end }} + {{- if .Values.prometheus.monitor.scheme }} + scheme: {{ .Values.prometheus.monitor.scheme }} + {{- end }} + {{- if .Values.prometheus.monitor.tlsConfig }} + tlsConfig: + {{- toYaml .Values.prometheus.monitor.tlsConfig | nindent 8 }} + {{- end }} + {{- if .Values.selfMonitor.enabled }} + - port: metrics + {{- if .Values.prometheus.monitor.interval }} + interval: {{ .Values.prometheus.monitor.interval }} + {{- end }} + {{- if .Values.prometheus.monitor.scrapeTimeout }} + scrapeTimeout: {{ .Values.prometheus.monitor.scrapeTimeout }} + {{- end }} + {{- if .Values.prometheus.monitor.proxyUrl }} + proxyUrl: {{ .Values.prometheus.monitor.proxyUrl}} + {{- end }} + {{- if .Values.prometheus.monitor.honorLabels }} + honorLabels: true + {{- end }} + {{- if .Values.prometheus.monitor.metricRelabelings }} + metricRelabelings: + {{- toYaml .Values.prometheus.monitor.metricRelabelings | nindent 8 }} + {{- end }} + {{- if .Values.prometheus.monitor.relabelings }} + relabelings: + {{- toYaml .Values.prometheus.monitor.relabelings | nindent 8 }} + {{- end }} + {{- if .Values.prometheus.monitor.scheme }} + scheme: {{ .Values.prometheus.monitor.scheme }} + {{- end }} + {{- if .Values.prometheus.monitor.tlsConfig }} + tlsConfig: + {{- toYaml .Values.prometheus.monitor.tlsConfig | nindent 8 }} + {{- end }} + {{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/stsdiscovery-role.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/stsdiscovery-role.yaml new file mode 100644 index 00000000..489de147 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/stsdiscovery-role.yaml @@ -0,0 +1,26 @@ +{{- if and .Values.autosharding.enabled .Values.rbac.create -}} +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: stsdiscovery-{{ template "kube-state-metrics.fullname" . }} + namespace: {{ template "kube-state-metrics.namespace" . }} + labels: + {{- include "kube-state-metrics.labels" . | indent 4 }} +rules: +- apiGroups: + - "" + resources: + - pods + verbs: + - get +- apiGroups: + - apps + resourceNames: + - {{ template "kube-state-metrics.fullname" . }} + resources: + - statefulsets + verbs: + - get + - list + - watch +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/stsdiscovery-rolebinding.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/stsdiscovery-rolebinding.yaml new file mode 100644 index 00000000..73b37a4f --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/stsdiscovery-rolebinding.yaml @@ -0,0 +1,17 @@ +{{- if and .Values.autosharding.enabled .Values.rbac.create -}} +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: stsdiscovery-{{ template "kube-state-metrics.fullname" . }} + namespace: {{ template "kube-state-metrics.namespace" . }} + labels: + {{- include "kube-state-metrics.labels" . | indent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: stsdiscovery-{{ template "kube-state-metrics.fullname" . }} +subjects: + - kind: ServiceAccount + name: {{ template "kube-state-metrics.serviceAccountName" . }} + namespace: {{ template "kube-state-metrics.namespace" . }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/verticalpodautoscaler.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/verticalpodautoscaler.yaml new file mode 100644 index 00000000..e825e5c8 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/templates/verticalpodautoscaler.yaml @@ -0,0 +1,34 @@ +{{- if and (.Capabilities.APIVersions.Has "autoscaling.k8s.io/v1") (.Values.verticalPodAutoscaler.enabled) }} +apiVersion: autoscaling.k8s.io/v1 +kind: VerticalPodAutoscaler +metadata: + name: {{ template "kube-state-metrics.fullname" . }} + namespace: {{ template "kube-state-metrics.namespace" . }} + labels: + {{- include "kube-state-metrics.labels" . | indent 4 }} +spec: + resourcePolicy: + containerPolicies: + - containerName: {{ template "kube-state-metrics.name" . }} + {{- if .Values.verticalPodAutoscaler.controlledResources }} + controlledResources: {{ .Values.verticalPodAutoscaler.controlledResources }} + {{- end }} + {{- if .Values.verticalPodAutoscaler.maxAllowed }} + maxAllowed: + {{ toYaml .Values.verticalPodAutoscaler.maxAllowed | nindent 8 }} + {{- end }} + {{- if .Values.verticalPodAutoscaler.minAllowed }} + minAllowed: + {{ toYaml .Values.verticalPodAutoscaler.minAllowed | nindent 8 }} + {{- end }} + targetRef: + apiVersion: apps/v1 + kind: Deployment + name: {{ template "kube-state-metrics.fullname" . }} + {{- if .Values.verticalPodAutoscaler.updatePolicy }} + updatePolicy: + {{- if .Values.verticalPodAutoscaler.updatePolicy.updateMode }} + updateMode: {{ .Values.verticalPodAutoscaler.updatePolicy.updateMode }} + {{- end }} + {{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/values.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/values.yaml new file mode 100644 index 00000000..d32b1232 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/kube-state-metrics/values.yaml @@ -0,0 +1,307 @@ +# Default values for kube-state-metrics. +prometheusScrape: true +image: + repository: registry.k8s.io/kube-state-metrics/kube-state-metrics + tag: v2.7.0 + sha: "" + pullPolicy: IfNotPresent + +imagePullSecrets: [] +# - name: "image-pull-secret" + +# If set to true, this will deploy kube-state-metrics as a StatefulSet and the data +# will be automatically sharded across <.Values.replicas> pods using the built-in +# autodiscovery feature: https://github.com/kubernetes/kube-state-metrics#automated-sharding +# This is an experimental feature and there are no stability guarantees. +autosharding: + enabled: false + +replicas: 1 + +# List of additional cli arguments to configure kube-state-metrics +# for example: --enable-gzip-encoding, --log-file, etc. +# all the possible args can be found here: https://github.com/kubernetes/kube-state-metrics/blob/master/docs/cli-arguments.md +extraArgs: [] + +service: + port: 8080 + # Default to clusterIP for backward compatibility + type: ClusterIP + nodePort: 0 + loadBalancerIP: "" + # Only allow access to the loadBalancerIP from these IPs + loadBalancerSourceRanges: [] + clusterIP: "" + annotations: {} + +## Additional labels to add to all resources +customLabels: {} + # app: kube-state-metrics + +## set to true to add the release label so scraping of the servicemonitor with kube-prometheus-stack works out of the box +releaseLabel: false + +hostNetwork: false + +rbac: + # If true, create & use RBAC resources + create: true + + # Set to a rolename to use existing role - skipping role creating - but still doing serviceaccount and rolebinding to it, rolename set here. + # useExistingRole: your-existing-role + + # If set to false - Run without Cluteradmin privs needed - ONLY works if namespace is also set (if useExistingRole is set this name is used as ClusterRole or Role to bind to) + useClusterRole: true + + # Add permissions for CustomResources' apiGroups in Role/ClusterRole. Should be used in conjunction with Custom Resource State Metrics configuration + # Example: + # - apiGroups: ["monitoring.coreos.com"] + # resources: ["prometheuses"] + # verbs: ["list", "watch"] + extraRules: [] + +serviceAccount: + # Specifies whether a ServiceAccount should be created, require rbac true + create: true + # The name of the ServiceAccount to use. + # If not set and create is true, a name is generated using the fullname template + name: + # Reference to one or more secrets to be used when pulling images + # ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ + imagePullSecrets: [] + # ServiceAccount annotations. + # Use case: AWS EKS IAM roles for service accounts + # ref: https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html + annotations: {} + +prometheus: + monitor: + enabled: false + additionalLabels: {} + namespace: "" + jobLabel: "" + interval: "" + ## SampleLimit defines per-scrape limit on number of scraped samples that will be accepted. + ## + sampleLimit: 0 + + ## TargetLimit defines a limit on the number of scraped targets that will be accepted. + ## + targetLimit: 0 + + ## Per-scrape limit on number of labels that will be accepted for a sample. Only valid in Prometheus versions 2.27.0 and newer. + ## + labelLimit: 0 + + ## Per-scrape limit on length of labels name that will be accepted for a sample. Only valid in Prometheus versions 2.27.0 and newer. + ## + labelNameLengthLimit: 0 + + ## Per-scrape limit on length of labels value that will be accepted for a sample. Only valid in Prometheus versions 2.27.0 and newer. + ## + labelValueLengthLimit: 0 + scrapeTimeout: "" + proxyUrl: "" + selectorOverride: {} + honorLabels: false + metricRelabelings: [] + relabelings: [] + scheme: "" + tlsConfig: {} + +## Specify if a Pod Security Policy for kube-state-metrics must be created +## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/ +## +podSecurityPolicy: + enabled: false + annotations: {} + ## Specify pod annotations + ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#apparmor + ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#seccomp + ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#sysctl + ## + # seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*' + # seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default' + # apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default' + + additionalVolumes: [] + +securityContext: + enabled: true + runAsGroup: 65534 + runAsUser: 65534 + fsGroup: 65534 + +## Specify security settings for a Container +## Allows overrides and additional options compared to (Pod) securityContext +## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container +containerSecurityContext: {} + +## Node labels for pod assignment +## Ref: https://kubernetes.io/docs/user-guide/node-selection/ +nodeSelector: {} + +## Affinity settings for pod assignment +## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ +affinity: {} + +## Tolerations for pod assignment +## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ +tolerations: [] + +## Topology spread constraints for pod assignment +## Ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/ +topologySpreadConstraints: [] + +# Annotations to be added to the deployment/statefulset +annotations: {} + +# Annotations to be added to the pod +podAnnotations: {} + +## Assign a PriorityClassName to pods if set +# priorityClassName: "" + +# Ref: https://kubernetes.io/docs/tasks/run-application/configure-pdb/ +podDisruptionBudget: {} + +# Comma-separated list of metrics to be exposed. +# This list comprises of exact metric names and/or regex patterns. +# The allowlist and denylist are mutually exclusive. +metricAllowlist: [] + +# Comma-separated list of metrics not to be enabled. +# This list comprises of exact metric names and/or regex patterns. +# The allowlist and denylist are mutually exclusive. +metricDenylist: [] + +# Comma-separated list of additional Kubernetes label keys that will be used in the resource's +# labels metric. By default the metric contains only name and namespace labels. +# To include additional labels, provide a list of resource names in their plural form and Kubernetes +# label keys you would like to allow for them (Example: '=namespaces=[k8s-label-1,k8s-label-n,...],pods=[app],...)'. +# A single '*' can be provided per resource instead to allow any labels, but that has +# severe performance implications (Example: '=pods=[*]'). +metricLabelsAllowlist: [] + # - namespaces=[k8s-label-1,k8s-label-n] + +# Comma-separated list of Kubernetes annotations keys that will be used in the resource' +# labels metric. By default the metric contains only name and namespace labels. +# To include additional annotations provide a list of resource names in their plural form and Kubernetes +# annotation keys you would like to allow for them (Example: '=namespaces=[kubernetes.io/team,...],pods=[kubernetes.io/team],...)'. +# A single '*' can be provided per resource instead to allow any annotations, but that has +# severe performance implications (Example: '=pods=[*]'). +metricAnnotationsAllowList: [] + # - pods=[k8s-annotation-1,k8s-annotation-n] + +# Available collectors for kube-state-metrics. +# By default, all available resources are enabled, comment out to disable. +collectors: + - certificatesigningrequests + - configmaps + - cronjobs + - daemonsets + - deployments + - endpoints + - horizontalpodautoscalers + - ingresses + - jobs + - leases + - limitranges + - mutatingwebhookconfigurations + - namespaces + - networkpolicies + - nodes + - persistentvolumeclaims + - persistentvolumes + - poddisruptionbudgets + - pods + - replicasets + - replicationcontrollers + - resourcequotas + - secrets + - services + - statefulsets + - storageclasses + - validatingwebhookconfigurations + - volumeattachments + # - verticalpodautoscalers # not a default resource, see also: https://github.com/kubernetes/kube-state-metrics#enabling-verticalpodautoscalers + +# Enabling kubeconfig will pass the --kubeconfig argument to the container +kubeconfig: + enabled: false + # base64 encoded kube-config file + secret: + +# Enable only the release namespace for collecting resources. By default all namespaces are collected. +# If releaseNamespace and namespaces are both set a merged list will be collected. +releaseNamespace: false + +# Comma-separated list(string) or yaml list of namespaces to be enabled for collecting resources. By default all namespaces are collected. +namespaces: "" + +# Comma-separated list of namespaces not to be enabled. If namespaces and namespaces-denylist are both set, +# only namespaces that are excluded in namespaces-denylist will be used. +namespacesDenylist: "" + +## Override the deployment namespace +## +namespaceOverride: "" + +resources: {} + # We usually recommend not to specify default resources and to leave this as a conscious + # choice for the user. This also increases chances charts run on environments with little + # resources, such as Minikube. If you do want to specify resources, uncomment the following + # lines, adjust them as necessary, and remove the curly braces after 'resources:'. + # limits: + # cpu: 100m + # memory: 64Mi + # requests: + # cpu: 10m + # memory: 32Mi + +## Provide a k8s version to define apiGroups for podSecurityPolicy Cluster Role. +## For example: kubeTargetVersionOverride: 1.14.9 +## +kubeTargetVersionOverride: "" + +# Enable self metrics configuration for service and Service Monitor +# Default values for telemetry configuration can be overridden +# If you set telemetryNodePort, you must also set service.type to NodePort +selfMonitor: + enabled: false + # telemetryHost: 0.0.0.0 + # telemetryPort: 8081 + # telemetryNodePort: 0 + +# Enable vertical pod autoscaler support for kube-state-metrics +verticalPodAutoscaler: + enabled: false + # List of resources that the vertical pod autoscaler can control. Defaults to cpu and memory + controlledResources: [] + + # Define the max allowed resources for the pod + maxAllowed: {} + # cpu: 200m + # memory: 100Mi + # Define the min allowed resources for the pod + minAllowed: {} + # cpu: 200m + # memory: 100Mi + + # updatePolicy: + # Specifies whether recommended updates are applied when a Pod is started and whether recommended updates + # are applied during the life of a Pod. Possible values are "Off", "Initial", "Recreate", and "Auto". + # updateMode: Auto + +# volumeMounts are used to add custom volume mounts to deployment. +# See example below +volumeMounts: [] +# - mountPath: /etc/config +# name: config-volume + +# volumes are used to add custom volumes to deployment +# See example below +volumes: [] +# - configMap: +# name: cm-for-volume +# name: config-volume diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/.helmignore b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/.helmignore new file mode 100644 index 00000000..f0c13194 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/.helmignore @@ -0,0 +1,21 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*~ +# Various IDEs +.project +.idea/ +*.tmproj diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/Chart.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/Chart.yaml new file mode 100644 index 00000000..51b2cede --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/Chart.yaml @@ -0,0 +1,18 @@ +apiVersion: v2 +appVersion: 1.5.0 +description: A Helm chart for prometheus node-exporter +home: https://github.com/prometheus/node_exporter/ +keywords: +- node-exporter +- prometheus +- exporter +maintainers: +- email: gianrubio@gmail.com + name: gianrubio +- email: zanhsieh@gmail.com + name: zanhsieh +name: prometheus-node-exporter +sources: +- https://github.com/prometheus/node_exporter/ +type: application +version: 4.8.1 diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/README.md b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/README.md new file mode 100644 index 00000000..02de7b14 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/README.md @@ -0,0 +1,77 @@ +# Prometheus `Node Exporter` + +Prometheus exporter for hardware and OS metrics exposed by *NIX kernels, written in Go with pluggable metric collectors. + +This chart bootstraps a prometheus [`Node Exporter`](http://github.com/prometheus/node_exporter) daemonset on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager. + +## Get Repository Info + +```console +helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +helm repo update +``` + +_See [`helm repo`](https://helm.sh/docs/helm/helm_repo/) for command documentation._ + +## Install Chart + +```console +helm install [RELEASE_NAME] prometheus-community/prometheus-node-exporter +``` + +_See [configuration](#configuring) below._ + +_See [helm install](https://helm.sh/docs/helm/helm_install/) for command documentation._ + +## Uninstall Chart + +```console +helm uninstall [RELEASE_NAME] +``` + +This removes all the Kubernetes components associated with the chart and deletes the release. + +_See [helm uninstall](https://helm.sh/docs/helm/helm_uninstall/) for command documentation._ + +## Upgrading Chart + +```console +helm upgrade [RELEASE_NAME] [CHART] --install +``` + +_See [helm upgrade](https://helm.sh/docs/helm/helm_upgrade/) for command documentation._ + +### 3.x to 4.x + +Starting from version 4.0.0, the `node exporter` chart is using the [Kubernetes recommended labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/). Therefore you have to delete the daemonset before you upgrade. + +```console +kubectl delete daemonset -l app=prometheus-node-exporter +helm upgrade -i prometheus-node-exporter prometheus-community/prometheus-node-exporter +``` + +If you use your own custom [ServiceMonitor](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#servicemonitor) or [PodMonitor](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#podmonitor), please ensure to upgrade their `selector` fields accordingly to the new labels. + +### From 2.x to 3.x + +Change the following: + +```yaml +hostRootFsMount: true +``` + +to: + +```yaml +hostRootFsMount: + enabled: true + mountPropagation: HostToContainer +``` + +## Configuring + +See [Customizing the Chart Before Installing](https://helm.sh/docs/intro/using_helm/#customizing-the-chart-before-installing). To see all configurable options with detailed comments, visit the chart's [values.yaml](./values.yaml), or run these configuration commands: + +```console +helm show values prometheus-community/prometheus-node-exporter +``` diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/ci/port-values.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/ci/port-values.yaml new file mode 100644 index 00000000..dbfb4b67 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/ci/port-values.yaml @@ -0,0 +1,3 @@ +service: + targetPort: 9102 + port: 9102 diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/NOTES.txt b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/NOTES.txt new file mode 100644 index 00000000..df05e3fb --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/NOTES.txt @@ -0,0 +1,15 @@ +1. Get the application URL by running these commands: +{{- if contains "NodePort" .Values.service.type }} + export NODE_PORT=$(kubectl get --namespace {{ template "prometheus-node-exporter.namespace" . }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ template "prometheus-node-exporter.fullname" . }}) + export NODE_IP=$(kubectl get nodes --namespace {{ template "prometheus-node-exporter.namespace" . }} -o jsonpath="{.items[0].status.addresses[0].address}") + echo http://$NODE_IP:$NODE_PORT +{{- else if contains "LoadBalancer" .Values.service.type }} + NOTE: It may take a few minutes for the LoadBalancer IP to be available. + You can watch the status of by running 'kubectl get svc -w {{ template "prometheus-node-exporter.fullname" . }}' + export SERVICE_IP=$(kubectl get svc --namespace {{ template "prometheus-node-exporter.namespace" . }} {{ template "prometheus-node-exporter.fullname" . }} -o jsonpath='{.status.loadBalancer.ingress[0].ip}') + echo http://$SERVICE_IP:{{ .Values.service.port }} +{{- else if contains "ClusterIP" .Values.service.type }} + export POD_NAME=$(kubectl get pods --namespace {{ template "prometheus-node-exporter.namespace" . }} -l "app.kubernetes.io/name={{ template "prometheus-node-exporter.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}") + echo "Visit http://127.0.0.1:9100 to use your application" + kubectl port-forward --namespace {{ template "prometheus-node-exporter.namespace" . }} $POD_NAME 9100 +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/_helpers.tpl b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/_helpers.tpl new file mode 100644 index 00000000..f5e2603e --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/_helpers.tpl @@ -0,0 +1,128 @@ +{{/* vim: set filetype=mustache: */}} +{{/* +Expand the name of the chart. +*/}} +{{- define "prometheus-node-exporter.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +If release name contains chart name it will be used as a full name. +*/}} +{{- define "prometheus-node-exporter.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "prometheus-node-exporter.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "prometheus-node-exporter.labels" -}} +helm.sh/chart: {{ include "prometheus-node-exporter.chart" . }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +app.kubernetes.io/component: metrics +app.kubernetes.io/part-of: {{ include "prometheus-node-exporter.name" . }} +{{ include "prometheus-node-exporter.selectorLabels" . }} +{{- with .Chart.AppVersion }} +app.kubernetes.io/version: {{ . | quote }} +{{- end }} +{{- with .Values.podLabels }} +{{ toYaml . }} +{{- end }} +{{- if .Values.releaseLabel }} +release: {{ .Release.Name }} +{{- end }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "prometheus-node-exporter.selectorLabels" -}} +app.kubernetes.io/name: {{ include "prometheus-node-exporter.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + + +{{/* +Create the name of the service account to use +*/}} +{{- define "prometheus-node-exporter.serviceAccountName" -}} +{{- if .Values.serviceAccount.create }} +{{- default (include "prometheus-node-exporter.fullname" .) .Values.serviceAccount.name }} +{{- else }} +{{- default "default" .Values.serviceAccount.name }} +{{- end }} +{{- end }} + +{{/* +The image to use +*/}} +{{- define "prometheus-node-exporter.image" -}} +{{- if .Values.image.sha }} +{{- printf "%s:%s@%s" .Values.image.repository (default (printf "v%s" .Chart.AppVersion) .Values.image.tag) .Values.image.sha }} +{{- else }} +{{- printf "%s:%s" .Values.image.repository (default (printf "v%s" .Chart.AppVersion) .Values.image.tag) }} +{{- end }} +{{- end }} + +{{/* +Allow the release namespace to be overridden for multi-namespace deployments in combined charts +*/}} +{{- define "prometheus-node-exporter.namespace" -}} +{{- if .Values.namespaceOverride }} +{{- .Values.namespaceOverride }} +{{- else }} +{{- .Release.Namespace }} +{{- end }} +{{- end }} + +{{/* +Create the namespace name of the service monitor +*/}} +{{- define "prometheus-node-exporter.monitor-namespace" -}} +{{- if .Values.namespaceOverride }} +{{- .Values.namespaceOverride }} +{{- else }} +{{- if .Values.prometheus.monitor.namespace }} +{{- .Values.prometheus.monitor.namespace }} +{{- else }} +{{- .Release.Namespace }} +{{- end }} +{{- end }} +{{- end }} + +{{/* Sets default scrape limits for servicemonitor */}} +{{- define "servicemonitor.scrapeLimits" -}} +{{- with .sampleLimit }} +sampleLimit: {{ . }} +{{- end }} +{{- with .targetLimit }} +targetLimit: {{ . }} +{{- end }} +{{- with .labelLimit }} +labelLimit: {{ . }} +{{- end }} +{{- with .labelNameLengthLimit }} +labelNameLengthLimit: {{ . }} +{{- end }} +{{- with .labelValueLengthLimit }} +labelValueLengthLimit: {{ . }} +{{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/daemonset.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/daemonset.yaml new file mode 100644 index 00000000..7603d947 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/daemonset.yaml @@ -0,0 +1,225 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: {{ include "prometheus-node-exporter.fullname" . }} + namespace: {{ include "prometheus-node-exporter.namespace" . }} + labels: + {{- include "prometheus-node-exporter.labels" . | nindent 4 }} + {{- with .Values.daemonsetAnnotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + selector: + matchLabels: + {{- include "prometheus-node-exporter.selectorLabels" . | nindent 6 }} + {{- with .Values.updateStrategy }} + updateStrategy: + {{- toYaml . | nindent 4 }} + {{- end }} + template: + metadata: + {{- with .Values.podAnnotations }} + annotations: + {{- toYaml . | nindent 8 }} + {{- end }} + labels: + {{- include "prometheus-node-exporter.labels" . | nindent 8 }} + spec: + automountServiceAccountToken: {{ .Values.serviceAccount.automountServiceAccountToken }} + {{- with .Values.securityContext }} + securityContext: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.priorityClassName }} + priorityClassName: {{ . }} + {{- end }} + {{- with .Values.extraInitContainers }} + initContainers: + {{- toYaml . | nindent 8 }} + {{- end }} + serviceAccountName: {{ include "prometheus-node-exporter.serviceAccountName" . }} + containers: + - name: node-exporter + image: {{ include "prometheus-node-exporter.image" . }} + imagePullPolicy: {{ .Values.image.pullPolicy }} + args: + - --path.procfs=/host/proc + - --path.sysfs=/host/sys + {{- if .Values.hostRootFsMount.enabled }} + - --path.rootfs=/host/root + {{- end }} + - --web.listen-address=[$(HOST_IP)]:{{ .Values.service.port }} + {{- with .Values.extraArgs }} + {{- toYaml . | nindent 12 }} + {{- end }} + {{- with .Values.containerSecurityContext }} + securityContext: + {{- toYaml . | nindent 12 }} + {{- end }} + env: + - name: HOST_IP + {{- if .Values.service.listenOnAllInterfaces }} + value: 0.0.0.0 + {{- else }} + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: status.hostIP + {{- end }} + {{- range $key, $value := .Values.env }} + - name: {{ $key }} + value: {{ $value | quote }} + {{- end }} + ports: + - name: {{ .Values.service.portName }} + containerPort: {{ .Values.service.port }} + protocol: TCP + livenessProbe: + failureThreshold: {{ .Values.livenessProbe.failureThreshold }} + httpGet: + httpHeaders: + {{- range $_, $header := .Values.livenessProbe.httpGet.httpHeaders }} + - name: {{ $header.name }} + value: {{ $header.value }} + {{- end }} + path: / + port: {{ .Values.service.port }} + scheme: {{ upper .Values.livenessProbe.httpGet.scheme }} + initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }} + periodSeconds: {{ .Values.livenessProbe.periodSeconds }} + successThreshold: {{ .Values.livenessProbe.successThreshold }} + timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }} + readinessProbe: + failureThreshold: {{ .Values.readinessProbe.failureThreshold }} + httpGet: + httpHeaders: + {{- range $_, $header := .Values.readinessProbe.httpGet.httpHeaders }} + - name: {{ $header.name }} + value: {{ $header.value }} + {{- end }} + path: / + port: {{ .Values.service.port }} + scheme: {{ upper .Values.readinessProbe.httpGet.scheme }} + initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }} + periodSeconds: {{ .Values.readinessProbe.periodSeconds }} + successThreshold: {{ .Values.readinessProbe.successThreshold }} + timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }} + {{- with .Values.resources }} + resources: + {{- toYaml . | nindent 12 }} + {{- end }} + volumeMounts: + - name: proc + mountPath: /host/proc + readOnly: true + - name: sys + mountPath: /host/sys + readOnly: true + {{- if .Values.hostRootFsMount.enabled }} + - name: root + mountPath: /host/root + {{- with .Values.hostRootFsMount.mountPropagation }} + mountPropagation: {{ . }} + {{- end }} + readOnly: true + {{- end }} + {{- range $_, $mount := .Values.extraHostVolumeMounts }} + - name: {{ $mount.name }} + mountPath: {{ $mount.mountPath }} + readOnly: {{ $mount.readOnly }} + {{- with $mount.mountPropagation }} + mountPropagation: {{ . }} + {{- end }} + {{- end }} + {{- range $_, $mount := .Values.sidecarVolumeMount }} + - name: {{ $mount.name }} + mountPath: {{ $mount.mountPath }} + readOnly: true + {{- end }} + {{- range $_, $mount := .Values.configmaps }} + - name: {{ $mount.name }} + mountPath: {{ $mount.mountPath }} + {{- end }} + {{- range $_, $mount := .Values.secrets }} + - name: {{ .name }} + mountPath: {{ .mountPath }} + {{- end }} + {{- with .Values.sidecars }} + {{- toYaml . | nindent 8 }} + {{- if or $.Values.sidecarVolumeMount $.Values.sidecarHostVolumeMounts }} + volumeMounts: + {{- range $_, $mount := $.Values.sidecarVolumeMount }} + - name: {{ $mount.name }} + mountPath: {{ $mount.mountPath }} + readOnly: {{ $mount.readOnly }} + {{- end }} + {{- range $_, $mount := $.Values.sidecarHostVolumeMounts }} + - name: {{ $mount.name }} + mountPath: {{ $mount.mountPath }} + readOnly: {{ $mount.readOnly }} + {{- if $mount.mountPropagation }} + mountPropagation: {{ $mount.mountPropagation }} + {{- end }} + {{- end }} + {{- end }} + {{- end }} + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{ toYaml . | nindent 8 }} + {{- end }} + hostNetwork: {{ .Values.hostNetwork }} + hostPID: {{ .Values.hostPID }} + {{- with .Values.affinity }} + affinity: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.dnsConfig }} + dnsConfig: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.nodeSelector }} + nodeSelector: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.tolerations }} + tolerations: + {{- toYaml . | nindent 8 }} + {{- end }} + volumes: + - name: proc + hostPath: + path: /proc + - name: sys + hostPath: + path: /sys + {{- if .Values.hostRootFsMount.enabled }} + - name: root + hostPath: + path: / + {{- end }} + {{- range $_, $mount := .Values.extraHostVolumeMounts }} + - name: {{ $mount.name }} + hostPath: + path: {{ $mount.hostPath }} + {{- end }} + {{- range $_, $mount := .Values.sidecarVolumeMount }} + - name: {{ $mount.name }} + emptyDir: + medium: Memory + {{- end }} + {{- range $_, $mount := .Values.sidecarHostVolumeMounts }} + - name: {{ $mount.name }} + hostPath: + path: {{ $mount.hostPath }} + {{- end }} + {{- range $_, $mount := .Values.configmaps }} + - name: {{ $mount.name }} + configMap: + name: {{ $mount.name }} + {{- end }} + {{- range $_, $mount := .Values.secrets }} + - name: {{ $mount.name }} + secret: + secretName: {{ $mount.name }} + {{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/endpoints.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/endpoints.yaml new file mode 100644 index 00000000..45eeb8d9 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/endpoints.yaml @@ -0,0 +1,18 @@ +{{- if .Values.endpoints }} +apiVersion: v1 +kind: Endpoints +metadata: + name: {{ include "prometheus-node-exporter.fullname" . }} + namespace: {{ include "prometheus-node-exporter.namespace" . }} + labels: + {{- include "prometheus-node-exporter.labels" . | nindent 4 }} +subsets: + - addresses: + {{- range .Values.endpoints }} + - ip: {{ . }} + {{- end }} + ports: + - name: {{ .Values.service.portName }} + port: 9100 + protocol: TCP +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/psp-clusterrole.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/psp-clusterrole.yaml new file mode 100644 index 00000000..89573172 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/psp-clusterrole.yaml @@ -0,0 +1,14 @@ +{{- if and .Values.rbac.create .Values.rbac.pspEnabled (.Capabilities.APIVersions.Has "policy/v1beta1/PodSecurityPolicy") }} +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: psp-{{ include "prometheus-node-exporter.fullname" . }} + labels: + {{- include "prometheus-node-exporter.labels" . | nindent 4 }} +rules: +- apiGroups: ['extensions'] + resources: ['podsecuritypolicies'] + verbs: ['use'] + resourceNames: + - {{ include "prometheus-node-exporter.fullname" . }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/psp-clusterrolebinding.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/psp-clusterrolebinding.yaml new file mode 100644 index 00000000..33337017 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/psp-clusterrolebinding.yaml @@ -0,0 +1,16 @@ +{{- if and .Values.rbac.create .Values.rbac.pspEnabled (.Capabilities.APIVersions.Has "policy/v1beta1/PodSecurityPolicy") }} +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: psp-{{ include "prometheus-node-exporter.fullname" . }} + labels: + {{- include "prometheus-node-exporter.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp-{{ include "prometheus-node-exporter.fullname" . }} +subjects: + - kind: ServiceAccount + name: {{ include "prometheus-node-exporter.fullname" . }} + namespace: {{ include "prometheus-node-exporter.namespace" . }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/psp.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/psp.yaml new file mode 100644 index 00000000..4896c84d --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/psp.yaml @@ -0,0 +1,49 @@ +{{- if and .Values.rbac.create .Values.rbac.pspEnabled (.Capabilities.APIVersions.Has "policy/v1beta1/PodSecurityPolicy") }} +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: {{ include "prometheus-node-exporter.fullname" . }} + namespace: {{ include "prometheus-node-exporter.namespace" . }} + labels: + {{- include "prometheus-node-exporter.labels" . | nindent 4 }} + {{- with .Values.rbac.pspAnnotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + privileged: false + # Allow core volume types. + volumes: + - 'configMap' + - 'emptyDir' + - 'projected' + - 'secret' + - 'downwardAPI' + - 'persistentVolumeClaim' + - 'hostPath' + hostNetwork: true + hostIPC: false + hostPID: true + hostPorts: + - min: 0 + max: 65535 + runAsUser: + # Permits the container to run with root privileges as well. + rule: 'RunAsAny' + seLinux: + # This policy assumes the nodes are using AppArmor rather than SELinux. + rule: 'RunAsAny' + supplementalGroups: + rule: 'MustRunAs' + ranges: + # Allow adding the root group. + - min: 0 + max: 65535 + fsGroup: + rule: 'MustRunAs' + ranges: + # Allow adding the root group. + - min: 0 + max: 65535 + readOnlyRootFilesystem: false +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/service.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/service.yaml new file mode 100644 index 00000000..c0129dbd --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/service.yaml @@ -0,0 +1,23 @@ +apiVersion: v1 +kind: Service +metadata: + name: {{ include "prometheus-node-exporter.fullname" . }} + namespace: {{ include "prometheus-node-exporter.namespace" . }} + labels: + {{- include "prometheus-node-exporter.labels" $ | nindent 4 }} + {{- with .Values.service.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + type: {{ .Values.service.type }} + ports: + - port: {{ .Values.service.port }} + {{- if ( and (eq .Values.service.type "NodePort" ) (not (empty .Values.service.nodePort)) ) }} + nodePort: {{ .Values.service.nodePort }} + {{- end }} + targetPort: {{ .Values.service.targetPort }} + protocol: TCP + name: {{ .Values.service.portName }} + selector: + {{- include "prometheus-node-exporter.selectorLabels" . | nindent 4 }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/serviceaccount.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/serviceaccount.yaml new file mode 100644 index 00000000..b82630ca --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/serviceaccount.yaml @@ -0,0 +1,17 @@ +{{- if and .Values.rbac.create .Values.serviceAccount.create -}} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "prometheus-node-exporter.serviceAccountName" . }} + namespace: {{ include "prometheus-node-exporter.namespace" . }} + labels: + {{- include "prometheus-node-exporter.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- with .Values.serviceAccount.imagePullSecrets }} +imagePullSecrets: + {{- toYaml . | nindent 2 }} +{{- end }} +{{- end -}} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/servicemonitor.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/servicemonitor.yaml new file mode 100644 index 00000000..073ce57a --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/servicemonitor.yaml @@ -0,0 +1,53 @@ +{{- if .Values.prometheus.monitor.enabled }} +apiVersion: {{ .Values.prometheus.monitor.apiVersion | default "monitoring.coreos.com/v1" }} +kind: ServiceMonitor +metadata: + name: {{ include "prometheus-node-exporter.fullname" . }} + namespace: {{ include "prometheus-node-exporter.monitor-namespace" . }} + labels: + {{- include "prometheus-node-exporter.labels" . | nindent 4 }} + {{- with .Values.prometheus.monitor.additionalLabels }} + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + jobLabel: {{ default "app.kubernetes.io/name" .Values.prometheus.monitor.jobLabel }} + {{- include "servicemonitor.scrapeLimits" .Values.prometheus.monitor | nindent 2 }} + selector: + matchLabels: + {{- with .Values.prometheus.monitor.selectorOverride }} + {{- toYaml . | nindent 6 }} + {{- else }} + {{- include "prometheus-node-exporter.selectorLabels" . | nindent 6 }} + {{- end }} + endpoints: + - port: {{ .Values.service.portName }} + scheme: {{ .Values.prometheus.monitor.scheme }} + {{- with .Values.prometheus.monitor.basicAuth }} + basicAuth: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.prometheus.monitor.bearerTokenFile }} + bearerTokenFile: {{ . }} + {{- end }} + {{- with .Values.prometheus.monitor.tlsConfig }} + tlsConfig: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.prometheus.monitor.proxyUrl }} + proxyUrl: {{ . }} + {{- end }} + {{- with .Values.prometheus.monitor.interval }} + interval: {{ . }} + {{- end }} + {{- with .Values.prometheus.monitor.scrapeTimeout }} + scrapeTimeout: {{ . }} + {{- end }} + {{- with .Values.prometheus.monitor.relabelings }} + relabelings: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.prometheus.monitor.metricRelabelings }} + metricRelabelings: + {{- toYaml . | nindent 8 }} + {{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/verticalpodautoscaler.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/verticalpodautoscaler.yaml new file mode 100644 index 00000000..ae8295d9 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/templates/verticalpodautoscaler.yaml @@ -0,0 +1,34 @@ +{{- if and (.Capabilities.APIVersions.Has "autoscaling.k8s.io/v1") (.Values.verticalPodAutoscaler.enabled) }} +apiVersion: autoscaling.k8s.io/v1 +kind: VerticalPodAutoscaler +metadata: + name: {{ include "prometheus-node-exporter.fullname" . }} + namespace: {{ include "prometheus-node-exporter.namespace" . }} + labels: + {{- include "prometheus-node-exporter.labels" . | nindent 4 }} +spec: + resourcePolicy: + containerPolicies: + - containerName: {{ include "prometheus-node-exporter.name" . }} + {{- with .Values.verticalPodAutoscaler.controlledResources }} + controlledResources: {{ . }} + {{- end }} + {{- with .Values.verticalPodAutoscaler.maxAllowed }} + maxAllowed: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.verticalPodAutoscaler.minAllowed }} + minAllowed: + {{- toYaml . | nindent 8 }} + {{- end }} + targetRef: + apiVersion: apps/v1 + kind: DaemonSet + name: {{ include "prometheus-node-exporter.fullname" . }} + {{- if .Values.verticalPodAutoscaler.updatePolicy }} + updatePolicy: + {{- with .Values.verticalPodAutoscaler.updatePolicy.updateMode }} + updateMode: {{ . }} + {{- end }} + {{- end }} +{{- end }} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/values.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/values.yaml new file mode 100644 index 00000000..f491bdfa --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/charts/prometheus-node-exporter/values.yaml @@ -0,0 +1,291 @@ +# Default values for prometheus-node-exporter. +# This is a YAML-formatted file. +# Declare variables to be passed into your templates. +image: + repository: quay.io/prometheus/node-exporter + # Overrides the image tag whose default is {{ printf "v%s" .Chart.AppVersion }} + tag: "" + pullPolicy: IfNotPresent + sha: "" + +imagePullSecrets: [] +# - name: "image-pull-secret" + +service: + type: ClusterIP + port: 9100 + targetPort: 9100 + nodePort: + portName: metrics + listenOnAllInterfaces: true + annotations: + prometheus.io/scrape: "true" + +# Additional environment variables that will be passed to the daemonset +env: {} +## env: +## VARIABLE: value + +prometheus: + monitor: + enabled: false + additionalLabels: {} + namespace: "" + + jobLabel: "" + + scheme: http + basicAuth: {} + bearerTokenFile: + tlsConfig: {} + + ## proxyUrl: URL of a proxy that should be used for scraping. + ## + proxyUrl: "" + + ## Override serviceMonitor selector + ## + selectorOverride: {} + + relabelings: [] + metricRelabelings: [] + interval: "" + scrapeTimeout: 10s + ## prometheus.monitor.apiVersion ApiVersion for the serviceMonitor Resource(defaults to "monitoring.coreos.com/v1") + apiVersion: "" + + ## SampleLimit defines per-scrape limit on number of scraped samples that will be accepted. + ## + sampleLimit: 0 + + ## TargetLimit defines a limit on the number of scraped targets that will be accepted. + ## + targetLimit: 0 + + ## Per-scrape limit on number of labels that will be accepted for a sample. Only valid in Prometheus versions 2.27.0 and newer. + ## + labelLimit: 0 + + ## Per-scrape limit on length of labels name that will be accepted for a sample. Only valid in Prometheus versions 2.27.0 and newer. + ## + labelNameLengthLimit: 0 + + ## Per-scrape limit on length of labels value that will be accepted for a sample. Only valid in Prometheus versions 2.27.0 and newer. + ## + labelValueLengthLimit: 0 + +## Customize the updateStrategy if set +updateStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + +resources: {} + # We usually recommend not to specify default resources and to leave this as a conscious + # choice for the user. This also increases chances charts run on environments with little + # resources, such as Minikube. If you do want to specify resources, uncomment the following + # lines, adjust them as necessary, and remove the curly braces after 'resources:'. + # limits: + # cpu: 200m + # memory: 50Mi + # requests: + # cpu: 100m + # memory: 30Mi + +serviceAccount: + # Specifies whether a ServiceAccount should be created + create: true + # The name of the ServiceAccount to use. + # If not set and create is true, a name is generated using the fullname template + name: + annotations: {} + imagePullSecrets: [] + automountServiceAccountToken: false + +securityContext: + fsGroup: 65534 + runAsGroup: 65534 + runAsNonRoot: true + runAsUser: 65534 + +containerSecurityContext: {} + # capabilities: + # add: + # - SYS_TIME + +rbac: + ## If true, create & use RBAC resources + ## + create: true + ## If true, create & use Pod Security Policy resources + ## https://kubernetes.io/docs/concepts/policy/pod-security-policy/ + pspEnabled: true + pspAnnotations: {} + +# for deployments that have node_exporter deployed outside of the cluster, list +# their addresses here +endpoints: [] + +# Expose the service to the host network +hostNetwork: true + +# Share the host process ID namespace +hostPID: true + +# Mount the node's root file system (/) at /host/root in the container +hostRootFsMount: + enabled: true + # Defines how new mounts in existing mounts on the node or in the container + # are propagated to the container or node, respectively. Possible values are + # None, HostToContainer, and Bidirectional. If this field is omitted, then + # None is used. More information on: + # https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation + mountPropagation: HostToContainer + +## Assign a group of affinity scheduling rules +## +affinity: {} +# nodeAffinity: +# requiredDuringSchedulingIgnoredDuringExecution: +# nodeSelectorTerms: +# - matchFields: +# - key: metadata.name +# operator: In +# values: +# - target-host-name + +# Annotations to be added to node exporter pods +podAnnotations: + # Fix for very slow GKE cluster upgrades + cluster-autoscaler.kubernetes.io/safe-to-evict: "true" + +# Extra labels to be added to node exporter pods +podLabels: {} + +# Annotations to be added to node exporter daemonset +daemonsetAnnotations: {} + +## set to true to add the release label so scraping of the servicemonitor with kube-prometheus-stack works out of the box +releaseLabel: false + +# Custom DNS configuration to be added to prometheus-node-exporter pods +dnsConfig: {} +# nameservers: +# - 1.2.3.4 +# searches: +# - ns1.svc.cluster-domain.example +# - my.dns.search.suffix +# options: +# - name: ndots +# value: "2" +# - name: edns0 + +## Assign a nodeSelector if operating a hybrid cluster +## +nodeSelector: {} +# beta.kubernetes.io/arch: amd64 +# beta.kubernetes.io/os: linux + +tolerations: + - effect: NoSchedule + operator: Exists + +## Assign a PriorityClassName to pods if set +# priorityClassName: "" + +## Additional container arguments +## +extraArgs: [] +# - --collector.diskstats.ignored-devices=^(ram|loop|fd|(h|s|v)d[a-z]|nvme\\d+n\\d+p)\\d+$ +# - --collector.textfile.directory=/run/prometheus + +## Additional mounts from the host to node-exporter container +## +extraHostVolumeMounts: [] +# - name: +# hostPath: +# mountPath: +# readOnly: true|false +# mountPropagation: None|HostToContainer|Bidirectional + +## Additional configmaps to be mounted. +## +configmaps: [] +# - name: +# mountPath: +secrets: [] +# - name: +# mountPath: +## Override the deployment namespace +## +namespaceOverride: "" + +## Additional containers for export metrics to text file +## +sidecars: [] +## - name: nvidia-dcgm-exporter +## image: nvidia/dcgm-exporter:1.4.3 + +## Volume for sidecar containers +## +sidecarVolumeMount: [] +## - name: collector-textfiles +## mountPath: /run/prometheus +## readOnly: false + +## Additional mounts from the host to sidecar containers +## +sidecarHostVolumeMounts: [] +# - name: +# hostPath: +# mountPath: +# readOnly: true|false +# mountPropagation: None|HostToContainer|Bidirectional + +## Additional InitContainers to initialize the pod +## +extraInitContainers: [] + +## Liveness probe +## +livenessProbe: + failureThreshold: 3 + httpGet: + httpHeaders: [] + scheme: http + initialDelaySeconds: 0 + periodSeconds: 10 + successThreshold: 1 + timeoutSeconds: 1 + +## Readiness probe +## +readinessProbe: + failureThreshold: 3 + httpGet: + httpHeaders: [] + scheme: http + initialDelaySeconds: 0 + periodSeconds: 10 + successThreshold: 1 + timeoutSeconds: 1 + +# Enable vertical pod autoscaler support for prometheus-node-exporter +verticalPodAutoscaler: + enabled: false + # List of resources that the vertical pod autoscaler can control. Defaults to cpu and memory + controlledResources: [] + + # Define the max allowed resources for the pod + maxAllowed: {} + # cpu: 200m + # memory: 100Mi + # Define the min allowed resources for the pod + minAllowed: {} + # cpu: 200m + # memory: 100Mi + + # updatePolicy: + # Specifies whether recommended updates are applied when a Pod is started and whether recommended updates + # are applied during the life of a Pod. Possible values are "Off", "Initial", "Recreate", and "Auto". + # updateMode: Auto diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-alertmanagerconfigs.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-alertmanagerconfigs.yaml new file mode 100644 index 00000000..858e8dcd --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-alertmanagerconfigs.yaml @@ -0,0 +1,4475 @@ +# https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.61.1/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.9.2 + creationTimestamp: null + name: alertmanagerconfigs.monitoring.coreos.com +spec: + group: monitoring.coreos.com + names: + categories: + - prometheus-operator + kind: AlertmanagerConfig + listKind: AlertmanagerConfigList + plural: alertmanagerconfigs + shortNames: + - amcfg + singular: alertmanagerconfig + scope: Namespaced + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: AlertmanagerConfig defines a namespaced AlertmanagerConfig to + be aggregated across multiple namespaces configuring one Alertmanager cluster. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: AlertmanagerConfigSpec is a specification of the desired + behavior of the Alertmanager configuration. By definition, the Alertmanager + configuration only applies to alerts for which the `namespace` label + is equal to the namespace of the AlertmanagerConfig resource. + properties: + inhibitRules: + description: List of inhibition rules. The rules will only apply to + alerts matching the resource's namespace. + items: + description: InhibitRule defines an inhibition rule that allows + to mute alerts when other alerts are already firing. See https://prometheus.io/docs/alerting/latest/configuration/#inhibit_rule + properties: + equal: + description: Labels that must have an equal value in the source + and target alert for the inhibition to take effect. + items: + type: string + type: array + sourceMatch: + description: Matchers for which one or more alerts have to exist + for the inhibition to take effect. The operator enforces that + the alert matches the resource's namespace. + items: + description: Matcher defines how to match on alert's labels. + properties: + matchType: + description: Match operation available with AlertManager + >= v0.22.0 and takes precedence over Regex (deprecated) + if non-empty. + enum: + - '!=' + - = + - =~ + - '!~' + type: string + name: + description: Label to match. + minLength: 1 + type: string + regex: + description: Whether to match on equality (false) or regular-expression + (true). Deprecated as of AlertManager >= v0.22.0 where + a user should use MatchType instead. + type: boolean + value: + description: Label value to match. + type: string + required: + - name + type: object + type: array + targetMatch: + description: Matchers that have to be fulfilled in the alerts + to be muted. The operator enforces that the alert matches + the resource's namespace. + items: + description: Matcher defines how to match on alert's labels. + properties: + matchType: + description: Match operation available with AlertManager + >= v0.22.0 and takes precedence over Regex (deprecated) + if non-empty. + enum: + - '!=' + - = + - =~ + - '!~' + type: string + name: + description: Label to match. + minLength: 1 + type: string + regex: + description: Whether to match on equality (false) or regular-expression + (true). Deprecated as of AlertManager >= v0.22.0 where + a user should use MatchType instead. + type: boolean + value: + description: Label value to match. + type: string + required: + - name + type: object + type: array + type: object + type: array + muteTimeIntervals: + description: List of MuteTimeInterval specifying when the routes should + be muted. + items: + description: MuteTimeInterval specifies the periods in time when + notifications will be muted + properties: + name: + description: Name of the time interval + type: string + timeIntervals: + description: TimeIntervals is a list of TimeInterval + items: + description: TimeInterval describes intervals of time + properties: + daysOfMonth: + description: DaysOfMonth is a list of DayOfMonthRange + items: + description: DayOfMonthRange is an inclusive range of + days of the month beginning at 1 + properties: + end: + description: End of the inclusive range + maximum: 31 + minimum: -31 + type: integer + start: + description: Start of the inclusive range + maximum: 31 + minimum: -31 + type: integer + type: object + type: array + months: + description: Months is a list of MonthRange + items: + description: MonthRange is an inclusive range of months + of the year beginning in January Months can be specified + by name (e.g 'January') by numerical month (e.g '1') + or as an inclusive range (e.g 'January:March', '1:3', + '1:March') + pattern: ^((?i)january|february|march|april|may|june|july|august|september|october|november|december|[1-12])(?:((:((?i)january|february|march|april|may|june|july|august|september|october|november|december|[1-12]))$)|$) + type: string + type: array + times: + description: Times is a list of TimeRange + items: + description: TimeRange defines a start and end time + in 24hr format + properties: + endTime: + description: EndTime is the end time in 24hr format. + pattern: ^((([01][0-9])|(2[0-3])):[0-5][0-9])$|(^24:00$) + type: string + startTime: + description: StartTime is the start time in 24hr + format. + pattern: ^((([01][0-9])|(2[0-3])):[0-5][0-9])$|(^24:00$) + type: string + type: object + type: array + weekdays: + description: Weekdays is a list of WeekdayRange + items: + description: WeekdayRange is an inclusive range of days + of the week beginning on Sunday Days can be specified + by name (e.g 'Sunday') or as an inclusive range (e.g + 'Monday:Friday') + pattern: ^((?i)sun|mon|tues|wednes|thurs|fri|satur)day(?:((:(sun|mon|tues|wednes|thurs|fri|satur)day)$)|$) + type: string + type: array + years: + description: Years is a list of YearRange + items: + description: YearRange is an inclusive range of years + pattern: ^2\d{3}(?::2\d{3}|$) + type: string + type: array + type: object + type: array + type: object + type: array + receivers: + description: List of receivers. + items: + description: Receiver defines one or more notification integrations. + properties: + emailConfigs: + description: List of Email configurations. + items: + description: EmailConfig configures notifications via Email. + properties: + authIdentity: + description: The identity to use for authentication. + type: string + authPassword: + description: The secret's key that contains the password + to use for authentication. The secret needs to be in + the same namespace as the AlertmanagerConfig object + and accessible by the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + authSecret: + description: The secret's key that contains the CRAM-MD5 + secret. The secret needs to be in the same namespace + as the AlertmanagerConfig object and accessible by the + Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + authUsername: + description: The username to use for authentication. + type: string + from: + description: The sender address. + type: string + headers: + description: Further headers email header key/value pairs. + Overrides any headers previously set by the notification + implementation. + items: + description: KeyValue defines a (key, value) tuple. + properties: + key: + description: Key of the tuple. + minLength: 1 + type: string + value: + description: Value of the tuple. + type: string + required: + - key + - value + type: object + type: array + hello: + description: The hostname to identify to the SMTP server. + type: string + html: + description: The HTML body of the email notification. + type: string + requireTLS: + description: The SMTP TLS requirement. Note that Go does + not support unencrypted connections to remote SMTP endpoints. + type: boolean + sendResolved: + description: Whether or not to notify about resolved alerts. + type: boolean + smarthost: + description: The SMTP host and port through which emails + are sent. E.g. example.com:25 + type: string + text: + description: The text body of the email notification. + type: string + tlsConfig: + description: TLS configuration + properties: + ca: + description: Certificate authority used when verifying + server certificates. + properties: + configMap: + description: ConfigMap containing data to use + for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for + the targets. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + cert: + description: Client certificate to present when doing + client-authentication. + properties: + configMap: + description: ConfigMap containing data to use + for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for + the targets. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + insecureSkipVerify: + description: Disable target certificate validation. + type: boolean + keySecret: + description: Secret containing the client key file + for the targets. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + serverName: + description: Used to verify the hostname for the targets. + type: string + type: object + to: + description: The email address to send notifications to. + type: string + type: object + type: array + name: + description: Name of the receiver. Must be unique across all + items from the list. + minLength: 1 + type: string + opsgenieConfigs: + description: List of OpsGenie configurations. + items: + description: OpsGenieConfig configures notifications via OpsGenie. + See https://prometheus.io/docs/alerting/latest/configuration/#opsgenie_config + properties: + actions: + description: Comma separated list of actions that will + be available for the alert. + type: string + apiKey: + description: The secret's key that contains the OpsGenie + API key. The secret needs to be in the same namespace + as the AlertmanagerConfig object and accessible by the + Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + apiURL: + description: The URL to send OpsGenie API requests to. + type: string + description: + description: Description of the incident. + type: string + details: + description: A set of arbitrary key/value pairs that provide + further detail about the incident. + items: + description: KeyValue defines a (key, value) tuple. + properties: + key: + description: Key of the tuple. + minLength: 1 + type: string + value: + description: Value of the tuple. + type: string + required: + - key + - value + type: object + type: array + entity: + description: Optional field that can be used to specify + which domain alert is related to. + type: string + httpConfig: + description: HTTP client configuration. + properties: + authorization: + description: Authorization header configuration for + the client. This is mutually exclusive with BasicAuth + and is only available starting from Alertmanager + v0.22+. + properties: + credentials: + description: The secret's key that contains the + credentials of the request + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: + description: Set the authentication type. Defaults + to Bearer, Basic will cause an error + type: string + type: object + basicAuth: + description: BasicAuth for the client. This is mutually + exclusive with Authorization. If both are defined, + BasicAuth takes precedence. + properties: + password: + description: The secret in the service monitor + namespace that contains the password for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + username: + description: The secret in the service monitor + namespace that contains the username for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + bearerTokenSecret: + description: The secret's key that contains the bearer + token to be used by the client for authentication. + The secret needs to be in the same namespace as + the AlertmanagerConfig object and accessible by + the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + followRedirects: + description: FollowRedirects specifies whether the + client should follow HTTP 3xx redirects. + type: boolean + oauth2: + description: OAuth2 client credentials used to fetch + a token for the targets. + properties: + clientId: + description: The secret or configmap containing + the OAuth2 client id + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + clientSecret: + description: The secret containing the OAuth2 + client secret + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + endpointParams: + additionalProperties: + type: string + description: Parameters to append to the token + URL + type: object + scopes: + description: OAuth2 scopes used for the token + request + items: + type: string + type: array + tokenUrl: + description: The URL to fetch the token from + minLength: 1 + type: string + required: + - clientId + - clientSecret + - tokenUrl + type: object + proxyURL: + description: Optional proxy URL. + type: string + tlsConfig: + description: TLS configuration for the client. + properties: + ca: + description: Certificate authority used when verifying + server certificates. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + cert: + description: Client certificate to present when + doing client-authentication. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + insecureSkipVerify: + description: Disable target certificate validation. + type: boolean + keySecret: + description: Secret containing the client key + file for the targets. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + serverName: + description: Used to verify the hostname for the + targets. + type: string + type: object + type: object + message: + description: Alert text limited to 130 characters. + type: string + note: + description: Additional alert note. + type: string + priority: + description: Priority level of alert. Possible values + are P1, P2, P3, P4, and P5. + type: string + responders: + description: List of responders responsible for notifications. + items: + description: OpsGenieConfigResponder defines a responder + to an incident. One of `id`, `name` or `username` + has to be defined. + properties: + id: + description: ID of the responder. + type: string + name: + description: Name of the responder. + type: string + type: + description: Type of responder. + enum: + - team + - teams + - user + - escalation + - schedule + minLength: 1 + type: string + username: + description: Username of the responder. + type: string + required: + - type + type: object + type: array + sendResolved: + description: Whether or not to notify about resolved alerts. + type: boolean + source: + description: Backlink to the sender of the notification. + type: string + tags: + description: Comma separated list of tags attached to + the notifications. + type: string + updateAlerts: + description: Whether to update message and description + of the alert in OpsGenie if it already exists By default, + the alert is never updated in OpsGenie, the new message + only appears in activity log. + type: boolean + type: object + type: array + pagerdutyConfigs: + description: List of PagerDuty configurations. + items: + description: PagerDutyConfig configures notifications via + PagerDuty. See https://prometheus.io/docs/alerting/latest/configuration/#pagerduty_config + properties: + class: + description: The class/type of the event. + type: string + client: + description: Client identification. + type: string + clientURL: + description: Backlink to the sender of notification. + type: string + component: + description: The part or component of the affected system + that is broken. + type: string + description: + description: Description of the incident. + type: string + details: + description: Arbitrary key/value pairs that provide further + detail about the incident. + items: + description: KeyValue defines a (key, value) tuple. + properties: + key: + description: Key of the tuple. + minLength: 1 + type: string + value: + description: Value of the tuple. + type: string + required: + - key + - value + type: object + type: array + group: + description: A cluster or grouping of sources. + type: string + httpConfig: + description: HTTP client configuration. + properties: + authorization: + description: Authorization header configuration for + the client. This is mutually exclusive with BasicAuth + and is only available starting from Alertmanager + v0.22+. + properties: + credentials: + description: The secret's key that contains the + credentials of the request + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: + description: Set the authentication type. Defaults + to Bearer, Basic will cause an error + type: string + type: object + basicAuth: + description: BasicAuth for the client. This is mutually + exclusive with Authorization. If both are defined, + BasicAuth takes precedence. + properties: + password: + description: The secret in the service monitor + namespace that contains the password for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + username: + description: The secret in the service monitor + namespace that contains the username for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + bearerTokenSecret: + description: The secret's key that contains the bearer + token to be used by the client for authentication. + The secret needs to be in the same namespace as + the AlertmanagerConfig object and accessible by + the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + followRedirects: + description: FollowRedirects specifies whether the + client should follow HTTP 3xx redirects. + type: boolean + oauth2: + description: OAuth2 client credentials used to fetch + a token for the targets. + properties: + clientId: + description: The secret or configmap containing + the OAuth2 client id + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + clientSecret: + description: The secret containing the OAuth2 + client secret + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + endpointParams: + additionalProperties: + type: string + description: Parameters to append to the token + URL + type: object + scopes: + description: OAuth2 scopes used for the token + request + items: + type: string + type: array + tokenUrl: + description: The URL to fetch the token from + minLength: 1 + type: string + required: + - clientId + - clientSecret + - tokenUrl + type: object + proxyURL: + description: Optional proxy URL. + type: string + tlsConfig: + description: TLS configuration for the client. + properties: + ca: + description: Certificate authority used when verifying + server certificates. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + cert: + description: Client certificate to present when + doing client-authentication. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + insecureSkipVerify: + description: Disable target certificate validation. + type: boolean + keySecret: + description: Secret containing the client key + file for the targets. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + serverName: + description: Used to verify the hostname for the + targets. + type: string + type: object + type: object + pagerDutyImageConfigs: + description: A list of image details to attach that provide + further detail about an incident. + items: + description: PagerDutyImageConfig attaches images to + an incident + properties: + alt: + description: Alt is the optional alternative text + for the image. + type: string + href: + description: Optional URL; makes the image a clickable + link. + type: string + src: + description: Src of the image being attached to + the incident + type: string + type: object + type: array + pagerDutyLinkConfigs: + description: A list of link details to attach that provide + further detail about an incident. + items: + description: PagerDutyLinkConfig attaches text links + to an incident + properties: + alt: + description: Text that describes the purpose of + the link, and can be used as the link's text. + type: string + href: + description: Href is the URL of the link to be attached + type: string + type: object + type: array + routingKey: + description: The secret's key that contains the PagerDuty + integration key (when using Events API v2). Either this + field or `serviceKey` needs to be defined. The secret + needs to be in the same namespace as the AlertmanagerConfig + object and accessible by the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + sendResolved: + description: Whether or not to notify about resolved alerts. + type: boolean + serviceKey: + description: The secret's key that contains the PagerDuty + service key (when using integration type "Prometheus"). + Either this field or `routingKey` needs to be defined. + The secret needs to be in the same namespace as the + AlertmanagerConfig object and accessible by the Prometheus + Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + severity: + description: Severity of the incident. + type: string + url: + description: The URL to send requests to. + type: string + type: object + type: array + pushoverConfigs: + description: List of Pushover configurations. + items: + description: PushoverConfig configures notifications via Pushover. + See https://prometheus.io/docs/alerting/latest/configuration/#pushover_config + properties: + expire: + description: How long your notification will continue + to be retried for, unless the user acknowledges the + notification. + pattern: ^(([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?$ + type: string + html: + description: Whether notification message is HTML or plain + text. + type: boolean + httpConfig: + description: HTTP client configuration. + properties: + authorization: + description: Authorization header configuration for + the client. This is mutually exclusive with BasicAuth + and is only available starting from Alertmanager + v0.22+. + properties: + credentials: + description: The secret's key that contains the + credentials of the request + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: + description: Set the authentication type. Defaults + to Bearer, Basic will cause an error + type: string + type: object + basicAuth: + description: BasicAuth for the client. This is mutually + exclusive with Authorization. If both are defined, + BasicAuth takes precedence. + properties: + password: + description: The secret in the service monitor + namespace that contains the password for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + username: + description: The secret in the service monitor + namespace that contains the username for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + bearerTokenSecret: + description: The secret's key that contains the bearer + token to be used by the client for authentication. + The secret needs to be in the same namespace as + the AlertmanagerConfig object and accessible by + the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + followRedirects: + description: FollowRedirects specifies whether the + client should follow HTTP 3xx redirects. + type: boolean + oauth2: + description: OAuth2 client credentials used to fetch + a token for the targets. + properties: + clientId: + description: The secret or configmap containing + the OAuth2 client id + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + clientSecret: + description: The secret containing the OAuth2 + client secret + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + endpointParams: + additionalProperties: + type: string + description: Parameters to append to the token + URL + type: object + scopes: + description: OAuth2 scopes used for the token + request + items: + type: string + type: array + tokenUrl: + description: The URL to fetch the token from + minLength: 1 + type: string + required: + - clientId + - clientSecret + - tokenUrl + type: object + proxyURL: + description: Optional proxy URL. + type: string + tlsConfig: + description: TLS configuration for the client. + properties: + ca: + description: Certificate authority used when verifying + server certificates. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + cert: + description: Client certificate to present when + doing client-authentication. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + insecureSkipVerify: + description: Disable target certificate validation. + type: boolean + keySecret: + description: Secret containing the client key + file for the targets. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + serverName: + description: Used to verify the hostname for the + targets. + type: string + type: object + type: object + message: + description: Notification message. + type: string + priority: + description: Priority, see https://pushover.net/api#priority + type: string + retry: + description: How often the Pushover servers will send + the same notification to the user. Must be at least + 30 seconds. + pattern: ^(([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?$ + type: string + sendResolved: + description: Whether or not to notify about resolved alerts. + type: boolean + sound: + description: The name of one of the sounds supported by + device clients to override the user's default sound + choice + type: string + title: + description: Notification title. + type: string + token: + description: The secret's key that contains the registered + application's API token, see https://pushover.net/apps. + The secret needs to be in the same namespace as the + AlertmanagerConfig object and accessible by the Prometheus + Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + url: + description: A supplementary URL shown alongside the message. + type: string + urlTitle: + description: A title for supplementary URL, otherwise + just the URL is shown + type: string + userKey: + description: The secret's key that contains the recipient + user's user key. The secret needs to be in the same + namespace as the AlertmanagerConfig object and accessible + by the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + type: object + type: array + slackConfigs: + description: List of Slack configurations. + items: + description: SlackConfig configures notifications via Slack. + See https://prometheus.io/docs/alerting/latest/configuration/#slack_config + properties: + actions: + description: A list of Slack actions that are sent with + each notification. + items: + description: SlackAction configures a single Slack action + that is sent with each notification. See https://api.slack.com/docs/message-attachments#action_fields + and https://api.slack.com/docs/message-buttons for + more information. + properties: + confirm: + description: SlackConfirmationField protect users + from destructive actions or particularly distinguished + decisions by asking them to confirm their button + click one more time. See https://api.slack.com/docs/interactive-message-field-guide#confirmation_fields + for more information. + properties: + dismissText: + type: string + okText: + type: string + text: + minLength: 1 + type: string + title: + type: string + required: + - text + type: object + name: + type: string + style: + type: string + text: + minLength: 1 + type: string + type: + minLength: 1 + type: string + url: + type: string + value: + type: string + required: + - text + - type + type: object + type: array + apiURL: + description: The secret's key that contains the Slack + webhook URL. The secret needs to be in the same namespace + as the AlertmanagerConfig object and accessible by the + Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + callbackId: + type: string + channel: + description: The channel or user to send notifications + to. + type: string + color: + type: string + fallback: + type: string + fields: + description: A list of Slack fields that are sent with + each notification. + items: + description: SlackField configures a single Slack field + that is sent with each notification. Each field must + contain a title, value, and optionally, a boolean + value to indicate if the field is short enough to + be displayed next to other fields designated as short. + See https://api.slack.com/docs/message-attachments#fields + for more information. + properties: + short: + type: boolean + title: + minLength: 1 + type: string + value: + minLength: 1 + type: string + required: + - title + - value + type: object + type: array + footer: + type: string + httpConfig: + description: HTTP client configuration. + properties: + authorization: + description: Authorization header configuration for + the client. This is mutually exclusive with BasicAuth + and is only available starting from Alertmanager + v0.22+. + properties: + credentials: + description: The secret's key that contains the + credentials of the request + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: + description: Set the authentication type. Defaults + to Bearer, Basic will cause an error + type: string + type: object + basicAuth: + description: BasicAuth for the client. This is mutually + exclusive with Authorization. If both are defined, + BasicAuth takes precedence. + properties: + password: + description: The secret in the service monitor + namespace that contains the password for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + username: + description: The secret in the service monitor + namespace that contains the username for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + bearerTokenSecret: + description: The secret's key that contains the bearer + token to be used by the client for authentication. + The secret needs to be in the same namespace as + the AlertmanagerConfig object and accessible by + the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + followRedirects: + description: FollowRedirects specifies whether the + client should follow HTTP 3xx redirects. + type: boolean + oauth2: + description: OAuth2 client credentials used to fetch + a token for the targets. + properties: + clientId: + description: The secret or configmap containing + the OAuth2 client id + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + clientSecret: + description: The secret containing the OAuth2 + client secret + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + endpointParams: + additionalProperties: + type: string + description: Parameters to append to the token + URL + type: object + scopes: + description: OAuth2 scopes used for the token + request + items: + type: string + type: array + tokenUrl: + description: The URL to fetch the token from + minLength: 1 + type: string + required: + - clientId + - clientSecret + - tokenUrl + type: object + proxyURL: + description: Optional proxy URL. + type: string + tlsConfig: + description: TLS configuration for the client. + properties: + ca: + description: Certificate authority used when verifying + server certificates. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + cert: + description: Client certificate to present when + doing client-authentication. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + insecureSkipVerify: + description: Disable target certificate validation. + type: boolean + keySecret: + description: Secret containing the client key + file for the targets. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + serverName: + description: Used to verify the hostname for the + targets. + type: string + type: object + type: object + iconEmoji: + type: string + iconURL: + type: string + imageURL: + type: string + linkNames: + type: boolean + mrkdwnIn: + items: + type: string + type: array + pretext: + type: string + sendResolved: + description: Whether or not to notify about resolved alerts. + type: boolean + shortFields: + type: boolean + text: + type: string + thumbURL: + type: string + title: + type: string + titleLink: + type: string + username: + type: string + type: object + type: array + snsConfigs: + description: List of SNS configurations + items: + description: SNSConfig configures notifications via AWS SNS. + See https://prometheus.io/docs/alerting/latest/configuration/#sns_configs + properties: + apiURL: + description: The SNS API URL i.e. https://sns.us-east-2.amazonaws.com. + If not specified, the SNS API URL from the SNS SDK will + be used. + type: string + attributes: + additionalProperties: + type: string + description: SNS message attributes. + type: object + httpConfig: + description: HTTP client configuration. + properties: + authorization: + description: Authorization header configuration for + the client. This is mutually exclusive with BasicAuth + and is only available starting from Alertmanager + v0.22+. + properties: + credentials: + description: The secret's key that contains the + credentials of the request + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: + description: Set the authentication type. Defaults + to Bearer, Basic will cause an error + type: string + type: object + basicAuth: + description: BasicAuth for the client. This is mutually + exclusive with Authorization. If both are defined, + BasicAuth takes precedence. + properties: + password: + description: The secret in the service monitor + namespace that contains the password for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + username: + description: The secret in the service monitor + namespace that contains the username for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + bearerTokenSecret: + description: The secret's key that contains the bearer + token to be used by the client for authentication. + The secret needs to be in the same namespace as + the AlertmanagerConfig object and accessible by + the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + followRedirects: + description: FollowRedirects specifies whether the + client should follow HTTP 3xx redirects. + type: boolean + oauth2: + description: OAuth2 client credentials used to fetch + a token for the targets. + properties: + clientId: + description: The secret or configmap containing + the OAuth2 client id + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + clientSecret: + description: The secret containing the OAuth2 + client secret + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + endpointParams: + additionalProperties: + type: string + description: Parameters to append to the token + URL + type: object + scopes: + description: OAuth2 scopes used for the token + request + items: + type: string + type: array + tokenUrl: + description: The URL to fetch the token from + minLength: 1 + type: string + required: + - clientId + - clientSecret + - tokenUrl + type: object + proxyURL: + description: Optional proxy URL. + type: string + tlsConfig: + description: TLS configuration for the client. + properties: + ca: + description: Certificate authority used when verifying + server certificates. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + cert: + description: Client certificate to present when + doing client-authentication. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + insecureSkipVerify: + description: Disable target certificate validation. + type: boolean + keySecret: + description: Secret containing the client key + file for the targets. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + serverName: + description: Used to verify the hostname for the + targets. + type: string + type: object + type: object + message: + description: The message content of the SNS notification. + type: string + phoneNumber: + description: Phone number if message is delivered via + SMS in E.164 format. If you don't specify this value, + you must specify a value for the TopicARN or TargetARN. + type: string + sendResolved: + description: Whether or not to notify about resolved alerts. + type: boolean + sigv4: + description: Configures AWS's Signature Verification 4 + signing process to sign requests. + properties: + accessKey: + description: AccessKey is the AWS API key. If blank, + the environment variable `AWS_ACCESS_KEY_ID` is + used. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + profile: + description: Profile is the named AWS profile used + to authenticate. + type: string + region: + description: Region is the AWS region. If blank, the + region from the default credentials chain used. + type: string + roleArn: + description: RoleArn is the named AWS profile used + to authenticate. + type: string + secretKey: + description: SecretKey is the AWS API secret. If blank, + the environment variable `AWS_SECRET_ACCESS_KEY` + is used. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + subject: + description: Subject line when the message is delivered + to email endpoints. + type: string + targetARN: + description: The mobile platform endpoint ARN if message + is delivered via mobile notifications. If you don't + specify this value, you must specify a value for the + topic_arn or PhoneNumber. + type: string + topicARN: + description: SNS topic ARN, i.e. arn:aws:sns:us-east-2:698519295917:My-Topic + If you don't specify this value, you must specify a + value for the PhoneNumber or TargetARN. + type: string + type: object + type: array + telegramConfigs: + description: List of Telegram configurations. + items: + description: TelegramConfig configures notifications via Telegram. + See https://prometheus.io/docs/alerting/latest/configuration/#telegram_config + properties: + apiURL: + description: The Telegram API URL i.e. https://api.telegram.org. + If not specified, default API URL will be used. + type: string + botToken: + description: Telegram bot token The secret needs to be + in the same namespace as the AlertmanagerConfig object + and accessible by the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + chatID: + description: The Telegram chat ID. + format: int64 + type: integer + disableNotifications: + description: Disable telegram notifications + type: boolean + httpConfig: + description: HTTP client configuration. + properties: + authorization: + description: Authorization header configuration for + the client. This is mutually exclusive with BasicAuth + and is only available starting from Alertmanager + v0.22+. + properties: + credentials: + description: The secret's key that contains the + credentials of the request + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: + description: Set the authentication type. Defaults + to Bearer, Basic will cause an error + type: string + type: object + basicAuth: + description: BasicAuth for the client. This is mutually + exclusive with Authorization. If both are defined, + BasicAuth takes precedence. + properties: + password: + description: The secret in the service monitor + namespace that contains the password for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + username: + description: The secret in the service monitor + namespace that contains the username for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + bearerTokenSecret: + description: The secret's key that contains the bearer + token to be used by the client for authentication. + The secret needs to be in the same namespace as + the AlertmanagerConfig object and accessible by + the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + followRedirects: + description: FollowRedirects specifies whether the + client should follow HTTP 3xx redirects. + type: boolean + oauth2: + description: OAuth2 client credentials used to fetch + a token for the targets. + properties: + clientId: + description: The secret or configmap containing + the OAuth2 client id + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + clientSecret: + description: The secret containing the OAuth2 + client secret + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + endpointParams: + additionalProperties: + type: string + description: Parameters to append to the token + URL + type: object + scopes: + description: OAuth2 scopes used for the token + request + items: + type: string + type: array + tokenUrl: + description: The URL to fetch the token from + minLength: 1 + type: string + required: + - clientId + - clientSecret + - tokenUrl + type: object + proxyURL: + description: Optional proxy URL. + type: string + tlsConfig: + description: TLS configuration for the client. + properties: + ca: + description: Certificate authority used when verifying + server certificates. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + cert: + description: Client certificate to present when + doing client-authentication. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + insecureSkipVerify: + description: Disable target certificate validation. + type: boolean + keySecret: + description: Secret containing the client key + file for the targets. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + serverName: + description: Used to verify the hostname for the + targets. + type: string + type: object + type: object + message: + description: Message template + type: string + parseMode: + description: Parse mode for telegram message + enum: + - MarkdownV2 + - Markdown + - HTML + type: string + sendResolved: + description: Whether to notify about resolved alerts. + type: boolean + type: object + type: array + victoropsConfigs: + description: List of VictorOps configurations. + items: + description: VictorOpsConfig configures notifications via + VictorOps. See https://prometheus.io/docs/alerting/latest/configuration/#victorops_config + properties: + apiKey: + description: The secret's key that contains the API key + to use when talking to the VictorOps API. The secret + needs to be in the same namespace as the AlertmanagerConfig + object and accessible by the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + apiUrl: + description: The VictorOps API URL. + type: string + customFields: + description: Additional custom fields for notification. + items: + description: KeyValue defines a (key, value) tuple. + properties: + key: + description: Key of the tuple. + minLength: 1 + type: string + value: + description: Value of the tuple. + type: string + required: + - key + - value + type: object + type: array + entityDisplayName: + description: Contains summary of the alerted problem. + type: string + httpConfig: + description: The HTTP client's configuration. + properties: + authorization: + description: Authorization header configuration for + the client. This is mutually exclusive with BasicAuth + and is only available starting from Alertmanager + v0.22+. + properties: + credentials: + description: The secret's key that contains the + credentials of the request + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: + description: Set the authentication type. Defaults + to Bearer, Basic will cause an error + type: string + type: object + basicAuth: + description: BasicAuth for the client. This is mutually + exclusive with Authorization. If both are defined, + BasicAuth takes precedence. + properties: + password: + description: The secret in the service monitor + namespace that contains the password for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + username: + description: The secret in the service monitor + namespace that contains the username for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + bearerTokenSecret: + description: The secret's key that contains the bearer + token to be used by the client for authentication. + The secret needs to be in the same namespace as + the AlertmanagerConfig object and accessible by + the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + followRedirects: + description: FollowRedirects specifies whether the + client should follow HTTP 3xx redirects. + type: boolean + oauth2: + description: OAuth2 client credentials used to fetch + a token for the targets. + properties: + clientId: + description: The secret or configmap containing + the OAuth2 client id + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + clientSecret: + description: The secret containing the OAuth2 + client secret + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + endpointParams: + additionalProperties: + type: string + description: Parameters to append to the token + URL + type: object + scopes: + description: OAuth2 scopes used for the token + request + items: + type: string + type: array + tokenUrl: + description: The URL to fetch the token from + minLength: 1 + type: string + required: + - clientId + - clientSecret + - tokenUrl + type: object + proxyURL: + description: Optional proxy URL. + type: string + tlsConfig: + description: TLS configuration for the client. + properties: + ca: + description: Certificate authority used when verifying + server certificates. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + cert: + description: Client certificate to present when + doing client-authentication. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + insecureSkipVerify: + description: Disable target certificate validation. + type: boolean + keySecret: + description: Secret containing the client key + file for the targets. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + serverName: + description: Used to verify the hostname for the + targets. + type: string + type: object + type: object + messageType: + description: Describes the behavior of the alert (CRITICAL, + WARNING, INFO). + type: string + monitoringTool: + description: The monitoring tool the state message is + from. + type: string + routingKey: + description: A key used to map the alert to a team. + type: string + sendResolved: + description: Whether or not to notify about resolved alerts. + type: boolean + stateMessage: + description: Contains long explanation of the alerted + problem. + type: string + type: object + type: array + webhookConfigs: + description: List of webhook configurations. + items: + description: WebhookConfig configures notifications via a + generic receiver supporting the webhook payload. See https://prometheus.io/docs/alerting/latest/configuration/#webhook_config + properties: + httpConfig: + description: HTTP client configuration. + properties: + authorization: + description: Authorization header configuration for + the client. This is mutually exclusive with BasicAuth + and is only available starting from Alertmanager + v0.22+. + properties: + credentials: + description: The secret's key that contains the + credentials of the request + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: + description: Set the authentication type. Defaults + to Bearer, Basic will cause an error + type: string + type: object + basicAuth: + description: BasicAuth for the client. This is mutually + exclusive with Authorization. If both are defined, + BasicAuth takes precedence. + properties: + password: + description: The secret in the service monitor + namespace that contains the password for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + username: + description: The secret in the service monitor + namespace that contains the username for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + bearerTokenSecret: + description: The secret's key that contains the bearer + token to be used by the client for authentication. + The secret needs to be in the same namespace as + the AlertmanagerConfig object and accessible by + the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + followRedirects: + description: FollowRedirects specifies whether the + client should follow HTTP 3xx redirects. + type: boolean + oauth2: + description: OAuth2 client credentials used to fetch + a token for the targets. + properties: + clientId: + description: The secret or configmap containing + the OAuth2 client id + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + clientSecret: + description: The secret containing the OAuth2 + client secret + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + endpointParams: + additionalProperties: + type: string + description: Parameters to append to the token + URL + type: object + scopes: + description: OAuth2 scopes used for the token + request + items: + type: string + type: array + tokenUrl: + description: The URL to fetch the token from + minLength: 1 + type: string + required: + - clientId + - clientSecret + - tokenUrl + type: object + proxyURL: + description: Optional proxy URL. + type: string + tlsConfig: + description: TLS configuration for the client. + properties: + ca: + description: Certificate authority used when verifying + server certificates. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + cert: + description: Client certificate to present when + doing client-authentication. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + insecureSkipVerify: + description: Disable target certificate validation. + type: boolean + keySecret: + description: Secret containing the client key + file for the targets. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + serverName: + description: Used to verify the hostname for the + targets. + type: string + type: object + type: object + maxAlerts: + description: Maximum number of alerts to be sent per webhook + message. When 0, all alerts are included. + format: int32 + minimum: 0 + type: integer + sendResolved: + description: Whether or not to notify about resolved alerts. + type: boolean + url: + description: The URL to send HTTP POST requests to. `urlSecret` + takes precedence over `url`. One of `urlSecret` and + `url` should be defined. + type: string + urlSecret: + description: The secret's key that contains the webhook + URL to send HTTP requests to. `urlSecret` takes precedence + over `url`. One of `urlSecret` and `url` should be defined. + The secret needs to be in the same namespace as the + AlertmanagerConfig object and accessible by the Prometheus + Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + type: object + type: array + wechatConfigs: + description: List of WeChat configurations. + items: + description: WeChatConfig configures notifications via WeChat. + See https://prometheus.io/docs/alerting/latest/configuration/#wechat_config + properties: + agentID: + type: string + apiSecret: + description: The secret's key that contains the WeChat + API key. The secret needs to be in the same namespace + as the AlertmanagerConfig object and accessible by the + Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + apiURL: + description: The WeChat API URL. + type: string + corpID: + description: The corp id for authentication. + type: string + httpConfig: + description: HTTP client configuration. + properties: + authorization: + description: Authorization header configuration for + the client. This is mutually exclusive with BasicAuth + and is only available starting from Alertmanager + v0.22+. + properties: + credentials: + description: The secret's key that contains the + credentials of the request + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: + description: Set the authentication type. Defaults + to Bearer, Basic will cause an error + type: string + type: object + basicAuth: + description: BasicAuth for the client. This is mutually + exclusive with Authorization. If both are defined, + BasicAuth takes precedence. + properties: + password: + description: The secret in the service monitor + namespace that contains the password for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + username: + description: The secret in the service monitor + namespace that contains the username for authentication. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + bearerTokenSecret: + description: The secret's key that contains the bearer + token to be used by the client for authentication. + The secret needs to be in the same namespace as + the AlertmanagerConfig object and accessible by + the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + followRedirects: + description: FollowRedirects specifies whether the + client should follow HTTP 3xx redirects. + type: boolean + oauth2: + description: OAuth2 client credentials used to fetch + a token for the targets. + properties: + clientId: + description: The secret or configmap containing + the OAuth2 client id + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + clientSecret: + description: The secret containing the OAuth2 + client secret + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + endpointParams: + additionalProperties: + type: string + description: Parameters to append to the token + URL + type: object + scopes: + description: OAuth2 scopes used for the token + request + items: + type: string + type: array + tokenUrl: + description: The URL to fetch the token from + minLength: 1 + type: string + required: + - clientId + - clientSecret + - tokenUrl + type: object + proxyURL: + description: Optional proxy URL. + type: string + tlsConfig: + description: TLS configuration for the client. + properties: + ca: + description: Certificate authority used when verifying + server certificates. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + cert: + description: Client certificate to present when + doing client-authentication. + properties: + configMap: + description: ConfigMap containing data to + use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use + for the targets. + properties: + key: + description: The key of the secret to + select from. Must be a valid secret + key. + type: string + name: + description: 'Name of the referent. More + info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + insecureSkipVerify: + description: Disable target certificate validation. + type: boolean + keySecret: + description: Secret containing the client key + file for the targets. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + serverName: + description: Used to verify the hostname for the + targets. + type: string + type: object + type: object + message: + description: API request data as defined by the WeChat + API. + type: string + messageType: + type: string + sendResolved: + description: Whether or not to notify about resolved alerts. + type: boolean + toParty: + type: string + toTag: + type: string + toUser: + type: string + type: object + type: array + required: + - name + type: object + type: array + route: + description: The Alertmanager route definition for alerts matching + the resource's namespace. If present, it will be added to the generated + Alertmanager configuration as a first-level route. + properties: + continue: + description: Boolean indicating whether an alert should continue + matching subsequent sibling nodes. It will always be overridden + to true for the first-level route by the Prometheus operator. + type: boolean + groupBy: + description: List of labels to group by. Labels must not be repeated + (unique list). Special label "..." (aggregate by all possible + labels), if provided, must be the only element in the list. + items: + type: string + type: array + groupInterval: + description: 'How long to wait before sending an updated notification. + Must match the regular expression`^(([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?$` + Example: "5m"' + type: string + groupWait: + description: 'How long to wait before sending the initial notification. + Must match the regular expression`^(([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?$` + Example: "30s"' + type: string + matchers: + description: 'List of matchers that the alert''s labels should + match. For the first level route, the operator removes any existing + equality and regexp matcher on the `namespace` label and adds + a `namespace: ` matcher.' + items: + description: Matcher defines how to match on alert's labels. + properties: + matchType: + description: Match operation available with AlertManager + >= v0.22.0 and takes precedence over Regex (deprecated) + if non-empty. + enum: + - '!=' + - = + - =~ + - '!~' + type: string + name: + description: Label to match. + minLength: 1 + type: string + regex: + description: Whether to match on equality (false) or regular-expression + (true). Deprecated as of AlertManager >= v0.22.0 where + a user should use MatchType instead. + type: boolean + value: + description: Label value to match. + type: string + required: + - name + type: object + type: array + muteTimeIntervals: + description: 'Note: this comment applies to the field definition + above but appears below otherwise it gets included in the generated + manifest. CRD schema doesn''t support self-referential types + for now (see https://github.com/kubernetes/kubernetes/issues/62872). + We have to use an alternative type to circumvent the limitation. + The downside is that the Kube API can''t validate the data beyond + the fact that it is a valid JSON representation. MuteTimeIntervals + is a list of MuteTimeInterval names that will mute this route + when matched,' + items: + type: string + type: array + receiver: + description: Name of the receiver for this route. If not empty, + it should be listed in the `receivers` field. + type: string + repeatInterval: + description: 'How long to wait before repeating the last notification. + Must match the regular expression`^(([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?$` + Example: "4h"' + type: string + routes: + description: Child routes. + items: + x-kubernetes-preserve-unknown-fields: true + type: array + type: object + type: object + required: + - spec + type: object + served: true + storage: true diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-alertmanagers.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-alertmanagers.yaml new file mode 100644 index 00000000..b2bc5bc9 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-alertmanagers.yaml @@ -0,0 +1,6851 @@ +# https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.61.1/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.9.2 + creationTimestamp: null + name: alertmanagers.monitoring.coreos.com +spec: + group: monitoring.coreos.com + names: + categories: + - prometheus-operator + kind: Alertmanager + listKind: AlertmanagerList + plural: alertmanagers + shortNames: + - am + singular: alertmanager + scope: Namespaced + versions: + - additionalPrinterColumns: + - description: The version of Alertmanager + jsonPath: .spec.version + name: Version + type: string + - description: The number of desired replicas + jsonPath: .spec.replicas + name: Replicas + type: integer + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + - description: Whether the resource reconciliation is paused or not + jsonPath: .status.paused + name: Paused + priority: 1 + type: boolean + name: v1 + schema: + openAPIV3Schema: + description: Alertmanager describes an Alertmanager cluster. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: 'Specification of the desired behavior of the Alertmanager + cluster. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#spec-and-status' + properties: + additionalPeers: + description: AdditionalPeers allows injecting a set of additional + Alertmanagers to peer with to form a highly available cluster. + items: + type: string + type: array + affinity: + description: If specified, the pod's scheduling constraints. + properties: + nodeAffinity: + description: Describes node affinity scheduling rules for the + pod. + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: The scheduler will prefer to schedule pods to + nodes that satisfy the affinity expressions specified by + this field, but it may choose a node that violates one or + more of the expressions. The node that is most preferred + is the one with the greatest sum of weights, i.e. for each + node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling affinity expressions, + etc.), compute a sum by iterating through the elements of + this field and adding "weight" to the sum if the node matches + the corresponding matchExpressions; the node(s) with the + highest sum are the most preferred. + items: + description: An empty preferred scheduling term matches + all objects with implicit weight 0 (i.e. it's a no-op). + A null preferred scheduling term matches no objects (i.e. + is also a no-op). + properties: + preference: + description: A node selector term, associated with the + corresponding weight. + properties: + matchExpressions: + description: A list of node selector requirements + by node's labels. + items: + description: A node selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: Represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists, DoesNotExist. Gt, and + Lt. + type: string + values: + description: An array of string values. If + the operator is In or NotIn, the values + array must be non-empty. If the operator + is Exists or DoesNotExist, the values array + must be empty. If the operator is Gt or + Lt, the values array must have a single + element, which will be interpreted as an + integer. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchFields: + description: A list of node selector requirements + by node's fields. + items: + description: A node selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: Represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists, DoesNotExist. Gt, and + Lt. + type: string + values: + description: An array of string values. If + the operator is In or NotIn, the values + array must be non-empty. If the operator + is Exists or DoesNotExist, the values array + must be empty. If the operator is Gt or + Lt, the values array must have a single + element, which will be interpreted as an + integer. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + type: object + x-kubernetes-map-type: atomic + weight: + description: Weight associated with matching the corresponding + nodeSelectorTerm, in the range 1-100. + format: int32 + type: integer + required: + - preference + - weight + type: object + type: array + requiredDuringSchedulingIgnoredDuringExecution: + description: If the affinity requirements specified by this + field are not met at scheduling time, the pod will not be + scheduled onto the node. If the affinity requirements specified + by this field cease to be met at some point during pod execution + (e.g. due to an update), the system may or may not try to + eventually evict the pod from its node. + properties: + nodeSelectorTerms: + description: Required. A list of node selector terms. + The terms are ORed. + items: + description: A null or empty node selector term matches + no objects. The requirements of them are ANDed. The + TopologySelectorTerm type implements a subset of the + NodeSelectorTerm. + properties: + matchExpressions: + description: A list of node selector requirements + by node's labels. + items: + description: A node selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: Represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists, DoesNotExist. Gt, and + Lt. + type: string + values: + description: An array of string values. If + the operator is In or NotIn, the values + array must be non-empty. If the operator + is Exists or DoesNotExist, the values array + must be empty. If the operator is Gt or + Lt, the values array must have a single + element, which will be interpreted as an + integer. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchFields: + description: A list of node selector requirements + by node's fields. + items: + description: A node selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: Represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists, DoesNotExist. Gt, and + Lt. + type: string + values: + description: An array of string values. If + the operator is In or NotIn, the values + array must be non-empty. If the operator + is Exists or DoesNotExist, the values array + must be empty. If the operator is Gt or + Lt, the values array must have a single + element, which will be interpreted as an + integer. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + type: object + x-kubernetes-map-type: atomic + type: array + required: + - nodeSelectorTerms + type: object + x-kubernetes-map-type: atomic + type: object + podAffinity: + description: Describes pod affinity scheduling rules (e.g. co-locate + this pod in the same node, zone, etc. as some other pod(s)). + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: The scheduler will prefer to schedule pods to + nodes that satisfy the affinity expressions specified by + this field, but it may choose a node that violates one or + more of the expressions. The node that is most preferred + is the one with the greatest sum of weights, i.e. for each + node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling affinity expressions, + etc.), compute a sum by iterating through the elements of + this field and adding "weight" to the sum if the node has + pods which matches the corresponding podAffinityTerm; the + node(s) with the highest sum are the most preferred. + items: + description: The weights of all of the matched WeightedPodAffinityTerm + fields are added per-node to find the most preferred node(s) + properties: + podAffinityTerm: + description: Required. A pod affinity term, associated + with the corresponding weight. + properties: + labelSelector: + description: A label query over a set of resources, + in this case pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaceSelector: + description: A label query over the set of namespaces + that the term applies to. The term is applied + to the union of the namespaces selected by this + field and the ones listed in the namespaces field. + null selector and null or empty namespaces list + means "this pod's namespace". An empty selector + ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: namespaces specifies a static list + of namespace names that the term applies to. The + term is applied to the union of the namespaces + listed in this field and the ones selected by + namespaceSelector. null or empty namespaces list + and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) + or not co-located (anti-affinity) with the pods + matching the labelSelector in the specified namespaces, + where co-located is defined as running on a node + whose value of the label with key topologyKey + matches that of any node on which any of the selected + pods is running. Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + weight: + description: weight associated with matching the corresponding + podAffinityTerm, in the range 1-100. + format: int32 + type: integer + required: + - podAffinityTerm + - weight + type: object + type: array + requiredDuringSchedulingIgnoredDuringExecution: + description: If the affinity requirements specified by this + field are not met at scheduling time, the pod will not be + scheduled onto the node. If the affinity requirements specified + by this field cease to be met at some point during pod execution + (e.g. due to a pod label update), the system may or may + not try to eventually evict the pod from its node. When + there are multiple elements, the lists of nodes corresponding + to each podAffinityTerm are intersected, i.e. all terms + must be satisfied. + items: + description: Defines a set of pods (namely those matching + the labelSelector relative to the given namespace(s)) + that this pod should be co-located (affinity) or not co-located + (anti-affinity) with, where co-located is defined as running + on a node whose value of the label with key + matches that of any node on which a pod of the set of + pods is running + properties: + labelSelector: + description: A label query over a set of resources, + in this case pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaceSelector: + description: A label query over the set of namespaces + that the term applies to. The term is applied to the + union of the namespaces selected by this field and + the ones listed in the namespaces field. null selector + and null or empty namespaces list means "this pod's + namespace". An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: namespaces specifies a static list of namespace + names that the term applies to. The term is applied + to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. null or + empty namespaces list and null namespaceSelector means + "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) + or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where + co-located is defined as running on a node whose value + of the label with key topologyKey matches that of + any node on which any of the selected pods is running. + Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + type: array + type: object + podAntiAffinity: + description: Describes pod anti-affinity scheduling rules (e.g. + avoid putting this pod in the same node, zone, etc. as some + other pod(s)). + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: The scheduler will prefer to schedule pods to + nodes that satisfy the anti-affinity expressions specified + by this field, but it may choose a node that violates one + or more of the expressions. The node that is most preferred + is the one with the greatest sum of weights, i.e. for each + node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling anti-affinity expressions, + etc.), compute a sum by iterating through the elements of + this field and adding "weight" to the sum if the node has + pods which matches the corresponding podAffinityTerm; the + node(s) with the highest sum are the most preferred. + items: + description: The weights of all of the matched WeightedPodAffinityTerm + fields are added per-node to find the most preferred node(s) + properties: + podAffinityTerm: + description: Required. A pod affinity term, associated + with the corresponding weight. + properties: + labelSelector: + description: A label query over a set of resources, + in this case pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaceSelector: + description: A label query over the set of namespaces + that the term applies to. The term is applied + to the union of the namespaces selected by this + field and the ones listed in the namespaces field. + null selector and null or empty namespaces list + means "this pod's namespace". An empty selector + ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: namespaces specifies a static list + of namespace names that the term applies to. The + term is applied to the union of the namespaces + listed in this field and the ones selected by + namespaceSelector. null or empty namespaces list + and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) + or not co-located (anti-affinity) with the pods + matching the labelSelector in the specified namespaces, + where co-located is defined as running on a node + whose value of the label with key topologyKey + matches that of any node on which any of the selected + pods is running. Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + weight: + description: weight associated with matching the corresponding + podAffinityTerm, in the range 1-100. + format: int32 + type: integer + required: + - podAffinityTerm + - weight + type: object + type: array + requiredDuringSchedulingIgnoredDuringExecution: + description: If the anti-affinity requirements specified by + this field are not met at scheduling time, the pod will + not be scheduled onto the node. If the anti-affinity requirements + specified by this field cease to be met at some point during + pod execution (e.g. due to a pod label update), the system + may or may not try to eventually evict the pod from its + node. When there are multiple elements, the lists of nodes + corresponding to each podAffinityTerm are intersected, i.e. + all terms must be satisfied. + items: + description: Defines a set of pods (namely those matching + the labelSelector relative to the given namespace(s)) + that this pod should be co-located (affinity) or not co-located + (anti-affinity) with, where co-located is defined as running + on a node whose value of the label with key + matches that of any node on which a pod of the set of + pods is running + properties: + labelSelector: + description: A label query over a set of resources, + in this case pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaceSelector: + description: A label query over the set of namespaces + that the term applies to. The term is applied to the + union of the namespaces selected by this field and + the ones listed in the namespaces field. null selector + and null or empty namespaces list means "this pod's + namespace". An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: namespaces specifies a static list of namespace + names that the term applies to. The term is applied + to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. null or + empty namespaces list and null namespaceSelector means + "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) + or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where + co-located is defined as running on a node whose value + of the label with key topologyKey matches that of + any node on which any of the selected pods is running. + Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + type: array + type: object + type: object + alertmanagerConfigMatcherStrategy: + description: The AlertmanagerConfigMatcherStrategy defines how AlertmanagerConfig + objects match the alerts. In the future more options may be added. + properties: + type: + default: OnNamespace + description: If set to `OnNamespace`, the operator injects a label + matcher matching the namespace of the AlertmanagerConfig object + for all its routes and inhibition rules. `None` will not add + any additional matchers other than the ones specified in the + AlertmanagerConfig. Default is `OnNamespace`. + enum: + - OnNamespace + - None + type: string + type: object + alertmanagerConfigNamespaceSelector: + description: Namespaces to be selected for AlertmanagerConfig discovery. + If nil, only check own namespace. + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. + The requirements are ANDed. + items: + description: A label selector requirement is a selector that + contains values, a key, and an operator that relates the key + and values. + properties: + key: + description: key is the label key that the selector applies + to. + type: string + operator: + description: operator represents a key's relationship to + a set of values. Valid operators are In, NotIn, Exists + and DoesNotExist. + type: string + values: + description: values is an array of string values. If the + operator is In or NotIn, the values array must be non-empty. + If the operator is Exists or DoesNotExist, the values + array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single + {key,value} in the matchLabels map is equivalent to an element + of matchExpressions, whose key field is "key", the operator + is "In", and the values array contains only "value". The requirements + are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + alertmanagerConfigSelector: + description: AlertmanagerConfigs to be selected for to merge and configure + Alertmanager with. + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. + The requirements are ANDed. + items: + description: A label selector requirement is a selector that + contains values, a key, and an operator that relates the key + and values. + properties: + key: + description: key is the label key that the selector applies + to. + type: string + operator: + description: operator represents a key's relationship to + a set of values. Valid operators are In, NotIn, Exists + and DoesNotExist. + type: string + values: + description: values is an array of string values. If the + operator is In or NotIn, the values array must be non-empty. + If the operator is Exists or DoesNotExist, the values + array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single + {key,value} in the matchLabels map is equivalent to an element + of matchExpressions, whose key field is "key", the operator + is "In", and the values array contains only "value". The requirements + are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + alertmanagerConfiguration: + description: 'EXPERIMENTAL: alertmanagerConfiguration specifies the + configuration of Alertmanager. If defined, it takes precedence over + the `configSecret` field. This field may change in future releases.' + properties: + global: + description: Defines the global parameters of the Alertmanager + configuration. + properties: + httpConfig: + description: HTTP client configuration. + properties: + authorization: + description: Authorization header configuration for the + client. This is mutually exclusive with BasicAuth and + is only available starting from Alertmanager v0.22+. + properties: + credentials: + description: The secret's key that contains the credentials + of the request + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: + description: Set the authentication type. Defaults + to Bearer, Basic will cause an error + type: string + type: object + basicAuth: + description: BasicAuth for the client. This is mutually + exclusive with Authorization. If both are defined, BasicAuth + takes precedence. + properties: + password: + description: The secret in the service monitor namespace + that contains the password for authentication. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + username: + description: The secret in the service monitor namespace + that contains the username for authentication. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + bearerTokenSecret: + description: The secret's key that contains the bearer + token to be used by the client for authentication. The + secret needs to be in the same namespace as the Alertmanager + object and accessible by the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + followRedirects: + description: FollowRedirects specifies whether the client + should follow HTTP 3xx redirects. + type: boolean + oauth2: + description: OAuth2 client credentials used to fetch a + token for the targets. + properties: + clientId: + description: The secret or configmap containing the + OAuth2 client id + properties: + configMap: + description: ConfigMap containing data to use + for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for + the targets. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + clientSecret: + description: The secret containing the OAuth2 client + secret + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + endpointParams: + additionalProperties: + type: string + description: Parameters to append to the token URL + type: object + scopes: + description: OAuth2 scopes used for the token request + items: + type: string + type: array + tokenUrl: + description: The URL to fetch the token from + minLength: 1 + type: string + required: + - clientId + - clientSecret + - tokenUrl + type: object + proxyURL: + description: Optional proxy URL. + type: string + tlsConfig: + description: TLS configuration for the client. + properties: + ca: + description: Certificate authority used when verifying + server certificates. + properties: + configMap: + description: ConfigMap containing data to use + for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for + the targets. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + cert: + description: Client certificate to present when doing + client-authentication. + properties: + configMap: + description: ConfigMap containing data to use + for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap + or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for + the targets. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + insecureSkipVerify: + description: Disable target certificate validation. + type: boolean + keySecret: + description: Secret containing the client key file + for the targets. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + serverName: + description: Used to verify the hostname for the targets. + type: string + type: object + type: object + resolveTimeout: + description: ResolveTimeout is the default value used by alertmanager + if the alert does not include EndsAt, after this time passes + it can declare the alert as resolved if it has not been + updated. This has no impact on alerts from Prometheus, as + they always include EndsAt. + pattern: ^(0|(([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?)$ + type: string + type: object + name: + description: The name of the AlertmanagerConfig resource which + is used to generate the Alertmanager configuration. It must + be defined in the same namespace as the Alertmanager object. + The operator will not enforce a `namespace` label for routes + and inhibition rules. + minLength: 1 + type: string + templates: + description: Custom notification templates. + items: + description: SecretOrConfigMap allows to specify data as a Secret + or ConfigMap. Fields are mutually exclusive. + properties: + configMap: + description: ConfigMap containing data to use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap or its key + must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for the targets. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + type: array + type: object + baseImage: + description: 'Base image that is used to deploy pods, without tag. + Deprecated: use ''image'' instead' + type: string + clusterAdvertiseAddress: + description: 'ClusterAdvertiseAddress is the explicit address to advertise + in cluster. Needs to be provided for non RFC1918 [1] (public) addresses. + [1] RFC1918: https://tools.ietf.org/html/rfc1918' + type: string + clusterGossipInterval: + description: Interval between gossip attempts. + pattern: ^(0|(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?)$ + type: string + clusterPeerTimeout: + description: Timeout for cluster peering. + pattern: ^(0|(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?)$ + type: string + clusterPushpullInterval: + description: Interval between pushpull attempts. + pattern: ^(0|(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?)$ + type: string + configMaps: + description: ConfigMaps is a list of ConfigMaps in the same namespace + as the Alertmanager object, which shall be mounted into the Alertmanager + Pods. Each ConfigMap is added to the StatefulSet definition as a + volume named `configmap-`. The ConfigMaps are mounted + into `/etc/alertmanager/configmaps/` in the 'alertmanager' + container. + items: + type: string + type: array + configSecret: + description: "ConfigSecret is the name of a Kubernetes Secret in the + same namespace as the Alertmanager object, which contains the configuration + for this Alertmanager instance. If empty, it defaults to `alertmanager-`. + \n The Alertmanager configuration should be available under the + `alertmanager.yaml` key. Additional keys from the original secret + are copied to the generated secret and mounted into the `/etc/alertmanager/config` + directory in the `alertmanager` container. \n If either the secret + or the `alertmanager.yaml` key is missing, the operator provisions + a minimal Alertmanager configuration with one empty receiver (effectively + dropping alert notifications)." + type: string + containers: + description: 'Containers allows injecting additional containers. This + is meant to allow adding an authentication proxy to an Alertmanager + pod. Containers described here modify an operator generated container + if they share the same name and modifications are done via a strategic + merge patch. The current container names are: `alertmanager` and + `config-reloader`. Overriding containers is entirely outside the + scope of what the maintainers will support and by doing so, you + accept that this behaviour may break at any time without notice.' + items: + description: A single application container that you want to run + within a pod. + properties: + args: + description: 'Arguments to the entrypoint. The container image''s + CMD is used if this is not provided. Variable references $(VAR_NAME) + are expanded using the container''s environment. If a variable + cannot be resolved, the reference in the input string will + be unchanged. Double $$ are reduced to a single $, which allows + for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will + produce the string literal "$(VAR_NAME)". Escaped references + will never be expanded, regardless of whether the variable + exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell' + items: + type: string + type: array + command: + description: 'Entrypoint array. Not executed within a shell. + The container image''s ENTRYPOINT is used if this is not provided. + Variable references $(VAR_NAME) are expanded using the container''s + environment. If a variable cannot be resolved, the reference + in the input string will be unchanged. Double $$ are reduced + to a single $, which allows for escaping the $(VAR_NAME) syntax: + i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". + Escaped references will never be expanded, regardless of whether + the variable exists or not. Cannot be updated. More info: + https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell' + items: + type: string + type: array + env: + description: List of environment variables to set in the container. + Cannot be updated. + items: + description: EnvVar represents an environment variable present + in a Container. + properties: + name: + description: Name of the environment variable. Must be + a C_IDENTIFIER. + type: string + value: + description: 'Variable references $(VAR_NAME) are expanded + using the previously defined environment variables in + the container and any service environment variables. + If a variable cannot be resolved, the reference in the + input string will be unchanged. Double $$ are reduced + to a single $, which allows for escaping the $(VAR_NAME) + syntax: i.e. "$$(VAR_NAME)" will produce the string + literal "$(VAR_NAME)". Escaped references will never + be expanded, regardless of whether the variable exists + or not. Defaults to "".' + type: string + valueFrom: + description: Source for the environment variable's value. + Cannot be used if value is not empty. + properties: + configMapKeyRef: + description: Selects a key of a ConfigMap. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the ConfigMap or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + fieldRef: + description: 'Selects a field of the pod: supports + metadata.name, metadata.namespace, `metadata.labels['''']`, + `metadata.annotations['''']`, spec.nodeName, + spec.serviceAccountName, status.hostIP, status.podIP, + status.podIPs.' + properties: + apiVersion: + description: Version of the schema the FieldPath + is written in terms of, defaults to "v1". + type: string + fieldPath: + description: Path of the field to select in the + specified API version. + type: string + required: + - fieldPath + type: object + x-kubernetes-map-type: atomic + resourceFieldRef: + description: 'Selects a resource of the container: + only resources limits and requests (limits.cpu, + limits.memory, limits.ephemeral-storage, requests.cpu, + requests.memory and requests.ephemeral-storage) + are currently supported.' + properties: + containerName: + description: 'Container name: required for volumes, + optional for env vars' + type: string + divisor: + anyOf: + - type: integer + - type: string + description: Specifies the output format of the + exposed resources, defaults to "1" + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + resource: + description: 'Required: resource to select' + type: string + required: + - resource + type: object + x-kubernetes-map-type: atomic + secretKeyRef: + description: Selects a key of a secret in the pod's + namespace + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + required: + - name + type: object + type: array + envFrom: + description: List of sources to populate environment variables + in the container. The keys defined within a source must be + a C_IDENTIFIER. All invalid keys will be reported as an event + when the container is starting. When a key exists in multiple + sources, the value associated with the last source will take + precedence. Values defined by an Env with a duplicate key + will take precedence. Cannot be updated. + items: + description: EnvFromSource represents the source of a set + of ConfigMaps + properties: + configMapRef: + description: The ConfigMap to select from + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the ConfigMap must be + defined + type: boolean + type: object + x-kubernetes-map-type: atomic + prefix: + description: An optional identifier to prepend to each + key in the ConfigMap. Must be a C_IDENTIFIER. + type: string + secretRef: + description: The Secret to select from + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret must be defined + type: boolean + type: object + x-kubernetes-map-type: atomic + type: object + type: array + image: + description: 'Container image name. More info: https://kubernetes.io/docs/concepts/containers/images + This field is optional to allow higher level config management + to default or override container images in workload controllers + like Deployments and StatefulSets.' + type: string + imagePullPolicy: + description: 'Image pull policy. One of Always, Never, IfNotPresent. + Defaults to Always if :latest tag is specified, or IfNotPresent + otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images' + type: string + lifecycle: + description: Actions that the management system should take + in response to container lifecycle events. Cannot be updated. + properties: + postStart: + description: 'PostStart is called immediately after a container + is created. If the handler fails, the container is terminated + and restarted according to its restart policy. Other management + of the container blocks until the hook completes. More + info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for + the command is root ('/') in the container's + filesystem. The command is simply exec'd, it is + not run inside a shell, so traditional shell instructions + ('|', etc) won't work. To use a shell, you need + to explicitly call out to that shell. Exit status + of 0 is treated as live/healthy and non-zero is + unhealthy. + items: + type: string + type: array + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to + the pod IP. You probably want to set "Host" in + httpHeaders instead. + type: string + httpHeaders: + description: Custom headers to set in the request. + HTTP allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access + on the container. Number must be in the range + 1 to 65535. Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the + host. Defaults to HTTP. + type: string + required: + - port + type: object + tcpSocket: + description: Deprecated. TCPSocket is NOT supported + as a LifecycleHandler and kept for the backward compatibility. + There are no validation of this field and lifecycle + hooks will fail in runtime when tcp handler is specified. + properties: + host: + description: 'Optional: Host name to connect to, + defaults to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access + on the container. Number must be in the range + 1 to 65535. Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + type: object + preStop: + description: 'PreStop is called immediately before a container + is terminated due to an API request or management event + such as liveness/startup probe failure, preemption, resource + contention, etc. The handler is not called if the container + crashes or exits. The Pod''s termination grace period + countdown begins before the PreStop hook is executed. + Regardless of the outcome of the handler, the container + will eventually terminate within the Pod''s termination + grace period (unless delayed by finalizers). Other management + of the container blocks until the hook completes or until + the termination grace period is reached. More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for + the command is root ('/') in the container's + filesystem. The command is simply exec'd, it is + not run inside a shell, so traditional shell instructions + ('|', etc) won't work. To use a shell, you need + to explicitly call out to that shell. Exit status + of 0 is treated as live/healthy and non-zero is + unhealthy. + items: + type: string + type: array + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to + the pod IP. You probably want to set "Host" in + httpHeaders instead. + type: string + httpHeaders: + description: Custom headers to set in the request. + HTTP allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access + on the container. Number must be in the range + 1 to 65535. Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the + host. Defaults to HTTP. + type: string + required: + - port + type: object + tcpSocket: + description: Deprecated. TCPSocket is NOT supported + as a LifecycleHandler and kept for the backward compatibility. + There are no validation of this field and lifecycle + hooks will fail in runtime when tcp handler is specified. + properties: + host: + description: 'Optional: Host name to connect to, + defaults to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access + on the container. Number must be in the range + 1 to 65535. Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + type: object + type: object + livenessProbe: + description: 'Periodic probe of container liveness. Container + will be restarted if the probe fails. Cannot be updated. More + info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for the + command is root ('/') in the container's filesystem. + The command is simply exec'd, it is not run inside + a shell, so traditional shell instructions ('|', etc) + won't work. To use a shell, you need to explicitly + call out to that shell. Exit status of 0 is treated + as live/healthy and non-zero is unhealthy. + items: + type: string + type: array + type: object + failureThreshold: + description: Minimum consecutive failures for the probe + to be considered failed after having succeeded. Defaults + to 3. Minimum value is 1. + format: int32 + type: integer + grpc: + description: GRPC specifies an action involving a GRPC port. + This is a beta field and requires enabling GRPCContainerProbe + feature gate. + properties: + port: + description: Port number of the gRPC service. Number + must be in the range 1 to 65535. + format: int32 + type: integer + service: + description: "Service is the name of the service to + place in the gRPC HealthCheckRequest (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md). + \n If this is not specified, the default behavior + is defined by gRPC." + type: string + required: + - port + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to the + pod IP. You probably want to set "Host" in httpHeaders + instead. + type: string + httpHeaders: + description: Custom headers to set in the request. HTTP + allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the host. + Defaults to HTTP. + type: string + required: + - port + type: object + initialDelaySeconds: + description: 'Number of seconds after the container has + started before liveness probes are initiated. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + periodSeconds: + description: How often (in seconds) to perform the probe. + Default to 10 seconds. Minimum value is 1. + format: int32 + type: integer + successThreshold: + description: Minimum consecutive successes for the probe + to be considered successful after having failed. Defaults + to 1. Must be 1 for liveness and startup. Minimum value + is 1. + format: int32 + type: integer + tcpSocket: + description: TCPSocket specifies an action involving a TCP + port. + properties: + host: + description: 'Optional: Host name to connect to, defaults + to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + terminationGracePeriodSeconds: + description: Optional duration in seconds the pod needs + to terminate gracefully upon probe failure. The grace + period is the duration in seconds after the processes + running in the pod are sent a termination signal and the + time when the processes are forcibly halted with a kill + signal. Set this value longer than the expected cleanup + time for your process. If this value is nil, the pod's + terminationGracePeriodSeconds will be used. Otherwise, + this value overrides the value provided by the pod spec. + Value must be non-negative integer. The value zero indicates + stop immediately via the kill signal (no opportunity to + shut down). This is a beta field and requires enabling + ProbeTerminationGracePeriod feature gate. Minimum value + is 1. spec.terminationGracePeriodSeconds is used if unset. + format: int64 + type: integer + timeoutSeconds: + description: 'Number of seconds after which the probe times + out. Defaults to 1 second. Minimum value is 1. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + type: object + name: + description: Name of the container specified as a DNS_LABEL. + Each container in a pod must have a unique name (DNS_LABEL). + Cannot be updated. + type: string + ports: + description: List of ports to expose from the container. Not + specifying a port here DOES NOT prevent that port from being + exposed. Any port which is listening on the default "0.0.0.0" + address inside a container will be accessible from the network. + Modifying this array with strategic merge patch may corrupt + the data. For more information See https://github.com/kubernetes/kubernetes/issues/108255. + Cannot be updated. + items: + description: ContainerPort represents a network port in a + single container. + properties: + containerPort: + description: Number of port to expose on the pod's IP + address. This must be a valid port number, 0 < x < 65536. + format: int32 + type: integer + hostIP: + description: What host IP to bind the external port to. + type: string + hostPort: + description: Number of port to expose on the host. If + specified, this must be a valid port number, 0 < x < + 65536. If HostNetwork is specified, this must match + ContainerPort. Most containers do not need this. + format: int32 + type: integer + name: + description: If specified, this must be an IANA_SVC_NAME + and unique within the pod. Each named port in a pod + must have a unique name. Name for the port that can + be referred to by services. + type: string + protocol: + default: TCP + description: Protocol for port. Must be UDP, TCP, or SCTP. + Defaults to "TCP". + type: string + required: + - containerPort + type: object + type: array + x-kubernetes-list-map-keys: + - containerPort + - protocol + x-kubernetes-list-type: map + readinessProbe: + description: 'Periodic probe of container service readiness. + Container will be removed from service endpoints if the probe + fails. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for the + command is root ('/') in the container's filesystem. + The command is simply exec'd, it is not run inside + a shell, so traditional shell instructions ('|', etc) + won't work. To use a shell, you need to explicitly + call out to that shell. Exit status of 0 is treated + as live/healthy and non-zero is unhealthy. + items: + type: string + type: array + type: object + failureThreshold: + description: Minimum consecutive failures for the probe + to be considered failed after having succeeded. Defaults + to 3. Minimum value is 1. + format: int32 + type: integer + grpc: + description: GRPC specifies an action involving a GRPC port. + This is a beta field and requires enabling GRPCContainerProbe + feature gate. + properties: + port: + description: Port number of the gRPC service. Number + must be in the range 1 to 65535. + format: int32 + type: integer + service: + description: "Service is the name of the service to + place in the gRPC HealthCheckRequest (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md). + \n If this is not specified, the default behavior + is defined by gRPC." + type: string + required: + - port + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to the + pod IP. You probably want to set "Host" in httpHeaders + instead. + type: string + httpHeaders: + description: Custom headers to set in the request. HTTP + allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the host. + Defaults to HTTP. + type: string + required: + - port + type: object + initialDelaySeconds: + description: 'Number of seconds after the container has + started before liveness probes are initiated. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + periodSeconds: + description: How often (in seconds) to perform the probe. + Default to 10 seconds. Minimum value is 1. + format: int32 + type: integer + successThreshold: + description: Minimum consecutive successes for the probe + to be considered successful after having failed. Defaults + to 1. Must be 1 for liveness and startup. Minimum value + is 1. + format: int32 + type: integer + tcpSocket: + description: TCPSocket specifies an action involving a TCP + port. + properties: + host: + description: 'Optional: Host name to connect to, defaults + to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + terminationGracePeriodSeconds: + description: Optional duration in seconds the pod needs + to terminate gracefully upon probe failure. The grace + period is the duration in seconds after the processes + running in the pod are sent a termination signal and the + time when the processes are forcibly halted with a kill + signal. Set this value longer than the expected cleanup + time for your process. If this value is nil, the pod's + terminationGracePeriodSeconds will be used. Otherwise, + this value overrides the value provided by the pod spec. + Value must be non-negative integer. The value zero indicates + stop immediately via the kill signal (no opportunity to + shut down). This is a beta field and requires enabling + ProbeTerminationGracePeriod feature gate. Minimum value + is 1. spec.terminationGracePeriodSeconds is used if unset. + format: int64 + type: integer + timeoutSeconds: + description: 'Number of seconds after which the probe times + out. Defaults to 1 second. Minimum value is 1. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + type: object + resources: + description: 'Compute Resources required by this container. + Cannot be updated. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + properties: + limits: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Limits describes the maximum amount of compute + resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + requests: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Requests describes the minimum amount of compute + resources required. If Requests is omitted for a container, + it defaults to Limits if that is explicitly specified, + otherwise to an implementation-defined value. More info: + https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + type: object + securityContext: + description: 'SecurityContext defines the security options the + container should be run with. If set, the fields of SecurityContext + override the equivalent fields of PodSecurityContext. More + info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/' + properties: + allowPrivilegeEscalation: + description: 'AllowPrivilegeEscalation controls whether + a process can gain more privileges than its parent process. + This bool directly controls if the no_new_privs flag will + be set on the container process. AllowPrivilegeEscalation + is true always when the container is: 1) run as Privileged + 2) has CAP_SYS_ADMIN Note that this field cannot be set + when spec.os.name is windows.' + type: boolean + capabilities: + description: The capabilities to add/drop when running containers. + Defaults to the default set of capabilities granted by + the container runtime. Note that this field cannot be + set when spec.os.name is windows. + properties: + add: + description: Added capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + drop: + description: Removed capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + type: object + privileged: + description: Run container in privileged mode. Processes + in privileged containers are essentially equivalent to + root on the host. Defaults to false. Note that this field + cannot be set when spec.os.name is windows. + type: boolean + procMount: + description: procMount denotes the type of proc mount to + use for the containers. The default is DefaultProcMount + which uses the container runtime defaults for readonly + paths and masked paths. This requires the ProcMountType + feature flag to be enabled. Note that this field cannot + be set when spec.os.name is windows. + type: string + readOnlyRootFilesystem: + description: Whether this container has a read-only root + filesystem. Default is false. Note that this field cannot + be set when spec.os.name is windows. + type: boolean + runAsGroup: + description: The GID to run the entrypoint of the container + process. Uses runtime default if unset. May also be set + in PodSecurityContext. If set in both SecurityContext + and PodSecurityContext, the value specified in SecurityContext + takes precedence. Note that this field cannot be set when + spec.os.name is windows. + format: int64 + type: integer + runAsNonRoot: + description: Indicates that the container must run as a + non-root user. If true, the Kubelet will validate the + image at runtime to ensure that it does not run as UID + 0 (root) and fail to start the container if it does. If + unset or false, no such validation will be performed. + May also be set in PodSecurityContext. If set in both + SecurityContext and PodSecurityContext, the value specified + in SecurityContext takes precedence. + type: boolean + runAsUser: + description: The UID to run the entrypoint of the container + process. Defaults to user specified in image metadata + if unspecified. May also be set in PodSecurityContext. If + set in both SecurityContext and PodSecurityContext, the + value specified in SecurityContext takes precedence. Note + that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + seLinuxOptions: + description: The SELinux context to be applied to the container. + If unspecified, the container runtime will allocate a + random SELinux context for each container. May also be + set in PodSecurityContext. If set in both SecurityContext + and PodSecurityContext, the value specified in SecurityContext + takes precedence. Note that this field cannot be set when + spec.os.name is windows. + properties: + level: + description: Level is SELinux level label that applies + to the container. + type: string + role: + description: Role is a SELinux role label that applies + to the container. + type: string + type: + description: Type is a SELinux type label that applies + to the container. + type: string + user: + description: User is a SELinux user label that applies + to the container. + type: string + type: object + seccompProfile: + description: The seccomp options to use by this container. + If seccomp options are provided at both the pod & container + level, the container options override the pod options. + Note that this field cannot be set when spec.os.name is + windows. + properties: + localhostProfile: + description: localhostProfile indicates a profile defined + in a file on the node should be used. The profile + must be preconfigured on the node to work. Must be + a descending path, relative to the kubelet's configured + seccomp profile location. Must only be set if type + is "Localhost". + type: string + type: + description: "type indicates which kind of seccomp profile + will be applied. Valid options are: \n Localhost - + a profile defined in a file on the node should be + used. RuntimeDefault - the container runtime default + profile should be used. Unconfined - no profile should + be applied." + type: string + required: + - type + type: object + windowsOptions: + description: The Windows specific settings applied to all + containers. If unspecified, the options from the PodSecurityContext + will be used. If set in both SecurityContext and PodSecurityContext, + the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is + linux. + properties: + gmsaCredentialSpec: + description: GMSACredentialSpec is where the GMSA admission + webhook (https://github.com/kubernetes-sigs/windows-gmsa) + inlines the contents of the GMSA credential spec named + by the GMSACredentialSpecName field. + type: string + gmsaCredentialSpecName: + description: GMSACredentialSpecName is the name of the + GMSA credential spec to use. + type: string + hostProcess: + description: HostProcess determines if a container should + be run as a 'Host Process' container. This field is + alpha-level and will only be honored by components + that enable the WindowsHostProcessContainers feature + flag. Setting this field without the feature flag + will result in errors when validating the Pod. All + of a Pod's containers must have the same effective + HostProcess value (it is not allowed to have a mix + of HostProcess containers and non-HostProcess containers). In + addition, if HostProcess is true then HostNetwork + must also be set to true. + type: boolean + runAsUserName: + description: The UserName in Windows to run the entrypoint + of the container process. Defaults to the user specified + in image metadata if unspecified. May also be set + in PodSecurityContext. If set in both SecurityContext + and PodSecurityContext, the value specified in SecurityContext + takes precedence. + type: string + type: object + type: object + startupProbe: + description: 'StartupProbe indicates that the Pod has successfully + initialized. If specified, no other probes are executed until + this completes successfully. If this probe fails, the Pod + will be restarted, just as if the livenessProbe failed. This + can be used to provide different probe parameters at the beginning + of a Pod''s lifecycle, when it might take a long time to load + data or warm a cache, than during steady-state operation. + This cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for the + command is root ('/') in the container's filesystem. + The command is simply exec'd, it is not run inside + a shell, so traditional shell instructions ('|', etc) + won't work. To use a shell, you need to explicitly + call out to that shell. Exit status of 0 is treated + as live/healthy and non-zero is unhealthy. + items: + type: string + type: array + type: object + failureThreshold: + description: Minimum consecutive failures for the probe + to be considered failed after having succeeded. Defaults + to 3. Minimum value is 1. + format: int32 + type: integer + grpc: + description: GRPC specifies an action involving a GRPC port. + This is a beta field and requires enabling GRPCContainerProbe + feature gate. + properties: + port: + description: Port number of the gRPC service. Number + must be in the range 1 to 65535. + format: int32 + type: integer + service: + description: "Service is the name of the service to + place in the gRPC HealthCheckRequest (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md). + \n If this is not specified, the default behavior + is defined by gRPC." + type: string + required: + - port + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to the + pod IP. You probably want to set "Host" in httpHeaders + instead. + type: string + httpHeaders: + description: Custom headers to set in the request. HTTP + allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the host. + Defaults to HTTP. + type: string + required: + - port + type: object + initialDelaySeconds: + description: 'Number of seconds after the container has + started before liveness probes are initiated. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + periodSeconds: + description: How often (in seconds) to perform the probe. + Default to 10 seconds. Minimum value is 1. + format: int32 + type: integer + successThreshold: + description: Minimum consecutive successes for the probe + to be considered successful after having failed. Defaults + to 1. Must be 1 for liveness and startup. Minimum value + is 1. + format: int32 + type: integer + tcpSocket: + description: TCPSocket specifies an action involving a TCP + port. + properties: + host: + description: 'Optional: Host name to connect to, defaults + to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + terminationGracePeriodSeconds: + description: Optional duration in seconds the pod needs + to terminate gracefully upon probe failure. The grace + period is the duration in seconds after the processes + running in the pod are sent a termination signal and the + time when the processes are forcibly halted with a kill + signal. Set this value longer than the expected cleanup + time for your process. If this value is nil, the pod's + terminationGracePeriodSeconds will be used. Otherwise, + this value overrides the value provided by the pod spec. + Value must be non-negative integer. The value zero indicates + stop immediately via the kill signal (no opportunity to + shut down). This is a beta field and requires enabling + ProbeTerminationGracePeriod feature gate. Minimum value + is 1. spec.terminationGracePeriodSeconds is used if unset. + format: int64 + type: integer + timeoutSeconds: + description: 'Number of seconds after which the probe times + out. Defaults to 1 second. Minimum value is 1. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + type: object + stdin: + description: Whether this container should allocate a buffer + for stdin in the container runtime. If this is not set, reads + from stdin in the container will always result in EOF. Default + is false. + type: boolean + stdinOnce: + description: Whether the container runtime should close the + stdin channel after it has been opened by a single attach. + When stdin is true the stdin stream will remain open across + multiple attach sessions. If stdinOnce is set to true, stdin + is opened on container start, is empty until the first client + attaches to stdin, and then remains open and accepts data + until the client disconnects, at which time stdin is closed + and remains closed until the container is restarted. If this + flag is false, a container processes that reads from stdin + will never receive an EOF. Default is false + type: boolean + terminationMessagePath: + description: 'Optional: Path at which the file to which the + container''s termination message will be written is mounted + into the container''s filesystem. Message written is intended + to be brief final status, such as an assertion failure message. + Will be truncated by the node if greater than 4096 bytes. + The total message length across all containers will be limited + to 12kb. Defaults to /dev/termination-log. Cannot be updated.' + type: string + terminationMessagePolicy: + description: Indicate how the termination message should be + populated. File will use the contents of terminationMessagePath + to populate the container status message on both success and + failure. FallbackToLogsOnError will use the last chunk of + container log output if the termination message file is empty + and the container exited with an error. The log output is + limited to 2048 bytes or 80 lines, whichever is smaller. Defaults + to File. Cannot be updated. + type: string + tty: + description: Whether this container should allocate a TTY for + itself, also requires 'stdin' to be true. Default is false. + type: boolean + volumeDevices: + description: volumeDevices is the list of block devices to be + used by the container. + items: + description: volumeDevice describes a mapping of a raw block + device within a container. + properties: + devicePath: + description: devicePath is the path inside of the container + that the device will be mapped to. + type: string + name: + description: name must match the name of a persistentVolumeClaim + in the pod + type: string + required: + - devicePath + - name + type: object + type: array + volumeMounts: + description: Pod volumes to mount into the container's filesystem. + Cannot be updated. + items: + description: VolumeMount describes a mounting of a Volume + within a container. + properties: + mountPath: + description: Path within the container at which the volume + should be mounted. Must not contain ':'. + type: string + mountPropagation: + description: mountPropagation determines how mounts are + propagated from the host to container and the other + way around. When not set, MountPropagationNone is used. + This field is beta in 1.10. + type: string + name: + description: This must match the Name of a Volume. + type: string + readOnly: + description: Mounted read-only if true, read-write otherwise + (false or unspecified). Defaults to false. + type: boolean + subPath: + description: Path within the volume from which the container's + volume should be mounted. Defaults to "" (volume's root). + type: string + subPathExpr: + description: Expanded path within the volume from which + the container's volume should be mounted. Behaves similarly + to SubPath but environment variable references $(VAR_NAME) + are expanded using the container's environment. Defaults + to "" (volume's root). SubPathExpr and SubPath are mutually + exclusive. + type: string + required: + - mountPath + - name + type: object + type: array + workingDir: + description: Container's working directory. If not specified, + the container runtime's default will be used, which might + be configured in the container image. Cannot be updated. + type: string + required: + - name + type: object + type: array + externalUrl: + description: The external URL the Alertmanager instances will be available + under. This is necessary to generate correct URLs. This is necessary + if Alertmanager is not served from root of a DNS name. + type: string + forceEnableClusterMode: + description: ForceEnableClusterMode ensures Alertmanager does not + deactivate the cluster mode when running with a single replica. + Use case is e.g. spanning an Alertmanager cluster across Kubernetes + clusters with a single replica in each. + type: boolean + hostAliases: + description: Pods' hostAliases configuration + items: + description: HostAlias holds the mapping between IP and hostnames + that will be injected as an entry in the pod's hosts file. + properties: + hostnames: + description: Hostnames for the above IP address. + items: + type: string + type: array + ip: + description: IP address of the host file entry. + type: string + required: + - hostnames + - ip + type: object + type: array + x-kubernetes-list-map-keys: + - ip + x-kubernetes-list-type: map + image: + description: Image if specified has precedence over baseImage, tag + and sha combinations. Specifying the version is still necessary + to ensure the Prometheus Operator knows what version of Alertmanager + is being configured. + type: string + imagePullSecrets: + description: An optional list of references to secrets in the same + namespace to use for pulling prometheus and alertmanager images + from registries see http://kubernetes.io/docs/user-guide/images#specifying-imagepullsecrets-on-a-pod + items: + description: LocalObjectReference contains enough information to + let you locate the referenced object inside the same namespace. + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + type: object + x-kubernetes-map-type: atomic + type: array + initContainers: + description: 'InitContainers allows adding initContainers to the pod + definition. Those can be used to e.g. fetch secrets for injection + into the Alertmanager configuration from external sources. Any errors + during the execution of an initContainer will lead to a restart + of the Pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ + Using initContainers for any use case other then secret fetching + is entirely outside the scope of what the maintainers will support + and by doing so, you accept that this behaviour may break at any + time without notice.' + items: + description: A single application container that you want to run + within a pod. + properties: + args: + description: 'Arguments to the entrypoint. The container image''s + CMD is used if this is not provided. Variable references $(VAR_NAME) + are expanded using the container''s environment. If a variable + cannot be resolved, the reference in the input string will + be unchanged. Double $$ are reduced to a single $, which allows + for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will + produce the string literal "$(VAR_NAME)". Escaped references + will never be expanded, regardless of whether the variable + exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell' + items: + type: string + type: array + command: + description: 'Entrypoint array. Not executed within a shell. + The container image''s ENTRYPOINT is used if this is not provided. + Variable references $(VAR_NAME) are expanded using the container''s + environment. If a variable cannot be resolved, the reference + in the input string will be unchanged. Double $$ are reduced + to a single $, which allows for escaping the $(VAR_NAME) syntax: + i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". + Escaped references will never be expanded, regardless of whether + the variable exists or not. Cannot be updated. More info: + https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell' + items: + type: string + type: array + env: + description: List of environment variables to set in the container. + Cannot be updated. + items: + description: EnvVar represents an environment variable present + in a Container. + properties: + name: + description: Name of the environment variable. Must be + a C_IDENTIFIER. + type: string + value: + description: 'Variable references $(VAR_NAME) are expanded + using the previously defined environment variables in + the container and any service environment variables. + If a variable cannot be resolved, the reference in the + input string will be unchanged. Double $$ are reduced + to a single $, which allows for escaping the $(VAR_NAME) + syntax: i.e. "$$(VAR_NAME)" will produce the string + literal "$(VAR_NAME)". Escaped references will never + be expanded, regardless of whether the variable exists + or not. Defaults to "".' + type: string + valueFrom: + description: Source for the environment variable's value. + Cannot be used if value is not empty. + properties: + configMapKeyRef: + description: Selects a key of a ConfigMap. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the ConfigMap or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + fieldRef: + description: 'Selects a field of the pod: supports + metadata.name, metadata.namespace, `metadata.labels['''']`, + `metadata.annotations['''']`, spec.nodeName, + spec.serviceAccountName, status.hostIP, status.podIP, + status.podIPs.' + properties: + apiVersion: + description: Version of the schema the FieldPath + is written in terms of, defaults to "v1". + type: string + fieldPath: + description: Path of the field to select in the + specified API version. + type: string + required: + - fieldPath + type: object + x-kubernetes-map-type: atomic + resourceFieldRef: + description: 'Selects a resource of the container: + only resources limits and requests (limits.cpu, + limits.memory, limits.ephemeral-storage, requests.cpu, + requests.memory and requests.ephemeral-storage) + are currently supported.' + properties: + containerName: + description: 'Container name: required for volumes, + optional for env vars' + type: string + divisor: + anyOf: + - type: integer + - type: string + description: Specifies the output format of the + exposed resources, defaults to "1" + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + resource: + description: 'Required: resource to select' + type: string + required: + - resource + type: object + x-kubernetes-map-type: atomic + secretKeyRef: + description: Selects a key of a secret in the pod's + namespace + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + required: + - name + type: object + type: array + envFrom: + description: List of sources to populate environment variables + in the container. The keys defined within a source must be + a C_IDENTIFIER. All invalid keys will be reported as an event + when the container is starting. When a key exists in multiple + sources, the value associated with the last source will take + precedence. Values defined by an Env with a duplicate key + will take precedence. Cannot be updated. + items: + description: EnvFromSource represents the source of a set + of ConfigMaps + properties: + configMapRef: + description: The ConfigMap to select from + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the ConfigMap must be + defined + type: boolean + type: object + x-kubernetes-map-type: atomic + prefix: + description: An optional identifier to prepend to each + key in the ConfigMap. Must be a C_IDENTIFIER. + type: string + secretRef: + description: The Secret to select from + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret must be defined + type: boolean + type: object + x-kubernetes-map-type: atomic + type: object + type: array + image: + description: 'Container image name. More info: https://kubernetes.io/docs/concepts/containers/images + This field is optional to allow higher level config management + to default or override container images in workload controllers + like Deployments and StatefulSets.' + type: string + imagePullPolicy: + description: 'Image pull policy. One of Always, Never, IfNotPresent. + Defaults to Always if :latest tag is specified, or IfNotPresent + otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images' + type: string + lifecycle: + description: Actions that the management system should take + in response to container lifecycle events. Cannot be updated. + properties: + postStart: + description: 'PostStart is called immediately after a container + is created. If the handler fails, the container is terminated + and restarted according to its restart policy. Other management + of the container blocks until the hook completes. More + info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for + the command is root ('/') in the container's + filesystem. The command is simply exec'd, it is + not run inside a shell, so traditional shell instructions + ('|', etc) won't work. To use a shell, you need + to explicitly call out to that shell. Exit status + of 0 is treated as live/healthy and non-zero is + unhealthy. + items: + type: string + type: array + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to + the pod IP. You probably want to set "Host" in + httpHeaders instead. + type: string + httpHeaders: + description: Custom headers to set in the request. + HTTP allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access + on the container. Number must be in the range + 1 to 65535. Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the + host. Defaults to HTTP. + type: string + required: + - port + type: object + tcpSocket: + description: Deprecated. TCPSocket is NOT supported + as a LifecycleHandler and kept for the backward compatibility. + There are no validation of this field and lifecycle + hooks will fail in runtime when tcp handler is specified. + properties: + host: + description: 'Optional: Host name to connect to, + defaults to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access + on the container. Number must be in the range + 1 to 65535. Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + type: object + preStop: + description: 'PreStop is called immediately before a container + is terminated due to an API request or management event + such as liveness/startup probe failure, preemption, resource + contention, etc. The handler is not called if the container + crashes or exits. The Pod''s termination grace period + countdown begins before the PreStop hook is executed. + Regardless of the outcome of the handler, the container + will eventually terminate within the Pod''s termination + grace period (unless delayed by finalizers). Other management + of the container blocks until the hook completes or until + the termination grace period is reached. More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for + the command is root ('/') in the container's + filesystem. The command is simply exec'd, it is + not run inside a shell, so traditional shell instructions + ('|', etc) won't work. To use a shell, you need + to explicitly call out to that shell. Exit status + of 0 is treated as live/healthy and non-zero is + unhealthy. + items: + type: string + type: array + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to + the pod IP. You probably want to set "Host" in + httpHeaders instead. + type: string + httpHeaders: + description: Custom headers to set in the request. + HTTP allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access + on the container. Number must be in the range + 1 to 65535. Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the + host. Defaults to HTTP. + type: string + required: + - port + type: object + tcpSocket: + description: Deprecated. TCPSocket is NOT supported + as a LifecycleHandler and kept for the backward compatibility. + There are no validation of this field and lifecycle + hooks will fail in runtime when tcp handler is specified. + properties: + host: + description: 'Optional: Host name to connect to, + defaults to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access + on the container. Number must be in the range + 1 to 65535. Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + type: object + type: object + livenessProbe: + description: 'Periodic probe of container liveness. Container + will be restarted if the probe fails. Cannot be updated. More + info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for the + command is root ('/') in the container's filesystem. + The command is simply exec'd, it is not run inside + a shell, so traditional shell instructions ('|', etc) + won't work. To use a shell, you need to explicitly + call out to that shell. Exit status of 0 is treated + as live/healthy and non-zero is unhealthy. + items: + type: string + type: array + type: object + failureThreshold: + description: Minimum consecutive failures for the probe + to be considered failed after having succeeded. Defaults + to 3. Minimum value is 1. + format: int32 + type: integer + grpc: + description: GRPC specifies an action involving a GRPC port. + This is a beta field and requires enabling GRPCContainerProbe + feature gate. + properties: + port: + description: Port number of the gRPC service. Number + must be in the range 1 to 65535. + format: int32 + type: integer + service: + description: "Service is the name of the service to + place in the gRPC HealthCheckRequest (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md). + \n If this is not specified, the default behavior + is defined by gRPC." + type: string + required: + - port + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to the + pod IP. You probably want to set "Host" in httpHeaders + instead. + type: string + httpHeaders: + description: Custom headers to set in the request. HTTP + allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the host. + Defaults to HTTP. + type: string + required: + - port + type: object + initialDelaySeconds: + description: 'Number of seconds after the container has + started before liveness probes are initiated. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + periodSeconds: + description: How often (in seconds) to perform the probe. + Default to 10 seconds. Minimum value is 1. + format: int32 + type: integer + successThreshold: + description: Minimum consecutive successes for the probe + to be considered successful after having failed. Defaults + to 1. Must be 1 for liveness and startup. Minimum value + is 1. + format: int32 + type: integer + tcpSocket: + description: TCPSocket specifies an action involving a TCP + port. + properties: + host: + description: 'Optional: Host name to connect to, defaults + to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + terminationGracePeriodSeconds: + description: Optional duration in seconds the pod needs + to terminate gracefully upon probe failure. The grace + period is the duration in seconds after the processes + running in the pod are sent a termination signal and the + time when the processes are forcibly halted with a kill + signal. Set this value longer than the expected cleanup + time for your process. If this value is nil, the pod's + terminationGracePeriodSeconds will be used. Otherwise, + this value overrides the value provided by the pod spec. + Value must be non-negative integer. The value zero indicates + stop immediately via the kill signal (no opportunity to + shut down). This is a beta field and requires enabling + ProbeTerminationGracePeriod feature gate. Minimum value + is 1. spec.terminationGracePeriodSeconds is used if unset. + format: int64 + type: integer + timeoutSeconds: + description: 'Number of seconds after which the probe times + out. Defaults to 1 second. Minimum value is 1. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + type: object + name: + description: Name of the container specified as a DNS_LABEL. + Each container in a pod must have a unique name (DNS_LABEL). + Cannot be updated. + type: string + ports: + description: List of ports to expose from the container. Not + specifying a port here DOES NOT prevent that port from being + exposed. Any port which is listening on the default "0.0.0.0" + address inside a container will be accessible from the network. + Modifying this array with strategic merge patch may corrupt + the data. For more information See https://github.com/kubernetes/kubernetes/issues/108255. + Cannot be updated. + items: + description: ContainerPort represents a network port in a + single container. + properties: + containerPort: + description: Number of port to expose on the pod's IP + address. This must be a valid port number, 0 < x < 65536. + format: int32 + type: integer + hostIP: + description: What host IP to bind the external port to. + type: string + hostPort: + description: Number of port to expose on the host. If + specified, this must be a valid port number, 0 < x < + 65536. If HostNetwork is specified, this must match + ContainerPort. Most containers do not need this. + format: int32 + type: integer + name: + description: If specified, this must be an IANA_SVC_NAME + and unique within the pod. Each named port in a pod + must have a unique name. Name for the port that can + be referred to by services. + type: string + protocol: + default: TCP + description: Protocol for port. Must be UDP, TCP, or SCTP. + Defaults to "TCP". + type: string + required: + - containerPort + type: object + type: array + x-kubernetes-list-map-keys: + - containerPort + - protocol + x-kubernetes-list-type: map + readinessProbe: + description: 'Periodic probe of container service readiness. + Container will be removed from service endpoints if the probe + fails. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for the + command is root ('/') in the container's filesystem. + The command is simply exec'd, it is not run inside + a shell, so traditional shell instructions ('|', etc) + won't work. To use a shell, you need to explicitly + call out to that shell. Exit status of 0 is treated + as live/healthy and non-zero is unhealthy. + items: + type: string + type: array + type: object + failureThreshold: + description: Minimum consecutive failures for the probe + to be considered failed after having succeeded. Defaults + to 3. Minimum value is 1. + format: int32 + type: integer + grpc: + description: GRPC specifies an action involving a GRPC port. + This is a beta field and requires enabling GRPCContainerProbe + feature gate. + properties: + port: + description: Port number of the gRPC service. Number + must be in the range 1 to 65535. + format: int32 + type: integer + service: + description: "Service is the name of the service to + place in the gRPC HealthCheckRequest (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md). + \n If this is not specified, the default behavior + is defined by gRPC." + type: string + required: + - port + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to the + pod IP. You probably want to set "Host" in httpHeaders + instead. + type: string + httpHeaders: + description: Custom headers to set in the request. HTTP + allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the host. + Defaults to HTTP. + type: string + required: + - port + type: object + initialDelaySeconds: + description: 'Number of seconds after the container has + started before liveness probes are initiated. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + periodSeconds: + description: How often (in seconds) to perform the probe. + Default to 10 seconds. Minimum value is 1. + format: int32 + type: integer + successThreshold: + description: Minimum consecutive successes for the probe + to be considered successful after having failed. Defaults + to 1. Must be 1 for liveness and startup. Minimum value + is 1. + format: int32 + type: integer + tcpSocket: + description: TCPSocket specifies an action involving a TCP + port. + properties: + host: + description: 'Optional: Host name to connect to, defaults + to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + terminationGracePeriodSeconds: + description: Optional duration in seconds the pod needs + to terminate gracefully upon probe failure. The grace + period is the duration in seconds after the processes + running in the pod are sent a termination signal and the + time when the processes are forcibly halted with a kill + signal. Set this value longer than the expected cleanup + time for your process. If this value is nil, the pod's + terminationGracePeriodSeconds will be used. Otherwise, + this value overrides the value provided by the pod spec. + Value must be non-negative integer. The value zero indicates + stop immediately via the kill signal (no opportunity to + shut down). This is a beta field and requires enabling + ProbeTerminationGracePeriod feature gate. Minimum value + is 1. spec.terminationGracePeriodSeconds is used if unset. + format: int64 + type: integer + timeoutSeconds: + description: 'Number of seconds after which the probe times + out. Defaults to 1 second. Minimum value is 1. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + type: object + resources: + description: 'Compute Resources required by this container. + Cannot be updated. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + properties: + limits: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Limits describes the maximum amount of compute + resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + requests: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Requests describes the minimum amount of compute + resources required. If Requests is omitted for a container, + it defaults to Limits if that is explicitly specified, + otherwise to an implementation-defined value. More info: + https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + type: object + securityContext: + description: 'SecurityContext defines the security options the + container should be run with. If set, the fields of SecurityContext + override the equivalent fields of PodSecurityContext. More + info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/' + properties: + allowPrivilegeEscalation: + description: 'AllowPrivilegeEscalation controls whether + a process can gain more privileges than its parent process. + This bool directly controls if the no_new_privs flag will + be set on the container process. AllowPrivilegeEscalation + is true always when the container is: 1) run as Privileged + 2) has CAP_SYS_ADMIN Note that this field cannot be set + when spec.os.name is windows.' + type: boolean + capabilities: + description: The capabilities to add/drop when running containers. + Defaults to the default set of capabilities granted by + the container runtime. Note that this field cannot be + set when spec.os.name is windows. + properties: + add: + description: Added capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + drop: + description: Removed capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + type: object + privileged: + description: Run container in privileged mode. Processes + in privileged containers are essentially equivalent to + root on the host. Defaults to false. Note that this field + cannot be set when spec.os.name is windows. + type: boolean + procMount: + description: procMount denotes the type of proc mount to + use for the containers. The default is DefaultProcMount + which uses the container runtime defaults for readonly + paths and masked paths. This requires the ProcMountType + feature flag to be enabled. Note that this field cannot + be set when spec.os.name is windows. + type: string + readOnlyRootFilesystem: + description: Whether this container has a read-only root + filesystem. Default is false. Note that this field cannot + be set when spec.os.name is windows. + type: boolean + runAsGroup: + description: The GID to run the entrypoint of the container + process. Uses runtime default if unset. May also be set + in PodSecurityContext. If set in both SecurityContext + and PodSecurityContext, the value specified in SecurityContext + takes precedence. Note that this field cannot be set when + spec.os.name is windows. + format: int64 + type: integer + runAsNonRoot: + description: Indicates that the container must run as a + non-root user. If true, the Kubelet will validate the + image at runtime to ensure that it does not run as UID + 0 (root) and fail to start the container if it does. If + unset or false, no such validation will be performed. + May also be set in PodSecurityContext. If set in both + SecurityContext and PodSecurityContext, the value specified + in SecurityContext takes precedence. + type: boolean + runAsUser: + description: The UID to run the entrypoint of the container + process. Defaults to user specified in image metadata + if unspecified. May also be set in PodSecurityContext. If + set in both SecurityContext and PodSecurityContext, the + value specified in SecurityContext takes precedence. Note + that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + seLinuxOptions: + description: The SELinux context to be applied to the container. + If unspecified, the container runtime will allocate a + random SELinux context for each container. May also be + set in PodSecurityContext. If set in both SecurityContext + and PodSecurityContext, the value specified in SecurityContext + takes precedence. Note that this field cannot be set when + spec.os.name is windows. + properties: + level: + description: Level is SELinux level label that applies + to the container. + type: string + role: + description: Role is a SELinux role label that applies + to the container. + type: string + type: + description: Type is a SELinux type label that applies + to the container. + type: string + user: + description: User is a SELinux user label that applies + to the container. + type: string + type: object + seccompProfile: + description: The seccomp options to use by this container. + If seccomp options are provided at both the pod & container + level, the container options override the pod options. + Note that this field cannot be set when spec.os.name is + windows. + properties: + localhostProfile: + description: localhostProfile indicates a profile defined + in a file on the node should be used. The profile + must be preconfigured on the node to work. Must be + a descending path, relative to the kubelet's configured + seccomp profile location. Must only be set if type + is "Localhost". + type: string + type: + description: "type indicates which kind of seccomp profile + will be applied. Valid options are: \n Localhost - + a profile defined in a file on the node should be + used. RuntimeDefault - the container runtime default + profile should be used. Unconfined - no profile should + be applied." + type: string + required: + - type + type: object + windowsOptions: + description: The Windows specific settings applied to all + containers. If unspecified, the options from the PodSecurityContext + will be used. If set in both SecurityContext and PodSecurityContext, + the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is + linux. + properties: + gmsaCredentialSpec: + description: GMSACredentialSpec is where the GMSA admission + webhook (https://github.com/kubernetes-sigs/windows-gmsa) + inlines the contents of the GMSA credential spec named + by the GMSACredentialSpecName field. + type: string + gmsaCredentialSpecName: + description: GMSACredentialSpecName is the name of the + GMSA credential spec to use. + type: string + hostProcess: + description: HostProcess determines if a container should + be run as a 'Host Process' container. This field is + alpha-level and will only be honored by components + that enable the WindowsHostProcessContainers feature + flag. Setting this field without the feature flag + will result in errors when validating the Pod. All + of a Pod's containers must have the same effective + HostProcess value (it is not allowed to have a mix + of HostProcess containers and non-HostProcess containers). In + addition, if HostProcess is true then HostNetwork + must also be set to true. + type: boolean + runAsUserName: + description: The UserName in Windows to run the entrypoint + of the container process. Defaults to the user specified + in image metadata if unspecified. May also be set + in PodSecurityContext. If set in both SecurityContext + and PodSecurityContext, the value specified in SecurityContext + takes precedence. + type: string + type: object + type: object + startupProbe: + description: 'StartupProbe indicates that the Pod has successfully + initialized. If specified, no other probes are executed until + this completes successfully. If this probe fails, the Pod + will be restarted, just as if the livenessProbe failed. This + can be used to provide different probe parameters at the beginning + of a Pod''s lifecycle, when it might take a long time to load + data or warm a cache, than during steady-state operation. + This cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for the + command is root ('/') in the container's filesystem. + The command is simply exec'd, it is not run inside + a shell, so traditional shell instructions ('|', etc) + won't work. To use a shell, you need to explicitly + call out to that shell. Exit status of 0 is treated + as live/healthy and non-zero is unhealthy. + items: + type: string + type: array + type: object + failureThreshold: + description: Minimum consecutive failures for the probe + to be considered failed after having succeeded. Defaults + to 3. Minimum value is 1. + format: int32 + type: integer + grpc: + description: GRPC specifies an action involving a GRPC port. + This is a beta field and requires enabling GRPCContainerProbe + feature gate. + properties: + port: + description: Port number of the gRPC service. Number + must be in the range 1 to 65535. + format: int32 + type: integer + service: + description: "Service is the name of the service to + place in the gRPC HealthCheckRequest (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md). + \n If this is not specified, the default behavior + is defined by gRPC." + type: string + required: + - port + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to the + pod IP. You probably want to set "Host" in httpHeaders + instead. + type: string + httpHeaders: + description: Custom headers to set in the request. HTTP + allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the host. + Defaults to HTTP. + type: string + required: + - port + type: object + initialDelaySeconds: + description: 'Number of seconds after the container has + started before liveness probes are initiated. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + periodSeconds: + description: How often (in seconds) to perform the probe. + Default to 10 seconds. Minimum value is 1. + format: int32 + type: integer + successThreshold: + description: Minimum consecutive successes for the probe + to be considered successful after having failed. Defaults + to 1. Must be 1 for liveness and startup. Minimum value + is 1. + format: int32 + type: integer + tcpSocket: + description: TCPSocket specifies an action involving a TCP + port. + properties: + host: + description: 'Optional: Host name to connect to, defaults + to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + terminationGracePeriodSeconds: + description: Optional duration in seconds the pod needs + to terminate gracefully upon probe failure. The grace + period is the duration in seconds after the processes + running in the pod are sent a termination signal and the + time when the processes are forcibly halted with a kill + signal. Set this value longer than the expected cleanup + time for your process. If this value is nil, the pod's + terminationGracePeriodSeconds will be used. Otherwise, + this value overrides the value provided by the pod spec. + Value must be non-negative integer. The value zero indicates + stop immediately via the kill signal (no opportunity to + shut down). This is a beta field and requires enabling + ProbeTerminationGracePeriod feature gate. Minimum value + is 1. spec.terminationGracePeriodSeconds is used if unset. + format: int64 + type: integer + timeoutSeconds: + description: 'Number of seconds after which the probe times + out. Defaults to 1 second. Minimum value is 1. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + type: object + stdin: + description: Whether this container should allocate a buffer + for stdin in the container runtime. If this is not set, reads + from stdin in the container will always result in EOF. Default + is false. + type: boolean + stdinOnce: + description: Whether the container runtime should close the + stdin channel after it has been opened by a single attach. + When stdin is true the stdin stream will remain open across + multiple attach sessions. If stdinOnce is set to true, stdin + is opened on container start, is empty until the first client + attaches to stdin, and then remains open and accepts data + until the client disconnects, at which time stdin is closed + and remains closed until the container is restarted. If this + flag is false, a container processes that reads from stdin + will never receive an EOF. Default is false + type: boolean + terminationMessagePath: + description: 'Optional: Path at which the file to which the + container''s termination message will be written is mounted + into the container''s filesystem. Message written is intended + to be brief final status, such as an assertion failure message. + Will be truncated by the node if greater than 4096 bytes. + The total message length across all containers will be limited + to 12kb. Defaults to /dev/termination-log. Cannot be updated.' + type: string + terminationMessagePolicy: + description: Indicate how the termination message should be + populated. File will use the contents of terminationMessagePath + to populate the container status message on both success and + failure. FallbackToLogsOnError will use the last chunk of + container log output if the termination message file is empty + and the container exited with an error. The log output is + limited to 2048 bytes or 80 lines, whichever is smaller. Defaults + to File. Cannot be updated. + type: string + tty: + description: Whether this container should allocate a TTY for + itself, also requires 'stdin' to be true. Default is false. + type: boolean + volumeDevices: + description: volumeDevices is the list of block devices to be + used by the container. + items: + description: volumeDevice describes a mapping of a raw block + device within a container. + properties: + devicePath: + description: devicePath is the path inside of the container + that the device will be mapped to. + type: string + name: + description: name must match the name of a persistentVolumeClaim + in the pod + type: string + required: + - devicePath + - name + type: object + type: array + volumeMounts: + description: Pod volumes to mount into the container's filesystem. + Cannot be updated. + items: + description: VolumeMount describes a mounting of a Volume + within a container. + properties: + mountPath: + description: Path within the container at which the volume + should be mounted. Must not contain ':'. + type: string + mountPropagation: + description: mountPropagation determines how mounts are + propagated from the host to container and the other + way around. When not set, MountPropagationNone is used. + This field is beta in 1.10. + type: string + name: + description: This must match the Name of a Volume. + type: string + readOnly: + description: Mounted read-only if true, read-write otherwise + (false or unspecified). Defaults to false. + type: boolean + subPath: + description: Path within the volume from which the container's + volume should be mounted. Defaults to "" (volume's root). + type: string + subPathExpr: + description: Expanded path within the volume from which + the container's volume should be mounted. Behaves similarly + to SubPath but environment variable references $(VAR_NAME) + are expanded using the container's environment. Defaults + to "" (volume's root). SubPathExpr and SubPath are mutually + exclusive. + type: string + required: + - mountPath + - name + type: object + type: array + workingDir: + description: Container's working directory. If not specified, + the container runtime's default will be used, which might + be configured in the container image. Cannot be updated. + type: string + required: + - name + type: object + type: array + listenLocal: + description: ListenLocal makes the Alertmanager server listen on loopback, + so that it does not bind against the Pod IP. Note this is only for + the Alertmanager UI, not the gossip communication. + type: boolean + logFormat: + description: Log format for Alertmanager to be configured with. + enum: + - "" + - logfmt + - json + type: string + logLevel: + description: Log level for Alertmanager to be configured with. + enum: + - "" + - debug + - info + - warn + - error + type: string + minReadySeconds: + description: Minimum number of seconds for which a newly created pod + should be ready without any of its container crashing for it to + be considered available. Defaults to 0 (pod will be considered available + as soon as it is ready) This is an alpha field and requires enabling + StatefulSetMinReadySeconds feature gate. + format: int32 + type: integer + nodeSelector: + additionalProperties: + type: string + description: Define which Nodes the Pods are scheduled on. + type: object + paused: + description: If set to true all actions on the underlying managed + objects are not goint to be performed, except for delete actions. + type: boolean + podMetadata: + description: PodMetadata configures Labels and Annotations which are + propagated to the alertmanager pods. + properties: + annotations: + additionalProperties: + type: string + description: 'Annotations is an unstructured key value map stored + with a resource that may be set by external tools to store and + retrieve arbitrary metadata. They are not queryable and should + be preserved when modifying objects. More info: http://kubernetes.io/docs/user-guide/annotations' + type: object + labels: + additionalProperties: + type: string + description: 'Map of string keys and values that can be used to + organize and categorize (scope and select) objects. May match + selectors of replication controllers and services. More info: + http://kubernetes.io/docs/user-guide/labels' + type: object + name: + description: 'Name must be unique within a namespace. Is required + when creating resources, although some resources may allow a + client to request the generation of an appropriate name automatically. + Name is primarily intended for creation idempotence and configuration + definition. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/identifiers#names' + type: string + type: object + portName: + description: Port name used for the pods and governing service. This + defaults to web + type: string + priorityClassName: + description: Priority class assigned to the Pods + type: string + replicas: + description: Size is the expected size of the alertmanager cluster. + The controller will eventually make the size of the running cluster + equal to the expected size. + format: int32 + type: integer + resources: + description: Define resources requests and limits for single Pods. + properties: + limits: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Limits describes the maximum amount of compute resources + allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + requests: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Requests describes the minimum amount of compute + resources required. If Requests is omitted for a container, + it defaults to Limits if that is explicitly specified, otherwise + to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + type: object + retention: + default: 120h + description: Time duration Alertmanager shall retain data for. Default + is '120h', and must match the regular expression `[0-9]+(ms|s|m|h)` + (milliseconds seconds minutes hours). + pattern: ^(0|(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?)$ + type: string + routePrefix: + description: The route prefix Alertmanager registers HTTP handlers + for. This is useful, if using ExternalURL and a proxy is rewriting + HTTP routes of a request, and the actual ExternalURL is still true, + but the server serves requests under a different route prefix. For + example for use with `kubectl proxy`. + type: string + secrets: + description: Secrets is a list of Secrets in the same namespace as + the Alertmanager object, which shall be mounted into the Alertmanager + Pods. Each Secret is added to the StatefulSet definition as a volume + named `secret-`. The Secrets are mounted into `/etc/alertmanager/secrets/` + in the 'alertmanager' container. + items: + type: string + type: array + securityContext: + description: SecurityContext holds pod-level security attributes and + common container settings. This defaults to the default PodSecurityContext. + properties: + fsGroup: + description: "A special supplemental group that applies to all + containers in a pod. Some volume types allow the Kubelet to + change the ownership of that volume to be owned by the pod: + \n 1. The owning GID will be the FSGroup 2. The setgid bit is + set (new files created in the volume will be owned by FSGroup) + 3. The permission bits are OR'd with rw-rw---- \n If unset, + the Kubelet will not modify the ownership and permissions of + any volume. Note that this field cannot be set when spec.os.name + is windows." + format: int64 + type: integer + fsGroupChangePolicy: + description: 'fsGroupChangePolicy defines behavior of changing + ownership and permission of the volume before being exposed + inside Pod. This field will only apply to volume types which + support fsGroup based ownership(and permissions). It will have + no effect on ephemeral volume types such as: secret, configmaps + and emptydir. Valid values are "OnRootMismatch" and "Always". + If not specified, "Always" is used. Note that this field cannot + be set when spec.os.name is windows.' + type: string + runAsGroup: + description: The GID to run the entrypoint of the container process. + Uses runtime default if unset. May also be set in SecurityContext. If + set in both SecurityContext and PodSecurityContext, the value + specified in SecurityContext takes precedence for that container. + Note that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + runAsNonRoot: + description: Indicates that the container must run as a non-root + user. If true, the Kubelet will validate the image at runtime + to ensure that it does not run as UID 0 (root) and fail to start + the container if it does. If unset or false, no such validation + will be performed. May also be set in SecurityContext. If set + in both SecurityContext and PodSecurityContext, the value specified + in SecurityContext takes precedence. + type: boolean + runAsUser: + description: The UID to run the entrypoint of the container process. + Defaults to user specified in image metadata if unspecified. + May also be set in SecurityContext. If set in both SecurityContext + and PodSecurityContext, the value specified in SecurityContext + takes precedence for that container. Note that this field cannot + be set when spec.os.name is windows. + format: int64 + type: integer + seLinuxOptions: + description: The SELinux context to be applied to all containers. + If unspecified, the container runtime will allocate a random + SELinux context for each container. May also be set in SecurityContext. If + set in both SecurityContext and PodSecurityContext, the value + specified in SecurityContext takes precedence for that container. + Note that this field cannot be set when spec.os.name is windows. + properties: + level: + description: Level is SELinux level label that applies to + the container. + type: string + role: + description: Role is a SELinux role label that applies to + the container. + type: string + type: + description: Type is a SELinux type label that applies to + the container. + type: string + user: + description: User is a SELinux user label that applies to + the container. + type: string + type: object + seccompProfile: + description: The seccomp options to use by the containers in this + pod. Note that this field cannot be set when spec.os.name is + windows. + properties: + localhostProfile: + description: localhostProfile indicates a profile defined + in a file on the node should be used. The profile must be + preconfigured on the node to work. Must be a descending + path, relative to the kubelet's configured seccomp profile + location. Must only be set if type is "Localhost". + type: string + type: + description: "type indicates which kind of seccomp profile + will be applied. Valid options are: \n Localhost - a profile + defined in a file on the node should be used. RuntimeDefault + - the container runtime default profile should be used. + Unconfined - no profile should be applied." + type: string + required: + - type + type: object + supplementalGroups: + description: A list of groups applied to the first process run + in each container, in addition to the container's primary GID. If + unspecified, no groups will be added to any container. Note + that this field cannot be set when spec.os.name is windows. + items: + format: int64 + type: integer + type: array + sysctls: + description: Sysctls hold a list of namespaced sysctls used for + the pod. Pods with unsupported sysctls (by the container runtime) + might fail to launch. Note that this field cannot be set when + spec.os.name is windows. + items: + description: Sysctl defines a kernel parameter to be set + properties: + name: + description: Name of a property to set + type: string + value: + description: Value of a property to set + type: string + required: + - name + - value + type: object + type: array + windowsOptions: + description: The Windows specific settings applied to all containers. + If unspecified, the options within a container's SecurityContext + will be used. If set in both SecurityContext and PodSecurityContext, + the value specified in SecurityContext takes precedence. Note + that this field cannot be set when spec.os.name is linux. + properties: + gmsaCredentialSpec: + description: GMSACredentialSpec is where the GMSA admission + webhook (https://github.com/kubernetes-sigs/windows-gmsa) + inlines the contents of the GMSA credential spec named by + the GMSACredentialSpecName field. + type: string + gmsaCredentialSpecName: + description: GMSACredentialSpecName is the name of the GMSA + credential spec to use. + type: string + hostProcess: + description: HostProcess determines if a container should + be run as a 'Host Process' container. This field is alpha-level + and will only be honored by components that enable the WindowsHostProcessContainers + feature flag. Setting this field without the feature flag + will result in errors when validating the Pod. All of a + Pod's containers must have the same effective HostProcess + value (it is not allowed to have a mix of HostProcess containers + and non-HostProcess containers). In addition, if HostProcess + is true then HostNetwork must also be set to true. + type: boolean + runAsUserName: + description: The UserName in Windows to run the entrypoint + of the container process. Defaults to the user specified + in image metadata if unspecified. May also be set in PodSecurityContext. + If set in both SecurityContext and PodSecurityContext, the + value specified in SecurityContext takes precedence. + type: string + type: object + type: object + serviceAccountName: + description: ServiceAccountName is the name of the ServiceAccount + to use to run the Prometheus Pods. + type: string + sha: + description: 'SHA of Alertmanager container image to be deployed. + Defaults to the value of `version`. Similar to a tag, but the SHA + explicitly deploys an immutable container image. Version and Tag + are ignored if SHA is set. Deprecated: use ''image'' instead. The + image digest can be specified as part of the image URL.' + type: string + storage: + description: Storage is the definition of how storage will be used + by the Alertmanager instances. + properties: + disableMountSubPath: + description: 'Deprecated: subPath usage will be disabled by default + in a future release, this option will become unnecessary. DisableMountSubPath + allows to remove any subPath usage in volume mounts.' + type: boolean + emptyDir: + description: 'EmptyDirVolumeSource to be used by the Prometheus + StatefulSets. If specified, used in place of any volumeClaimTemplate. + More info: https://kubernetes.io/docs/concepts/storage/volumes/#emptydir' + properties: + medium: + description: 'medium represents what type of storage medium + should back this directory. The default is "" which means + to use the node''s default medium. Must be an empty string + (default) or Memory. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir' + type: string + sizeLimit: + anyOf: + - type: integer + - type: string + description: 'sizeLimit is the total amount of local storage + required for this EmptyDir volume. The size limit is also + applicable for memory medium. The maximum usage on memory + medium EmptyDir would be the minimum value between the SizeLimit + specified here and the sum of memory limits of all containers + in a pod. The default is nil which means that the limit + is undefined. More info: http://kubernetes.io/docs/user-guide/volumes#emptydir' + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: object + ephemeral: + description: 'EphemeralVolumeSource to be used by the Prometheus + StatefulSets. This is a beta field in k8s 1.21, for lower versions, + starting with k8s 1.19, it requires enabling the GenericEphemeralVolume + feature gate. More info: https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/#generic-ephemeral-volumes' + properties: + volumeClaimTemplate: + description: "Will be used to create a stand-alone PVC to + provision the volume. The pod in which this EphemeralVolumeSource + is embedded will be the owner of the PVC, i.e. the PVC will + be deleted together with the pod. The name of the PVC will + be `-` where `` is the + name from the `PodSpec.Volumes` array entry. Pod validation + will reject the pod if the concatenated name is not valid + for a PVC (for example, too long). \n An existing PVC with + that name that is not owned by the pod will *not* be used + for the pod to avoid using an unrelated volume by mistake. + Starting the pod is then blocked until the unrelated PVC + is removed. If such a pre-created PVC is meant to be used + by the pod, the PVC has to updated with an owner reference + to the pod once the pod exists. Normally this should not + be necessary, but it may be useful when manually reconstructing + a broken cluster. \n This field is read-only and no changes + will be made by Kubernetes to the PVC after it has been + created. \n Required, must not be nil." + properties: + metadata: + description: May contain labels and annotations that will + be copied into the PVC when creating it. No other fields + are allowed and will be rejected during validation. + type: object + spec: + description: The specification for the PersistentVolumeClaim. + The entire content is copied unchanged into the PVC + that gets created from this template. The same fields + as in a PersistentVolumeClaim are also valid here. + properties: + accessModes: + description: 'accessModes contains the desired access + modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1' + items: + type: string + type: array + dataSource: + description: 'dataSource field can be used to specify + either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) + * An existing PVC (PersistentVolumeClaim) If the + provisioner or an external controller can support + the specified data source, it will create a new + volume based on the contents of the specified data + source. If the AnyVolumeDataSource feature gate + is enabled, this field will always have the same + contents as the DataSourceRef field.' + properties: + apiGroup: + description: APIGroup is the group for the resource + being referenced. If APIGroup is not specified, + the specified Kind must be in the core API group. + For any other third-party types, APIGroup is + required. + type: string + kind: + description: Kind is the type of resource being + referenced + type: string + name: + description: Name is the name of resource being + referenced + type: string + required: + - kind + - name + type: object + x-kubernetes-map-type: atomic + dataSourceRef: + description: 'dataSourceRef specifies the object from + which to populate the volume with data, if a non-empty + volume is desired. This may be any local object + from a non-empty API group (non core object) or + a PersistentVolumeClaim object. When this field + is specified, volume binding will only succeed if + the type of the specified object matches some installed + volume populator or dynamic provisioner. This field + will replace the functionality of the DataSource + field and as such if both fields are non-empty, + they must have the same value. For backwards compatibility, + both fields (DataSource and DataSourceRef) will + be set to the same value automatically if one of + them is empty and the other is non-empty. There + are two important differences between DataSource + and DataSourceRef: * While DataSource only allows + two specific types of objects, DataSourceRef allows + any non-core object, as well as PersistentVolumeClaim + objects. * While DataSource ignores disallowed values + (dropping them), DataSourceRef preserves all values, + and generates an error if a disallowed value is + specified. (Beta) Using this field requires the + AnyVolumeDataSource feature gate to be enabled.' + properties: + apiGroup: + description: APIGroup is the group for the resource + being referenced. If APIGroup is not specified, + the specified Kind must be in the core API group. + For any other third-party types, APIGroup is + required. + type: string + kind: + description: Kind is the type of resource being + referenced + type: string + name: + description: Name is the name of resource being + referenced + type: string + required: + - kind + - name + type: object + x-kubernetes-map-type: atomic + resources: + description: 'resources represents the minimum resources + the volume should have. If RecoverVolumeExpansionFailure + feature is enabled users are allowed to specify + resource requirements that are lower than previous + value but must still be higher than capacity recorded + in the status field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources' + properties: + limits: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Limits describes the maximum amount + of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + requests: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Requests describes the minimum amount + of compute resources required. If Requests is + omitted for a container, it defaults to Limits + if that is explicitly specified, otherwise to + an implementation-defined value. More info: + https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + type: object + selector: + description: selector is a label query over volumes + to consider for binding. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement is + a selector that contains values, a key, and + an operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If + the operator is Exists or DoesNotExist, + the values array must be empty. This array + is replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + storageClassName: + description: 'storageClassName is the name of the + StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1' + type: string + volumeMode: + description: volumeMode defines what type of volume + is required by the claim. Value of Filesystem is + implied when not included in claim spec. + type: string + volumeName: + description: volumeName is the binding reference to + the PersistentVolume backing this claim. + type: string + type: object + required: + - spec + type: object + type: object + volumeClaimTemplate: + description: A PVC spec to be used by the Prometheus StatefulSets. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this + representation of an object. Servers should convert recognized + schemas to the latest internal value, and may reject unrecognized + values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST + resource this object represents. Servers may infer this + from the endpoint the client submits requests to. Cannot + be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + description: EmbeddedMetadata contains metadata relevant to + an EmbeddedResource. + properties: + annotations: + additionalProperties: + type: string + description: 'Annotations is an unstructured key value + map stored with a resource that may be set by external + tools to store and retrieve arbitrary metadata. They + are not queryable and should be preserved when modifying + objects. More info: http://kubernetes.io/docs/user-guide/annotations' + type: object + labels: + additionalProperties: + type: string + description: 'Map of string keys and values that can be + used to organize and categorize (scope and select) objects. + May match selectors of replication controllers and services. + More info: http://kubernetes.io/docs/user-guide/labels' + type: object + name: + description: 'Name must be unique within a namespace. + Is required when creating resources, although some resources + may allow a client to request the generation of an appropriate + name automatically. Name is primarily intended for creation + idempotence and configuration definition. Cannot be + updated. More info: http://kubernetes.io/docs/user-guide/identifiers#names' + type: string + type: object + spec: + description: 'Spec defines the desired characteristics of + a volume requested by a pod author. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims' + properties: + accessModes: + description: 'accessModes contains the desired access + modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1' + items: + type: string + type: array + dataSource: + description: 'dataSource field can be used to specify + either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) + * An existing PVC (PersistentVolumeClaim) If the provisioner + or an external controller can support the specified + data source, it will create a new volume based on the + contents of the specified data source. If the AnyVolumeDataSource + feature gate is enabled, this field will always have + the same contents as the DataSourceRef field.' + properties: + apiGroup: + description: APIGroup is the group for the resource + being referenced. If APIGroup is not specified, + the specified Kind must be in the core API group. + For any other third-party types, APIGroup is required. + type: string + kind: + description: Kind is the type of resource being referenced + type: string + name: + description: Name is the name of resource being referenced + type: string + required: + - kind + - name + type: object + x-kubernetes-map-type: atomic + dataSourceRef: + description: 'dataSourceRef specifies the object from + which to populate the volume with data, if a non-empty + volume is desired. This may be any local object from + a non-empty API group (non core object) or a PersistentVolumeClaim + object. When this field is specified, volume binding + will only succeed if the type of the specified object + matches some installed volume populator or dynamic provisioner. + This field will replace the functionality of the DataSource + field and as such if both fields are non-empty, they + must have the same value. For backwards compatibility, + both fields (DataSource and DataSourceRef) will be set + to the same value automatically if one of them is empty + and the other is non-empty. There are two important + differences between DataSource and DataSourceRef: * + While DataSource only allows two specific types of objects, + DataSourceRef allows any non-core object, as well as + PersistentVolumeClaim objects. * While DataSource ignores + disallowed values (dropping them), DataSourceRef preserves + all values, and generates an error if a disallowed value + is specified. (Beta) Using this field requires the AnyVolumeDataSource + feature gate to be enabled.' + properties: + apiGroup: + description: APIGroup is the group for the resource + being referenced. If APIGroup is not specified, + the specified Kind must be in the core API group. + For any other third-party types, APIGroup is required. + type: string + kind: + description: Kind is the type of resource being referenced + type: string + name: + description: Name is the name of resource being referenced + type: string + required: + - kind + - name + type: object + x-kubernetes-map-type: atomic + resources: + description: 'resources represents the minimum resources + the volume should have. If RecoverVolumeExpansionFailure + feature is enabled users are allowed to specify resource + requirements that are lower than previous value but + must still be higher than capacity recorded in the status + field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources' + properties: + limits: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Limits describes the maximum amount + of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + requests: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Requests describes the minimum amount + of compute resources required. If Requests is omitted + for a container, it defaults to Limits if that is + explicitly specified, otherwise to an implementation-defined + value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + type: object + selector: + description: selector is a label query over volumes to + consider for binding. + properties: + matchExpressions: + description: matchExpressions is a list of label selector + requirements. The requirements are ANDed. + items: + description: A label selector requirement is a selector + that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that the selector + applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are In, + NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string values. + If the operator is In or NotIn, the values + array must be non-empty. If the operator is + Exists or DoesNotExist, the values array must + be empty. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. + A single {key,value} in the matchLabels map is equivalent + to an element of matchExpressions, whose key field + is "key", the operator is "In", and the values array + contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + storageClassName: + description: 'storageClassName is the name of the StorageClass + required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1' + type: string + volumeMode: + description: volumeMode defines what type of volume is + required by the claim. Value of Filesystem is implied + when not included in claim spec. + type: string + volumeName: + description: volumeName is the binding reference to the + PersistentVolume backing this claim. + type: string + type: object + status: + description: 'Status represents the current information/status + of a persistent volume claim. Read-only. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims' + properties: + accessModes: + description: 'accessModes contains the actual access modes + the volume backing the PVC has. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1' + items: + type: string + type: array + allocatedResources: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: allocatedResources is the storage resource + within AllocatedResources tracks the capacity allocated + to a PVC. It may be larger than the actual capacity + when a volume expansion operation is requested. For + storage quota, the larger value from allocatedResources + and PVC.spec.resources is used. If allocatedResources + is not set, PVC.spec.resources alone is used for quota + calculation. If a volume expansion capacity request + is lowered, allocatedResources is only lowered if there + are no expansion operations in progress and if the actual + volume capacity is equal or lower than the requested + capacity. This is an alpha field and requires enabling + RecoverVolumeExpansionFailure feature. + type: object + capacity: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: capacity represents the actual resources + of the underlying volume. + type: object + conditions: + description: conditions is the current Condition of persistent + volume claim. If underlying persistent volume is being + resized then the Condition will be set to 'ResizeStarted'. + items: + description: PersistentVolumeClaimCondition contails + details about state of pvc + properties: + lastProbeTime: + description: lastProbeTime is the time we probed + the condition. + format: date-time + type: string + lastTransitionTime: + description: lastTransitionTime is the time the + condition transitioned from one status to another. + format: date-time + type: string + message: + description: message is the human-readable message + indicating details about last transition. + type: string + reason: + description: reason is a unique, this should be + a short, machine understandable string that gives + the reason for condition's last transition. If + it reports "ResizeStarted" that means the underlying + persistent volume is being resized. + type: string + status: + type: string + type: + description: PersistentVolumeClaimConditionType + is a valid value of PersistentVolumeClaimCondition.Type + type: string + required: + - status + - type + type: object + type: array + phase: + description: phase represents the current phase of PersistentVolumeClaim. + type: string + resizeStatus: + description: resizeStatus stores status of resize operation. + ResizeStatus is not set by default but when expansion + is complete resizeStatus is set to empty string by resize + controller or kubelet. This is an alpha field and requires + enabling RecoverVolumeExpansionFailure feature. + type: string + type: object + type: object + type: object + tag: + description: 'Tag of Alertmanager container image to be deployed. + Defaults to the value of `version`. Version is ignored if Tag is + set. Deprecated: use ''image'' instead. The image tag can be specified + as part of the image URL.' + type: string + tolerations: + description: If specified, the pod's tolerations. + items: + description: The pod this Toleration is attached to tolerates any + taint that matches the triple using the matching + operator . + properties: + effect: + description: Effect indicates the taint effect to match. Empty + means match all taint effects. When specified, allowed values + are NoSchedule, PreferNoSchedule and NoExecute. + type: string + key: + description: Key is the taint key that the toleration applies + to. Empty means match all taint keys. If the key is empty, + operator must be Exists; this combination means to match all + values and all keys. + type: string + operator: + description: Operator represents a key's relationship to the + value. Valid operators are Exists and Equal. Defaults to Equal. + Exists is equivalent to wildcard for value, so that a pod + can tolerate all taints of a particular category. + type: string + tolerationSeconds: + description: TolerationSeconds represents the period of time + the toleration (which must be of effect NoExecute, otherwise + this field is ignored) tolerates the taint. By default, it + is not set, which means tolerate the taint forever (do not + evict). Zero and negative values will be treated as 0 (evict + immediately) by the system. + format: int64 + type: integer + value: + description: Value is the taint value the toleration matches + to. If the operator is Exists, the value should be empty, + otherwise just a regular string. + type: string + type: object + type: array + topologySpreadConstraints: + description: If specified, the pod's topology spread constraints. + items: + description: TopologySpreadConstraint specifies how to spread matching + pods among the given topology. + properties: + labelSelector: + description: LabelSelector is used to find matching pods. Pods + that match this label selector are counted to determine the + number of pods in their corresponding topology domain. + properties: + matchExpressions: + description: matchExpressions is a list of label selector + requirements. The requirements are ANDed. + items: + description: A label selector requirement is a selector + that contains values, a key, and an operator that relates + the key and values. + properties: + key: + description: key is the label key that the selector + applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are In, NotIn, + Exists and DoesNotExist. + type: string + values: + description: values is an array of string values. + If the operator is In or NotIn, the values array + must be non-empty. If the operator is Exists or + DoesNotExist, the values array must be empty. This + array is replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. + A single {key,value} in the matchLabels map is equivalent + to an element of matchExpressions, whose key field is + "key", the operator is "In", and the values array contains + only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + matchLabelKeys: + description: MatchLabelKeys is a set of pod label keys to select + the pods over which spreading will be calculated. The keys + are used to lookup values from the incoming pod labels, those + key-value labels are ANDed with labelSelector to select the + group of existing pods over which spreading will be calculated + for the incoming pod. Keys that don't exist in the incoming + pod labels will be ignored. A null or empty list means only + match against labelSelector. + items: + type: string + type: array + x-kubernetes-list-type: atomic + maxSkew: + description: 'MaxSkew describes the degree to which pods may + be unevenly distributed. When `whenUnsatisfiable=DoNotSchedule`, + it is the maximum permitted difference between the number + of matching pods in the target topology and the global minimum. + The global minimum is the minimum number of matching pods + in an eligible domain or zero if the number of eligible domains + is less than MinDomains. For example, in a 3-zone cluster, + MaxSkew is set to 1, and pods with the same labelSelector + spread as 2/2/1: In this case, the global minimum is 1. | + zone1 | zone2 | zone3 | | P P | P P | P | - if MaxSkew + is 1, incoming pod can only be scheduled to zone3 to become + 2/2/2; scheduling it onto zone1(zone2) would make the ActualSkew(3-1) + on zone1(zone2) violate MaxSkew(1). - if MaxSkew is 2, incoming + pod can be scheduled onto any zone. When `whenUnsatisfiable=ScheduleAnyway`, + it is used to give higher precedence to topologies that satisfy + it. It''s a required field. Default value is 1 and 0 is not + allowed.' + format: int32 + type: integer + minDomains: + description: "MinDomains indicates a minimum number of eligible + domains. When the number of eligible domains with matching + topology keys is less than minDomains, Pod Topology Spread + treats \"global minimum\" as 0, and then the calculation of + Skew is performed. And when the number of eligible domains + with matching topology keys equals or greater than minDomains, + this value has no effect on scheduling. As a result, when + the number of eligible domains is less than minDomains, scheduler + won't schedule more than maxSkew Pods to those domains. If + value is nil, the constraint behaves as if MinDomains is equal + to 1. Valid values are integers greater than 0. When value + is not nil, WhenUnsatisfiable must be DoNotSchedule. \n For + example, in a 3-zone cluster, MaxSkew is set to 2, MinDomains + is set to 5 and pods with the same labelSelector spread as + 2/2/2: | zone1 | zone2 | zone3 | | P P | P P | P P | + The number of domains is less than 5(MinDomains), so \"global + minimum\" is treated as 0. In this situation, new pod with + the same labelSelector cannot be scheduled, because computed + skew will be 3(3 - 0) if new Pod is scheduled to any of the + three zones, it will violate MaxSkew. \n This is a beta field + and requires the MinDomainsInPodTopologySpread feature gate + to be enabled (enabled by default)." + format: int32 + type: integer + nodeAffinityPolicy: + description: "NodeAffinityPolicy indicates how we will treat + Pod's nodeAffinity/nodeSelector when calculating pod topology + spread skew. Options are: - Honor: only nodes matching nodeAffinity/nodeSelector + are included in the calculations. - Ignore: nodeAffinity/nodeSelector + are ignored. All nodes are included in the calculations. \n + If this value is nil, the behavior is equivalent to the Honor + policy. This is a alpha-level feature enabled by the NodeInclusionPolicyInPodTopologySpread + feature flag." + type: string + nodeTaintsPolicy: + description: "NodeTaintsPolicy indicates how we will treat node + taints when calculating pod topology spread skew. Options + are: - Honor: nodes without taints, along with tainted nodes + for which the incoming pod has a toleration, are included. + - Ignore: node taints are ignored. All nodes are included. + \n If this value is nil, the behavior is equivalent to the + Ignore policy. This is a alpha-level feature enabled by the + NodeInclusionPolicyInPodTopologySpread feature flag." + type: string + topologyKey: + description: TopologyKey is the key of node labels. Nodes that + have a label with this key and identical values are considered + to be in the same topology. We consider each + as a "bucket", and try to put balanced number of pods into + each bucket. We define a domain as a particular instance of + a topology. Also, we define an eligible domain as a domain + whose nodes meet the requirements of nodeAffinityPolicy and + nodeTaintsPolicy. e.g. If TopologyKey is "kubernetes.io/hostname", + each Node is a domain of that topology. And, if TopologyKey + is "topology.kubernetes.io/zone", each zone is a domain of + that topology. It's a required field. + type: string + whenUnsatisfiable: + description: 'WhenUnsatisfiable indicates how to deal with a + pod if it doesn''t satisfy the spread constraint. - DoNotSchedule + (default) tells the scheduler not to schedule it. - ScheduleAnyway + tells the scheduler to schedule the pod in any location, but + giving higher precedence to topologies that would help reduce + the skew. A constraint is considered "Unsatisfiable" for an + incoming pod if and only if every possible node assignment + for that pod would violate "MaxSkew" on some topology. For + example, in a 3-zone cluster, MaxSkew is set to 1, and pods + with the same labelSelector spread as 3/1/1: | zone1 | zone2 + | zone3 | | P P P | P | P | If WhenUnsatisfiable is + set to DoNotSchedule, incoming pod can only be scheduled to + zone2(zone3) to become 3/2/1(3/1/2) as ActualSkew(2-1) on + zone2(zone3) satisfies MaxSkew(1). In other words, the cluster + can still be imbalanced, but scheduler won''t make it *more* + imbalanced. It''s a required field.' + type: string + required: + - maxSkew + - topologyKey + - whenUnsatisfiable + type: object + type: array + version: + description: Version the cluster should be on. + type: string + volumeMounts: + description: VolumeMounts allows configuration of additional VolumeMounts + on the output StatefulSet definition. VolumeMounts specified will + be appended to other VolumeMounts in the alertmanager container, + that are generated as a result of StorageSpec objects. + items: + description: VolumeMount describes a mounting of a Volume within + a container. + properties: + mountPath: + description: Path within the container at which the volume should + be mounted. Must not contain ':'. + type: string + mountPropagation: + description: mountPropagation determines how mounts are propagated + from the host to container and the other way around. When + not set, MountPropagationNone is used. This field is beta + in 1.10. + type: string + name: + description: This must match the Name of a Volume. + type: string + readOnly: + description: Mounted read-only if true, read-write otherwise + (false or unspecified). Defaults to false. + type: boolean + subPath: + description: Path within the volume from which the container's + volume should be mounted. Defaults to "" (volume's root). + type: string + subPathExpr: + description: Expanded path within the volume from which the + container's volume should be mounted. Behaves similarly to + SubPath but environment variable references $(VAR_NAME) are + expanded using the container's environment. Defaults to "" + (volume's root). SubPathExpr and SubPath are mutually exclusive. + type: string + required: + - mountPath + - name + type: object + type: array + volumes: + description: Volumes allows configuration of additional volumes on + the output StatefulSet definition. Volumes specified will be appended + to other volumes that are generated as a result of StorageSpec objects. + items: + description: Volume represents a named volume in a pod that may + be accessed by any container in the pod. + properties: + awsElasticBlockStore: + description: 'awsElasticBlockStore represents an AWS Disk resource + that is attached to a kubelet''s host machine and then exposed + to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore' + properties: + fsType: + description: 'fsType is the filesystem type of the volume + that you want to mount. Tip: Ensure that the filesystem + type is supported by the host operating system. Examples: + "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" + if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore + TODO: how do we prevent errors in the filesystem from + compromising the machine' + type: string + partition: + description: 'partition is the partition in the volume that + you want to mount. If omitted, the default is to mount + by volume name. Examples: For volume /dev/sda1, you specify + the partition as "1". Similarly, the volume partition + for /dev/sda is "0" (or you can leave the property empty).' + format: int32 + type: integer + readOnly: + description: 'readOnly value true will force the readOnly + setting in VolumeMounts. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore' + type: boolean + volumeID: + description: 'volumeID is unique ID of the persistent disk + resource in AWS (Amazon EBS volume). More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore' + type: string + required: + - volumeID + type: object + azureDisk: + description: azureDisk represents an Azure Data Disk mount on + the host and bind mount to the pod. + properties: + cachingMode: + description: 'cachingMode is the Host Caching mode: None, + Read Only, Read Write.' + type: string + diskName: + description: diskName is the Name of the data disk in the + blob storage + type: string + diskURI: + description: diskURI is the URI of data disk in the blob + storage + type: string + fsType: + description: fsType is Filesystem type to mount. Must be + a filesystem type supported by the host operating system. + Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" + if unspecified. + type: string + kind: + description: 'kind expected values are Shared: multiple + blob disks per storage account Dedicated: single blob + disk per storage account Managed: azure managed data + disk (only in managed availability set). defaults to shared' + type: string + readOnly: + description: readOnly Defaults to false (read/write). ReadOnly + here will force the ReadOnly setting in VolumeMounts. + type: boolean + required: + - diskName + - diskURI + type: object + azureFile: + description: azureFile represents an Azure File Service mount + on the host and bind mount to the pod. + properties: + readOnly: + description: readOnly defaults to false (read/write). ReadOnly + here will force the ReadOnly setting in VolumeMounts. + type: boolean + secretName: + description: secretName is the name of secret that contains + Azure Storage Account Name and Key + type: string + shareName: + description: shareName is the azure share Name + type: string + required: + - secretName + - shareName + type: object + cephfs: + description: cephFS represents a Ceph FS mount on the host that + shares a pod's lifetime + properties: + monitors: + description: 'monitors is Required: Monitors is a collection + of Ceph monitors More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it' + items: + type: string + type: array + path: + description: 'path is Optional: Used as the mounted root, + rather than the full Ceph tree, default is /' + type: string + readOnly: + description: 'readOnly is Optional: Defaults to false (read/write). + ReadOnly here will force the ReadOnly setting in VolumeMounts. + More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it' + type: boolean + secretFile: + description: 'secretFile is Optional: SecretFile is the + path to key ring for User, default is /etc/ceph/user.secret + More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it' + type: string + secretRef: + description: 'secretRef is Optional: SecretRef is reference + to the authentication secret for User, default is empty. + More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it' + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + type: object + x-kubernetes-map-type: atomic + user: + description: 'user is optional: User is the rados user name, + default is admin More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it' + type: string + required: + - monitors + type: object + cinder: + description: 'cinder represents a cinder volume attached and + mounted on kubelets host machine. More info: https://examples.k8s.io/mysql-cinder-pd/README.md' + properties: + fsType: + description: 'fsType is the filesystem type to mount. Must + be a filesystem type supported by the host operating system. + Examples: "ext4", "xfs", "ntfs". Implicitly inferred to + be "ext4" if unspecified. More info: https://examples.k8s.io/mysql-cinder-pd/README.md' + type: string + readOnly: + description: 'readOnly defaults to false (read/write). ReadOnly + here will force the ReadOnly setting in VolumeMounts. + More info: https://examples.k8s.io/mysql-cinder-pd/README.md' + type: boolean + secretRef: + description: 'secretRef is optional: points to a secret + object containing parameters used to connect to OpenStack.' + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + type: object + x-kubernetes-map-type: atomic + volumeID: + description: 'volumeID used to identify the volume in cinder. + More info: https://examples.k8s.io/mysql-cinder-pd/README.md' + type: string + required: + - volumeID + type: object + configMap: + description: configMap represents a configMap that should populate + this volume + properties: + defaultMode: + description: 'defaultMode is optional: mode bits used to + set permissions on created files by default. Must be an + octal value between 0000 and 0777 or a decimal value between + 0 and 511. YAML accepts both octal and decimal values, + JSON requires decimal values for mode bits. Defaults to + 0644. Directories within the path are not affected by + this setting. This might be in conflict with other options + that affect the file mode, like fsGroup, and the result + can be other mode bits set.' + format: int32 + type: integer + items: + description: items if unspecified, each key-value pair in + the Data field of the referenced ConfigMap will be projected + into the volume as a file whose name is the key and content + is the value. If specified, the listed keys will be projected + into the specified paths, and unlisted keys will not be + present. If a key is specified which is not present in + the ConfigMap, the volume setup will error unless it is + marked optional. Paths must be relative and may not contain + the '..' path or start with '..'. + items: + description: Maps a string key to a path within a volume. + properties: + key: + description: key is the key to project. + type: string + mode: + description: 'mode is Optional: mode bits used to + set permissions on this file. Must be an octal value + between 0000 and 0777 or a decimal value between + 0 and 511. YAML accepts both octal and decimal values, + JSON requires decimal values for mode bits. If not + specified, the volume defaultMode will be used. + This might be in conflict with other options that + affect the file mode, like fsGroup, and the result + can be other mode bits set.' + format: int32 + type: integer + path: + description: path is the relative path of the file + to map the key to. May not be an absolute path. + May not contain the path element '..'. May not start + with the string '..'. + type: string + required: + - key + - path + type: object + type: array + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: optional specify whether the ConfigMap or its + keys must be defined + type: boolean + type: object + x-kubernetes-map-type: atomic + csi: + description: csi (Container Storage Interface) represents ephemeral + storage that is handled by certain external CSI drivers (Beta + feature). + properties: + driver: + description: driver is the name of the CSI driver that handles + this volume. Consult with your admin for the correct name + as registered in the cluster. + type: string + fsType: + description: fsType to mount. Ex. "ext4", "xfs", "ntfs". + If not provided, the empty value is passed to the associated + CSI driver which will determine the default filesystem + to apply. + type: string + nodePublishSecretRef: + description: nodePublishSecretRef is a reference to the + secret object containing sensitive information to pass + to the CSI driver to complete the CSI NodePublishVolume + and NodeUnpublishVolume calls. This field is optional, + and may be empty if no secret is required. If the secret + object contains more than one secret, all secret references + are passed. + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + type: object + x-kubernetes-map-type: atomic + readOnly: + description: readOnly specifies a read-only configuration + for the volume. Defaults to false (read/write). + type: boolean + volumeAttributes: + additionalProperties: + type: string + description: volumeAttributes stores driver-specific properties + that are passed to the CSI driver. Consult your driver's + documentation for supported values. + type: object + required: + - driver + type: object + downwardAPI: + description: downwardAPI represents downward API about the pod + that should populate this volume + properties: + defaultMode: + description: 'Optional: mode bits to use on created files + by default. Must be a Optional: mode bits used to set + permissions on created files by default. Must be an octal + value between 0000 and 0777 or a decimal value between + 0 and 511. YAML accepts both octal and decimal values, + JSON requires decimal values for mode bits. Defaults to + 0644. Directories within the path are not affected by + this setting. This might be in conflict with other options + that affect the file mode, like fsGroup, and the result + can be other mode bits set.' + format: int32 + type: integer + items: + description: Items is a list of downward API volume file + items: + description: DownwardAPIVolumeFile represents information + to create the file containing the pod field + properties: + fieldRef: + description: 'Required: Selects a field of the pod: + only annotations, labels, name and namespace are + supported.' + properties: + apiVersion: + description: Version of the schema the FieldPath + is written in terms of, defaults to "v1". + type: string + fieldPath: + description: Path of the field to select in the + specified API version. + type: string + required: + - fieldPath + type: object + x-kubernetes-map-type: atomic + mode: + description: 'Optional: mode bits used to set permissions + on this file, must be an octal value between 0000 + and 0777 or a decimal value between 0 and 511. YAML + accepts both octal and decimal values, JSON requires + decimal values for mode bits. If not specified, + the volume defaultMode will be used. This might + be in conflict with other options that affect the + file mode, like fsGroup, and the result can be other + mode bits set.' + format: int32 + type: integer + path: + description: 'Required: Path is the relative path + name of the file to be created. Must not be absolute + or contain the ''..'' path. Must be utf-8 encoded. + The first item of the relative path must not start + with ''..''' + type: string + resourceFieldRef: + description: 'Selects a resource of the container: + only resources limits and requests (limits.cpu, + limits.memory, requests.cpu and requests.memory) + are currently supported.' + properties: + containerName: + description: 'Container name: required for volumes, + optional for env vars' + type: string + divisor: + anyOf: + - type: integer + - type: string + description: Specifies the output format of the + exposed resources, defaults to "1" + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + resource: + description: 'Required: resource to select' + type: string + required: + - resource + type: object + x-kubernetes-map-type: atomic + required: + - path + type: object + type: array + type: object + emptyDir: + description: 'emptyDir represents a temporary directory that + shares a pod''s lifetime. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir' + properties: + medium: + description: 'medium represents what type of storage medium + should back this directory. The default is "" which means + to use the node''s default medium. Must be an empty string + (default) or Memory. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir' + type: string + sizeLimit: + anyOf: + - type: integer + - type: string + description: 'sizeLimit is the total amount of local storage + required for this EmptyDir volume. The size limit is also + applicable for memory medium. The maximum usage on memory + medium EmptyDir would be the minimum value between the + SizeLimit specified here and the sum of memory limits + of all containers in a pod. The default is nil which means + that the limit is undefined. More info: http://kubernetes.io/docs/user-guide/volumes#emptydir' + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: object + ephemeral: + description: "ephemeral represents a volume that is handled + by a cluster storage driver. The volume's lifecycle is tied + to the pod that defines it - it will be created before the + pod starts, and deleted when the pod is removed. \n Use this + if: a) the volume is only needed while the pod runs, b) features + of normal volumes like restoring from snapshot or capacity + tracking are needed, c) the storage driver is specified through + a storage class, and d) the storage driver supports dynamic + volume provisioning through a PersistentVolumeClaim (see EphemeralVolumeSource + for more information on the connection between this volume + type and PersistentVolumeClaim). \n Use PersistentVolumeClaim + or one of the vendor-specific APIs for volumes that persist + for longer than the lifecycle of an individual pod. \n Use + CSI for light-weight local ephemeral volumes if the CSI driver + is meant to be used that way - see the documentation of the + driver for more information. \n A pod can use both types of + ephemeral volumes and persistent volumes at the same time." + properties: + volumeClaimTemplate: + description: "Will be used to create a stand-alone PVC to + provision the volume. The pod in which this EphemeralVolumeSource + is embedded will be the owner of the PVC, i.e. the PVC + will be deleted together with the pod. The name of the + PVC will be `-` where `` is the name from the `PodSpec.Volumes` array entry. + Pod validation will reject the pod if the concatenated + name is not valid for a PVC (for example, too long). \n + An existing PVC with that name that is not owned by the + pod will *not* be used for the pod to avoid using an unrelated + volume by mistake. Starting the pod is then blocked until + the unrelated PVC is removed. If such a pre-created PVC + is meant to be used by the pod, the PVC has to updated + with an owner reference to the pod once the pod exists. + Normally this should not be necessary, but it may be useful + when manually reconstructing a broken cluster. \n This + field is read-only and no changes will be made by Kubernetes + to the PVC after it has been created. \n Required, must + not be nil." + properties: + metadata: + description: May contain labels and annotations that + will be copied into the PVC when creating it. No other + fields are allowed and will be rejected during validation. + type: object + spec: + description: The specification for the PersistentVolumeClaim. + The entire content is copied unchanged into the PVC + that gets created from this template. The same fields + as in a PersistentVolumeClaim are also valid here. + properties: + accessModes: + description: 'accessModes contains the desired access + modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1' + items: + type: string + type: array + dataSource: + description: 'dataSource field can be used to specify + either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) + * An existing PVC (PersistentVolumeClaim) If the + provisioner or an external controller can support + the specified data source, it will create a new + volume based on the contents of the specified + data source. If the AnyVolumeDataSource feature + gate is enabled, this field will always have the + same contents as the DataSourceRef field.' + properties: + apiGroup: + description: APIGroup is the group for the resource + being referenced. If APIGroup is not specified, + the specified Kind must be in the core API + group. For any other third-party types, APIGroup + is required. + type: string + kind: + description: Kind is the type of resource being + referenced + type: string + name: + description: Name is the name of resource being + referenced + type: string + required: + - kind + - name + type: object + x-kubernetes-map-type: atomic + dataSourceRef: + description: 'dataSourceRef specifies the object + from which to populate the volume with data, if + a non-empty volume is desired. This may be any + local object from a non-empty API group (non core + object) or a PersistentVolumeClaim object. When + this field is specified, volume binding will only + succeed if the type of the specified object matches + some installed volume populator or dynamic provisioner. + This field will replace the functionality of the + DataSource field and as such if both fields are + non-empty, they must have the same value. For + backwards compatibility, both fields (DataSource + and DataSourceRef) will be set to the same value + automatically if one of them is empty and the + other is non-empty. There are two important differences + between DataSource and DataSourceRef: * While + DataSource only allows two specific types of objects, + DataSourceRef allows any non-core object, as well + as PersistentVolumeClaim objects. * While DataSource + ignores disallowed values (dropping them), DataSourceRef + preserves all values, and generates an error if + a disallowed value is specified. (Beta) Using + this field requires the AnyVolumeDataSource feature + gate to be enabled.' + properties: + apiGroup: + description: APIGroup is the group for the resource + being referenced. If APIGroup is not specified, + the specified Kind must be in the core API + group. For any other third-party types, APIGroup + is required. + type: string + kind: + description: Kind is the type of resource being + referenced + type: string + name: + description: Name is the name of resource being + referenced + type: string + required: + - kind + - name + type: object + x-kubernetes-map-type: atomic + resources: + description: 'resources represents the minimum resources + the volume should have. If RecoverVolumeExpansionFailure + feature is enabled users are allowed to specify + resource requirements that are lower than previous + value but must still be higher than capacity recorded + in the status field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources' + properties: + limits: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Limits describes the maximum amount + of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + requests: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Requests describes the minimum + amount of compute resources required. If Requests + is omitted for a container, it defaults to + Limits if that is explicitly specified, otherwise + to an implementation-defined value. More info: + https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + type: object + selector: + description: selector is a label query over volumes + to consider for binding. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + storageClassName: + description: 'storageClassName is the name of the + StorageClass required by the claim. More info: + https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1' + type: string + volumeMode: + description: volumeMode defines what type of volume + is required by the claim. Value of Filesystem + is implied when not included in claim spec. + type: string + volumeName: + description: volumeName is the binding reference + to the PersistentVolume backing this claim. + type: string + type: object + required: + - spec + type: object + type: object + fc: + description: fc represents a Fibre Channel resource that is + attached to a kubelet's host machine and then exposed to the + pod. + properties: + fsType: + description: 'fsType is the filesystem type to mount. Must + be a filesystem type supported by the host operating system. + Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" + if unspecified. TODO: how do we prevent errors in the + filesystem from compromising the machine' + type: string + lun: + description: 'lun is Optional: FC target lun number' + format: int32 + type: integer + readOnly: + description: 'readOnly is Optional: Defaults to false (read/write). + ReadOnly here will force the ReadOnly setting in VolumeMounts.' + type: boolean + targetWWNs: + description: 'targetWWNs is Optional: FC target worldwide + names (WWNs)' + items: + type: string + type: array + wwids: + description: 'wwids Optional: FC volume world wide identifiers + (wwids) Either wwids or combination of targetWWNs and + lun must be set, but not both simultaneously.' + items: + type: string + type: array + type: object + flexVolume: + description: flexVolume represents a generic volume resource + that is provisioned/attached using an exec based plugin. + properties: + driver: + description: driver is the name of the driver to use for + this volume. + type: string + fsType: + description: fsType is the filesystem type to mount. Must + be a filesystem type supported by the host operating system. + Ex. "ext4", "xfs", "ntfs". The default filesystem depends + on FlexVolume script. + type: string + options: + additionalProperties: + type: string + description: 'options is Optional: this field holds extra + command options if any.' + type: object + readOnly: + description: 'readOnly is Optional: defaults to false (read/write). + ReadOnly here will force the ReadOnly setting in VolumeMounts.' + type: boolean + secretRef: + description: 'secretRef is Optional: secretRef is reference + to the secret object containing sensitive information + to pass to the plugin scripts. This may be empty if no + secret object is specified. If the secret object contains + more than one secret, all secrets are passed to the plugin + scripts.' + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + type: object + x-kubernetes-map-type: atomic + required: + - driver + type: object + flocker: + description: flocker represents a Flocker volume attached to + a kubelet's host machine. This depends on the Flocker control + service being running + properties: + datasetName: + description: datasetName is Name of the dataset stored as + metadata -> name on the dataset for Flocker should be + considered as deprecated + type: string + datasetUUID: + description: datasetUUID is the UUID of the dataset. This + is unique identifier of a Flocker dataset + type: string + type: object + gcePersistentDisk: + description: 'gcePersistentDisk represents a GCE Disk resource + that is attached to a kubelet''s host machine and then exposed + to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk' + properties: + fsType: + description: 'fsType is filesystem type of the volume that + you want to mount. Tip: Ensure that the filesystem type + is supported by the host operating system. Examples: "ext4", + "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. + More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk + TODO: how do we prevent errors in the filesystem from + compromising the machine' + type: string + partition: + description: 'partition is the partition in the volume that + you want to mount. If omitted, the default is to mount + by volume name. Examples: For volume /dev/sda1, you specify + the partition as "1". Similarly, the volume partition + for /dev/sda is "0" (or you can leave the property empty). + More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk' + format: int32 + type: integer + pdName: + description: 'pdName is unique name of the PD resource in + GCE. Used to identify the disk in GCE. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk' + type: string + readOnly: + description: 'readOnly here will force the ReadOnly setting + in VolumeMounts. Defaults to false. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk' + type: boolean + required: + - pdName + type: object + gitRepo: + description: 'gitRepo represents a git repository at a particular + revision. DEPRECATED: GitRepo is deprecated. To provision + a container with a git repo, mount an EmptyDir into an InitContainer + that clones the repo using git, then mount the EmptyDir into + the Pod''s container.' + properties: + directory: + description: directory is the target directory name. Must + not contain or start with '..'. If '.' is supplied, the + volume directory will be the git repository. Otherwise, + if specified, the volume will contain the git repository + in the subdirectory with the given name. + type: string + repository: + description: repository is the URL + type: string + revision: + description: revision is the commit hash for the specified + revision. + type: string + required: + - repository + type: object + glusterfs: + description: 'glusterfs represents a Glusterfs mount on the + host that shares a pod''s lifetime. More info: https://examples.k8s.io/volumes/glusterfs/README.md' + properties: + endpoints: + description: 'endpoints is the endpoint name that details + Glusterfs topology. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod' + type: string + path: + description: 'path is the Glusterfs volume path. More info: + https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod' + type: string + readOnly: + description: 'readOnly here will force the Glusterfs volume + to be mounted with read-only permissions. Defaults to + false. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod' + type: boolean + required: + - endpoints + - path + type: object + hostPath: + description: 'hostPath represents a pre-existing file or directory + on the host machine that is directly exposed to the container. + This is generally used for system agents or other privileged + things that are allowed to see the host machine. Most containers + will NOT need this. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath + --- TODO(jonesdl) We need to restrict who can use host directory + mounts and who can/can not mount host directories as read/write.' + properties: + path: + description: 'path of the directory on the host. If the + path is a symlink, it will follow the link to the real + path. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath' + type: string + type: + description: 'type for HostPath Volume Defaults to "" More + info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath' + type: string + required: + - path + type: object + iscsi: + description: 'iscsi represents an ISCSI Disk resource that is + attached to a kubelet''s host machine and then exposed to + the pod. More info: https://examples.k8s.io/volumes/iscsi/README.md' + properties: + chapAuthDiscovery: + description: chapAuthDiscovery defines whether support iSCSI + Discovery CHAP authentication + type: boolean + chapAuthSession: + description: chapAuthSession defines whether support iSCSI + Session CHAP authentication + type: boolean + fsType: + description: 'fsType is the filesystem type of the volume + that you want to mount. Tip: Ensure that the filesystem + type is supported by the host operating system. Examples: + "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" + if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#iscsi + TODO: how do we prevent errors in the filesystem from + compromising the machine' + type: string + initiatorName: + description: initiatorName is the custom iSCSI Initiator + Name. If initiatorName is specified with iscsiInterface + simultaneously, new iSCSI interface : will be created for the connection. + type: string + iqn: + description: iqn is the target iSCSI Qualified Name. + type: string + iscsiInterface: + description: iscsiInterface is the interface Name that uses + an iSCSI transport. Defaults to 'default' (tcp). + type: string + lun: + description: lun represents iSCSI Target Lun number. + format: int32 + type: integer + portals: + description: portals is the iSCSI Target Portal List. The + portal is either an IP or ip_addr:port if the port is + other than default (typically TCP ports 860 and 3260). + items: + type: string + type: array + readOnly: + description: readOnly here will force the ReadOnly setting + in VolumeMounts. Defaults to false. + type: boolean + secretRef: + description: secretRef is the CHAP Secret for iSCSI target + and initiator authentication + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + type: object + x-kubernetes-map-type: atomic + targetPortal: + description: targetPortal is iSCSI Target Portal. The Portal + is either an IP or ip_addr:port if the port is other than + default (typically TCP ports 860 and 3260). + type: string + required: + - iqn + - lun + - targetPortal + type: object + name: + description: 'name of the volume. Must be a DNS_LABEL and unique + within the pod. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + nfs: + description: 'nfs represents an NFS mount on the host that shares + a pod''s lifetime More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs' + properties: + path: + description: 'path that is exported by the NFS server. More + info: https://kubernetes.io/docs/concepts/storage/volumes#nfs' + type: string + readOnly: + description: 'readOnly here will force the NFS export to + be mounted with read-only permissions. Defaults to false. + More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs' + type: boolean + server: + description: 'server is the hostname or IP address of the + NFS server. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs' + type: string + required: + - path + - server + type: object + persistentVolumeClaim: + description: 'persistentVolumeClaimVolumeSource represents a + reference to a PersistentVolumeClaim in the same namespace. + More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims' + properties: + claimName: + description: 'claimName is the name of a PersistentVolumeClaim + in the same namespace as the pod using this volume. More + info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims' + type: string + readOnly: + description: readOnly Will force the ReadOnly setting in + VolumeMounts. Default false. + type: boolean + required: + - claimName + type: object + photonPersistentDisk: + description: photonPersistentDisk represents a PhotonController + persistent disk attached and mounted on kubelets host machine + properties: + fsType: + description: fsType is the filesystem type to mount. Must + be a filesystem type supported by the host operating system. + Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" + if unspecified. + type: string + pdID: + description: pdID is the ID that identifies Photon Controller + persistent disk + type: string + required: + - pdID + type: object + portworxVolume: + description: portworxVolume represents a portworx volume attached + and mounted on kubelets host machine + properties: + fsType: + description: fSType represents the filesystem type to mount + Must be a filesystem type supported by the host operating + system. Ex. "ext4", "xfs". Implicitly inferred to be "ext4" + if unspecified. + type: string + readOnly: + description: readOnly defaults to false (read/write). ReadOnly + here will force the ReadOnly setting in VolumeMounts. + type: boolean + volumeID: + description: volumeID uniquely identifies a Portworx volume + type: string + required: + - volumeID + type: object + projected: + description: projected items for all in one resources secrets, + configmaps, and downward API + properties: + defaultMode: + description: defaultMode are the mode bits used to set permissions + on created files by default. Must be an octal value between + 0000 and 0777 or a decimal value between 0 and 511. YAML + accepts both octal and decimal values, JSON requires decimal + values for mode bits. Directories within the path are + not affected by this setting. This might be in conflict + with other options that affect the file mode, like fsGroup, + and the result can be other mode bits set. + format: int32 + type: integer + sources: + description: sources is the list of volume projections + items: + description: Projection that may be projected along with + other supported volume types + properties: + configMap: + description: configMap information about the configMap + data to project + properties: + items: + description: items if unspecified, each key-value + pair in the Data field of the referenced ConfigMap + will be projected into the volume as a file + whose name is the key and content is the value. + If specified, the listed keys will be projected + into the specified paths, and unlisted keys + will not be present. If a key is specified which + is not present in the ConfigMap, the volume + setup will error unless it is marked optional. + Paths must be relative and may not contain the + '..' path or start with '..'. + items: + description: Maps a string key to a path within + a volume. + properties: + key: + description: key is the key to project. + type: string + mode: + description: 'mode is Optional: mode bits + used to set permissions on this file. + Must be an octal value between 0000 and + 0777 or a decimal value between 0 and + 511. YAML accepts both octal and decimal + values, JSON requires decimal values for + mode bits. If not specified, the volume + defaultMode will be used. This might be + in conflict with other options that affect + the file mode, like fsGroup, and the result + can be other mode bits set.' + format: int32 + type: integer + path: + description: path is the relative path of + the file to map the key to. May not be + an absolute path. May not contain the + path element '..'. May not start with + the string '..'. + type: string + required: + - key + - path + type: object + type: array + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: optional specify whether the ConfigMap + or its keys must be defined + type: boolean + type: object + x-kubernetes-map-type: atomic + downwardAPI: + description: downwardAPI information about the downwardAPI + data to project + properties: + items: + description: Items is a list of DownwardAPIVolume + file + items: + description: DownwardAPIVolumeFile represents + information to create the file containing + the pod field + properties: + fieldRef: + description: 'Required: Selects a field + of the pod: only annotations, labels, + name and namespace are supported.' + properties: + apiVersion: + description: Version of the schema the + FieldPath is written in terms of, + defaults to "v1". + type: string + fieldPath: + description: Path of the field to select + in the specified API version. + type: string + required: + - fieldPath + type: object + x-kubernetes-map-type: atomic + mode: + description: 'Optional: mode bits used to + set permissions on this file, must be + an octal value between 0000 and 0777 or + a decimal value between 0 and 511. YAML + accepts both octal and decimal values, + JSON requires decimal values for mode + bits. If not specified, the volume defaultMode + will be used. This might be in conflict + with other options that affect the file + mode, like fsGroup, and the result can + be other mode bits set.' + format: int32 + type: integer + path: + description: 'Required: Path is the relative + path name of the file to be created. Must + not be absolute or contain the ''..'' + path. Must be utf-8 encoded. The first + item of the relative path must not start + with ''..''' + type: string + resourceFieldRef: + description: 'Selects a resource of the + container: only resources limits and requests + (limits.cpu, limits.memory, requests.cpu + and requests.memory) are currently supported.' + properties: + containerName: + description: 'Container name: required + for volumes, optional for env vars' + type: string + divisor: + anyOf: + - type: integer + - type: string + description: Specifies the output format + of the exposed resources, defaults + to "1" + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + resource: + description: 'Required: resource to + select' + type: string + required: + - resource + type: object + x-kubernetes-map-type: atomic + required: + - path + type: object + type: array + type: object + secret: + description: secret information about the secret data + to project + properties: + items: + description: items if unspecified, each key-value + pair in the Data field of the referenced Secret + will be projected into the volume as a file + whose name is the key and content is the value. + If specified, the listed keys will be projected + into the specified paths, and unlisted keys + will not be present. If a key is specified which + is not present in the Secret, the volume setup + will error unless it is marked optional. Paths + must be relative and may not contain the '..' + path or start with '..'. + items: + description: Maps a string key to a path within + a volume. + properties: + key: + description: key is the key to project. + type: string + mode: + description: 'mode is Optional: mode bits + used to set permissions on this file. + Must be an octal value between 0000 and + 0777 or a decimal value between 0 and + 511. YAML accepts both octal and decimal + values, JSON requires decimal values for + mode bits. If not specified, the volume + defaultMode will be used. This might be + in conflict with other options that affect + the file mode, like fsGroup, and the result + can be other mode bits set.' + format: int32 + type: integer + path: + description: path is the relative path of + the file to map the key to. May not be + an absolute path. May not contain the + path element '..'. May not start with + the string '..'. + type: string + required: + - key + - path + type: object + type: array + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: optional field specify whether the + Secret or its key must be defined + type: boolean + type: object + x-kubernetes-map-type: atomic + serviceAccountToken: + description: serviceAccountToken is information about + the serviceAccountToken data to project + properties: + audience: + description: audience is the intended audience + of the token. A recipient of a token must identify + itself with an identifier specified in the audience + of the token, and otherwise should reject the + token. The audience defaults to the identifier + of the apiserver. + type: string + expirationSeconds: + description: expirationSeconds is the requested + duration of validity of the service account + token. As the token approaches expiration, the + kubelet volume plugin will proactively rotate + the service account token. The kubelet will + start trying to rotate the token if the token + is older than 80 percent of its time to live + or if the token is older than 24 hours.Defaults + to 1 hour and must be at least 10 minutes. + format: int64 + type: integer + path: + description: path is the path relative to the + mount point of the file to project the token + into. + type: string + required: + - path + type: object + type: object + type: array + type: object + quobyte: + description: quobyte represents a Quobyte mount on the host + that shares a pod's lifetime + properties: + group: + description: group to map volume access to Default is no + group + type: string + readOnly: + description: readOnly here will force the Quobyte volume + to be mounted with read-only permissions. Defaults to + false. + type: boolean + registry: + description: registry represents a single or multiple Quobyte + Registry services specified as a string as host:port pair + (multiple entries are separated with commas) which acts + as the central registry for volumes + type: string + tenant: + description: tenant owning the given Quobyte volume in the + Backend Used with dynamically provisioned Quobyte volumes, + value is set by the plugin + type: string + user: + description: user to map volume access to Defaults to serivceaccount + user + type: string + volume: + description: volume is a string that references an already + created Quobyte volume by name. + type: string + required: + - registry + - volume + type: object + rbd: + description: 'rbd represents a Rados Block Device mount on the + host that shares a pod''s lifetime. More info: https://examples.k8s.io/volumes/rbd/README.md' + properties: + fsType: + description: 'fsType is the filesystem type of the volume + that you want to mount. Tip: Ensure that the filesystem + type is supported by the host operating system. Examples: + "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" + if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#rbd + TODO: how do we prevent errors in the filesystem from + compromising the machine' + type: string + image: + description: 'image is the rados image name. More info: + https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it' + type: string + keyring: + description: 'keyring is the path to key ring for RBDUser. + Default is /etc/ceph/keyring. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it' + type: string + monitors: + description: 'monitors is a collection of Ceph monitors. + More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it' + items: + type: string + type: array + pool: + description: 'pool is the rados pool name. Default is rbd. + More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it' + type: string + readOnly: + description: 'readOnly here will force the ReadOnly setting + in VolumeMounts. Defaults to false. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it' + type: boolean + secretRef: + description: 'secretRef is name of the authentication secret + for RBDUser. If provided overrides keyring. Default is + nil. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it' + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + type: object + x-kubernetes-map-type: atomic + user: + description: 'user is the rados user name. Default is admin. + More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it' + type: string + required: + - image + - monitors + type: object + scaleIO: + description: scaleIO represents a ScaleIO persistent volume + attached and mounted on Kubernetes nodes. + properties: + fsType: + description: fsType is the filesystem type to mount. Must + be a filesystem type supported by the host operating system. + Ex. "ext4", "xfs", "ntfs". Default is "xfs". + type: string + gateway: + description: gateway is the host address of the ScaleIO + API Gateway. + type: string + protectionDomain: + description: protectionDomain is the name of the ScaleIO + Protection Domain for the configured storage. + type: string + readOnly: + description: readOnly Defaults to false (read/write). ReadOnly + here will force the ReadOnly setting in VolumeMounts. + type: boolean + secretRef: + description: secretRef references to the secret for ScaleIO + user and other sensitive information. If this is not provided, + Login operation will fail. + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + type: object + x-kubernetes-map-type: atomic + sslEnabled: + description: sslEnabled Flag enable/disable SSL communication + with Gateway, default false + type: boolean + storageMode: + description: storageMode indicates whether the storage for + a volume should be ThickProvisioned or ThinProvisioned. + Default is ThinProvisioned. + type: string + storagePool: + description: storagePool is the ScaleIO Storage Pool associated + with the protection domain. + type: string + system: + description: system is the name of the storage system as + configured in ScaleIO. + type: string + volumeName: + description: volumeName is the name of a volume already + created in the ScaleIO system that is associated with + this volume source. + type: string + required: + - gateway + - secretRef + - system + type: object + secret: + description: 'secret represents a secret that should populate + this volume. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret' + properties: + defaultMode: + description: 'defaultMode is Optional: mode bits used to + set permissions on created files by default. Must be an + octal value between 0000 and 0777 or a decimal value between + 0 and 511. YAML accepts both octal and decimal values, + JSON requires decimal values for mode bits. Defaults to + 0644. Directories within the path are not affected by + this setting. This might be in conflict with other options + that affect the file mode, like fsGroup, and the result + can be other mode bits set.' + format: int32 + type: integer + items: + description: items If unspecified, each key-value pair in + the Data field of the referenced Secret will be projected + into the volume as a file whose name is the key and content + is the value. If specified, the listed keys will be projected + into the specified paths, and unlisted keys will not be + present. If a key is specified which is not present in + the Secret, the volume setup will error unless it is marked + optional. Paths must be relative and may not contain the + '..' path or start with '..'. + items: + description: Maps a string key to a path within a volume. + properties: + key: + description: key is the key to project. + type: string + mode: + description: 'mode is Optional: mode bits used to + set permissions on this file. Must be an octal value + between 0000 and 0777 or a decimal value between + 0 and 511. YAML accepts both octal and decimal values, + JSON requires decimal values for mode bits. If not + specified, the volume defaultMode will be used. + This might be in conflict with other options that + affect the file mode, like fsGroup, and the result + can be other mode bits set.' + format: int32 + type: integer + path: + description: path is the relative path of the file + to map the key to. May not be an absolute path. + May not contain the path element '..'. May not start + with the string '..'. + type: string + required: + - key + - path + type: object + type: array + optional: + description: optional field specify whether the Secret or + its keys must be defined + type: boolean + secretName: + description: 'secretName is the name of the secret in the + pod''s namespace to use. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret' + type: string + type: object + storageos: + description: storageOS represents a StorageOS volume attached + and mounted on Kubernetes nodes. + properties: + fsType: + description: fsType is the filesystem type to mount. Must + be a filesystem type supported by the host operating system. + Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" + if unspecified. + type: string + readOnly: + description: readOnly defaults to false (read/write). ReadOnly + here will force the ReadOnly setting in VolumeMounts. + type: boolean + secretRef: + description: secretRef specifies the secret to use for obtaining + the StorageOS API credentials. If not specified, default + values will be attempted. + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + type: object + x-kubernetes-map-type: atomic + volumeName: + description: volumeName is the human-readable name of the + StorageOS volume. Volume names are only unique within + a namespace. + type: string + volumeNamespace: + description: volumeNamespace specifies the scope of the + volume within StorageOS. If no namespace is specified + then the Pod's namespace will be used. This allows the + Kubernetes name scoping to be mirrored within StorageOS + for tighter integration. Set VolumeName to any name to + override the default behaviour. Set to "default" if you + are not using namespaces within StorageOS. Namespaces + that do not pre-exist within StorageOS will be created. + type: string + type: object + vsphereVolume: + description: vsphereVolume represents a vSphere volume attached + and mounted on kubelets host machine + properties: + fsType: + description: fsType is filesystem type to mount. Must be + a filesystem type supported by the host operating system. + Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" + if unspecified. + type: string + storagePolicyID: + description: storagePolicyID is the storage Policy Based + Management (SPBM) profile ID associated with the StoragePolicyName. + type: string + storagePolicyName: + description: storagePolicyName is the storage Policy Based + Management (SPBM) profile name. + type: string + volumePath: + description: volumePath is the path that identifies vSphere + volume vmdk + type: string + required: + - volumePath + type: object + required: + - name + type: object + type: array + web: + description: Defines the web command line flags when starting Alertmanager. + properties: + httpConfig: + description: Defines HTTP parameters for web server. + properties: + headers: + description: List of headers that can be added to HTTP responses. + properties: + contentSecurityPolicy: + description: Set the Content-Security-Policy header to + HTTP responses. Unset if blank. + type: string + strictTransportSecurity: + description: Set the Strict-Transport-Security header + to HTTP responses. Unset if blank. Please make sure + that you use this with care as this header might force + browsers to load Prometheus and the other applications + hosted on the same domain and subdomains over HTTPS. + https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security + type: string + xContentTypeOptions: + description: Set the X-Content-Type-Options header to + HTTP responses. Unset if blank. Accepted value is nosniff. + https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options + enum: + - "" + - NoSniff + type: string + xFrameOptions: + description: Set the X-Frame-Options header to HTTP responses. + Unset if blank. Accepted values are deny and sameorigin. + https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options + enum: + - "" + - Deny + - SameOrigin + type: string + xXSSProtection: + description: Set the X-XSS-Protection header to all responses. + Unset if blank. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection + type: string + type: object + http2: + description: Enable HTTP/2 support. Note that HTTP/2 is only + supported with TLS. When TLSConfig is not configured, HTTP/2 + will be disabled. Whenever the value of the field changes, + a rolling update will be triggered. + type: boolean + type: object + tlsConfig: + description: Defines the TLS parameters for HTTPS. + properties: + cert: + description: Contains the TLS certificate for the server. + properties: + configMap: + description: ConfigMap containing data to use for the + targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the ConfigMap or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for the targets. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + cipherSuites: + description: 'List of supported cipher suites for TLS versions + up to TLS 1.2. If empty, Go default cipher suites are used. + Available cipher suites are documented in the go documentation: + https://golang.org/pkg/crypto/tls/#pkg-constants' + items: + type: string + type: array + client_ca: + description: Contains the CA certificate for client certificate + authentication to the server. + properties: + configMap: + description: ConfigMap containing data to use for the + targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the ConfigMap or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for the targets. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + clientAuthType: + description: 'Server policy for client authentication. Maps + to ClientAuth Policies. For more detail on clientAuth options: + https://golang.org/pkg/crypto/tls/#ClientAuthType' + type: string + curvePreferences: + description: 'Elliptic curves that will be used in an ECDHE + handshake, in preference order. Available curves are documented + in the go documentation: https://golang.org/pkg/crypto/tls/#CurveID' + items: + type: string + type: array + keySecret: + description: Secret containing the TLS key for the server. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + maxVersion: + description: Maximum TLS version that is acceptable. Defaults + to TLS13. + type: string + minVersion: + description: Minimum TLS version that is acceptable. Defaults + to TLS12. + type: string + preferServerCipherSuites: + description: Controls whether the server selects the client's + most preferred cipher suite, or the server's most preferred + cipher suite. If true then the server's preference, as expressed + in the order of elements in cipherSuites, is used. + type: boolean + required: + - cert + - keySecret + type: object + type: object + type: object + status: + description: 'Most recent observed status of the Alertmanager cluster. + Read-only. Not included when requesting from the apiserver, only from + the Prometheus Operator API itself. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#spec-and-status' + properties: + availableReplicas: + description: Total number of available pods (ready for at least minReadySeconds) + targeted by this Alertmanager cluster. + format: int32 + type: integer + paused: + description: Represents whether any actions on the underlying managed + objects are being performed. Only delete actions will be performed. + type: boolean + replicas: + description: Total number of non-terminated pods targeted by this + Alertmanager cluster (their labels match the selector). + format: int32 + type: integer + unavailableReplicas: + description: Total number of unavailable pods targeted by this Alertmanager + cluster. + format: int32 + type: integer + updatedReplicas: + description: Total number of non-terminated pods targeted by this + Alertmanager cluster that have the desired version spec. + format: int32 + type: integer + required: + - availableReplicas + - paused + - replicas + - unavailableReplicas + - updatedReplicas + type: object + required: + - spec + type: object + served: true + storage: true + subresources: {} diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-podmonitors.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-podmonitors.yaml new file mode 100644 index 00000000..d39a5527 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-podmonitors.yaml @@ -0,0 +1,666 @@ +# https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.61.1/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.9.2 + creationTimestamp: null + name: podmonitors.monitoring.coreos.com +spec: + group: monitoring.coreos.com + names: + categories: + - prometheus-operator + kind: PodMonitor + listKind: PodMonitorList + plural: podmonitors + shortNames: + - pmon + singular: podmonitor + scope: Namespaced + versions: + - name: v1 + schema: + openAPIV3Schema: + description: PodMonitor defines monitoring for a set of pods. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: Specification of desired Pod selection for target discovery + by Prometheus. + properties: + attachMetadata: + description: Attaches node metadata to discovered targets. Requires + Prometheus v2.35.0 and above. + properties: + node: + description: When set to true, Prometheus must have permissions + to get Nodes. + type: boolean + type: object + jobLabel: + description: The label to use to retrieve the job name from. + type: string + labelLimit: + description: Per-scrape limit on number of labels that will be accepted + for a sample. Only valid in Prometheus versions 2.27.0 and newer. + format: int64 + type: integer + labelNameLengthLimit: + description: Per-scrape limit on length of labels name that will be + accepted for a sample. Only valid in Prometheus versions 2.27.0 + and newer. + format: int64 + type: integer + labelValueLengthLimit: + description: Per-scrape limit on length of labels value that will + be accepted for a sample. Only valid in Prometheus versions 2.27.0 + and newer. + format: int64 + type: integer + namespaceSelector: + description: Selector to select which namespaces the Endpoints objects + are discovered from. + properties: + any: + description: Boolean describing whether all namespaces are selected + in contrast to a list restricting them. + type: boolean + matchNames: + description: List of namespace names to select from. + items: + type: string + type: array + type: object + podMetricsEndpoints: + description: A list of endpoints allowed as part of this PodMonitor. + items: + description: PodMetricsEndpoint defines a scrapeable endpoint of + a Kubernetes Pod serving Prometheus metrics. + properties: + authorization: + description: Authorization section for this endpoint + properties: + credentials: + description: The secret's key that contains the credentials + of the request + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: + description: Set the authentication type. Defaults to Bearer, + Basic will cause an error + type: string + type: object + basicAuth: + description: 'BasicAuth allow an endpoint to authenticate over + basic authentication. More info: https://prometheus.io/docs/operating/configuration/#endpoint' + properties: + password: + description: The secret in the service monitor namespace + that contains the password for authentication. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + username: + description: The secret in the service monitor namespace + that contains the username for authentication. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + bearerTokenSecret: + description: Secret to mount to read bearer token for scraping + targets. The secret needs to be in the same namespace as the + pod monitor and accessible by the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + enableHttp2: + description: Whether to enable HTTP2. + type: boolean + filterRunning: + description: 'Drop pods that are not running. (Failed, Succeeded). + Enabled by default. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase' + type: boolean + followRedirects: + description: FollowRedirects configures whether scrape requests + follow HTTP 3xx redirects. + type: boolean + honorLabels: + description: HonorLabels chooses the metric's labels on collisions + with target labels. + type: boolean + honorTimestamps: + description: HonorTimestamps controls whether Prometheus respects + the timestamps present in scraped data. + type: boolean + interval: + description: Interval at which metrics should be scraped If + not specified Prometheus' global scrape interval is used. + pattern: ^(0|(([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?)$ + type: string + metricRelabelings: + description: MetricRelabelConfigs to apply to samples before + ingestion. + items: + description: 'RelabelConfig allows dynamic rewriting of the + label set, being applied to samples before ingestion. It + defines ``-section of Prometheus + configuration. More info: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs' + properties: + action: + default: replace + description: Action to perform based on regex matching. + Default is 'replace'. uppercase and lowercase actions + require Prometheus >= 2.36. + enum: + - replace + - Replace + - keep + - Keep + - drop + - Drop + - hashmod + - HashMod + - labelmap + - LabelMap + - labeldrop + - LabelDrop + - labelkeep + - LabelKeep + - lowercase + - Lowercase + - uppercase + - Uppercase + type: string + modulus: + description: Modulus to take of the hash of the source + label values. + format: int64 + type: integer + regex: + description: Regular expression against which the extracted + value is matched. Default is '(.*)' + type: string + replacement: + description: Replacement value against which a regex replace + is performed if the regular expression matches. Regex + capture groups are available. Default is '$1' + type: string + separator: + description: Separator placed between concatenated source + label values. default is ';'. + type: string + sourceLabels: + description: The source labels select values from existing + labels. Their content is concatenated using the configured + separator and matched against the configured regular + expression for the replace, keep, and drop actions. + items: + description: LabelName is a valid Prometheus label name + which may only contain ASCII letters, numbers, as + well as underscores. + pattern: ^[a-zA-Z_][a-zA-Z0-9_]*$ + type: string + type: array + targetLabel: + description: Label to which the resulting value is written + in a replace action. It is mandatory for replace actions. + Regex capture groups are available. + type: string + type: object + type: array + oauth2: + description: OAuth2 for the URL. Only valid in Prometheus versions + 2.27.0 and newer. + properties: + clientId: + description: The secret or configmap containing the OAuth2 + client id + properties: + configMap: + description: ConfigMap containing data to use for the + targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the ConfigMap or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for the targets. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + clientSecret: + description: The secret containing the OAuth2 client secret + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + endpointParams: + additionalProperties: + type: string + description: Parameters to append to the token URL + type: object + scopes: + description: OAuth2 scopes used for the token request + items: + type: string + type: array + tokenUrl: + description: The URL to fetch the token from + minLength: 1 + type: string + required: + - clientId + - clientSecret + - tokenUrl + type: object + params: + additionalProperties: + items: + type: string + type: array + description: Optional HTTP URL parameters + type: object + path: + description: HTTP path to scrape for metrics. If empty, Prometheus + uses the default value (e.g. `/metrics`). + type: string + port: + description: Name of the pod port this endpoint refers to. Mutually + exclusive with targetPort. + type: string + proxyUrl: + description: ProxyURL eg http://proxyserver:2195 Directs scrapes + to proxy through this endpoint. + type: string + relabelings: + description: 'RelabelConfigs to apply to samples before scraping. + Prometheus Operator automatically adds relabelings for a few + standard Kubernetes fields. The original scrape job''s name + is available via the `__tmp_prometheus_job_name` label. More + info: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config' + items: + description: 'RelabelConfig allows dynamic rewriting of the + label set, being applied to samples before ingestion. It + defines ``-section of Prometheus + configuration. More info: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs' + properties: + action: + default: replace + description: Action to perform based on regex matching. + Default is 'replace'. uppercase and lowercase actions + require Prometheus >= 2.36. + enum: + - replace + - Replace + - keep + - Keep + - drop + - Drop + - hashmod + - HashMod + - labelmap + - LabelMap + - labeldrop + - LabelDrop + - labelkeep + - LabelKeep + - lowercase + - Lowercase + - uppercase + - Uppercase + type: string + modulus: + description: Modulus to take of the hash of the source + label values. + format: int64 + type: integer + regex: + description: Regular expression against which the extracted + value is matched. Default is '(.*)' + type: string + replacement: + description: Replacement value against which a regex replace + is performed if the regular expression matches. Regex + capture groups are available. Default is '$1' + type: string + separator: + description: Separator placed between concatenated source + label values. default is ';'. + type: string + sourceLabels: + description: The source labels select values from existing + labels. Their content is concatenated using the configured + separator and matched against the configured regular + expression for the replace, keep, and drop actions. + items: + description: LabelName is a valid Prometheus label name + which may only contain ASCII letters, numbers, as + well as underscores. + pattern: ^[a-zA-Z_][a-zA-Z0-9_]*$ + type: string + type: array + targetLabel: + description: Label to which the resulting value is written + in a replace action. It is mandatory for replace actions. + Regex capture groups are available. + type: string + type: object + type: array + scheme: + description: HTTP scheme to use for scraping. + type: string + scrapeTimeout: + description: Timeout after which the scrape is ended If not + specified, the Prometheus global scrape interval is used. + pattern: ^(0|(([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?)$ + type: string + targetPort: + anyOf: + - type: integer + - type: string + description: 'Deprecated: Use ''port'' instead.' + x-kubernetes-int-or-string: true + tlsConfig: + description: TLS configuration to use when scraping the endpoint. + properties: + ca: + description: Certificate authority used when verifying server + certificates. + properties: + configMap: + description: ConfigMap containing data to use for the + targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the ConfigMap or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for the targets. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + cert: + description: Client certificate to present when doing client-authentication. + properties: + configMap: + description: ConfigMap containing data to use for the + targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the ConfigMap or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for the targets. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + insecureSkipVerify: + description: Disable target certificate validation. + type: boolean + keySecret: + description: Secret containing the client key file for the + targets. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + serverName: + description: Used to verify the hostname for the targets. + type: string + type: object + type: object + type: array + podTargetLabels: + description: PodTargetLabels transfers labels on the Kubernetes Pod + onto the target. + items: + type: string + type: array + sampleLimit: + description: SampleLimit defines per-scrape limit on number of scraped + samples that will be accepted. + format: int64 + type: integer + selector: + description: Selector to select Pod objects. + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. + The requirements are ANDed. + items: + description: A label selector requirement is a selector that + contains values, a key, and an operator that relates the key + and values. + properties: + key: + description: key is the label key that the selector applies + to. + type: string + operator: + description: operator represents a key's relationship to + a set of values. Valid operators are In, NotIn, Exists + and DoesNotExist. + type: string + values: + description: values is an array of string values. If the + operator is In or NotIn, the values array must be non-empty. + If the operator is Exists or DoesNotExist, the values + array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single + {key,value} in the matchLabels map is equivalent to an element + of matchExpressions, whose key field is "key", the operator + is "In", and the values array contains only "value". The requirements + are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + targetLimit: + description: TargetLimit defines a limit on the number of scraped + targets that will be accepted. + format: int64 + type: integer + required: + - podMetricsEndpoints + - selector + type: object + required: + - spec + type: object + served: true + storage: true diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-probes.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-probes.yaml new file mode 100644 index 00000000..d82abd8f --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-probes.yaml @@ -0,0 +1,705 @@ +# https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.61.1/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.9.2 + creationTimestamp: null + name: probes.monitoring.coreos.com +spec: + group: monitoring.coreos.com + names: + categories: + - prometheus-operator + kind: Probe + listKind: ProbeList + plural: probes + shortNames: + - prb + singular: probe + scope: Namespaced + versions: + - name: v1 + schema: + openAPIV3Schema: + description: Probe defines monitoring for a set of static targets or ingresses. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: Specification of desired Ingress selection for target discovery + by Prometheus. + properties: + authorization: + description: Authorization section for this endpoint + properties: + credentials: + description: The secret's key that contains the credentials of + the request + properties: + key: + description: The key of the secret to select from. Must be + a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must be + defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: + description: Set the authentication type. Defaults to Bearer, + Basic will cause an error + type: string + type: object + basicAuth: + description: 'BasicAuth allow an endpoint to authenticate over basic + authentication. More info: https://prometheus.io/docs/operating/configuration/#endpoint' + properties: + password: + description: The secret in the service monitor namespace that + contains the password for authentication. + properties: + key: + description: The key of the secret to select from. Must be + a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must be + defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + username: + description: The secret in the service monitor namespace that + contains the username for authentication. + properties: + key: + description: The key of the secret to select from. Must be + a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must be + defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + bearerTokenSecret: + description: Secret to mount to read bearer token for scraping targets. + The secret needs to be in the same namespace as the probe and accessible + by the Prometheus Operator. + properties: + key: + description: The key of the secret to select from. Must be a + valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + interval: + description: Interval at which targets are probed using the configured + prober. If not specified Prometheus' global scrape interval is used. + pattern: ^(0|(([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?)$ + type: string + jobName: + description: The job name assigned to scraped metrics by default. + type: string + labelLimit: + description: Per-scrape limit on number of labels that will be accepted + for a sample. Only valid in Prometheus versions 2.27.0 and newer. + format: int64 + type: integer + labelNameLengthLimit: + description: Per-scrape limit on length of labels name that will be + accepted for a sample. Only valid in Prometheus versions 2.27.0 + and newer. + format: int64 + type: integer + labelValueLengthLimit: + description: Per-scrape limit on length of labels value that will + be accepted for a sample. Only valid in Prometheus versions 2.27.0 + and newer. + format: int64 + type: integer + metricRelabelings: + description: MetricRelabelConfigs to apply to samples before ingestion. + items: + description: 'RelabelConfig allows dynamic rewriting of the label + set, being applied to samples before ingestion. It defines ``-section + of Prometheus configuration. More info: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs' + properties: + action: + default: replace + description: Action to perform based on regex matching. Default + is 'replace'. uppercase and lowercase actions require Prometheus + >= 2.36. + enum: + - replace + - Replace + - keep + - Keep + - drop + - Drop + - hashmod + - HashMod + - labelmap + - LabelMap + - labeldrop + - LabelDrop + - labelkeep + - LabelKeep + - lowercase + - Lowercase + - uppercase + - Uppercase + type: string + modulus: + description: Modulus to take of the hash of the source label + values. + format: int64 + type: integer + regex: + description: Regular expression against which the extracted + value is matched. Default is '(.*)' + type: string + replacement: + description: Replacement value against which a regex replace + is performed if the regular expression matches. Regex capture + groups are available. Default is '$1' + type: string + separator: + description: Separator placed between concatenated source label + values. default is ';'. + type: string + sourceLabels: + description: The source labels select values from existing labels. + Their content is concatenated using the configured separator + and matched against the configured regular expression for + the replace, keep, and drop actions. + items: + description: LabelName is a valid Prometheus label name which + may only contain ASCII letters, numbers, as well as underscores. + pattern: ^[a-zA-Z_][a-zA-Z0-9_]*$ + type: string + type: array + targetLabel: + description: Label to which the resulting value is written in + a replace action. It is mandatory for replace actions. Regex + capture groups are available. + type: string + type: object + type: array + module: + description: 'The module to use for probing specifying how to probe + the target. Example module configuring in the blackbox exporter: + https://github.com/prometheus/blackbox_exporter/blob/master/example.yml' + type: string + oauth2: + description: OAuth2 for the URL. Only valid in Prometheus versions + 2.27.0 and newer. + properties: + clientId: + description: The secret or configmap containing the OAuth2 client + id + properties: + configMap: + description: ConfigMap containing data to use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap or its key + must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for the targets. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + clientSecret: + description: The secret containing the OAuth2 client secret + properties: + key: + description: The key of the secret to select from. Must be + a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must be + defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + endpointParams: + additionalProperties: + type: string + description: Parameters to append to the token URL + type: object + scopes: + description: OAuth2 scopes used for the token request + items: + type: string + type: array + tokenUrl: + description: The URL to fetch the token from + minLength: 1 + type: string + required: + - clientId + - clientSecret + - tokenUrl + type: object + prober: + description: Specification for the prober to use for probing targets. + The prober.URL parameter is required. Targets cannot be probed if + left empty. + properties: + path: + default: /probe + description: Path to collect metrics from. Defaults to `/probe`. + type: string + proxyUrl: + description: Optional ProxyURL. + type: string + scheme: + description: HTTP scheme to use for scraping. Defaults to `http`. + type: string + url: + description: Mandatory URL of the prober. + type: string + required: + - url + type: object + sampleLimit: + description: SampleLimit defines per-scrape limit on number of scraped + samples that will be accepted. + format: int64 + type: integer + scrapeTimeout: + description: Timeout for scraping metrics from the Prometheus exporter. + If not specified, the Prometheus global scrape interval is used. + pattern: ^(0|(([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?)$ + type: string + targetLimit: + description: TargetLimit defines a limit on the number of scraped + targets that will be accepted. + format: int64 + type: integer + targets: + description: Targets defines a set of static or dynamically discovered + targets to probe. + properties: + ingress: + description: ingress defines the Ingress objects to probe and + the relabeling configuration. If `staticConfig` is also defined, + `staticConfig` takes precedence. + properties: + namespaceSelector: + description: From which namespaces to select Ingress objects. + properties: + any: + description: Boolean describing whether all namespaces + are selected in contrast to a list restricting them. + type: boolean + matchNames: + description: List of namespace names to select from. + items: + type: string + type: array + type: object + relabelingConfigs: + description: 'RelabelConfigs to apply to the label set of + the target before it gets scraped. The original ingress + address is available via the `__tmp_prometheus_ingress_address` + label. It can be used to customize the probed URL. The original + scrape job''s name is available via the `__tmp_prometheus_job_name` + label. More info: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config' + items: + description: 'RelabelConfig allows dynamic rewriting of + the label set, being applied to samples before ingestion. + It defines ``-section of Prometheus + configuration. More info: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs' + properties: + action: + default: replace + description: Action to perform based on regex matching. + Default is 'replace'. uppercase and lowercase actions + require Prometheus >= 2.36. + enum: + - replace + - Replace + - keep + - Keep + - drop + - Drop + - hashmod + - HashMod + - labelmap + - LabelMap + - labeldrop + - LabelDrop + - labelkeep + - LabelKeep + - lowercase + - Lowercase + - uppercase + - Uppercase + type: string + modulus: + description: Modulus to take of the hash of the source + label values. + format: int64 + type: integer + regex: + description: Regular expression against which the extracted + value is matched. Default is '(.*)' + type: string + replacement: + description: Replacement value against which a regex + replace is performed if the regular expression matches. + Regex capture groups are available. Default is '$1' + type: string + separator: + description: Separator placed between concatenated source + label values. default is ';'. + type: string + sourceLabels: + description: The source labels select values from existing + labels. Their content is concatenated using the configured + separator and matched against the configured regular + expression for the replace, keep, and drop actions. + items: + description: LabelName is a valid Prometheus label + name which may only contain ASCII letters, numbers, + as well as underscores. + pattern: ^[a-zA-Z_][a-zA-Z0-9_]*$ + type: string + type: array + targetLabel: + description: Label to which the resulting value is written + in a replace action. It is mandatory for replace actions. + Regex capture groups are available. + type: string + type: object + type: array + selector: + description: Selector to select the Ingress objects. + properties: + matchExpressions: + description: matchExpressions is a list of label selector + requirements. The requirements are ANDed. + items: + description: A label selector requirement is a selector + that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that the selector + applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are In, NotIn, + Exists and DoesNotExist. + type: string + values: + description: values is an array of string values. + If the operator is In or NotIn, the values array + must be non-empty. If the operator is Exists or + DoesNotExist, the values array must be empty. + This array is replaced during a strategic merge + patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. + A single {key,value} in the matchLabels map is equivalent + to an element of matchExpressions, whose key field is + "key", the operator is "In", and the values array contains + only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + type: object + staticConfig: + description: 'staticConfig defines the static list of targets + to probe and the relabeling configuration. If `ingress` is also + defined, `staticConfig` takes precedence. More info: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#static_config.' + properties: + labels: + additionalProperties: + type: string + description: Labels assigned to all metrics scraped from the + targets. + type: object + relabelingConfigs: + description: 'RelabelConfigs to apply to the label set of + the targets before it gets scraped. More info: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config' + items: + description: 'RelabelConfig allows dynamic rewriting of + the label set, being applied to samples before ingestion. + It defines ``-section of Prometheus + configuration. More info: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs' + properties: + action: + default: replace + description: Action to perform based on regex matching. + Default is 'replace'. uppercase and lowercase actions + require Prometheus >= 2.36. + enum: + - replace + - Replace + - keep + - Keep + - drop + - Drop + - hashmod + - HashMod + - labelmap + - LabelMap + - labeldrop + - LabelDrop + - labelkeep + - LabelKeep + - lowercase + - Lowercase + - uppercase + - Uppercase + type: string + modulus: + description: Modulus to take of the hash of the source + label values. + format: int64 + type: integer + regex: + description: Regular expression against which the extracted + value is matched. Default is '(.*)' + type: string + replacement: + description: Replacement value against which a regex + replace is performed if the regular expression matches. + Regex capture groups are available. Default is '$1' + type: string + separator: + description: Separator placed between concatenated source + label values. default is ';'. + type: string + sourceLabels: + description: The source labels select values from existing + labels. Their content is concatenated using the configured + separator and matched against the configured regular + expression for the replace, keep, and drop actions. + items: + description: LabelName is a valid Prometheus label + name which may only contain ASCII letters, numbers, + as well as underscores. + pattern: ^[a-zA-Z_][a-zA-Z0-9_]*$ + type: string + type: array + targetLabel: + description: Label to which the resulting value is written + in a replace action. It is mandatory for replace actions. + Regex capture groups are available. + type: string + type: object + type: array + static: + description: The list of hosts to probe. + items: + type: string + type: array + type: object + type: object + tlsConfig: + description: TLS configuration to use when scraping the endpoint. + properties: + ca: + description: Certificate authority used when verifying server + certificates. + properties: + configMap: + description: ConfigMap containing data to use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap or its key + must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for the targets. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + cert: + description: Client certificate to present when doing client-authentication. + properties: + configMap: + description: ConfigMap containing data to use for the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap or its key + must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for the targets. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + insecureSkipVerify: + description: Disable target certificate validation. + type: boolean + keySecret: + description: Secret containing the client key file for the targets. + properties: + key: + description: The key of the secret to select from. Must be + a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must be + defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + serverName: + description: Used to verify the hostname for the targets. + type: string + type: object + type: object + required: + - spec + type: object + served: true + storage: true diff --git a/manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-prometheuses.yaml b/manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-prometheuses.yaml new file mode 100644 index 00000000..e908b4f0 --- /dev/null +++ b/manifests/observability/prometheus/charts/kube-prometheus-stack/crds/crd-prometheuses.yaml @@ -0,0 +1,8865 @@ +# https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.61.1/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.9.2 + creationTimestamp: null + name: prometheuses.monitoring.coreos.com +spec: + group: monitoring.coreos.com + names: + categories: + - prometheus-operator + kind: Prometheus + listKind: PrometheusList + plural: prometheuses + shortNames: + - prom + singular: prometheus + scope: Namespaced + versions: + - additionalPrinterColumns: + - description: The version of Prometheus + jsonPath: .spec.version + name: Version + type: string + - description: The number of desired replicas + jsonPath: .spec.replicas + name: Desired + type: integer + - description: The number of ready replicas + jsonPath: .status.availableReplicas + name: Ready + type: integer + - jsonPath: .status.conditions[?(@.type == 'Reconciled')].status + name: Reconciled + type: string + - jsonPath: .status.conditions[?(@.type == 'Available')].status + name: Available + type: string + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + - description: Whether the resource reconciliation is paused or not + jsonPath: .status.paused + name: Paused + priority: 1 + type: boolean + name: v1 + schema: + openAPIV3Schema: + description: Prometheus defines a Prometheus deployment. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: 'Specification of the desired behavior of the Prometheus + cluster. More info: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#spec-and-status' + properties: + additionalAlertManagerConfigs: + description: 'AdditionalAlertManagerConfigs allows specifying a key + of a Secret containing additional Prometheus AlertManager configurations. + AlertManager configurations specified are appended to the configurations + generated by the Prometheus Operator. Job configurations specified + must have the form as specified in the official Prometheus documentation: + https://prometheus.io/docs/prometheus/latest/configuration/configuration/#alertmanager_config. + As AlertManager configs are appended, the user is responsible to + make sure it is valid. Note that using this feature may expose the + possibility to break upgrades of Prometheus. It is advised to review + Prometheus release notes to ensure that no incompatible AlertManager + configs are going to break Prometheus after the upgrade.' + properties: + key: + description: The key of the secret to select from. Must be a + valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + additionalAlertRelabelConfigs: + description: 'AdditionalAlertRelabelConfigs allows specifying a key + of a Secret containing additional Prometheus alert relabel configurations. + Alert relabel configurations specified are appended to the configurations + generated by the Prometheus Operator. Alert relabel configurations + specified must have the form as specified in the official Prometheus + documentation: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#alert_relabel_configs. + As alert relabel configs are appended, the user is responsible to + make sure it is valid. Note that using this feature may expose the + possibility to break upgrades of Prometheus. It is advised to review + Prometheus release notes to ensure that no incompatible alert relabel + configs are going to break Prometheus after the upgrade.' + properties: + key: + description: The key of the secret to select from. Must be a + valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + additionalArgs: + description: AdditionalArgs allows setting additional arguments for + the Prometheus container. It is intended for e.g. activating hidden + flags which are not supported by the dedicated configuration options + yet. The arguments are passed as-is to the Prometheus container + which may cause issues if they are invalid or not supported by the + given Prometheus version. In case of an argument conflict (e.g. + an argument which is already set by the operator itself) or when + providing an invalid argument the reconciliation will fail and an + error will be logged. + items: + description: Argument as part of the AdditionalArgs list. + properties: + name: + description: Name of the argument, e.g. "scrape.discovery-reload-interval". + minLength: 1 + type: string + value: + description: Argument value, e.g. 30s. Can be empty for name-only + arguments (e.g. --storage.tsdb.no-lockfile) + type: string + required: + - name + type: object + type: array + additionalScrapeConfigs: + description: 'AdditionalScrapeConfigs allows specifying a key of a + Secret containing additional Prometheus scrape configurations. Scrape + configurations specified are appended to the configurations generated + by the Prometheus Operator. Job configurations specified must have + the form as specified in the official Prometheus documentation: + https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config. + As scrape configs are appended, the user is responsible to make + sure it is valid. Note that using this feature may expose the possibility + to break upgrades of Prometheus. It is advised to review Prometheus + release notes to ensure that no incompatible scrape configs are + going to break Prometheus after the upgrade.' + properties: + key: + description: The key of the secret to select from. Must be a + valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + affinity: + description: If specified, the pod's scheduling constraints. + properties: + nodeAffinity: + description: Describes node affinity scheduling rules for the + pod. + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: The scheduler will prefer to schedule pods to + nodes that satisfy the affinity expressions specified by + this field, but it may choose a node that violates one or + more of the expressions. The node that is most preferred + is the one with the greatest sum of weights, i.e. for each + node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling affinity expressions, + etc.), compute a sum by iterating through the elements of + this field and adding "weight" to the sum if the node matches + the corresponding matchExpressions; the node(s) with the + highest sum are the most preferred. + items: + description: An empty preferred scheduling term matches + all objects with implicit weight 0 (i.e. it's a no-op). + A null preferred scheduling term matches no objects (i.e. + is also a no-op). + properties: + preference: + description: A node selector term, associated with the + corresponding weight. + properties: + matchExpressions: + description: A list of node selector requirements + by node's labels. + items: + description: A node selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: Represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists, DoesNotExist. Gt, and + Lt. + type: string + values: + description: An array of string values. If + the operator is In or NotIn, the values + array must be non-empty. If the operator + is Exists or DoesNotExist, the values array + must be empty. If the operator is Gt or + Lt, the values array must have a single + element, which will be interpreted as an + integer. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchFields: + description: A list of node selector requirements + by node's fields. + items: + description: A node selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: Represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists, DoesNotExist. Gt, and + Lt. + type: string + values: + description: An array of string values. If + the operator is In or NotIn, the values + array must be non-empty. If the operator + is Exists or DoesNotExist, the values array + must be empty. If the operator is Gt or + Lt, the values array must have a single + element, which will be interpreted as an + integer. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + type: object + x-kubernetes-map-type: atomic + weight: + description: Weight associated with matching the corresponding + nodeSelectorTerm, in the range 1-100. + format: int32 + type: integer + required: + - preference + - weight + type: object + type: array + requiredDuringSchedulingIgnoredDuringExecution: + description: If the affinity requirements specified by this + field are not met at scheduling time, the pod will not be + scheduled onto the node. If the affinity requirements specified + by this field cease to be met at some point during pod execution + (e.g. due to an update), the system may or may not try to + eventually evict the pod from its node. + properties: + nodeSelectorTerms: + description: Required. A list of node selector terms. + The terms are ORed. + items: + description: A null or empty node selector term matches + no objects. The requirements of them are ANDed. The + TopologySelectorTerm type implements a subset of the + NodeSelectorTerm. + properties: + matchExpressions: + description: A list of node selector requirements + by node's labels. + items: + description: A node selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: Represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists, DoesNotExist. Gt, and + Lt. + type: string + values: + description: An array of string values. If + the operator is In or NotIn, the values + array must be non-empty. If the operator + is Exists or DoesNotExist, the values array + must be empty. If the operator is Gt or + Lt, the values array must have a single + element, which will be interpreted as an + integer. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchFields: + description: A list of node selector requirements + by node's fields. + items: + description: A node selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: Represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists, DoesNotExist. Gt, and + Lt. + type: string + values: + description: An array of string values. If + the operator is In or NotIn, the values + array must be non-empty. If the operator + is Exists or DoesNotExist, the values array + must be empty. If the operator is Gt or + Lt, the values array must have a single + element, which will be interpreted as an + integer. This array is replaced during a + strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + type: object + x-kubernetes-map-type: atomic + type: array + required: + - nodeSelectorTerms + type: object + x-kubernetes-map-type: atomic + type: object + podAffinity: + description: Describes pod affinity scheduling rules (e.g. co-locate + this pod in the same node, zone, etc. as some other pod(s)). + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: The scheduler will prefer to schedule pods to + nodes that satisfy the affinity expressions specified by + this field, but it may choose a node that violates one or + more of the expressions. The node that is most preferred + is the one with the greatest sum of weights, i.e. for each + node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling affinity expressions, + etc.), compute a sum by iterating through the elements of + this field and adding "weight" to the sum if the node has + pods which matches the corresponding podAffinityTerm; the + node(s) with the highest sum are the most preferred. + items: + description: The weights of all of the matched WeightedPodAffinityTerm + fields are added per-node to find the most preferred node(s) + properties: + podAffinityTerm: + description: Required. A pod affinity term, associated + with the corresponding weight. + properties: + labelSelector: + description: A label query over a set of resources, + in this case pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaceSelector: + description: A label query over the set of namespaces + that the term applies to. The term is applied + to the union of the namespaces selected by this + field and the ones listed in the namespaces field. + null selector and null or empty namespaces list + means "this pod's namespace". An empty selector + ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: namespaces specifies a static list + of namespace names that the term applies to. The + term is applied to the union of the namespaces + listed in this field and the ones selected by + namespaceSelector. null or empty namespaces list + and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) + or not co-located (anti-affinity) with the pods + matching the labelSelector in the specified namespaces, + where co-located is defined as running on a node + whose value of the label with key topologyKey + matches that of any node on which any of the selected + pods is running. Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + weight: + description: weight associated with matching the corresponding + podAffinityTerm, in the range 1-100. + format: int32 + type: integer + required: + - podAffinityTerm + - weight + type: object + type: array + requiredDuringSchedulingIgnoredDuringExecution: + description: If the affinity requirements specified by this + field are not met at scheduling time, the pod will not be + scheduled onto the node. If the affinity requirements specified + by this field cease to be met at some point during pod execution + (e.g. due to a pod label update), the system may or may + not try to eventually evict the pod from its node. When + there are multiple elements, the lists of nodes corresponding + to each podAffinityTerm are intersected, i.e. all terms + must be satisfied. + items: + description: Defines a set of pods (namely those matching + the labelSelector relative to the given namespace(s)) + that this pod should be co-located (affinity) or not co-located + (anti-affinity) with, where co-located is defined as running + on a node whose value of the label with key + matches that of any node on which a pod of the set of + pods is running + properties: + labelSelector: + description: A label query over a set of resources, + in this case pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaceSelector: + description: A label query over the set of namespaces + that the term applies to. The term is applied to the + union of the namespaces selected by this field and + the ones listed in the namespaces field. null selector + and null or empty namespaces list means "this pod's + namespace". An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: namespaces specifies a static list of namespace + names that the term applies to. The term is applied + to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. null or + empty namespaces list and null namespaceSelector means + "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) + or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where + co-located is defined as running on a node whose value + of the label with key topologyKey matches that of + any node on which any of the selected pods is running. + Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + type: array + type: object + podAntiAffinity: + description: Describes pod anti-affinity scheduling rules (e.g. + avoid putting this pod in the same node, zone, etc. as some + other pod(s)). + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: The scheduler will prefer to schedule pods to + nodes that satisfy the anti-affinity expressions specified + by this field, but it may choose a node that violates one + or more of the expressions. The node that is most preferred + is the one with the greatest sum of weights, i.e. for each + node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling anti-affinity expressions, + etc.), compute a sum by iterating through the elements of + this field and adding "weight" to the sum if the node has + pods which matches the corresponding podAffinityTerm; the + node(s) with the highest sum are the most preferred. + items: + description: The weights of all of the matched WeightedPodAffinityTerm + fields are added per-node to find the most preferred node(s) + properties: + podAffinityTerm: + description: Required. A pod affinity term, associated + with the corresponding weight. + properties: + labelSelector: + description: A label query over a set of resources, + in this case pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaceSelector: + description: A label query over the set of namespaces + that the term applies to. The term is applied + to the union of the namespaces selected by this + field and the ones listed in the namespaces field. + null selector and null or empty namespaces list + means "this pod's namespace". An empty selector + ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: A label selector requirement + is a selector that contains values, a key, + and an operator that relates the key and + values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: operator represents a key's + relationship to a set of values. Valid + operators are In, NotIn, Exists and + DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. + If the operator is Exists or DoesNotExist, + the values array must be empty. This + array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is + "In", and the values array contains only "value". + The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: namespaces specifies a static list + of namespace names that the term applies to. The + term is applied to the union of the namespaces + listed in this field and the ones selected by + namespaceSelector. null or empty namespaces list + and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) + or not co-located (anti-affinity) with the pods + matching the labelSelector in the specified namespaces, + where co-located is defined as running on a node + whose value of the label with key topologyKey + matches that of any node on which any of the selected + pods is running. Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + weight: + description: weight associated with matching the corresponding + podAffinityTerm, in the range 1-100. + format: int32 + type: integer + required: + - podAffinityTerm + - weight + type: object + type: array + requiredDuringSchedulingIgnoredDuringExecution: + description: If the anti-affinity requirements specified by + this field are not met at scheduling time, the pod will + not be scheduled onto the node. If the anti-affinity requirements + specified by this field cease to be met at some point during + pod execution (e.g. due to a pod label update), the system + may or may not try to eventually evict the pod from its + node. When there are multiple elements, the lists of nodes + corresponding to each podAffinityTerm are intersected, i.e. + all terms must be satisfied. + items: + description: Defines a set of pods (namely those matching + the labelSelector relative to the given namespace(s)) + that this pod should be co-located (affinity) or not co-located + (anti-affinity) with, where co-located is defined as running + on a node whose value of the label with key + matches that of any node on which a pod of the set of + pods is running + properties: + labelSelector: + description: A label query over a set of resources, + in this case pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaceSelector: + description: A label query over the set of namespaces + that the term applies to. The term is applied to the + union of the namespaces selected by this field and + the ones listed in the namespaces field. null selector + and null or empty namespaces list means "this pod's + namespace". An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: namespaces specifies a static list of namespace + names that the term applies to. The term is applied + to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. null or + empty namespaces list and null namespaceSelector means + "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) + or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where + co-located is defined as running on a node whose value + of the label with key topologyKey matches that of + any node on which any of the selected pods is running. + Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + type: array + type: object + type: object + alerting: + description: Define details regarding alerting. + properties: + alertmanagers: + description: AlertmanagerEndpoints Prometheus should fire alerts + against. + items: + description: AlertmanagerEndpoints defines a selection of a + single Endpoints object containing alertmanager IPs to fire + alerts against. + properties: + apiVersion: + description: Version of the Alertmanager API that Prometheus + uses to send alerts. It can be "v1" or "v2". + type: string + authorization: + description: Authorization section for this alertmanager + endpoint + properties: + credentials: + description: The secret's key that contains the credentials + of the request + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: + description: Set the authentication type. Defaults to + Bearer, Basic will cause an error + type: string + type: object + bearerTokenFile: + description: BearerTokenFile to read from filesystem to + use when authenticating to Alertmanager. + type: string + enableHttp2: + description: Whether to enable HTTP2. + type: boolean + name: + description: Name of Endpoints object in Namespace. + type: string + namespace: + description: Namespace of Endpoints object. + type: string + pathPrefix: + description: Prefix for the HTTP path alerts are pushed + to. + type: string + port: + anyOf: + - type: integer + - type: string + description: Port the Alertmanager API is exposed on. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use when firing alerts. + type: string + timeout: + description: Timeout is a per-target Alertmanager timeout + when pushing alerts. + pattern: ^(0|(([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?)$ + type: string + tlsConfig: + description: TLS Config to use for alertmanager connection. + properties: + ca: + description: Certificate authority used when verifying + server certificates. + properties: + configMap: + description: ConfigMap containing data to use for + the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for the + targets. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + caFile: + description: Path to the CA cert in the Prometheus container + to use for the targets. + type: string + cert: + description: Client certificate to present when doing + client-authentication. + properties: + configMap: + description: ConfigMap containing data to use for + the targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the ConfigMap or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for the + targets. + properties: + key: + description: The key of the secret to select + from. Must be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, + kind, uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + certFile: + description: Path to the client cert file in the Prometheus + container for the targets. + type: string + insecureSkipVerify: + description: Disable target certificate validation. + type: boolean + keyFile: + description: Path to the client key file in the Prometheus + container for the targets. + type: string + keySecret: + description: Secret containing the client key file for + the targets. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + serverName: + description: Used to verify the hostname for the targets. + type: string + type: object + required: + - name + - namespace + - port + type: object + type: array + required: + - alertmanagers + type: object + allowOverlappingBlocks: + description: AllowOverlappingBlocks enables vertical compaction and + vertical query merge in Prometheus. This is still experimental in + Prometheus so it may change in any upcoming release. + type: boolean + apiserverConfig: + description: APIServerConfig allows specifying a host and auth methods + to access apiserver. If left empty, Prometheus is assumed to run + inside of the cluster and will discover API servers automatically + and use the pod's CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount/. + properties: + authorization: + description: Authorization section for accessing apiserver + properties: + credentials: + description: The secret's key that contains the credentials + of the request + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + credentialsFile: + description: File to read a secret from, mutually exclusive + with Credentials (from SafeAuthorization) + type: string + type: + description: Set the authentication type. Defaults to Bearer, + Basic will cause an error + type: string + type: object + basicAuth: + description: BasicAuth allow an endpoint to authenticate over + basic authentication + properties: + password: + description: The secret in the service monitor namespace that + contains the password for authentication. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + username: + description: The secret in the service monitor namespace that + contains the username for authentication. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + bearerToken: + description: Bearer token for accessing apiserver. + type: string + bearerTokenFile: + description: File to read bearer token for accessing apiserver. + type: string + host: + description: Host of apiserver. A valid string consisting of a + hostname or IP followed by an optional port number + type: string + tlsConfig: + description: TLS Config to use for accessing apiserver. + properties: + ca: + description: Certificate authority used when verifying server + certificates. + properties: + configMap: + description: ConfigMap containing data to use for the + targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the ConfigMap or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for the targets. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + caFile: + description: Path to the CA cert in the Prometheus container + to use for the targets. + type: string + cert: + description: Client certificate to present when doing client-authentication. + properties: + configMap: + description: ConfigMap containing data to use for the + targets. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the ConfigMap or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + secret: + description: Secret containing data to use for the targets. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its key + must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + certFile: + description: Path to the client cert file in the Prometheus + container for the targets. + type: string + insecureSkipVerify: + description: Disable target certificate validation. + type: boolean + keyFile: + description: Path to the client key file in the Prometheus + container for the targets. + type: string + keySecret: + description: Secret containing the client key file for the + targets. + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + optional: + description: Specify whether the Secret or its key must + be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + serverName: + description: Used to verify the hostname for the targets. + type: string + type: object + required: + - host + type: object + arbitraryFSAccessThroughSMs: + description: ArbitraryFSAccessThroughSMs configures whether configuration + based on a service monitor can access arbitrary files on the file + system of the Prometheus container e.g. bearer token files. + properties: + deny: + type: boolean + type: object + baseImage: + description: 'Base image to use for a Prometheus deployment. Deprecated: + use ''image'' instead' + type: string + configMaps: + description: ConfigMaps is a list of ConfigMaps in the same namespace + as the Prometheus object, which shall be mounted into the Prometheus + Pods. Each ConfigMap is added to the StatefulSet definition as a + volume named `configmap-`. The ConfigMaps are mounted + into /etc/prometheus/configmaps/ in the 'prometheus' + container. + items: + type: string + type: array + containers: + description: 'Containers allows injecting additional containers or + modifying operator generated containers. This can be used to allow + adding an authentication proxy to a Prometheus pod or to change + the behavior of an operator generated container. Containers described + here modify an operator generated container if they share the same + name and modifications are done via a strategic merge patch. The + current container names are: `prometheus`, `config-reloader`, and + `thanos-sidecar`. Overriding containers is entirely outside the + scope of what the maintainers will support and by doing so, you + accept that this behaviour may break at any time without notice.' + items: + description: A single application container that you want to run + within a pod. + properties: + args: + description: 'Arguments to the entrypoint. The container image''s + CMD is used if this is not provided. Variable references $(VAR_NAME) + are expanded using the container''s environment. If a variable + cannot be resolved, the reference in the input string will + be unchanged. Double $$ are reduced to a single $, which allows + for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will + produce the string literal "$(VAR_NAME)". Escaped references + will never be expanded, regardless of whether the variable + exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell' + items: + type: string + type: array + command: + description: 'Entrypoint array. Not executed within a shell. + The container image''s ENTRYPOINT is used if this is not provided. + Variable references $(VAR_NAME) are expanded using the container''s + environment. If a variable cannot be resolved, the reference + in the input string will be unchanged. Double $$ are reduced + to a single $, which allows for escaping the $(VAR_NAME) syntax: + i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". + Escaped references will never be expanded, regardless of whether + the variable exists or not. Cannot be updated. More info: + https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell' + items: + type: string + type: array + env: + description: List of environment variables to set in the container. + Cannot be updated. + items: + description: EnvVar represents an environment variable present + in a Container. + properties: + name: + description: Name of the environment variable. Must be + a C_IDENTIFIER. + type: string + value: + description: 'Variable references $(VAR_NAME) are expanded + using the previously defined environment variables in + the container and any service environment variables. + If a variable cannot be resolved, the reference in the + input string will be unchanged. Double $$ are reduced + to a single $, which allows for escaping the $(VAR_NAME) + syntax: i.e. "$$(VAR_NAME)" will produce the string + literal "$(VAR_NAME)". Escaped references will never + be expanded, regardless of whether the variable exists + or not. Defaults to "".' + type: string + valueFrom: + description: Source for the environment variable's value. + Cannot be used if value is not empty. + properties: + configMapKeyRef: + description: Selects a key of a ConfigMap. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the ConfigMap or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + fieldRef: + description: 'Selects a field of the pod: supports + metadata.name, metadata.namespace, `metadata.labels['''']`, + `metadata.annotations['''']`, spec.nodeName, + spec.serviceAccountName, status.hostIP, status.podIP, + status.podIPs.' + properties: + apiVersion: + description: Version of the schema the FieldPath + is written in terms of, defaults to "v1". + type: string + fieldPath: + description: Path of the field to select in the + specified API version. + type: string + required: + - fieldPath + type: object + x-kubernetes-map-type: atomic + resourceFieldRef: + description: 'Selects a resource of the container: + only resources limits and requests (limits.cpu, + limits.memory, limits.ephemeral-storage, requests.cpu, + requests.memory and requests.ephemeral-storage) + are currently supported.' + properties: + containerName: + description: 'Container name: required for volumes, + optional for env vars' + type: string + divisor: + anyOf: + - type: integer + - type: string + description: Specifies the output format of the + exposed resources, defaults to "1" + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + resource: + description: 'Required: resource to select' + type: string + required: + - resource + type: object + x-kubernetes-map-type: atomic + secretKeyRef: + description: Selects a key of a secret in the pod's + namespace + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + required: + - name + type: object + type: array + envFrom: + description: List of sources to populate environment variables + in the container. The keys defined within a source must be + a C_IDENTIFIER. All invalid keys will be reported as an event + when the container is starting. When a key exists in multiple + sources, the value associated with the last source will take + precedence. Values defined by an Env with a duplicate key + will take precedence. Cannot be updated. + items: + description: EnvFromSource represents the source of a set + of ConfigMaps + properties: + configMapRef: + description: The ConfigMap to select from + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the ConfigMap must be + defined + type: boolean + type: object + x-kubernetes-map-type: atomic + prefix: + description: An optional identifier to prepend to each + key in the ConfigMap. Must be a C_IDENTIFIER. + type: string + secretRef: + description: The Secret to select from + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret must be defined + type: boolean + type: object + x-kubernetes-map-type: atomic + type: object + type: array + image: + description: 'Container image name. More info: https://kubernetes.io/docs/concepts/containers/images + This field is optional to allow higher level config management + to default or override container images in workload controllers + like Deployments and StatefulSets.' + type: string + imagePullPolicy: + description: 'Image pull policy. One of Always, Never, IfNotPresent. + Defaults to Always if :latest tag is specified, or IfNotPresent + otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images' + type: string + lifecycle: + description: Actions that the management system should take + in response to container lifecycle events. Cannot be updated. + properties: + postStart: + description: 'PostStart is called immediately after a container + is created. If the handler fails, the container is terminated + and restarted according to its restart policy. Other management + of the container blocks until the hook completes. More + info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for + the command is root ('/') in the container's + filesystem. The command is simply exec'd, it is + not run inside a shell, so traditional shell instructions + ('|', etc) won't work. To use a shell, you need + to explicitly call out to that shell. Exit status + of 0 is treated as live/healthy and non-zero is + unhealthy. + items: + type: string + type: array + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to + the pod IP. You probably want to set "Host" in + httpHeaders instead. + type: string + httpHeaders: + description: Custom headers to set in the request. + HTTP allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access + on the container. Number must be in the range + 1 to 65535. Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the + host. Defaults to HTTP. + type: string + required: + - port + type: object + tcpSocket: + description: Deprecated. TCPSocket is NOT supported + as a LifecycleHandler and kept for the backward compatibility. + There are no validation of this field and lifecycle + hooks will fail in runtime when tcp handler is specified. + properties: + host: + description: 'Optional: Host name to connect to, + defaults to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access + on the container. Number must be in the range + 1 to 65535. Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + type: object + preStop: + description: 'PreStop is called immediately before a container + is terminated due to an API request or management event + such as liveness/startup probe failure, preemption, resource + contention, etc. The handler is not called if the container + crashes or exits. The Pod''s termination grace period + countdown begins before the PreStop hook is executed. + Regardless of the outcome of the handler, the container + will eventually terminate within the Pod''s termination + grace period (unless delayed by finalizers). Other management + of the container blocks until the hook completes or until + the termination grace period is reached. More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for + the command is root ('/') in the container's + filesystem. The command is simply exec'd, it is + not run inside a shell, so traditional shell instructions + ('|', etc) won't work. To use a shell, you need + to explicitly call out to that shell. Exit status + of 0 is treated as live/healthy and non-zero is + unhealthy. + items: + type: string + type: array + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to + the pod IP. You probably want to set "Host" in + httpHeaders instead. + type: string + httpHeaders: + description: Custom headers to set in the request. + HTTP allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access + on the container. Number must be in the range + 1 to 65535. Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the + host. Defaults to HTTP. + type: string + required: + - port + type: object + tcpSocket: + description: Deprecated. TCPSocket is NOT supported + as a LifecycleHandler and kept for the backward compatibility. + There are no validation of this field and lifecycle + hooks will fail in runtime when tcp handler is specified. + properties: + host: + description: 'Optional: Host name to connect to, + defaults to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access + on the container. Number must be in the range + 1 to 65535. Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + type: object + type: object + livenessProbe: + description: 'Periodic probe of container liveness. Container + will be restarted if the probe fails. Cannot be updated. More + info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for the + command is root ('/') in the container's filesystem. + The command is simply exec'd, it is not run inside + a shell, so traditional shell instructions ('|', etc) + won't work. To use a shell, you need to explicitly + call out to that shell. Exit status of 0 is treated + as live/healthy and non-zero is unhealthy. + items: + type: string + type: array + type: object + failureThreshold: + description: Minimum consecutive failures for the probe + to be considered failed after having succeeded. Defaults + to 3. Minimum value is 1. + format: int32 + type: integer + grpc: + description: GRPC specifies an action involving a GRPC port. + This is a beta field and requires enabling GRPCContainerProbe + feature gate. + properties: + port: + description: Port number of the gRPC service. Number + must be in the range 1 to 65535. + format: int32 + type: integer + service: + description: "Service is the name of the service to + place in the gRPC HealthCheckRequest (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md). + \n If this is not specified, the default behavior + is defined by gRPC." + type: string + required: + - port + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to the + pod IP. You probably want to set "Host" in httpHeaders + instead. + type: string + httpHeaders: + description: Custom headers to set in the request. HTTP + allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the host. + Defaults to HTTP. + type: string + required: + - port + type: object + initialDelaySeconds: + description: 'Number of seconds after the container has + started before liveness probes are initiated. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + periodSeconds: + description: How often (in seconds) to perform the probe. + Default to 10 seconds. Minimum value is 1. + format: int32 + type: integer + successThreshold: + description: Minimum consecutive successes for the probe + to be considered successful after having failed. Defaults + to 1. Must be 1 for liveness and startup. Minimum value + is 1. + format: int32 + type: integer + tcpSocket: + description: TCPSocket specifies an action involving a TCP + port. + properties: + host: + description: 'Optional: Host name to connect to, defaults + to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + terminationGracePeriodSeconds: + description: Optional duration in seconds the pod needs + to terminate gracefully upon probe failure. The grace + period is the duration in seconds after the processes + running in the pod are sent a termination signal and the + time when the processes are forcibly halted with a kill + signal. Set this value longer than the expected cleanup + time for your process. If this value is nil, the pod's + terminationGracePeriodSeconds will be used. Otherwise, + this value overrides the value provided by the pod spec. + Value must be non-negative integer. The value zero indicates + stop immediately via the kill signal (no opportunity to + shut down). This is a beta field and requires enabling + ProbeTerminationGracePeriod feature gate. Minimum value + is 1. spec.terminationGracePeriodSeconds is used if unset. + format: int64 + type: integer + timeoutSeconds: + description: 'Number of seconds after which the probe times + out. Defaults to 1 second. Minimum value is 1. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + type: object + name: + description: Name of the container specified as a DNS_LABEL. + Each container in a pod must have a unique name (DNS_LABEL). + Cannot be updated. + type: string + ports: + description: List of ports to expose from the container. Not + specifying a port here DOES NOT prevent that port from being + exposed. Any port which is listening on the default "0.0.0.0" + address inside a container will be accessible from the network. + Modifying this array with strategic merge patch may corrupt + the data. For more information See https://github.com/kubernetes/kubernetes/issues/108255. + Cannot be updated. + items: + description: ContainerPort represents a network port in a + single container. + properties: + containerPort: + description: Number of port to expose on the pod's IP + address. This must be a valid port number, 0 < x < 65536. + format: int32 + type: integer + hostIP: + description: What host IP to bind the external port to. + type: string + hostPort: + description: Number of port to expose on the host. If + specified, this must be a valid port number, 0 < x < + 65536. If HostNetwork is specified, this must match + ContainerPort. Most containers do not need this. + format: int32 + type: integer + name: + description: If specified, this must be an IANA_SVC_NAME + and unique within the pod. Each named port in a pod + must have a unique name. Name for the port that can + be referred to by services. + type: string + protocol: + default: TCP + description: Protocol for port. Must be UDP, TCP, or SCTP. + Defaults to "TCP". + type: string + required: + - containerPort + type: object + type: array + x-kubernetes-list-map-keys: + - containerPort + - protocol + x-kubernetes-list-type: map + readinessProbe: + description: 'Periodic probe of container service readiness. + Container will be removed from service endpoints if the probe + fails. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for the + command is root ('/') in the container's filesystem. + The command is simply exec'd, it is not run inside + a shell, so traditional shell instructions ('|', etc) + won't work. To use a shell, you need to explicitly + call out to that shell. Exit status of 0 is treated + as live/healthy and non-zero is unhealthy. + items: + type: string + type: array + type: object + failureThreshold: + description: Minimum consecutive failures for the probe + to be considered failed after having succeeded. Defaults + to 3. Minimum value is 1. + format: int32 + type: integer + grpc: + description: GRPC specifies an action involving a GRPC port. + This is a beta field and requires enabling GRPCContainerProbe + feature gate. + properties: + port: + description: Port number of the gRPC service. Number + must be in the range 1 to 65535. + format: int32 + type: integer + service: + description: "Service is the name of the service to + place in the gRPC HealthCheckRequest (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md). + \n If this is not specified, the default behavior + is defined by gRPC." + type: string + required: + - port + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to the + pod IP. You probably want to set "Host" in httpHeaders + instead. + type: string + httpHeaders: + description: Custom headers to set in the request. HTTP + allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the host. + Defaults to HTTP. + type: string + required: + - port + type: object + initialDelaySeconds: + description: 'Number of seconds after the container has + started before liveness probes are initiated. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + periodSeconds: + description: How often (in seconds) to perform the probe. + Default to 10 seconds. Minimum value is 1. + format: int32 + type: integer + successThreshold: + description: Minimum consecutive successes for the probe + to be considered successful after having failed. Defaults + to 1. Must be 1 for liveness and startup. Minimum value + is 1. + format: int32 + type: integer + tcpSocket: + description: TCPSocket specifies an action involving a TCP + port. + properties: + host: + description: 'Optional: Host name to connect to, defaults + to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + terminationGracePeriodSeconds: + description: Optional duration in seconds the pod needs + to terminate gracefully upon probe failure. The grace + period is the duration in seconds after the processes + running in the pod are sent a termination signal and the + time when the processes are forcibly halted with a kill + signal. Set this value longer than the expected cleanup + time for your process. If this value is nil, the pod's + terminationGracePeriodSeconds will be used. Otherwise, + this value overrides the value provided by the pod spec. + Value must be non-negative integer. The value zero indicates + stop immediately via the kill signal (no opportunity to + shut down). This is a beta field and requires enabling + ProbeTerminationGracePeriod feature gate. Minimum value + is 1. spec.terminationGracePeriodSeconds is used if unset. + format: int64 + type: integer + timeoutSeconds: + description: 'Number of seconds after which the probe times + out. Defaults to 1 second. Minimum value is 1. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + type: object + resources: + description: 'Compute Resources required by this container. + Cannot be updated. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + properties: + limits: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Limits describes the maximum amount of compute + resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + requests: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Requests describes the minimum amount of compute + resources required. If Requests is omitted for a container, + it defaults to Limits if that is explicitly specified, + otherwise to an implementation-defined value. More info: + https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + type: object + securityContext: + description: 'SecurityContext defines the security options the + container should be run with. If set, the fields of SecurityContext + override the equivalent fields of PodSecurityContext. More + info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/' + properties: + allowPrivilegeEscalation: + description: 'AllowPrivilegeEscalation controls whether + a process can gain more privileges than its parent process. + This bool directly controls if the no_new_privs flag will + be set on the container process. AllowPrivilegeEscalation + is true always when the container is: 1) run as Privileged + 2) has CAP_SYS_ADMIN Note that this field cannot be set + when spec.os.name is windows.' + type: boolean + capabilities: + description: The capabilities to add/drop when running containers. + Defaults to the default set of capabilities granted by + the container runtime. Note that this field cannot be + set when spec.os.name is windows. + properties: + add: + description: Added capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + drop: + description: Removed capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + type: object + privileged: + description: Run container in privileged mode. Processes + in privileged containers are essentially equivalent to + root on the host. Defaults to false. Note that this field + cannot be set when spec.os.name is windows. + type: boolean + procMount: + description: procMount denotes the type of proc mount to + use for the containers. The default is DefaultProcMount + which uses the container runtime defaults for readonly + paths and masked paths. This requires the ProcMountType + feature flag to be enabled. Note that this field cannot + be set when spec.os.name is windows. + type: string + readOnlyRootFilesystem: + description: Whether this container has a read-only root + filesystem. Default is false. Note that this field cannot + be set when spec.os.name is windows. + type: boolean + runAsGroup: + description: The GID to run the entrypoint of the container + process. Uses runtime default if unset. May also be set + in PodSecurityContext. If set in both SecurityContext + and PodSecurityContext, the value specified in SecurityContext + takes precedence. Note that this field cannot be set when + spec.os.name is windows. + format: int64 + type: integer + runAsNonRoot: + description: Indicates that the container must run as a + non-root user. If true, the Kubelet will validate the + image at runtime to ensure that it does not run as UID + 0 (root) and fail to start the container if it does. If + unset or false, no such validation will be performed. + May also be set in PodSecurityContext. If set in both + SecurityContext and PodSecurityContext, the value specified + in SecurityContext takes precedence. + type: boolean + runAsUser: + description: The UID to run the entrypoint of the container + process. Defaults to user specified in image metadata + if unspecified. May also be set in PodSecurityContext. If + set in both SecurityContext and PodSecurityContext, the + value specified in SecurityContext takes precedence. Note + that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + seLinuxOptions: + description: The SELinux context to be applied to the container. + If unspecified, the container runtime will allocate a + random SELinux context for each container. May also be + set in PodSecurityContext. If set in both SecurityContext + and PodSecurityContext, the value specified in SecurityContext + takes precedence. Note that this field cannot be set when + spec.os.name is windows. + properties: + level: + description: Level is SELinux level label that applies + to the container. + type: string + role: + description: Role is a SELinux role label that applies + to the container. + type: string + type: + description: Type is a SELinux type label that applies + to the container. + type: string + user: + description: User is a SELinux user label that applies + to the container. + type: string + type: object + seccompProfile: + description: The seccomp options to use by this container. + If seccomp options are provided at both the pod & container + level, the container options override the pod options. + Note that this field cannot be set when spec.os.name is + windows. + properties: + localhostProfile: + description: localhostProfile indicates a profile defined + in a file on the node should be used. The profile + must be preconfigured on the node to work. Must be + a descending path, relative to the kubelet's configured + seccomp profile location. Must only be set if type + is "Localhost". + type: string + type: + description: "type indicates which kind of seccomp profile + will be applied. Valid options are: \n Localhost - + a profile defined in a file on the node should be + used. RuntimeDefault - the container runtime default + profile should be used. Unconfined - no profile should + be applied." + type: string + required: + - type + type: object + windowsOptions: + description: The Windows specific settings applied to all + containers. If unspecified, the options from the PodSecurityContext + will be used. If set in both SecurityContext and PodSecurityContext, + the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is + linux. + properties: + gmsaCredentialSpec: + description: GMSACredentialSpec is where the GMSA admission + webhook (https://github.com/kubernetes-sigs/windows-gmsa) + inlines the contents of the GMSA credential spec named + by the GMSACredentialSpecName field. + type: string + gmsaCredentialSpecName: + description: GMSACredentialSpecName is the name of the + GMSA credential spec to use. + type: string + hostProcess: + description: HostProcess determines if a container should + be run as a 'Host Process' container. This field is + alpha-level and will only be honored by components + that enable the WindowsHostProcessContainers feature + flag. Setting this field without the feature flag + will result in errors when validating the Pod. All + of a Pod's containers must have the same effective + HostProcess value (it is not allowed to have a mix + of HostProcess containers and non-HostProcess containers). In + addition, if HostProcess is true then HostNetwork + must also be set to true. + type: boolean + runAsUserName: + description: The UserName in Windows to run the entrypoint + of the container process. Defaults to the user specified + in image metadata if unspecified. May also be set + in PodSecurityContext. If set in both SecurityContext + and PodSecurityContext, the value specified in SecurityContext + takes precedence. + type: string + type: object + type: object + startupProbe: + description: 'StartupProbe indicates that the Pod has successfully + initialized. If specified, no other probes are executed until + this completes successfully. If this probe fails, the Pod + will be restarted, just as if the livenessProbe failed. This + can be used to provide different probe parameters at the beginning + of a Pod''s lifecycle, when it might take a long time to load + data or warm a cache, than during steady-state operation. + This cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for the + command is root ('/') in the container's filesystem. + The command is simply exec'd, it is not run inside + a shell, so traditional shell instructions ('|', etc) + won't work. To use a shell, you need to explicitly + call out to that shell. Exit status of 0 is treated + as live/healthy and non-zero is unhealthy. + items: + type: string + type: array + type: object + failureThreshold: + description: Minimum consecutive failures for the probe + to be considered failed after having succeeded. Defaults + to 3. Minimum value is 1. + format: int32 + type: integer + grpc: + description: GRPC specifies an action involving a GRPC port. + This is a beta field and requires enabling GRPCContainerProbe + feature gate. + properties: + port: + description: Port number of the gRPC service. Number + must be in the range 1 to 65535. + format: int32 + type: integer + service: + description: "Service is the name of the service to + place in the gRPC HealthCheckRequest (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md). + \n If this is not specified, the default behavior + is defined by gRPC." + type: string + required: + - port + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to the + pod IP. You probably want to set "Host" in httpHeaders + instead. + type: string + httpHeaders: + description: Custom headers to set in the request. HTTP + allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the host. + Defaults to HTTP. + type: string + required: + - port + type: object + initialDelaySeconds: + description: 'Number of seconds after the container has + started before liveness probes are initiated. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + periodSeconds: + description: How often (in seconds) to perform the probe. + Default to 10 seconds. Minimum value is 1. + format: int32 + type: integer + successThreshold: + description: Minimum consecutive successes for the probe + to be considered successful after having failed. Defaults + to 1. Must be 1 for liveness and startup. Minimum value + is 1. + format: int32 + type: integer + tcpSocket: + description: TCPSocket specifies an action involving a TCP + port. + properties: + host: + description: 'Optional: Host name to connect to, defaults + to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + terminationGracePeriodSeconds: + description: Optional duration in seconds the pod needs + to terminate gracefully upon probe failure. The grace + period is the duration in seconds after the processes + running in the pod are sent a termination signal and the + time when the processes are forcibly halted with a kill + signal. Set this value longer than the expected cleanup + time for your process. If this value is nil, the pod's + terminationGracePeriodSeconds will be used. Otherwise, + this value overrides the value provided by the pod spec. + Value must be non-negative integer. The value zero indicates + stop immediately via the kill signal (no opportunity to + shut down). This is a beta field and requires enabling + ProbeTerminationGracePeriod feature gate. Minimum value + is 1. spec.terminationGracePeriodSeconds is used if unset. + format: int64 + type: integer + timeoutSeconds: + description: 'Number of seconds after which the probe times + out. Defaults to 1 second. Minimum value is 1. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + type: object + stdin: + description: Whether this container should allocate a buffer + for stdin in the container runtime. If this is not set, reads + from stdin in the container will always result in EOF. Default + is false. + type: boolean + stdinOnce: + description: Whether the container runtime should close the + stdin channel after it has been opened by a single attach. + When stdin is true the stdin stream will remain open across + multiple attach sessions. If stdinOnce is set to true, stdin + is opened on container start, is empty until the first client + attaches to stdin, and then remains open and accepts data + until the client disconnects, at which time stdin is closed + and remains closed until the container is restarted. If this + flag is false, a container processes that reads from stdin + will never receive an EOF. Default is false + type: boolean + terminationMessagePath: + description: 'Optional: Path at which the file to which the + container''s termination message will be written is mounted + into the container''s filesystem. Message written is intended + to be brief final status, such as an assertion failure message. + Will be truncated by the node if greater than 4096 bytes. + The total message length across all containers will be limited + to 12kb. Defaults to /dev/termination-log. Cannot be updated.' + type: string + terminationMessagePolicy: + description: Indicate how the termination message should be + populated. File will use the contents of terminationMessagePath + to populate the container status message on both success and + failure. FallbackToLogsOnError will use the last chunk of + container log output if the termination message file is empty + and the container exited with an error. The log output is + limited to 2048 bytes or 80 lines, whichever is smaller. Defaults + to File. Cannot be updated. + type: string + tty: + description: Whether this container should allocate a TTY for + itself, also requires 'stdin' to be true. Default is false. + type: boolean + volumeDevices: + description: volumeDevices is the list of block devices to be + used by the container. + items: + description: volumeDevice describes a mapping of a raw block + device within a container. + properties: + devicePath: + description: devicePath is the path inside of the container + that the device will be mapped to. + type: string + name: + description: name must match the name of a persistentVolumeClaim + in the pod + type: string + required: + - devicePath + - name + type: object + type: array + volumeMounts: + description: Pod volumes to mount into the container's filesystem. + Cannot be updated. + items: + description: VolumeMount describes a mounting of a Volume + within a container. + properties: + mountPath: + description: Path within the container at which the volume + should be mounted. Must not contain ':'. + type: string + mountPropagation: + description: mountPropagation determines how mounts are + propagated from the host to container and the other + way around. When not set, MountPropagationNone is used. + This field is beta in 1.10. + type: string + name: + description: This must match the Name of a Volume. + type: string + readOnly: + description: Mounted read-only if true, read-write otherwise + (false or unspecified). Defaults to false. + type: boolean + subPath: + description: Path within the volume from which the container's + volume should be mounted. Defaults to "" (volume's root). + type: string + subPathExpr: + description: Expanded path within the volume from which + the container's volume should be mounted. Behaves similarly + to SubPath but environment variable references $(VAR_NAME) + are expanded using the container's environment. Defaults + to "" (volume's root). SubPathExpr and SubPath are mutually + exclusive. + type: string + required: + - mountPath + - name + type: object + type: array + workingDir: + description: Container's working directory. If not specified, + the container runtime's default will be used, which might + be configured in the container image. Cannot be updated. + type: string + required: + - name + type: object + type: array + disableCompaction: + description: Disable prometheus compaction. + type: boolean + enableAdminAPI: + description: 'Enable access to prometheus web admin API. Defaults + to the value of `false`. WARNING: Enabling the admin APIs enables + mutating endpoints, to delete data, shutdown Prometheus, and more. + Enabling this should be done with care and the user is advised to + add additional authentication authorization via a proxy to ensure + only clients authorized to perform these actions can do so. For + more information see https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-admin-apis' + type: boolean + enableFeatures: + description: Enable access to Prometheus disabled features. By default, + no features are enabled. Enabling disabled features is entirely + outside the scope of what the maintainers will support and by doing + so, you accept that this behaviour may break at any time without + notice. For more information see https://prometheus.io/docs/prometheus/latest/disabled_features/ + items: + type: string + type: array + enableRemoteWriteReceiver: + description: 'Enable Prometheus to be used as a receiver for the Prometheus + remote write protocol. Defaults to the value of `false`. WARNING: + This is not considered an efficient way of ingesting samples. Use + it with caution for specific low-volume use cases. It is not suitable + for replacing the ingestion via scraping and turning Prometheus + into a push-based metrics collection system. For more information + see https://prometheus.io/docs/prometheus/latest/querying/api/#remote-write-receiver + Only valid in Prometheus versions 2.33.0 and newer.' + type: boolean + enforcedBodySizeLimit: + description: 'EnforcedBodySizeLimit defines the maximum size of uncompressed + response body that will be accepted by Prometheus. Targets responding + with a body larger than this many bytes will cause the scrape to + fail. Example: 100MB. If defined, the limit will apply to all service/pod + monitors and probes. This is an experimental feature, this behaviour + could change or be removed in the future. Only valid in Prometheus + versions 2.28.0 and newer.' + pattern: (^0|([0-9]*[.])?[0-9]+((K|M|G|T|E|P)i?)?B)$ + type: string + enforcedLabelLimit: + description: Per-scrape limit on number of labels that will be accepted + for a sample. If more than this number of labels are present post + metric-relabeling, the entire scrape will be treated as failed. + 0 means no limit. Only valid in Prometheus versions 2.27.0 and newer. + format: int64 + type: integer + enforcedLabelNameLengthLimit: + description: Per-scrape limit on length of labels name that will be + accepted for a sample. If a label name is longer than this number + post metric-relabeling, the entire scrape will be treated as failed. + 0 means no limit. Only valid in Prometheus versions 2.27.0 and newer. + format: int64 + type: integer + enforcedLabelValueLengthLimit: + description: Per-scrape limit on length of labels value that will + be accepted for a sample. If a label value is longer than this number + post metric-relabeling, the entire scrape will be treated as failed. + 0 means no limit. Only valid in Prometheus versions 2.27.0 and newer. + format: int64 + type: integer + enforcedNamespaceLabel: + description: "EnforcedNamespaceLabel If set, a label will be added + to \n 1. all user-metrics (created by `ServiceMonitor`, `PodMonitor` + and `Probe` objects) and 2. in all `PrometheusRule` objects (except + the ones excluded in `prometheusRulesExcludedFromEnforce`) to * + alerting & recording rules and * the metrics used in their expressions + (`expr`). \n Label name is this field's value. Label value is the + namespace of the created object (mentioned above)." + type: string + enforcedSampleLimit: + description: EnforcedSampleLimit defines global limit on number of + scraped samples that will be accepted. This overrides any SampleLimit + set per ServiceMonitor or/and PodMonitor. It is meant to be used + by admins to enforce the SampleLimit to keep overall number of samples/series + under the desired limit. Note that if SampleLimit is lower that + value will be taken instead. + format: int64 + type: integer + enforcedTargetLimit: + description: EnforcedTargetLimit defines a global limit on the number + of scraped targets. This overrides any TargetLimit set per ServiceMonitor + or/and PodMonitor. It is meant to be used by admins to enforce + the TargetLimit to keep the overall number of targets under the + desired limit. Note that if TargetLimit is lower, that value will + be taken instead, except if either value is zero, in which case + the non-zero value will be used. If both values are zero, no limit + is enforced. + format: int64 + type: integer + evaluationInterval: + default: 30s + description: 'Interval between consecutive evaluations. Default: `30s`' + pattern: ^(0|(([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?)$ + type: string + excludedFromEnforcement: + description: List of references to PodMonitor, ServiceMonitor, Probe + and PrometheusRule objects to be excluded from enforcing a namespace + label of origin. Applies only if enforcedNamespaceLabel set to true. + items: + description: ObjectReference references a PodMonitor, ServiceMonitor, + Probe or PrometheusRule object. + properties: + group: + default: monitoring.coreos.com + description: Group of the referent. When not specified, it defaults + to `monitoring.coreos.com` + enum: + - monitoring.coreos.com + type: string + name: + description: Name of the referent. When not set, all resources + are matched. + type: string + namespace: + description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/' + minLength: 1 + type: string + resource: + description: Resource of the referent. + enum: + - prometheusrules + - servicemonitors + - podmonitors + - probes + type: string + required: + - namespace + - resource + type: object + type: array + exemplars: + description: Exemplars related settings that are runtime reloadable. + It requires to enable the exemplar storage feature to be effective. + properties: + maxSize: + description: Maximum number of exemplars stored in memory for + all series. If not set, Prometheus uses its default value. A + value of zero or less than zero disables the storage. + format: int64 + type: integer + type: object + externalLabels: + additionalProperties: + type: string + description: The labels to add to any time series or alerts when communicating + with external systems (federation, remote storage, Alertmanager). + type: object + externalUrl: + description: The external URL the Prometheus instances will be available + under. This is necessary to generate correct URLs. This is necessary + if Prometheus is not served from root of a DNS name. + type: string + hostAliases: + description: Pods' hostAliases configuration + items: + description: HostAlias holds the mapping between IP and hostnames + that will be injected as an entry in the pod's hosts file. + properties: + hostnames: + description: Hostnames for the above IP address. + items: + type: string + type: array + ip: + description: IP address of the host file entry. + type: string + required: + - hostnames + - ip + type: object + type: array + x-kubernetes-list-map-keys: + - ip + x-kubernetes-list-type: map + hostNetwork: + description: Use the host's network namespace if true. Make sure to + understand the security implications if you want to enable it. When + hostNetwork is enabled, this will set dnsPolicy to ClusterFirstWithHostNet + automatically. + type: boolean + ignoreNamespaceSelectors: + description: IgnoreNamespaceSelectors if set to true will ignore NamespaceSelector + settings from all PodMonitor, ServiceMonitor and Probe objects. + They will only discover endpoints within the namespace of the PodMonitor, + ServiceMonitor and Probe objects. Defaults to false. + type: boolean + image: + description: Image if specified has precedence over baseImage, tag + and sha combinations. Specifying the version is still necessary + to ensure the Prometheus Operator knows what version of Prometheus + is being configured. + type: string + imagePullSecrets: + description: An optional list of references to secrets in the same + namespace to use for pulling prometheus and alertmanager images + from registries see http://kubernetes.io/docs/user-guide/images#specifying-imagepullsecrets-on-a-pod + items: + description: LocalObjectReference contains enough information to + let you locate the referenced object inside the same namespace. + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, uid?' + type: string + type: object + x-kubernetes-map-type: atomic + type: array + initContainers: + description: 'InitContainers allows adding initContainers to the pod + definition. Those can be used to e.g. fetch secrets for injection + into the Prometheus configuration from external sources. Any errors + during the execution of an initContainer will lead to a restart + of the Pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ + InitContainers described here modify an operator generated init + containers if they share the same name and modifications are done + via a strategic merge patch. The current init container name is: + `init-config-reloader`. Overriding init containers is entirely outside + the scope of what the maintainers will support and by doing so, + you accept that this behaviour may break at any time without notice.' + items: + description: A single application container that you want to run + within a pod. + properties: + args: + description: 'Arguments to the entrypoint. The container image''s + CMD is used if this is not provided. Variable references $(VAR_NAME) + are expanded using the container''s environment. If a variable + cannot be resolved, the reference in the input string will + be unchanged. Double $$ are reduced to a single $, which allows + for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will + produce the string literal "$(VAR_NAME)". Escaped references + will never be expanded, regardless of whether the variable + exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell' + items: + type: string + type: array + command: + description: 'Entrypoint array. Not executed within a shell. + The container image''s ENTRYPOINT is used if this is not provided. + Variable references $(VAR_NAME) are expanded using the container''s + environment. If a variable cannot be resolved, the reference + in the input string will be unchanged. Double $$ are reduced + to a single $, which allows for escaping the $(VAR_NAME) syntax: + i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". + Escaped references will never be expanded, regardless of whether + the variable exists or not. Cannot be updated. More info: + https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell' + items: + type: string + type: array + env: + description: List of environment variables to set in the container. + Cannot be updated. + items: + description: EnvVar represents an environment variable present + in a Container. + properties: + name: + description: Name of the environment variable. Must be + a C_IDENTIFIER. + type: string + value: + description: 'Variable references $(VAR_NAME) are expanded + using the previously defined environment variables in + the container and any service environment variables. + If a variable cannot be resolved, the reference in the + input string will be unchanged. Double $$ are reduced + to a single $, which allows for escaping the $(VAR_NAME) + syntax: i.e. "$$(VAR_NAME)" will produce the string + literal "$(VAR_NAME)". Escaped references will never + be expanded, regardless of whether the variable exists + or not. Defaults to "".' + type: string + valueFrom: + description: Source for the environment variable's value. + Cannot be used if value is not empty. + properties: + configMapKeyRef: + description: Selects a key of a ConfigMap. + properties: + key: + description: The key to select. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the ConfigMap or + its key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + fieldRef: + description: 'Selects a field of the pod: supports + metadata.name, metadata.namespace, `metadata.labels['''']`, + `metadata.annotations['''']`, spec.nodeName, + spec.serviceAccountName, status.hostIP, status.podIP, + status.podIPs.' + properties: + apiVersion: + description: Version of the schema the FieldPath + is written in terms of, defaults to "v1". + type: string + fieldPath: + description: Path of the field to select in the + specified API version. + type: string + required: + - fieldPath + type: object + x-kubernetes-map-type: atomic + resourceFieldRef: + description: 'Selects a resource of the container: + only resources limits and requests (limits.cpu, + limits.memory, limits.ephemeral-storage, requests.cpu, + requests.memory and requests.ephemeral-storage) + are currently supported.' + properties: + containerName: + description: 'Container name: required for volumes, + optional for env vars' + type: string + divisor: + anyOf: + - type: integer + - type: string + description: Specifies the output format of the + exposed resources, defaults to "1" + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + resource: + description: 'Required: resource to select' + type: string + required: + - resource + type: object + x-kubernetes-map-type: atomic + secretKeyRef: + description: Selects a key of a secret in the pod's + namespace + properties: + key: + description: The key of the secret to select from. Must + be a valid secret key. + type: string + name: + description: 'Name of the referent. More info: + https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret or its + key must be defined + type: boolean + required: + - key + type: object + x-kubernetes-map-type: atomic + type: object + required: + - name + type: object + type: array + envFrom: + description: List of sources to populate environment variables + in the container. The keys defined within a source must be + a C_IDENTIFIER. All invalid keys will be reported as an event + when the container is starting. When a key exists in multiple + sources, the value associated with the last source will take + precedence. Values defined by an Env with a duplicate key + will take precedence. Cannot be updated. + items: + description: EnvFromSource represents the source of a set + of ConfigMaps + properties: + configMapRef: + description: The ConfigMap to select from + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the ConfigMap must be + defined + type: boolean + type: object + x-kubernetes-map-type: atomic + prefix: + description: An optional identifier to prepend to each + key in the ConfigMap. Must be a C_IDENTIFIER. + type: string + secretRef: + description: The Secret to select from + properties: + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + TODO: Add other useful fields. apiVersion, kind, + uid?' + type: string + optional: + description: Specify whether the Secret must be defined + type: boolean + type: object + x-kubernetes-map-type: atomic + type: object + type: array + image: + description: 'Container image name. More info: https://kubernetes.io/docs/concepts/containers/images + This field is optional to allow higher level config management + to default or override container images in workload controllers + like Deployments and StatefulSets.' + type: string + imagePullPolicy: + description: 'Image pull policy. One of Always, Never, IfNotPresent. + Defaults to Always if :latest tag is specified, or IfNotPresent + otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images' + type: string + lifecycle: + description: Actions that the management system should take + in response to container lifecycle events. Cannot be updated. + properties: + postStart: + description: 'PostStart is called immediately after a container + is created. If the handler fails, the container is terminated + and restarted according to its restart policy. Other management + of the container blocks until the hook completes. More + info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for + the command is root ('/') in the container's + filesystem. The command is simply exec'd, it is + not run inside a shell, so traditional shell instructions + ('|', etc) won't work. To use a shell, you need + to explicitly call out to that shell. Exit status + of 0 is treated as live/healthy and non-zero is + unhealthy. + items: + type: string + type: array + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to + the pod IP. You probably want to set "Host" in + httpHeaders instead. + type: string + httpHeaders: + description: Custom headers to set in the request. + HTTP allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access + on the container. Number must be in the range + 1 to 65535. Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the + host. Defaults to HTTP. + type: string + required: + - port + type: object + tcpSocket: + description: Deprecated. TCPSocket is NOT supported + as a LifecycleHandler and kept for the backward compatibility. + There are no validation of this field and lifecycle + hooks will fail in runtime when tcp handler is specified. + properties: + host: + description: 'Optional: Host name to connect to, + defaults to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access + on the container. Number must be in the range + 1 to 65535. Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + type: object + preStop: + description: 'PreStop is called immediately before a container + is terminated due to an API request or management event + such as liveness/startup probe failure, preemption, resource + contention, etc. The handler is not called if the container + crashes or exits. The Pod''s termination grace period + countdown begins before the PreStop hook is executed. + Regardless of the outcome of the handler, the container + will eventually terminate within the Pod''s termination + grace period (unless delayed by finalizers). Other management + of the container blocks until the hook completes or until + the termination grace period is reached. More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for + the command is root ('/') in the container's + filesystem. The command is simply exec'd, it is + not run inside a shell, so traditional shell instructions + ('|', etc) won't work. To use a shell, you need + to explicitly call out to that shell. Exit status + of 0 is treated as live/healthy and non-zero is + unhealthy. + items: + type: string + type: array + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to + the pod IP. You probably want to set "Host" in + httpHeaders instead. + type: string + httpHeaders: + description: Custom headers to set in the request. + HTTP allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access + on the container. Number must be in the range + 1 to 65535. Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the + host. Defaults to HTTP. + type: string + required: + - port + type: object + tcpSocket: + description: Deprecated. TCPSocket is NOT supported + as a LifecycleHandler and kept for the backward compatibility. + There are no validation of this field and lifecycle + hooks will fail in runtime when tcp handler is specified. + properties: + host: + description: 'Optional: Host name to connect to, + defaults to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access + on the container. Number must be in the range + 1 to 65535. Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + type: object + type: object + livenessProbe: + description: 'Periodic probe of container liveness. Container + will be restarted if the probe fails. Cannot be updated. More + info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for the + command is root ('/') in the container's filesystem. + The command is simply exec'd, it is not run inside + a shell, so traditional shell instructions ('|', etc) + won't work. To use a shell, you need to explicitly + call out to that shell. Exit status of 0 is treated + as live/healthy and non-zero is unhealthy. + items: + type: string + type: array + type: object + failureThreshold: + description: Minimum consecutive failures for the probe + to be considered failed after having succeeded. Defaults + to 3. Minimum value is 1. + format: int32 + type: integer + grpc: + description: GRPC specifies an action involving a GRPC port. + This is a beta field and requires enabling GRPCContainerProbe + feature gate. + properties: + port: + description: Port number of the gRPC service. Number + must be in the range 1 to 65535. + format: int32 + type: integer + service: + description: "Service is the name of the service to + place in the gRPC HealthCheckRequest (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md). + \n If this is not specified, the default behavior + is defined by gRPC." + type: string + required: + - port + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to the + pod IP. You probably want to set "Host" in httpHeaders + instead. + type: string + httpHeaders: + description: Custom headers to set in the request. HTTP + allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the host. + Defaults to HTTP. + type: string + required: + - port + type: object + initialDelaySeconds: + description: 'Number of seconds after the container has + started before liveness probes are initiated. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + periodSeconds: + description: How often (in seconds) to perform the probe. + Default to 10 seconds. Minimum value is 1. + format: int32 + type: integer + successThreshold: + description: Minimum consecutive successes for the probe + to be considered successful after having failed. Defaults + to 1. Must be 1 for liveness and startup. Minimum value + is 1. + format: int32 + type: integer + tcpSocket: + description: TCPSocket specifies an action involving a TCP + port. + properties: + host: + description: 'Optional: Host name to connect to, defaults + to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + terminationGracePeriodSeconds: + description: Optional duration in seconds the pod needs + to terminate gracefully upon probe failure. The grace + period is the duration in seconds after the processes + running in the pod are sent a termination signal and the + time when the processes are forcibly halted with a kill + signal. Set this value longer than the expected cleanup + time for your process. If this value is nil, the pod's + terminationGracePeriodSeconds will be used. Otherwise, + this value overrides the value provided by the pod spec. + Value must be non-negative integer. The value zero indicates + stop immediately via the kill signal (no opportunity to + shut down). This is a beta field and requires enabling + ProbeTerminationGracePeriod feature gate. Minimum value + is 1. spec.terminationGracePeriodSeconds is used if unset. + format: int64 + type: integer + timeoutSeconds: + description: 'Number of seconds after which the probe times + out. Defaults to 1 second. Minimum value is 1. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + type: object + name: + description: Name of the container specified as a DNS_LABEL. + Each container in a pod must have a unique name (DNS_LABEL). + Cannot be updated. + type: string + ports: + description: List of ports to expose from the container. Not + specifying a port here DOES NOT prevent that port from being + exposed. Any port which is listening on the default "0.0.0.0" + address inside a container will be accessible from the network. + Modifying this array with strategic merge patch may corrupt + the data. For more information See https://github.com/kubernetes/kubernetes/issues/108255. + Cannot be updated. + items: + description: ContainerPort represents a network port in a + single container. + properties: + containerPort: + description: Number of port to expose on the pod's IP + address. This must be a valid port number, 0 < x < 65536. + format: int32 + type: integer + hostIP: + description: What host IP to bind the external port to. + type: string + hostPort: + description: Number of port to expose on the host. If + specified, this must be a valid port number, 0 < x < + 65536. If HostNetwork is specified, this must match + ContainerPort. Most containers do not need this. + format: int32 + type: integer + name: + description: If specified, this must be an IANA_SVC_NAME + and unique within the pod. Each named port in a pod + must have a unique name. Name for the port that can + be referred to by services. + type: string + protocol: + default: TCP + description: Protocol for port. Must be UDP, TCP, or SCTP. + Defaults to "TCP". + type: string + required: + - containerPort + type: object + type: array + x-kubernetes-list-map-keys: + - containerPort + - protocol + x-kubernetes-list-type: map + readinessProbe: + description: 'Periodic probe of container service readiness. + Container will be removed from service endpoints if the probe + fails. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for the + command is root ('/') in the container's filesystem. + The command is simply exec'd, it is not run inside + a shell, so traditional shell instructions ('|', etc) + won't work. To use a shell, you need to explicitly + call out to that shell. Exit status of 0 is treated + as live/healthy and non-zero is unhealthy. + items: + type: string + type: array + type: object + failureThreshold: + description: Minimum consecutive failures for the probe + to be considered failed after having succeeded. Defaults + to 3. Minimum value is 1. + format: int32 + type: integer + grpc: + description: GRPC specifies an action involving a GRPC port. + This is a beta field and requires enabling GRPCContainerProbe + feature gate. + properties: + port: + description: Port number of the gRPC service. Number + must be in the range 1 to 65535. + format: int32 + type: integer + service: + description: "Service is the name of the service to + place in the gRPC HealthCheckRequest (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md). + \n If this is not specified, the default behavior + is defined by gRPC." + type: string + required: + - port + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to the + pod IP. You probably want to set "Host" in httpHeaders + instead. + type: string + httpHeaders: + description: Custom headers to set in the request. HTTP + allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the host. + Defaults to HTTP. + type: string + required: + - port + type: object + initialDelaySeconds: + description: 'Number of seconds after the container has + started before liveness probes are initiated. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + periodSeconds: + description: How often (in seconds) to perform the probe. + Default to 10 seconds. Minimum value is 1. + format: int32 + type: integer + successThreshold: + description: Minimum consecutive successes for the probe + to be considered successful after having failed. Defaults + to 1. Must be 1 for liveness and startup. Minimum value + is 1. + format: int32 + type: integer + tcpSocket: + description: TCPSocket specifies an action involving a TCP + port. + properties: + host: + description: 'Optional: Host name to connect to, defaults + to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + terminationGracePeriodSeconds: + description: Optional duration in seconds the pod needs + to terminate gracefully upon probe failure. The grace + period is the duration in seconds after the processes + running in the pod are sent a termination signal and the + time when the processes are forcibly halted with a kill + signal. Set this value longer than the expected cleanup + time for your process. If this value is nil, the pod's + terminationGracePeriodSeconds will be used. Otherwise, + this value overrides the value provided by the pod spec. + Value must be non-negative integer. The value zero indicates + stop immediately via the kill signal (no opportunity to + shut down). This is a beta field and requires enabling + ProbeTerminationGracePeriod feature gate. Minimum value + is 1. spec.terminationGracePeriodSeconds is used if unset. + format: int64 + type: integer + timeoutSeconds: + description: 'Number of seconds after which the probe times + out. Defaults to 1 second. Minimum value is 1. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + type: object + resources: + description: 'Compute Resources required by this container. + Cannot be updated. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + properties: + limits: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Limits describes the maximum amount of compute + resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + requests: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: 'Requests describes the minimum amount of compute + resources required. If Requests is omitted for a container, + it defaults to Limits if that is explicitly specified, + otherwise to an implementation-defined value. More info: + https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + type: object + securityContext: + description: 'SecurityContext defines the security options the + container should be run with. If set, the fields of SecurityContext + override the equivalent fields of PodSecurityContext. More + info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/' + properties: + allowPrivilegeEscalation: + description: 'AllowPrivilegeEscalation controls whether + a process can gain more privileges than its parent process. + This bool directly controls if the no_new_privs flag will + be set on the container process. AllowPrivilegeEscalation + is true always when the container is: 1) run as Privileged + 2) has CAP_SYS_ADMIN Note that this field cannot be set + when spec.os.name is windows.' + type: boolean + capabilities: + description: The capabilities to add/drop when running containers. + Defaults to the default set of capabilities granted by + the container runtime. Note that this field cannot be + set when spec.os.name is windows. + properties: + add: + description: Added capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + drop: + description: Removed capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + type: object + privileged: + description: Run container in privileged mode. Processes + in privileged containers are essentially equivalent to + root on the host. Defaults to false. Note that this field + cannot be set when spec.os.name is windows. + type: boolean + procMount: + description: procMount denotes the type of proc mount to + use for the containers. The default is DefaultProcMount + which uses the container runtime defaults for readonly + paths and masked paths. This requires the ProcMountType + feature flag to be enabled. Note that this field cannot + be set when spec.os.name is windows. + type: string + readOnlyRootFilesystem: + description: Whether this container has a read-only root + filesystem. Default is false. Note that this field cannot + be set when spec.os.name is windows. + type: boolean + runAsGroup: + description: The GID to run the entrypoint of the container + process. Uses runtime default if unset. May also be set + in PodSecurityContext. If set in both SecurityContext + and PodSecurityContext, the value specified in SecurityContext + takes precedence. Note that this field cannot be set when + spec.os.name is windows. + format: int64 + type: integer + runAsNonRoot: + description: Indicates that the container must run as a + non-root user. If true, the Kubelet will validate the + image at runtime to ensure that it does not run as UID + 0 (root) and fail to start the container if it does. If + unset or false, no such validation will be performed. + May also be set in PodSecurityContext. If set in both + SecurityContext and PodSecurityContext, the value specified + in SecurityContext takes precedence. + type: boolean + runAsUser: + description: The UID to run the entrypoint of the container + process. Defaults to user specified in image metadata + if unspecified. May also be set in PodSecurityContext. If + set in both SecurityContext and PodSecurityContext, the + value specified in SecurityContext takes precedence. Note + that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + seLinuxOptions: + description: The SELinux context to be applied to the container. + If unspecified, the container runtime will allocate a + random SELinux context for each container. May also be + set in PodSecurityContext. If set in both SecurityContext + and PodSecurityContext, the value specified in SecurityContext + takes precedence. Note that this field cannot be set when + spec.os.name is windows. + properties: + level: + description: Level is SELinux level label that applies + to the container. + type: string + role: + description: Role is a SELinux role label that applies + to the container. + type: string + type: + description: Type is a SELinux type label that applies + to the container. + type: string + user: + description: User is a SELinux user label that applies + to the container. + type: string + type: object + seccompProfile: + description: The seccomp options to use by this container. + If seccomp options are provided at both the pod & container + level, the container options override the pod options. + Note that this field cannot be set when spec.os.name is + windows. + properties: + localhostProfile: + description: localhostProfile indicates a profile defined + in a file on the node should be used. The profile + must be preconfigured on the node to work. Must be + a descending path, relative to the kubelet's configured + seccomp profile location. Must only be set if type + is "Localhost". + type: string + type: + description: "type indicates which kind of seccomp profile + will be applied. Valid options are: \n Localhost - + a profile defined in a file on the node should be + used. RuntimeDefault - the container runtime default + profile should be used. Unconfined - no profile should + be applied." + type: string + required: + - type + type: object + windowsOptions: + description: The Windows specific settings applied to all + containers. If unspecified, the options from the PodSecurityContext + will be used. If set in both SecurityContext and PodSecurityContext, + the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is + linux. + properties: + gmsaCredentialSpec: + description: GMSACredentialSpec is where the GMSA admission + webhook (https://github.com/kubernetes-sigs/windows-gmsa) + inlines the contents of the GMSA credential spec named + by the GMSACredentialSpecName field. + type: string + gmsaCredentialSpecName: + description: GMSACredentialSpecName is the name of the + GMSA credential spec to use. + type: string + hostProcess: + description: HostProcess determines if a container should + be run as a 'Host Process' container. This field is + alpha-level and will only be honored by components + that enable the WindowsHostProcessContainers feature + flag. Setting this field without the feature flag + will result in errors when validating the Pod. All + of a Pod's containers must have the same effective + HostProcess value (it is not allowed to have a mix + of HostProcess containers and non-HostProcess containers). In + addition, if HostProcess is true then HostNetwork + must also be set to true. + type: boolean + runAsUserName: + description: The UserName in Windows to run the entrypoint + of the container process. Defaults to the user specified + in image metadata if unspecified. May also be set + in PodSecurityContext. If set in both SecurityContext + and PodSecurityContext, the value specified in SecurityContext + takes precedence. + type: string + type: object + type: object + startupProbe: + description: 'StartupProbe indicates that the Pod has successfully + initialized. If specified, no other probes are executed until + this completes successfully. If this probe fails, the Pod + will be restarted, just as if the livenessProbe failed. This + can be used to provide different probe parameters at the beginning + of a Pod''s lifecycle, when it might take a long time to load + data or warm a cache, than during steady-state operation. + This cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + properties: + exec: + description: Exec specifies the action to take. + properties: + command: + description: Command is the command line to execute + inside the container, the working directory for the + command is root ('/') in the container's filesystem. + The command is simply exec'd, it is not run inside + a shell, so traditional shell instructions ('|', etc) + won't work. To use a shell, you need to explicitly + call out to that shell. Exit status of 0 is treated + as live/healthy and non-zero is unhealthy. + items: + type: string + type: array + type: object + failureThreshold: + description: Minimum consecutive failures for the probe + to be considered failed after having succeeded. Defaults + to 3. Minimum value is 1. + format: int32 + type: integer + grpc: + description: GRPC specifies an action involving a GRPC port. + This is a beta field and requires enabling GRPCContainerProbe + feature gate. + properties: + port: + description: Port number of the gRPC service. Number + must be in the range 1 to 65535. + format: int32 + type: integer + service: + description: "Service is the name of the service to + place in the gRPC HealthCheckRequest (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md). + \n If this is not specified, the default behavior + is defined by gRPC." + type: string + required: + - port + type: object + httpGet: + description: HTTPGet specifies the http request to perform. + properties: + host: + description: Host name to connect to, defaults to the + pod IP. You probably want to set "Host" in httpHeaders + instead. + type: string + httpHeaders: + description: Custom headers to set in the request. HTTP + allows repeated headers. + items: + description: HTTPHeader describes a custom header + to be used in HTTP probes + properties: + name: + description: The header field name + type: string + value: + description: The header field value + type: string + required: + - name + - value + type: object + type: array + path: + description: Path to access on the HTTP server. + type: string + port: + anyOf: + - type: integer + - type: string + description: Name or number of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + scheme: + description: Scheme to use for connecting to the host. + Defaults to HTTP. + type: string + required: + - port + type: object + initialDelaySeconds: + description: 'Number of seconds after the container has + started before liveness probes are initiated. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + periodSeconds: + description: How often (in seconds) to perform the probe. + Default to 10 seconds. Minimum value is 1. + format: int32 + type: integer + successThreshold: + description: Minimum consecutive successes for the probe + to be considered successful after having failed. Defaults + to 1. Must be 1 for liveness and startup. Minimum value + is 1. + format: int32 + type: integer + tcpSocket: + description: TCPSocket specifies an action involving a TCP + port. + properties: + host: + description: 'Optional: Host name to connect to, defaults + to the pod IP.' + type: string + port: + anyOf: + - type: integer + - type: string + description: Number or name of the port to access on + the container. Number must be in the range 1 to 65535. + Name must be an IANA_SVC_NAME. + x-kubernetes-int-or-string: true + required: + - port + type: object + terminationGracePeriodSeconds: + description: Optional duration in seconds the pod needs + to terminate gracefully upon probe failure. The grace + period is the duration in seconds after the processes + running in the pod are sent a termination signal and the + time when the processes are forcibly halted with a kill + signal. Set this value longer than the expected cleanup + time for your process. If this value is nil, the pod's + terminationGracePeriodSeconds will be used. Otherwise, + this value overrides the value provided by the pod spec. + Value must be non-negative integer. The value zero indicates + stop immediately via the kill signal (no opportunity to + shut down). This is a beta field and requires enabling + ProbeTerminationGracePeriod feature gate. Minimum value + is 1. spec.terminationGracePeriodSeconds is used if unset. + format: int64 + type: integer + timeoutSeconds: + description: 'Number of seconds after which the probe times + out. Defaults to 1 second. Minimum value is 1. More info: + https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' + format: int32 + type: integer + type: object + stdin: + description: Whether this container should allocate a buffer + for stdin in the container runtime. If this is not set, reads + from stdin in the container will always result in EOF. Default + is false. + type: boolean + stdinOnce: + description: Whether the container runtime should close the + stdin channel after it has been opened by a single attach. + When stdin is true the stdin stream will remain open across + multiple attach sessions. If stdinOnce is set to true, stdin + is opened on container start, is empty until the first client + attaches to stdin, and then remains open and accepts data + until the client disconnects, at which time stdin is closed + and remains closed until the container is restarted. If this + flag is false, a container processes that reads from stdin + will never receive an EOF. Default is false + type: boolean + terminationMessagePath: + description: 'Optional: Path at which the file to which the + container''s termination message will be written is mounted + into the container''s filesystem. Message written is intended + to be brief final status, such as an assertion failure message. + Will be truncated by the node if greater than 4096 bytes. + The total message length across all containers will be limited + to 12kb. Defaults to /dev/termination-log. Cannot be updated.' + type: string + terminationMessagePolicy: + description: Indicate how the termination message should be + populated. File will use the contents of terminationMessagePath + to populate the container status message on both success and + failure. FallbackToLogsOnError will use the last chunk of + container log output if the termination message file is empty + and the container exited with an error. The log output is + limited to 2048 bytes or 80 lines, whichever is smaller. Defaults + to File. Cannot be updated. + type: string + tty: + description: Whether this container should allocate a TTY for + itself, also requires 'stdin' to be true. Default is false. + type: boolean + volumeDevices: + description: volumeDevices is the list of block devices to be + used by the container. + items: + description: volumeDevice describes a mapping of a raw block + device within a container. + properties: + devicePath: + description: devicePath is the path inside of the container + that the device will be mapped to. + type: string + name: + description: name must match the name of a persistentVolumeClaim + in the pod + type: string + required: + - devicePath + - name + type: object + type: array + volumeMounts: + description: Pod volumes to mount into the container's filesystem. + Cannot be updated. + items: + description: VolumeMount describes a mounting of a Volume + within a container. + properties: + mountPath: + description: Path within the container at which the volume + should be mounted. Must not contain ':'. + type: string + mountPropagation: + description: mountPropagation determines how mounts are + propagated from the host to container and the other + way around. When not set, MountPropagationNone is used. + This field is beta in 1.10. + type: string + name: + description: This must match the Name of a Volume. + type: string + readOnly: + description: Mounted read-only if true, read-write otherwise + (false or unspecified). Defaults to false. + type: boolean + subPath: + description: Path within the volume from which the container's + volume should be mounted. Defaults to "" (volume's root). + type: string + subPathExpr: + description: Expanded path within the volume from which + the container's volume should be mounted. Behaves similarly + to SubPath but environment variable references $(VAR_NAME) + are expanded using the container's environment. Defaults + to "" (volume's root). SubPathExpr and SubPath are mutually + exclusive. + type: string + required: + - mountPath + - name + type: object + type: array + workingDir: + description: Container's working directory. If not specified, + the container runtime's default will be used, which might + be configured in the container image. Cannot be updated. + type: string + required: + - name + type: object + type: array + listenLocal: + description: ListenLocal makes the Prometheus server listen on loopback, + so that it does not bind against the Pod IP. + type: boolean + logFormat: + description: Log format for Prometheus to be configured with. + enum: + - "" + - logfmt + - json + type: string + logLevel: + description: Log level for Prometheus to be configured with. + enum: + - "" + - debug + - info + - warn + - error + type: string + minReadySeconds: + description: Minimum number of seconds for which a newly created pod + should be ready without any of its container crashing for it to + be considered available. Defaults to 0 (pod will be considered available + as soon as it is ready) This is an alpha field and requires enabling + StatefulSetMinReadySeconds feature gate. + format: int32 + type: integer + nodeSelector: + additionalProperties: + type: string + description: Define which Nodes the Pods are scheduled on. + type: object + overrideHonorLabels: + description: When true, Prometheus resolves label conflicts by renaming + the labels in the scraped data to "exported_