Merge pull request #278 from stakater/add-tronador-workflow

Add tronador workflow and troubleshooting

.vuepress/config.js

@@ -237,6 +237,8 @@ module.exports = {
             "/content/sre/tronador/environment_provisioner.md",
             "/content/sre/tronador/config_file.md",
             "/content/sre/tronador/cluster_task.md",
+            "/content/sre/tronador/workflow.md",
+            "/content/sre/tronador/troubleshooting.md",
          ]
       },
       {

content/sre/tronador/troubleshooting.md

@@ -0,0 +1,76 @@
+# Troubleshooting
+
+
+Please note: The guide below is written as a SAAP customer's point of view. If you use Tronador independently, the guide below might not be applicable to you. However, you can still use this guide to get an idea of how the process works.
+
+## Developer support
+
+Environment provisioning takes a few minutes to complete, since there is a lot of steps involved that can take some time (image creation, argocd sync, Helm Operator reconcile, etc). Please take a look at the [workflow for Tronador](./workflow.html) to see how the process works. If the environment is still not provisioned after a few minutes, you can use the following steps to get an idea of why it is taking so long:
+
+* Check your cluster's PipelineRun resource to check its status. If any task fails, you can look at the logs of the failed task to see what went wrong.
+
+<figure>
+  <img
+  src="./images/pipeline-ts.png"
+  alt="pipeline">
+  <figcaption>Pipeline guide</figcaption>
+</figure>
+
+<figure>
+  <img
+  src="./images/pipeline-success.png"
+  alt="pipeline">
+  <figcaption>A successful pipeline run</figcaption>
+</figure>
+
+* View your gitops repository to verify that the Environment Provisioner CR is pushed to it.
+
+<figure>
+  <img
+  src="./images/gitops.png"
+  alt="pipeline">
+  <figcaption>Environment Provisioner pushed</figcaption>
+</figure>
+
+* Confirm that the Environment Provisioner CR is pushed to your cluster by checking the status of the ArgoCD Application managing its sync.
+
+<figure>
+  <img
+  src="./images/argocd.png"
+  alt="pipeline">
+  <figcaption>Relevant ArgoCD Application synced</figcaption>
+</figure>
+
+* Check the status of the Helm Release to see if it is in a good state.
+
+<figure>
+  <img
+  src="./images/hr-ts.png"
+  alt="pipeline">
+  <figcaption>Verify that HR is created</figcaption>
+</figure>
+
+<figure>
+  <img
+  src="./images/hr-status-ts.png"
+  alt="pipeline">
+  <figcaption>Check status</figcaption>
+</figure>
+
+* If everything above looks good, then the pods might be in a failing state. Check the pods deployed to your provisioned namespace and view their events to see if there are any failures, and why. Most likely the issue is a lack of imagePullSecrets in the provisioned namespace. Those can be added by [Tenant Operator's](../tenant-operator/overview.html) [TemplateGroupInstance](../tenant-operator/customresources.html#_5-templategroupinstance) by setting the proper label in your tronador config file.
+
+
+## Cluster Admin support
+
+* In case Helm Release keeps failing, you will need to check Helm Operator's logs to get more information about the failure.
+  * If you see constant helm release failures with the error message `Could not resolve host: github.com` or similar, you might need to change the `dnsConfig` within the template's spec for the Helm Operator's deployment with the following details:
+
+    ```yaml
+      dnsPolicy: "ClusterFirst"
+      dnsConfig:
+        options:
+          - name: ndots
+            value: "1"
+    ```
+
+    The Helm Operator uses an Alpine base image, which has a bad address issue where it cannot resolve addresses unless it has a dot `.` at the end of it. The above workaround will fix this issue.

content/sre/tronador/workflow.md

@@ -0,0 +1,187 @@
+# Workflow guide for Tronador
+
+Please note: The guide below is written as a SAAP customer's point of view. If you use Tronador independently, the guide below might not be applicable to you. However, you can still use this guide to get an idea of how the process works.
+
+For tronador to work, you need to add support for it in your Git Repository by adding a [Tronador config file](./config_file.html). Afterwards, a Tekton pipeline needs to be setup with the [Tronador cluster task](./cluster_task.html), and a cluster task that pushes the output EP to your gitops repository. Test environments should then be created automatically every time a PR is created or updated. The entire Dynamic Test Environment (DTE) creation process is described below.
+
+
+<figure>
+  <img
+  src="./images/workflow.png"
+  alt="pipeline-git">
+  <figcaption>Tronador Workflow</figcaption>
+</figure>
+
+### Tronador config file
+
+The [Tronador config file](./config_file.html) is placed in the github repository and is used to configure the EnvironmentProvisioner CR. Environment Provisioning will only work if the config file exists.
+
+```yaml
+application:
+  chart_path: deploy
+  namespaceLabels:
+    kind: pr
+    team: gabbar
+    stakater.com/tenant: veeru
+  value_overrides:
+    application:
+      deployment:
+        image:
+          repository: {{APPLICATION_IMAGE_NAME}}
+          tag: {{APPLICATION_IMAGE_TAG}}
+```
+
+### Github Event
+ Github (or any other repository management system) events are used to trigger the Tronador pipeline. The pipeline is triggered whenever a PR is opened for a repository that supports tronador.
+
+### Tekton Pipeline
+
+The Tekton Pipeline needs to watch your repo where tronador is configured to be used, and when a PR is created or updated, it will trigger the Tekton pipeline. The pipeline will first create a docker image from the changes in your PR. It will then create an EnvironmentProvisioner CR by using the details in the config file, and replacing its `APPLICATION_IMAGE_NAME` and `APPLICATION_IMAGE_TAG` variables in the config file with the details about the created docker image. Finally it will push the Environment Provisioner to your gitops repository.
+
+<figure>
+  <img
+  src="./images/pipeline-git.png"
+  alt="pipeline-git">
+  <figcaption>Created docker image</figcaption>
+</figure>
+
+<br>
+
+<figure>
+  <img
+  src="./images/pipeline.png"
+  alt="pipeline">
+  <figcaption>A successful pipeline run</figcaption>
+</figure>
+
+### GitOps Repository
+
+A successful pipeline run will create a new Environment Provisioner CR, and push it to your gitops repository.
+
+<figure>
+  <img
+  src="./images/gitops.png"
+  alt="pipeline">
+  <figcaption>Environment Provisioner pushed</figcaption>
+</figure>
+
+### ArgoCD push
+
+ArgoCD is a tool that can be used to watch a repo and push the changes to your cluster. In this case, it will push the Environment Provisioner CR from your gitops repository to your cluster.
+
+<figure>
+  <img
+  src="./images/argocd.png"
+  alt="pipeline">
+  <figcaption>Relevant ArgoCD Application synced</figcaption>
+</figure>
+
+### Environment Provisioner
+
+The EnvironmentProvisioner Custom Resource does the actual work of creating the test environment. It is responsible for creating the test environment, and for updating it when the image is updated. The process for this is to first create a namespace, then create a helm release within the namespace using details about the image to be deployed. These details are gotten from the pipeline and the tronador config file.
+
+```yaml
+apiVersion: tronador.stakater.com/v1alpha1
+kind: EnvironmentProvisioner
+metadata:
+  name: stakater-nordmart-inventory-74
+spec:
+  application:
+    release:
+      chart:
+        git: 'https://github.com/stakater-lab/stakater-nordmart-inventory'
+        path: deploy
+        ref: run-pipeline
+      releaseName: run-pipeline
+      values:
+        application:
+          deployment:
+            image:
+              repository: >-
+                nexus-docker-stakater-nexus.apps.binero-test.8sdzwd1l.kubeapp.cloud/stakater-nordmart-inventory
+              tag: SNAPSHOT-PR-74-6f50ff2f
+  namespaceLabels:
+    kind: pr
+    stakater.com/tenant: veeru
+    team: gabbar
+status:
+  applicationStatus: deployed
+  applicationStatusMsg: >-
+    Release was successful for Helm release 'run-pipeline' in
+    'stakater-nordmart-inventory-74'.
+  currentPhase: Waiting For Next Reconciliation Loop
+  namespaceLabels:
+    kind: pr
+    stakater.com/tenant: veeru
+    team: gabbar
+```
+
+### Helm Release
+
+Helm Releases are watched by the Helm Operator, which manages the creation of all resources listed in a referenced helm chart. This chart is usually placed within the repo itself, and in this case tronador passes along the reference to this chart from the Environment Provisioner to the Helm Release. The only thing that is changed is the image to be deployed, which is passed along from the pipeline with each update to the PR and updated within the helm release automatically.
+
+```yaml
+apiVersion: helm.fluxcd.io/v1
+kind: HelmRelease
+metadata:
+  name: run-pipeline
+  namespace: stakater-nordmart-inventory-74
+  labels:
+    app.kubernetes.io/instance: run-pipeline
+    app.kubernetes.io/managed-by: tronador.stakater.com
+spec:
+  chart:
+    git: 'https://github.com/stakater-lab/stakater-nordmart-inventory'
+    path: deploy
+    ref: run-pipeline
+  releaseName: run-pipeline
+  rollback: {}
+  test: {}
+  values:
+    application:
+      deployment:
+        image:
+          repository: >-
+            nexus-docker-stakater-nexus.apps.binero-test.8sdzwd1l.kubeapp.cloud/stakater-nordmart-inventory
+          tag: SNAPSHOT-PR-74-6f50ff2f
+status:
+  conditions:
+    - lastTransitionTime: '2022-03-21T11:46:19Z'
+      lastUpdateTime: '2022-03-21T11:46:19Z'
+      message: >-
+        Chart fetch failed for Helm release 'run-pipeline' in
+        'stakater-nordmart-inventory-74'.
+      reason: ChartFetchFailed
+      status: 'False'
+      type: ChartFetched
+    - lastTransitionTime: '2022-03-21T11:46:40Z'
+      lastUpdateTime: '2022-03-21T11:46:40Z'
+      message: >-
+        Installation or upgrade succeeded for Helm release 'run-pipeline' in
+        'stakater-nordmart-inventory-74'.
+      reason: Deployed
+      status: 'True'
+      type: Deployed
+    - lastTransitionTime: '2022-03-21T11:46:40Z'
+      lastUpdateTime: '2022-03-21T11:48:01Z'
+      message: >-
+        Release was successful for Helm release 'run-pipeline' in
+        'stakater-nordmart-inventory-74'.
+      reason: Succeeded
+      status: 'True'
+      type: Released
+  lastAttemptedRevision: 6f50ff2ff844f5f7c2b6627e3d7319e4717ac847
+  observedGeneration: 1
+  phase: Succeeded
+  releaseName: run-pipeline
+  releaseStatus: deployed
+  revision: 6f50ff2ff844f5f7c2b6627e3d7319e4717ac847
+```
+
+### Secrets management
+
+Secrets for the helm chart to be deployed are currently passed along from the tronador config file, to the helm release. You can also use [Tenant Operator's](../tenant-operator/overview.html) [TemplateGroupInstance](../tenant-operator/customresources.html#_5-templategroupinstance) to pass secrets to the namespace that will be provisioned by the EnvironmentProvisioner by setting the proper label in your tronador config file.
+
+### Pods Deployed
+
+After the entire above process is completed, the Helm Release will create a deployment that will deploy the pods for the application and its changes mentioned in the PR to the namespace created by the Environment Provisioner.