Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chore(Test): Addition of E2E test for Azure Experiments #354

Merged
merged 16 commits into from
Jan 15, 2022
Prev Previous commit
Next Next commit
Added disk loss e2e test
Signed-off-by: Akash Shrivastava <akash@chaosnative.com>
  • Loading branch information
Akash Shrivastava committed Jan 6, 2022
commit e2c068efbc2064079066e639ba4a4e29db730a64
102 changes: 102 additions & 0 deletions .github/workflows/azure-experiment-pipeline.yml
Original file line number Diff line number Diff line change
@@ -4,15 +4,25 @@ on:
workflow_dispatch:
inputs:
goExperimentImage:
description: "Go Experiment Image"
default: "litmuschaos/go-runner:ci"
required: true
operatorImage:
description: "Operator Image"
default: "litmuschaos/chaos-operator:ci"
required: true
runnerImage:
description: "Runner Image"
default: "litmuschaos/chaos-runner:ci"
required: true
chaosNamespace:
description: "Chaos Namespace"
default: "default"
required: true
experimentImagePullPolicy:
description: "Experiment Image Pull Policy"
default: "Always"
required: true

jobs:
Azure_VM_Instance_Stop:
@@ -99,3 +109,95 @@ jobs:
- name: Deleting K3S cluster
if: always()
run: /usr/local/bin/k3s-uninstall.sh

Azure_VM_Disk_Loss:
runs-on: ubuntu-latest
steps:

#Install and configure a k3s cluster
- name: Installing Prerequisites (K3S Cluster)
env:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
run: |
curl -sfL https://get.k3s.io | sh -s - --docker --write-kubeconfig-mode 664
kubectl wait node --all --for condition=ready --timeout=90s
mkdir -p $HOME/.kube && cat /etc/rancher/k3s/k3s.yaml > $HOME/.kube/config
kubectl get nodes

- uses: actions/checkout@v2

- uses: actions/setup-go@v2
with:
go-version: '1.14'
avaakash marked this conversation as resolved.
Show resolved Hide resolved

- name: Create azure auth file from secrets
run: echo "${{ secrets.AZURE_AUTH_FILE }}" > azure.auth

- name: Create Kubernetes secret for azure experiment
if: always()
env:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
run: |
kubectl create secret generic cloud-secret --from-file=azure.auth

- name: Azure Login
uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}

- name: Create a Azure VM Instance with target Disk Volumes
if: always()
run: |
az group create -n ${{ secrets.AZURE_RESOURCE_GROUP }} -l eastus && \
az vm create -g ${{ secrets.AZURE_RESOURCE_GROUP }} -n ${{ secrets.INSTANCE_ONE }} --image UbuntuLTS --generate-ssh-keys && \
az disk create -n ${{ secrets.DISK_ONE }} -g ${{ secrets.AZURE_RESOURCE_GROUP }} --size-gb=4 && \
az disk create -n ${{ secrets.DISK_TWO }} -g ${{ secrets.AZURE_RESOURCE_GROUP }} --size-gb=4 && \
az vm disk attach -g ${{ secrets.AZURE_RESOURCE_GROUP }} --vm-name ${{ secrets.INSTANCE_ONE }} -n ${{ secrets.DISK_ONE }} && \
az vm disk attach -g ${{ secrets.AZURE_RESOURCE_GROUP }} --vm-name ${{ secrets.INSTANCE_ONE }} -n ${{ secrets.DISK_TWO }}
- name: Litmus Infra Setup
if: always()
run: make build-litmus
env:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
OPERATOR_IMAGE: "${{ github.event.inputs.operatorImage }}"
RUNNER_IMAGE: "${{ github.event.inputs.runnerImage }}"

- name: Run Azure VM Disk Loss experiment in serial & parallel mode
if: always()
env:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
AZURE_INSTANCE_NAME: "${{ secrets.INSTANCE_ONE }},${{ secrets.INSTANCE_TWO }}"
AZURE_DISK_NAME: "${{ secrets.DISK_ONE }},${{ secrets.DISK_TWO }}"
AZURE_RESOURCE_GROUP: "${{ secrets.AZURE_RESOURCE_GROUP }}"
GO_EXPERIMENT_IMAGE: "${{ github.event.inputs.goExperimentImage }}"
EXPERIMENT_IMAGE_PULL_POLICY: "${{ github.event.inputs.experimentImagePullPolicy }}"
CHAOS_NAMESPACE: "${{ github.event.inputs.chaosNamespace }}"
run: make azure-disk-loss

- name: Delete the VM Instance and target Disk Volumes
if: always()
run: |
az vm disk detach -g ${{ secrets.AZURE_RESOURCE_GROUP }} --vm-name ${{ secrets.INSTANCE_ONE }} -n ${{ secrets.DISK_ONE }} && \
az vm disk detach -g ${{ secrets.AZURE_RESOURCE_GROUP }} --vm-name ${{ secrets.INSTANCE_ONE }} -n ${{ secrets.DISK_TWO }} && \
az disk delete -n ${{ secrets.DISK_ONE }} -g ${{ secrets.AZURE_RESOURCE_GROUP }} --yes && \
az disk delete -n ${{ secrets.DISK_TWO }} -g ${{ secrets.AZURE_RESOURCE_GROUP }} --yes && \
az vm delete --ids $(az vm list -g ${{ secrets.AZURE_RESOURCE_GROUP }} --query "[].id" -o tsv) --yes
az group delete -n ${{ secrets.AZURE_RESOURCE_GROUP }} --yes

- name: "[Debug]: check chaos resources"
if: ${{ failure() }}
continue-on-error: true
run: |
bash <(curl -s https://raw.githubusercontent.com/litmuschaos/litmus-e2e/master/build/debug.sh)

- name: "[Debug]: check operator logs"
if: ${{ failure() }}
continue-on-error: true
run: |
operator_name=$(kubectl get pods -n litmus -l app.kubernetes.io/component=operator --no-headers | awk '{print$1}')
kubectl logs $operator_name -n litmus > logs.txt
cat logs.txt

- name: Deleting K3S cluster
if: always()
run: /usr/local/bin/k3s-uninstall.sh
92 changes: 92 additions & 0 deletions .github/workflows/nightly-azure-experiment-pipeline.yml
Original file line number Diff line number Diff line change
@@ -89,3 +89,95 @@ jobs:
- name: Deleting K3S cluster
if: always()
run: /usr/local/bin/k3s-uninstall.sh

Azure_VM_Disk_Loss:
runs-on: ubuntu-latest
steps:

#Install and configure a k3s cluster
- name: Installing Prerequisites (K3S Cluster)
env:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
run: |
curl -sfL https://get.k3s.io | sh -s - --docker --write-kubeconfig-mode 664
kubectl wait node --all --for condition=ready --timeout=90s
mkdir -p $HOME/.kube && cat /etc/rancher/k3s/k3s.yaml > $HOME/.kube/config
kubectl get nodes

- uses: actions/checkout@v2

- uses: actions/setup-go@v2
with:
go-version: '1.14'
avaakash marked this conversation as resolved.
Show resolved Hide resolved

- name: Create azure auth file from secrets
run: echo "${{ secrets.AZURE_AUTH_FILE }}" > azure.auth

- name: Create Kubernetes secret for azure experiment
if: always()
env:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
run: |
kubectl create secret generic cloud-secret --from-file=azure.auth

- name: Azure Login
uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}

- name: Create a Azure VM Instance with target Disk Volumes
if: always()
run: |
az group create -n ${{ secrets.AZURE_RESOURCE_GROUP }} -l eastus && \
az vm create -g ${{ secrets.AZURE_RESOURCE_GROUP }} -n ${{ secrets.INSTANCE_ONE }} --image UbuntuLTS --generate-ssh-keys && \
az disk create -n ${{ secrets.DISK_ONE }} -g ${{ secrets.AZURE_RESOURCE_GROUP }} --size-gb=4 && \
az disk create -n ${{ secrets.DISK_TWO }} -g ${{ secrets.AZURE_RESOURCE_GROUP }} --size-gb=4 && \
az vm disk attach -g ${{ secrets.AZURE_RESOURCE_GROUP }} --vm-name ${{ secrets.INSTANCE_ONE }} -n ${{ secrets.DISK_ONE }} && \
az vm disk attach -g ${{ secrets.AZURE_RESOURCE_GROUP }} --vm-name ${{ secrets.INSTANCE_ONE }} -n ${{ secrets.DISK_TWO }}
- name: Litmus Infra Setup
if: always()
run: make build-litmus
env:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
OPERATOR_IMAGE: "${{ github.event.inputs.operatorImage }}"
RUNNER_IMAGE: "${{ github.event.inputs.runnerImage }}"

- name: Run Azure VM Disk Loss experiment in serial & parallel mode
if: always()
env:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
AZURE_INSTANCE_NAME: "${{ secrets.INSTANCE_ONE }},${{ secrets.INSTANCE_TWO }}"
AZURE_DISK_NAME: "${{ secrets.DISK_ONE }},${{ secrets.DISK_TWO }}"
AZURE_RESOURCE_GROUP: "${{ secrets.AZURE_RESOURCE_GROUP }}"
GO_EXPERIMENT_IMAGE: "${{ github.event.inputs.goExperimentImage }}"
EXPERIMENT_IMAGE_PULL_POLICY: "${{ github.event.inputs.experimentImagePullPolicy }}"
CHAOS_NAMESPACE: "${{ github.event.inputs.chaosNamespace }}"
run: make azure-disk-loss

- name: Delete the VM Instance and target Disk Volumes
if: always()
run: |
az vm disk detach -g ${{ secrets.AZURE_RESOURCE_GROUP }} --vm-name ${{ secrets.INSTANCE_ONE }} -n ${{ secrets.DISK_ONE }} && \
az vm disk detach -g ${{ secrets.AZURE_RESOURCE_GROUP }} --vm-name ${{ secrets.INSTANCE_ONE }} -n ${{ secrets.DISK_TWO }} && \
az disk delete -n ${{ secrets.DISK_ONE }} -g ${{ secrets.AZURE_RESOURCE_GROUP }} --yes && \
az disk delete -n ${{ secrets.DISK_TWO }} -g ${{ secrets.AZURE_RESOURCE_GROUP }} --yes && \
az vm delete --ids $(az vm list -g ${{ secrets.AZURE_RESOURCE_GROUP }} --query "[].id" -o tsv) --yes
az group delete -n ${{ secrets.AZURE_RESOURCE_GROUP }} --yes

- name: "[Debug]: check chaos resources"
if: ${{ failure() }}
continue-on-error: true
run: |
bash <(curl -s https://raw.githubusercontent.com/litmuschaos/litmus-e2e/master/build/debug.sh)

- name: "[Debug]: check operator logs"
if: ${{ failure() }}
continue-on-error: true
run: |
operator_name=$(kubectl get pods -n litmus -l app.kubernetes.io/component=operator --no-headers | awk '{print$1}')
kubectl logs $operator_name -n litmus > logs.txt
cat logs.txt

- name: Deleting K3S cluster
if: always()
run: /usr/local/bin/k3s-uninstall.sh