Skip to content
This repository has been archived by the owner on Feb 3, 2021. It is now read-only.

Commit

Permalink
Merge pull request #1 from syakir-uzair/master
Browse files Browse the repository at this point in the history 
feat: Add support for file upload and improve error handling
  • Loading branch information
flowerinthenight authored Feb 3, 2021
2 parents d7189d7 + 59a1bbc commit 1aa48fc
Show file tree
Hide file tree
Showing 4 changed files with 140 additions and 101 deletions.
140 changes: 75 additions & 65 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,10 +1,15 @@
# OOPS

![main](https://github.com/flowerinthenight/oops/workflows/main/badge.svg)

## Overview

`oops` is an automation-friendly, highly-scalable, and scriptable API/generic testing tool built to run on [Kubernetes](https://kubernetes.io/). It accepts and runs file-based test cases, or 'scenarios' written in [YAML](https://yaml.org/).

## Running in a local environment

You can install the binary and run local scenario file(s). This is useful for testing your scenario file(s) before deployment.

```bash
# Install the binary.
$ brew tap flowerinthenight/tap
Expand All @@ -23,6 +28,7 @@ $ oops --dir ./examples/
```

## Deploying to Kubernetes

To scale the testing workload, this tool will attempt to distribute all scenario files to all worker pods using pub/sub messaging (currently supports SNS+SQS, and GCP PubSub). At the moment, it needs to be triggered first before the actual execution starts. The trigger payload is `{"code":"start"}`.

It is recommended that your scenario files are isolated at the file level. That means that as much as possible, a single scenario file is standalone. For integration tests, a single scenario file could contain multiple related test cases (i.e. create, inspect, delete type of tests) in it.
Expand All @@ -32,20 +38,22 @@ Although this tool was built to run on k8s, it will work just fine in any enviro
An example [`deployment.yaml`](https://github.com/flowerinthenight/oops/blob/master/deployment.yaml) for k8s using GCP PubSub is provided for reference. Make sure to update the relevant values for your own setup.

## Scenario file

The following is the specification of a valid scenario file. All scenario files must have a `.yaml` extension.

```yaml
tags:
# Tag(s) for this scenario file. If these tag combinations match with what is
# provided in the --tags flag (if any), this scenario file is allowed to run.
key1: value1
key2: value2

env:
# These key/values will be added to the environment variables in your local
# or in the pod oops will be running on.
ENV_KEY1: value1
ENV_KEY2: value2

# Any value that starts with '#!' (i.e. #!/bin/bash) will be written to disk as
# an executable script file and the resulting output combined from stdout & stderr
# will become the final evaluated value. This is useful if you chain http calls,
Expand All @@ -68,69 +76,69 @@ prepare: |
# A list of http requests to perform sequentially. This tool will continue running
# all the list entries even if failure occurs during the execution.
run:
- http:
method: POST

# Filename: <tempdir>/<scenario-filename>.yaml_run<index>_url
# Example: /tmp/scenario01.yaml_run0_url
url: "https://service.alphaus.cloud/users"

# Filename: <tempdir>/<scenario-filename>.yaml_run<index>_hdr.<key>
# Example: /tmp/scenario01.yaml_run0_hdr.Authorization
headers:
Authorization: |
#!/bin/bash
echo -n "Bearer $TOKEN"
Content-Type: application/json

# Filename: <tempdir>/<scenario-filename>.yaml_run<index>_qparams.<key>
# Example: /tmp/scenario01.yaml_run0_qparams.key2
query_params:
key1: value1
key2: |
#!/bin/bash
echo -n "$KEY2"
# Filename: <tempdir>/<scenario-filename>.yaml_run<index>_forms.<key>
# Example: /tmp/scenario01.yaml_run0_forms.key2
forms:
key1: value1
key2: |
#!/bin/bash
echo -n "$KEY2"
# Filename: <tempdir>/<scenario-filename>.yaml_run<index>_payload
# Example: /tmp/scenario01.yaml_run0_payload
payload: |
{"key1":"value1","key2":"value2"}
# If response payload is not empty, its contents will be written in the
# file below. Useful if you want to refer to it under 'asserts.script'
# and/or 'check'.
response_out: /tmp/out.json

asserts:
# The expected http status code. Indicates a failure if not equal.
status_code: 200

# JSON validation using https://github.com/xeipuuv/gojsonschema package.
validate_json: |
{
"type": "object",
"properties": {
...
- http:
method: POST

# Filename: <tempdir>/<scenario-filename>.yaml_run<index>_url
# Example: /tmp/scenario01.yaml_run0_url
url: "https://service.alphaus.cloud/users"

# Filename: <tempdir>/<scenario-filename>.yaml_run<index>_hdr.<key>
# Example: /tmp/scenario01.yaml_run0_hdr.Authorization
headers:
Authorization: |
#!/bin/bash
echo -n "Bearer $TOKEN"
Content-Type: application/json

# Filename: <tempdir>/<scenario-filename>.yaml_run<index>_qparams.<key>
# Example: /tmp/scenario01.yaml_run0_qparams.key2
query_params:
key1: value1
key2: |
#!/bin/bash
echo -n "$KEY2"
# Filename: <tempdir>/<scenario-filename>.yaml_run<index>_forms.<key>
# Example: /tmp/scenario01.yaml_run0_forms.key2
forms:
key1: value1
key2: |
#!/bin/bash
echo -n "$KEY2"
# Filename: <tempdir>/<scenario-filename>.yaml_run<index>_payload
# Example: /tmp/scenario01.yaml_run0_payload
payload: |
{"key1":"value1","key2":"value2"}
# If response payload is not empty, its contents will be written in the
# file below. Useful if you want to refer to it under 'asserts.script'
# and/or 'check'.
response_out: /tmp/out.json

asserts:
# The expected http status code. Indicates a failure if not equal.
status_code: 200

# JSON validation using https://github.com/xeipuuv/gojsonschema package.
validate_json: |
{
"type": "object",
"properties": {
...
}
}
}
# A non-zero return value indicates a failure.
# Filename: <tempdir>/<scenario-filename>.yaml_run<index>_assertscript
# Example: /tmp/scenario01.yaml_run0_assertscript
script: |
#!/bin/bash
if [[ "$(cat /tmp/out.json | jq -r .username)" != "user01" ]]; then
echo "try fail"
exit 1
fi
# A non-zero return value indicates a failure.
# Filename: <tempdir>/<scenario-filename>.yaml_run<index>_assertscript
# Example: /tmp/scenario01.yaml_run0_assertscript
script: |
#!/bin/bash
if [[ "$(cat /tmp/out.json | jq -r .username)" != "user01" ]]; then
echo "try fail"
exit 1
fi
# A script to run after 'run', if present. Useful also as a standalone script
# in itself, if 'run' is empty. A non-zero return value indicates a failure.
Expand All @@ -143,10 +151,12 @@ check: |
Example [scenario files](https://github.com/flowerinthenight/oops/tree/master/examples) are provided for reference as well. You can run them as is.
----
---
## TODO
PR's are welcome!
- [ ] Parsing and assertions for response JSON payloads
- [x] Labels/tags for filtering what tests to run
- [x] Support for other scripting engines other than `bash/sh`, i.e. Python
Expand Down
22 changes: 16 additions & 6 deletions main.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ type cmd struct {

// To identify a batch. Sent by the initiator together with
// the 'process' code.
Id string `json:"id"`
ID string `json:"id"`

// The file to process. Sent together with the 'process' code.
Scenario string `json:"scenario"`
Expand Down Expand Up @@ -97,8 +97,17 @@ func combineFilesAndDir() []string {
})

var final []string
for k, _ := range tmp {
final = append(final, k)
for k := range tmp {
_, err := os.Stat(k)
if os.IsNotExist(err) {
log.Printf("File does not exist: %v", k)
} else {
final = append(final, k)
}
}

if len(final) == 0 {
log.Fatal("No files found. Please recheck directory.")
}

return final
Expand All @@ -110,7 +119,7 @@ func distributePubsub(app *appctx) {
for _, f := range final {
nc := cmd{
Code: "process",
Id: id,
ID: id,
Scenario: f,
}

Expand Down Expand Up @@ -141,7 +150,7 @@ func distributeSQS(app *appctx) {
for _, f := range final {
nc := cmd{
Code: "process",
Id: id,
ID: id,
Scenario: f,
}

Expand Down Expand Up @@ -249,7 +258,8 @@ func run(ctx context.Context, done chan error) {
}

app := &appctx{mtx: &sync.Mutex{}}
ctx0, _ := context.WithCancel(ctx)
ctx0, cancelCtx0 := context.WithCancel(ctx)
defer cancelCtx0()
done0 := make(chan error, 1)

switch {
Expand Down
Loading

0 comments on commit 1aa48fc

Please sign in to comment.