Skip to content

A set of load tests for OCM's clusters-service, based on vegeta.

License

Notifications You must be signed in to change notification settings

cloud-bulldozer/ocm-api-load

Folders and files

NameName
Last commit message
Last commit date
Jul 4, 2024
Nov 22, 2022
Jul 1, 2024
Jul 15, 2022
Aug 9, 2022
Jul 4, 2024
Jul 1, 2024
Dec 7, 2022
Aug 19, 2021
May 13, 2022
Sep 19, 2022
Aug 8, 2022
Sep 19, 2022
Jul 1, 2024
Jul 1, 2024
Oct 20, 2021

Repository files navigation

OCM API LOAD TEST

Go Version Python Version

A set of load tests for OCM's clusters-service, based on vegeta.

Requirements

  • Go >= 1.15

To get all modules to local cache run

go mod dowload

How to run?

Compile using make and run as a simple binary:

./ocm-load-test --test-id=foo --ocm-token=$OCM_TOKEN --duration=20m --rate=5/s --output-path=./results --test-names="<test_name>[,...]"
./ocm-load-test --config-file /path/to/config

if your config-file is named config.yaml and it in the same path as your binary, it will be autodetected and you could run by just calling it:

./ocm-load-test

Flags

Note: Flags always take precedence over config file.

Default values don't count for precedence.

      --aws-access-key string      AWS access key
      --aws-access-secret string   AWS access secret
      --aws-account-id string      AWS Account ID, is the 12-digit account number.
      --aws-region string          AWS region (default "us-west-1")
      --config-file string         config file (default "config.yaml")
      --cooldown int               Cooldown time between tests in seconds. (default 10)
      --duration int               Duration of each individual run in minutes. (default 1)
      --elastic-index string       Elasticsearch index to store the documents
      --elastic-password string    Elasticsearch Password for authentication
      --elastic-server string      Elasticsearch cluster URL
      --elastic-user string        Elasticsearch User for authentication
      --elastic-insecure-skip-verify bool        Elasticsearch skip tls verifcation during authentication
      --end-rate int               Ending request per second rate. (E.g.: 5 would be 5 req/s)
      --gateway-url string         Gateway url to perform the test against (default "https://api.integration.openshift.com")
  -h, --help                       help for ocm-api-load
      --ocm-token string           OCM Authorization token
      --ocm-token-url string       Token URL (default "https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token")
      --output-path string         Output directory for result and report files (default "results")
      --ramp-duration int          Duration of ramp in minutes, before normal execution. (default 0)
      --ramp-steps int             Number of stepts to get from start rate to end rate. (Minimum 2 steps)
      --ramp-type string           Type of ramp to use for all tests. (linear, exponential)
      --rate string                Rate of the attack. Format example 5/s. (Available units 'ns', 'us', 'ms', 's', 'm', 'h') (default "1/s")
      --start-rate int             Starting request per second rate. (E.g.: 5 would be 5 req/s)
      --test-id string             Unique ID to identify the test run. UUID is recommended (default "c160dab1-7fa3-4965-9797-47da16e5c1b9")
      --test-names strings         Names for the tests to be run.
  -v, --verbose                    set this flag to activate verbose logging.

Tests

Name Endpoint Method
self-access-token /api/accounts_mgmt/v1/access_token POST
list-subscriptions /api/accounts_mgmt/v1/subscriptions GET
access-review /api/authorizations/v1/access_review POST
register-new-cluster /api/accounts_mgmt/v1/cluster_registrations POST
register-existing-cluster /api/accounts_mgmt/v1/cluster_registrations POST
create-cluster /api/clusters_mgmt/v1/clusters POST
list-clusters /api/clusters_mgmt/v1/clusters GET
get-current-account /api/accounts_mgmt/v1/current_account GET
quota-cost /api/accounts_mgmt/v1/organizations/{orgId}/quota_cost GET
resource-review /api/authorizations/v1/resource_review POST
cluster-authorizations /api/accounts_mgmt/v1/cluster_authorizations POST
self-terms-review /api/authorizations/v1/self_terms_review POST
certificates /api/accounts_mgmt/v1/certificates POST
create-services /api/service_mgmt/v1/services POST
get-services /api/service_mgmt/v1/services GET
patch-services /api/service_mgmt/v1/services/{srvcId} PATCH
-- -- --

Config file

Global options

  • ocm-token: OCM Authorization token
  • ocm-token-url: Token URL (default "https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token")
  • gateway-url: Gateway url to perform the test against (default "https://api.integration.openshift.com")
  • client:
    • id: OpenID client identifier.
    • secret: OpenID client secret.
  • aws:
    • region: AWS region (default "us-west-1")
    • access-key: AWS access key
    • secret-access-key: AWS access secret
    • account-id: AWS Account ID, is the 12-digit account number
    • account-name: AWS Account Name to be used in the requests
  • output-path: Path to output results.
  • duration: Duration of each individual run in minutes. (default 1)
  • cooldown: Cooldown time between tests in seconds. (default 10 s)
  • rate: Rate of the attack. Format example 5/s. (Available units 'ns', 'us', 'ms', 's', 'm', 'h') (default "1/s")
  • test-id: Unique ID to identify the test run. UUID is recommended (default "dc049b1d-92b4-420c-9eb7-34f30229ef46")
  • ramp-type: Type of ramp to use for all tests. (linear, exponential)
  • ramp-duration: Duration of ramp in minutes, before normal execution. (default 0)
  • start-rate: Starting request per second rate. (E.g.: 5 would be 5 req/s)
  • end-rate: Ending request per second rate. (E.g.: 5 would be 5 req/s)
  • ramp-steps: Number of stepts to get from start rate to end rate. (Minimum 2 steps)
  • tests: List of the tests to run. Empty list means all.
  • elastic:
    • server: Elasticsearch cluster URL
    • user: Elasticsearch User for authentication
    • password: Elasticsearch Password for authentication
    • index: Elasticsearch index to store the documents

Test options

Each test can contain this options:

  • rate: Rate of the attack. Format example 5/s. (Available units 'ns', 'us', 'ms', 's', 'm', 'h') (default "1/s")
  • duration: Override duration for the test. (A positive integer accompanied of a valid unit)

Ramping functionality

Each test can have a specific configuration for ranmping up the rate, inthis case the following options must be provided.

  • duration: in minutes
  • ramp-type: Type of ramp to use for all tests. (linear, exponential)
  • ramp-duration: Duration of ramp in minutes, before normal execution. (default 0)
  • start-rate: Starting request per second rate. (E.g.: 5 would be 5 req/s)
  • end-rate: Ending request per second rate. (E.g.: 5 would be 5 req/s)
  • ramp-steps: Number of stepts to get from start rate to end rate. (Minimum 2 steps)

rate option is not needed for this.

Example
  cluster-authorizations:
    duration: 30
    ramp-type: exponential
    ramp-duration: 10
    start-rate: 1
    end-rate: 50
    ramp-steps: 6

Obligatory options

  • ocm-token
  • gateway-url

Defaults

  • output-path: set to ./results
  • duration: set to 1 minute

Minimal yaml config file

---
token: xxxXXXyyyYYYzzzZZZ
gateway-url: https://api.my-env.openshift.com

Full yaml config file

See example

Reporting

Requirements

External

vegeta executable is necessary

$ go get -u github.com/tsenart/vegeta

python requirements

$ python3 -m venv env
$ . ./env/bin/activate
$ pip3 install -r requirements.txt

Usage

To generate the report run the following command:

python3 automation.py --dir /tests/2021-05-18

The first argument should be the path to the folder where the results folder is located.

Graph a specific file

python3 automation.py graph --dir /tests/2021-05-18/results/ --filename access_review.json

This should open the browser with an interactive Graph for access review.

Generate vegeta reports

python3 automation.py report --dir /tests/2021-05-18

This will generate all the vegeta report files for each result file

When done deactivate virtual environment

$ deactivate

How to release

Steps:

  • Add your GitHub token to the env variable GITHUB_TOKEN

  • Make sure you have github-release installed

  • Be sure you are in the latest version of main branch and have bumped the version

  • Now you are ready to run make release this will build the binary and generate the tarfiles that contain all the needed files

Ramping Up Theory

The test will run a number with a running time of / rounded for each step, this can sometimes make the test last more or less than the expected duration, but we want to have a even distribution of times.

As each step finishes it will increase the rate according to a delta that is calculated with the parameters:

For both types of ramps we have common behaviour:

  • First rate: is always start-rate
  • Last rate: is always end-rate
  • Since we cannot use float values for rates, we round all the rates to it's closest integer.

For a linear ramp it will use this formula

delta = ( end-rate - start-rate ) / ( ramp-steps - 1 )

ramp-steps, has always have to be greater than 1

So the new rate will be:

newRate = oldRate + delta

For an exponential distribution

We are using the exponential formula f(t)= x * <coeff> ^ t

the coeff is calculated with this formula

coeff = (end-rate / start-rate) ^ (1 / ramp-steps)

So the new rate will be:

newRate = start-rate * coeff ^ <# of step>

duration vs ramp-duration

The duration is the number of minutes the test is going to run. The ramp-duration is the number of minutes the ramp is going to last.

  • If ramp-duration is not set, the ramp will take the whole duration.
  • If ramp-duration is set, it will run the ramp for that long and then run the remaining of the duration at the end-rate.
    • E.g.: if duration is 30 minutes and ramp-duration is 20 minutes. The test will run a ramp for 20 minutes and keep running at end-rate for the remaining 10 minutes. So it will run end-rate for duration - ramp-duration minutes.
  • If ramp-duration is greater than duration it will just perform a ramp for ramp-duration minutes.

Overrides for the values work the same, localized test values take priority over global values.