As HAC-dev is communicating with a lot of different components, we want to make sure that the API is valid. That's why we introduced contract tests to our repo. We're using the Pact framework for our tests. This documentation is describing Pact tests in the context of HAC-dev and RHTAP. It is not going to the very details of a Pact functionality, so consider visiting the official Pact documentation. Once the tests are set up and running, we don't expect them to change often. They would be there mainly to raise our confidence that API is not broken in any way and notify us if some breaking change is coming.
- HAC-dev contract testing using Pact
The main idea of contract testing is to ensure that the communication between two components is not broken. To do that, the consumer specifies the contract and the provider makes sure it's not breaking it with code changes.
In our case, HAC-dev is the only consumer so far. But several providers are talking with it (HAS, SPI...). In the future, it may be possible to add some other custom consumers and their contracts to the already existing ones.
The advantage of those tests is that they run together with unit tests, which means:
- they are cheap on resources - they use the same setup as unit tests
- they are relatively fast, compared with the e2e tests
- they don't need both consumer and provider running during the verification
- the history is versioned and stored in one place
That said, we make sure that the API between components isn't broken with recent changes right after the developer makes a change locally - no need to wait for the complex and expensive tests (integration tests, e2e tests).
A developer making a change on a provider is ensured that an existing contract is still valid by simply running unit tests.
The consumer creates the contract, it specifies the request it sends and the expected fields it should get back in the response. For those fields, the consumer may specify just their type (e.g. field "id" is a "string"), exact data it should have for that particular request (e.g. "type" should be "Created"), or create a matcher (e.g. "date" should be in format "mm-dd-yyyy"). When the contract is created, the consumer pushes it to the Pact broker.
Pact broker is an app, running on a cluster (cluster used for HAC-dev PR checks, maintained by HAC-dev QEs). It stores all contracts, their versions (from both the consumer and provider side), and verification results. Consumer pushes generated contracts to the Pact broker and the provider download contracts from there to verify them.
The provider is not creating any contracts, it just makes sure that it's not breaking any existing ones. During verification on the provider side, the contracts are downloaded and verified. The status of the verification is sent back to the Pact broker.
The tests are executed during different phases of a product lifecycle. They can be executed during unit tests or local development and can be also used as a gating during promotion to different environments.
A contract is stored in a json file. The name of a contract has this structure: <consumer>-<provider>.json. It contains three parts - the example response with actual data, the matchers for the fields where some other matcher was used then the exact value of the field, and the metadata of the contract.
Apart from the request and expected response, a contract also specifies a state. A state describes a setup of a system before the request is executed. This is a crucial part of a test and the main work for the dev/qe to properly define and implement the state. More information about the state can be found in the official documentation.
Let's demonstrate that on the Get application test. The contract specifies the GET request (URL, body, etc.) and what the expected response is (e.g. body includes a kind field with the text value Application). The contract also specifies the state, which can be something like App MyApp exists and has component MyComponent. And the job is done for the consumer.
Now it's the provider's turn to interpret this state. In our example, the provider would have StateHandler defined with the description same as in the consumer and the actual code, that has to be done to fulfill this state. In the code, it can look like this:
pactTypes.StateHandlers{
"App MyApp exists and has component MyComp": createAppAndComponents(myAppNamespace, "MyApp", "MyComp"),
}With the state and logic defined, Pact knows what to execute before that particular Pact verification.
Pact provides a simple API to define arbitrary states along with any parameters imaginable. However, in order to keep a comprehensive list of all states and parameters in a single place, we have slightly extended this functionality.
In the pact-tests/states folder you will find two files. state-params.ts is where we define types/interfaces for different kinds of state parameters, to ensure type safety.
states.ts serves as the single source of truth for provider states of our consumer. As such it contains:
ProviderStatesenum: this is the list of state descriptions, when a new state needs to be defined, add it herestateParamsrecord: this maps entries fromProviderStatesto their sample parameters. State parameters may be any JSON orundefined, but we do prefer them being cast to a more specific type (such as those defined instate-params.ts). When adding a new state, make sure to add a sample with parameters here, since it also serves as basic validation of the params at runtime.setStatefunction: extended version of pact provider state API. We recommend using this function to declare provider states in pact tests, since it will check your state against the existingProviderStatesand enforce any new state be added there.
For example, given we have the following state defined in our states.ts file:
export enum ProviderStates {
appExists = "Application exists",
}
const stateParams: Record<ProviderStates, JsonMap | undefined> = {
"Application exists": { appName: 'app', namespace: 'default' } as ApplicationParams,
}We can then utilize it in the tests as follows, using the setState function:
pactWith({ consumer: 'HACdev', provider: 'HAS' }, (interaction) => {
interaction('Getting application', ({ provider, execute }) => {
beforeEach(() => {
setState(provider, ProviderStates.appExists, { appName: 'x', namespace: 'foo' });
setState(provider, ProviderStates.appExists, { appName: 'y', namespace: 'foo' })
.uponReceiving('Get app with its components.')
.withRequest(contract.request)
.willRespondWith(contract.response);
});Here we set two states for different applications to exist. Pact is going to combine any declared states together, until it encounters the withRequest and willRespondWith calls. At that point, the interaction is complete.
The tables below describe when the tests are running and what is tested/published.
Consumer (HAC-dev):
| Event | What is checked | Pushed to Pact broker | Implemented |
|---|---|---|---|
Locally running yarn pact |
Contract is generated, no verification is running | No | Yes |
| PR update | Contract is generated, no verification is running | Yes commit SHA is a version tagged by PR number |
Yes job |
| PR merge | Contract is generated, no verification is running | Yes commit SHA is a version tagged by branch "main" |
Yes job |
Provider:
| Event | What is checked | Pushed to Pact broker |
|---|---|---|
| Locally as part of unit tests | Runs verification against consumer "main" branch |
No |
| PR update | Runs verification against consumer "main" branch and all environments |
No* |
| PR merge | TBD | Yes commit SHA is a version tagged by branch "main" |
* The idea was to push also those tags, but for now, nothing is pushed as we don't have access to the secrets from this GH action.
Pact tests should be also used as a gating during the promotion of a change to staging or production environments. Pact has a can-i-deploy functionality, that checks specified versions of consumer and provider and responds with the information, whether those changes are compatible or not. This part is not implemented yet.
Pact broker is an application running on the cluster maintained by HAC-dev QEs. You can read more about it in official documentation. If you want to know more about the actual setup, see the HAC-dev QE infra-deployments fork documentation.
The rest of this documentation is dedicated to the implementation details of HAC-dev participant. If you're interested in seeing detailed information about implementation on providers, follow the appropriate link: HAS
HAC-dev is a consumer, so in this repo, we generate the contracts, that are pushed to the Pact broker.
The tests are stored in the pact-tests folder and end with *.pact.spec.ts. The rest of the files are test support.
The generated pacts are stored in the pact/pacts folder.
The whole test is wrapped in the pactWith function. Make sure to import the function from pact v3, as per this example:
import { pactWith } from 'jest-pact/dist/v3';
pactWith({ consumer: 'HACdev', provider: 'HAS' }, (interaction) => {
interaction('Application creation', ({ provider, execute }) => {
beforeEach(() => {
provider
.uponReceiving('Create an application.')
.withRequest(createContract.request)
.willRespondWith(createContract.response);
});
execute('Create an application.', async (mockserver) => {
const product = await mockK8sCreateResource(createContract, mockserver);
expect(product.kind).toEqual(createContract.model.kind);
});
});
});The general shape of a pact spec:
interactionserves as the suite descriptionproviderAPI lets us declare the state that provider side needs to implement using StateHandlers, as well as what request is going to be sent, and the expected response to the requestexecuteconstitutes a test case, this is where the actual request needs to be made, in our case we will be making the request to themockserverprovided by pact
To generate the contract, the test executes the method that calls the request and captured our expected response to the contract. HAC-dev is using @openshift/dynamic-plugin-sdk-utils for the request execution, so we have to mock its behavior. This is an additional step and is usually not required for Pact tests themselves.
To make writing new pacts a little easier, we introduced a couple of conventions and helper functions.
- We recommend defining the contract details in a separate file in the
contractsfolder. If for example, I create a spec file calledapplication.pact.spec.tsand would like to test getting the application, then the contract would be defined incontracts/application-service/application-get.ts. Please note that spec files are created per tested resource, meanwhile the contract files are created per interaction, which should increase the readability. Test file names respect<resource>.spec.tsnaming convention and contract files respect<resourece>-<interaction_type>.tsone. Application-service denoting the provider for this contract. - Make use of the
PactContractinterface defined incontracts/contracts.tswhen defining a contract. This interface includes all the necessary information for pact tests: name and namespace of the resource in question, request definition, and expected response in pact-friendly format. It is also generic to anyK8sResourceCommontype, ensuring type safety with the objects being handled. - Use the helper methods for mocking requests.
contracts/contracts.tsalso exports helper methods for the basic CRUD operations on k8s resources. - When importing pact resources, make sure they come from the
v3package to avoid version mismatches.
For example, the contract for application creation could look something like this:
import { ApplicationGroupVersionKind, ApplicationModel } from '../../../src/models/application';
import { ApplicationKind } from '../../../src/types';
import { getUrlPath, PactContract } from '../contracts';
export const contract: PactContract<ApplicationKind> = {
namespace,
groupVersionKind: ApplicationGroupVersionKind,
resourceName: app,
request: {
method: 'POST',
path: getUrlPath(ApplicationModel, namespace, app),
body: <<requestBody>>,
headers: { 'Content-Type': 'application/json' },
},
response: {
status: 201,
body: <<expectedResponse>>,
},
model: ApplicationModel,
};We are interested in Applications here so we are using ApplicationModel and ApplicationKind from hac-dev sources for type safety. We also use the getUrlPath helper to extract the URL segments from the model. The rest is completely up to our definition.
Now we can write a spec file as described above. Don't forget to import the contract object we've exported from our contract definition file, since we will be using it throughout the spec:
interaction('API Pact test', ({ provider, execute }) => {
beforeEach(() => {
provider
.uponReceiving('Create an application.')
.withRequest(contract.request)
.willRespondWith(contract.response);
});
execute('App is created', async (mockserver) => {
const product = await mockK8sCreateResource(contract, mockserver);
expect(product.kind).toEqual(contract.model.kind);
});
});Take note of two things:
- we are utilizing the
contractobject as much as possible to fill in the values for pact - we are using the
mockK8sCreateResourcemethod to send the request and we pass thecontractandmockserverto it- since we are interested in Application creation, we are using the
createmock method contractcontains all the information to mock and construct the request, as well as the expected responsemockserveris provided by pact, it is being used as the target of the request
- since we are interested in Application creation, we are using the
In same cases we may need to specify a state for the interaction. Usually, the state is setup by using given method on provider like this:
beforeEach(() => {
provider
.given("Application exists", {appName: "someAppName", namespace: "default"})
.uponReceiving('Create an application.')
.withRequest(contract.request)
.willRespondWith(contract.response);
});To make sure we won't do typo in state and use correct parameters, there is a helper method setState() that should be used instead. It allows you to specify the state by choosing one from the existing states and then force you to use the corresponding parameters - if any:
beforeEach(() => {
setState(provider, ProviderStates.appExists, getAppParams)
.uponReceiving('Get app with its components')
.withRequest(getContract.request)
.willRespondWith(getContract.response);
});In case we need to add more then one state to the interaction, it is possible to call multiple setState() methods:
beforeEach(() => {
setState(provider, ProviderStates.appExists, getAppParams);
setState(provider, ProviderStates.appHasComponent, compParams)
.uponReceiving('Get app with its components')
.withRequest(getContract.request)
.willRespondWith(getContract.response);
});The consumer has to specify a field name that should be checked. There are several ways how to check this field's value. Let's take a short example of code and explain the different matchers used in HAC-dev tests. For more information, take a look at the official documentation.
const expectedResponse = {
kind: ApplicationGroupVersionKind.kind,
metadata: {
creationTimestamp: regex(
matchers.dateAndTime,
'2022-01-21T13:36:30Z'
),
generation: like(1),
name: app,
}
}The exact value is checked when the value of a field is directly the value the response should return. In the example, the exact value check is done for the kind or name field.
Like matchers verifies that the value is something like the given one. In the example, the generation field is checked to have a value with an integer type.
Regex matchers match the value against a regex. They take 2 arguments: the regex pattern and an example string. In our tests, the regex is kept separated in matchers.ts file.
The example code with given matchers will then generate this snippet in the contract file:
"response": {
"body": {
"kind": "Application",
"metadata": {
"creationTimestamp": "2022-01-21T13:36:30Z",
"generation": 1,
"name": "myapp",
}
},
"matchingRules": {
"$.body.metadata.creationTimestamp": {
"match": "regex",
"regex": "^(-?(?:[1-9][0-9]*)?[0-9]{4})-(1[0-2]|0[1-9])-(3[01]|0[1-9]|[12][0-9])T(2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9])(.[0-9]+)?(Z|[+-](?:2[0-3]|[01][0-9]):[0-5][0-9])?$"
},
"$.body.metadata.generation": {
"match": "type"
}
}Pushing to the Pact broker is done via pact-broker CLI, but you can also use curl if needed. To push the contract, you need to have a file with the contract, -a tag specifying a contract version, -b tag for the Pact broker URL, and -u and -p tag for username and password. If you need to push contracts there but don't have creds, ping kfoniok.
pact-broker publish \
"$(pwd)/pact/pacts/HACdev-HAS.json" \
-a ${SHA:0:7} \
-b $PACT_BROKER_BASE_URL \
-u $PACT_BROKER_USERNAME \
-p $PACT_BROKER_PASSWORDTo add a new contract, create a new test file, new contract file and new states, if needed. You can use some existing test as a template.
- in a
pact-testsfolder, create a new file named<resource>.pact.spec.ts- add new
pactWith()block including your desired interactions - setup the provider in
beforeEach()method - call request by
execute(), using mocked@openshift/dynamic-plugin-sdk-utilsmethods - include some simple check in the
execute()method to be sure that API worked, e.g.expect(response.kind).toEqual(createContract.model.kind);
- add new
- create new file in
pact-tests/contracts/<provider>folder named<resource>-<interaction_type>.ts- specify the contract using
PactContractinterface - set all parameters needed for the state setup, if any
- specify the contract using
- run a new test locally by
yarn pact - once the test passes, you can check the contract file stored in the
pact/pactsfolder - create a PR and see the pact job passing
- when the PR is merged, the contract should be saved in a Pact Broker on a QE cluster
If you have any problems, reach out to kfoniok.