Using portman-->k6

[chyde80]

Hi @thim81 ,

When you are using portman to convert the OAI spec to a collection and then using postman-to-k6 - do you even bother running newman after using portman, or do you just use k6 for your contract/functional/fuzz/etc AND performance tests? Personally I'd like to separate the two; run contract/functional/etc on newman using portman, and run only perf tests on k6 using postman-to-k6 but it doesn't seem there is an easy separation. Do you have ideas, and what have you done practically?

[thim81]

hi @chyde80

My addvice is use a pipeline that executes it as follow-up stages:

So first run the contracts tests generated by Portman, via Newman
If these all pass, run performance tests via K6.

The contract tests purpose is to validate the contracts of the API responses.
The K6 tests purpose is to validate only the performance of the API, so that is why in K6 we only validate if the response times are within a threshold and if the responses match a 2xx response. We don't test through K6 the detailed content of the response or schema validations.

This a diagram of how I typically use OpenAPI as the source for various outputs that are being generated.

I use different Portman configs (YAML files) to generate compact (for K6) or extended (Newman) test suites.
In the CI/CD pipeline, I pass along different CLI options using --cliOptionsFile, which reference the different Portman configs.

[chyde80]

This is exactly the type of information I was looking for, and how I wanted to slice it up. Thank you so much! You should put this somewhere in your repo/readme/etc.

[thim81]

I talk about it in a podcast with the K6 team https://www.youtube.com/watch?v=Be66Db4wHLA

I Hope to find some time to write up a blog post.

[chyde80]

Do you have examples of how you "slice it up" for the compact (k6) and extended (newman) suites?

[chyde80]

Hi @thim81 did you see my question above? Also, did you write a blog post/etc? Thanks!