Skip to content

Releases: razorpay/devstack

Improved helmfile debugging with werf

04 Nov 13:08
@himanshu-garg-razorpay himanshu-garg-razorpay
werf
e8a8408
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Improved helmfile debugging with werf Latest
Latest

This release adds a new script allowing us to (among other things) install all the tools required by devstack with a single command. This note will demo an upgrade from existing tools, showcasing the benefits of using werf over helm in handling and reporting error cases improving the debugging experience for helm releases.

Problem
We analysed all the issues that landed in our support channel in the last quarter and realised that a lot of these had one thing in common, developers didn't understand what went wrong. Even when our team started looking into a support issue, we had to jump through a few hoops to figure out the root cause, like events, logs and descriptions for all the deployments, jobs and hooks. And even before we did that, we had to list out all these resources from the helm chart. The underlying problems would usually be something like ImagePullBackoff (ErrImagePull), ContainerConfigError, CrashLoopBackoff etc. But helm doesn't exit the release in case of such an error, instead it keeps on waiting for success till the end of timeout. We only get very opaque errors from helm like timed out waiting for condition and that too after exhausting the entire timeout which is around 15-20 min of wait time.

Screenshot 2022-11-04 at 6 09 59 PM

Screenshot 2022-11-04 at 6 27 04 PM

Solution
We needed something that could track the status of all the resources being built in the release with full view of relevant events, logs etc, exit the release in case of errors and report it transparently.

Well werf has all that (and more)
Checkout how to install this at the end of this doc.

Demo
ErrImagePull - The docker image mentioned in the deployment doesn't exist

Screen.Recording.2022-11-04.at.6.21.27.PM.mov

CreateContainerConfigError

Screen.Recording.2022-11-09.at.11.16.11.AM.mov

Readiness Probe Failed

ReadinessProbeFailed.mov

Bad DNS Policy

BadDNSPolicy.mov

Easy Installation
Copy and run the following command in your terminal which installs werf. It also configures helmfile to work with it. And boom..! now every deployment you make, has nice logs.

Please, remember to open a new terminal after running this.

sh -euo pipefail -c "$(curl 'https://raw.githubusercontent.com/razorpay/devstack/master/setup-helper.sh'); setup_tools_only"
Screen.Recording.2022-11-04.at.6.31.20.PM.mov
Assets 2
Loading