# Set rollout image (containers contains 'initContainer', 'container', 'ephemeralContainer')
kubectl argo rollouts set image my-rollout containerName=imageName
+
+# Set rollout image for all containers
+kubectl argo rollouts set image my-rollout *=imageName
-h, --help help for image
diff --git a/search/search_index.json b/search/search_index.json
index e7ed06d7e8..8bcf659ad5 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Argo Rollouts - Kubernetes Progressive Delivery Controller","text":""},{"location":"#what-is-argo-rollouts","title":"What is Argo Rollouts?","text":"Argo Rollouts is a Kubernetes controller and set of CRDs which provide advanced deployment capabilities such as blue-green, canary, canary analysis, experimentation, and progressive delivery features to Kubernetes.
Argo Rollouts (optionally) integrates with ingress controllers and service meshes, leveraging their traffic shaping abilities to gradually shift traffic to the new version during an update. Additionally, Rollouts can query and interpret metrics from various providers to verify key KPIs and drive automated promotion or rollback during an update.
Here is a demonstration video (click to watch on Youtube):
"},{"location":"#why-argo-rollouts","title":"Why Argo Rollouts?","text":"The native Kubernetes Deployment Object supports the RollingUpdate strategy which provides a basic set of safety guarantees (readiness probes) during an update. However the rolling update strategy faces many limitations:
- Few controls over the speed of the rollout
- Inability to control traffic flow to the new version
- Readiness probes are unsuitable for deeper, stress, or one-time checks
- No ability to query external metrics to verify an update
- Can halt the progression, but unable to automatically abort and rollback the update
For these reasons, in large scale high-volume production environments, a rolling update is often considered too risky of an update procedure since it provides no control over the blast radius, may rollout too aggressively, and provides no automated rollback upon failures.
"},{"location":"#controller-features","title":"Controller Features","text":" - Blue-Green update strategy
- Canary update strategy
- Fine-grained, weighted traffic shifting
- Automated rollbacks and promotions
- Manual judgement
- Customizable metric queries and analysis of business KPIs
- Ingress controller integration: NGINX, ALB, Apache APISIX
- Service Mesh integration: Istio, Linkerd, SMI
- Simultaneous usage of multiple providers: SMI + NGINX, Istio + ALB, etc.
- Metric provider integration: Prometheus, Wavefront, Kayenta, Web, Kubernetes Jobs, Datadog, New Relic, Graphite, InfluxDB
"},{"location":"#quick-start","title":"Quick Start","text":"kubectl create namespace argo-rollouts\nkubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml\n
Follow the full getting started guide to walk through creating and then updating a rollout object.
"},{"location":"#how-does-it-work","title":"How does it work?","text":"Similar to the deployment object, the Argo Rollouts controller will manage the creation, scaling, and deletion of ReplicaSets. These ReplicaSets are defined by the spec.template field inside the Rollout resource, which uses the same pod template as the deployment object.
When the spec.template is changed, that signals to the Argo Rollouts controller that a new ReplicaSet will be introduced. The controller will use the strategy set within the spec.strategy field in order to determine how the rollout will progress from the old ReplicaSet to the new ReplicaSet. Once that new ReplicaSet is scaled up (and optionally passes an Analysis), the controller will mark it as \"stable\".
If another change occurs in the spec.template during a transition from a stable ReplicaSet to a new ReplicaSet (i.e. you change the application version in the middle of a rollout), then the previously new ReplicaSet will be scaled down, and the controller will try to progress the ReplicasSet that reflects the updated spec.template field. There is more information on the behaviors of each strategy in the spec section.
"},{"location":"#use-cases-of-argo-rollouts","title":"Use cases of Argo Rollouts","text":" -
A user wants to run last-minute functional tests on the new version before it starts to serve production traffic. With the BlueGreen strategy, Argo Rollouts allows users to specify a preview service and an active service. The Rollout will configure the preview service to send traffic to the new version while the active service continues to receive production traffic. Once a user is satisfied, they can promote the preview service to be the new active service. (example)
-
Before a new version starts receiving live traffic, a generic set of steps need to be executed beforehand. With the BlueGreen Strategy, the user can bring up the new version without it receiving traffic from the active service. Once those steps finish executing, the rollout can cut over traffic to the new version.
-
A user wants to give a small percentage of the production traffic to a new version of their application for a couple of hours. Afterward, they want to scale down the new version and look at some metrics to determine if the new version is performant compared to the old version. Then they will decide if they want to roll out the new version for all of the production traffic or stick with the current version. With the canary strategy, the rollout can scale up a ReplicaSet with the new version to receive a specified percentage of traffic, wait for a specified amount of time, set the percentage back to 0, and then wait to rollout out to service all of the traffic once the user is satisfied. (example)
-
A user wants to slowly give the new version more production traffic. They start by giving it a small percentage of the live traffic and wait a while before giving the new version more traffic. Eventually, the new version will receive all the production traffic. With the canary strategy, the user specifies the percentages they want the new version to receive and the amount of time to wait between percentages. (example)
-
A user wants to use the normal Rolling Update strategy from the deployment. If a user uses the canary strategy with no steps, the rollout will use the max surge and max unavailable values to roll to the new version. (example)
"},{"location":"#examples","title":"Examples","text":"You can see more examples of Rollouts at:
- The example directory
- The Argo Rollouts Demo application
"},{"location":"CONTRIBUTING/","title":"Contributing","text":""},{"location":"CONTRIBUTING/#before-you-start","title":"Before You Start","text":"Argo Rollouts is written in Golang. If you do not have a good grounding in Go, try out the tutorial.
"},{"location":"CONTRIBUTING/#pre-requisites","title":"Pre-requisites","text":"Install:
- docker
- golang
- kubectl
- kustomize >= 4.5.5
- k3d recommended
Kustomize is required for unit tests (make test is using it), so you must install it locally if you wish to make code contributions to Argo Rollouts.
Argo Rollout additionally uses the following tools
golangci-lint to lint the project.protoc and swagger-codegen to generate proto related filesyarn to build the UI
Run the following commands to install them:
# macOS\nbrew install golangci-lint\n\n# linux\ngo get -u github.com/golangci/golangci-lint/cmd/golangci-lint\n
Brew users can quickly install the lot:
brew install go kubectl kustomize golangci-lint protobuf swagger-codegen k3d\n
Set up environment variables (e.g. is ~/.bashrc):
export GOPATH=~/go\nexport PATH=$PATH:$GOPATH/bin\n
Checkout the code:
go get -u github.com/argoproj/argo-rollouts\ncd ~/go/src/github.com/argoproj/argo-rollouts\n
"},{"location":"CONTRIBUTING/#building","title":"Building","text":"go.mod is used, so the go build/test commands automatically install the needed dependencies
The make controller command will build the controller.
-
make install-tools-local - Runs scripts to install codegen utility CLIs necessary for codegen.
-
make codegen - Runs the code generator that creates the informers, client, lister, and deepcopies from the types.go and modifies the open-api spec.
"},{"location":"CONTRIBUTING/#running-controller-locally","title":"Running Controller Locally","text":"It is much easier to run and debug if you run Argo Rollout in your local machine than in the Kubernetes cluster.
cd ~/go/src/github.com/argoproj/argo-rollouts\ngo run ./cmd/rollouts-controller/main.go\n
When running locally it will connect to whatever kubernetes cluster you have configured in your kubeconfig. You will need to make sure to install the Argo Rollout CRDs into your local cluster, and have the argo-rollouts namespace.
"},{"location":"CONTRIBUTING/#running-unit-tests","title":"Running Unit Tests","text":"To run unit tests:
make test\n
"},{"location":"CONTRIBUTING/#running-e2e-tests","title":"Running E2E tests","text":"The end-to-end tests need to run against a kubernetes cluster with the Argo Rollouts controller running. The rollout controller can be started with the command:
make start-e2e\n
Start and prepare your cluster for e2e tests:
k3d cluster create\nkubectl create ns argo-rollouts\nkubectl apply -k manifests/crds\nkubectl apply -f test/e2e/crds\n
Then run the e2e tests:
make test-e2e\n
To run a subset of e2e tests, you need to specify the suite with -run, and the specific test regex with -testify.m.
E2E_TEST_OPTIONS=\"-run 'TestCanarySuite' -testify.m 'TestCanaryScaleDownOnAbortNoTrafficRouting'\" make test-e2e\n
"},{"location":"CONTRIBUTING/#running-the-ui","title":"Running the UI","text":"If you'd like to run the UI locally, you first need a running Rollouts controller. This can be a locally running controller with a k3d cluster, as described above, or a controller running in a remote Kubernetes cluster.
In order for the local React app to communicate with the controller and Kubernetes API, run the following to open a port forward to the dashboard:
kubectl argo rollouts dashboard\n
Note that you can also build the API server and run this instead,
make plugin\n./dist/kubectl-argo-rollouts dashboard\n
In another terminal, run the following to start the UI:
cd ui\nyarn install\nyarn start\n
"},{"location":"CONTRIBUTING/#controller-architecture","title":"Controller architecture","text":"Argo Rollouts is actually a collection of individual controllers that handle a specific aspect of Progressive Delivery.
The controllers are:
- Rollout Controller
- Service Controller
- Ingress Controller
- Experiment Controller
- AnalysisRun Controller
"},{"location":"CONTRIBUTING/#tips","title":"Tips","text":" - You can run the tests using a different kubeconfig by setting the
KUBECONFIG environment variable:
KUBECONFIG=~/.kube/minikube make start-e2e\nKUBECONFIG=~/.kube/minikube make test-e2e\n
- To run a specific e2e test, set the
E2E_TEST_OPTIONS environment variable to specify the test (or test regex):
make test-e2e E2E_TEST_OPTIONS=\"-testify.m ^TestRolloutRestart$\"\n
- The e2e tests are designed to run as quickly as possible, eliminating readiness and termination delays. However, it is often desired to artificially slow down the tests for debugging purposes, as well as to understand what the test is doing. To delay startup and termination of pods, set the
E2E_POD_DELAY to an integer value in seconds. This environment variable is often coupled with E2E_TEST_OPTIONS to debug and slow down a specific test.
make test-e2e E2E_POD_DELAY=10\n
- Increasing the timeout. The E2E tests time out waiting on conditions to be met within 60 seconds. If debugging the rollout controller, it may be useful to increase this timeout while say sitting at a debugger breakpoint:
make test-e2e E2E_WAIT_TIMEOUT=999999\n
- The e2e tests leverage a feature of the controller allowing the controller to be sharded with a user-specific \"instance id\" label. This allows the tests to operate only on rollouts with the specified label, and prevents any other controllers (including the system rollout controller), from also operating on the same set of rollouts. This value can be changed (from the default of
argo-rollouts-e2e), using the E2E_INSTANCE_ID environment variable:
make start-e2e E2E_INSTANCE_ID=foo\nmake test-e2e E2E_INSTANCE_ID=foo\n
Alternatively, the e2e tests can be run against the system controller (i.e. without an instance id):
make start-e2e E2E_INSTANCE_ID=''\n
- Working on CRDs? While editing them directly works when you are finding the shape of things you want, the final CRDs are autogenerated. Make sure to regenerate them before submitting PRs. They are controlled by the relevant annotations in the types file:
eg: Analysis Templates are controlled by annotations in pkg/apis/rollouts/v1alpha1/analysis_types.go.
Refer to https://book-v1.book.kubebuilder.io/beyond_basics/generating_crd.html and https://book.kubebuilder.io/reference/markers/crd-validation.html for more info on annotations you can use.
"},{"location":"CONTRIBUTING/#running-local-containers","title":"Running Local Containers","text":"You may need to run containers locally, so here's how:
Create login to Docker Hub, then login.
docker login\n
Add your username as the environment variable, e.g. to your ~/.bash_profile:
export IMAGE_NAMESPACE=argoproj\n
Build the images:
DOCKER_PUSH=true make image\n
Update the manifests:
make manifests\n
Install the manifests:
kubectl -n argo-rollouts apply -f manifests/install.yaml\n
"},{"location":"CONTRIBUTING/#upgrading-kubernetes-libraries","title":"Upgrading Kubernetes Libraries","text":"Argo Rollouts has a dependency on the kubernetes/kubernetes repo for some of the functionality that has not been pushed into the other kubernetes repositories yet. In order to import the kubernetes/kubernetes repo, all of the associated repos have to pinned to the correct version specified by the kubernetes/kubernetes release. The ./hack/update-k8s-dependencies.sh updates all the dependencies to the those correct versions.
"},{"location":"CONTRIBUTING/#upgrading-notifications-engine","title":"Upgrading Notifications Engine","text":"Argo Rollouts has a dependency on the argoproj/notifications-engines repo for the notifications functionality and related documentation.
This is updated by upgrading the Go library in go.mod by running the commands:
go get github.com/argoproj/notifications-engine@LATEST_COMMIT_HASH\ngo mod tidy\n
Next the latest notifications documentation can be imported by running:
make docs\n
"},{"location":"CONTRIBUTING/#documentation-changes","title":"Documentation Changes","text":"Modify contents in docs/ directory.
Preview changes in your browser by visiting http://localhost:8000 after running:
make serve-docs\n
To publish changes, run:
make release-docs\n
"},{"location":"FAQ/","title":"FAQ","text":""},{"location":"FAQ/#general","title":"General","text":""},{"location":"FAQ/#does-argo-rollouts-depend-on-argo-cd-or-any-other-argo-project","title":"Does Argo Rollouts depend on Argo CD or any other Argo project?","text":"Argo Rollouts is a standalone project. Even though it works great with Argo CD and other Argo projects, it can be used on its own for Progressive Delivery scenarios. More specifically, Argo Rollouts does NOT require that you also have installed Argo CD on the same cluster.
"},{"location":"FAQ/#how-does-argo-rollouts-integrate-with-argo-cd","title":"How does Argo Rollouts integrate with Argo CD?","text":"Argo CD understands the health of Argo Rollouts resources via Argo CD\u2019s Lua health check. These Health checks understand when the Argo Rollout objects are Progressing, Suspended, Degraded, or Healthy. Additionally, Argo CD has Lua based Resource Actions that can mutate an Argo Rollouts resource (i.e. unpause a Rollout).
As a result, an operator can build automation to react to the states of the Argo Rollouts resources. For example, if a Rollout created by Argo CD is paused, Argo CD detects that and marks the Application as suspended. Once the new version is verified to be good, the operator can use Argo CD\u2019s resume resource action to unpause the Rollout so it can continue to make progress.
"},{"location":"FAQ/#can-we-run-the-argo-rollouts-kubectl-plugin-commands-via-argo-cd","title":"Can we run the Argo Rollouts kubectl plugin commands via Argo CD?","text":"Argo CD supports running Lua scripts to modify resource kinds (i.e. suspending a CronJob by setting the .spec.suspend to true). These Lua Scripts can be configured in the argocd-cm ConfigMap or upstreamed to the Argo CD's resource_customizations directory. These custom actions have two Lua scripts: one to modify the said resource and another to detect if the action can be executed (i.e. A user should not be able to resuming a unpaused Rollout). Argo CD allows users to execute these actions via the UI or CLI.
In the CLI, a user (or a CI system) can run
argocd app actions run <APP_NAME> <ACTION> \n
This command executes the action listed on the application listed. In the UI, a user can click the hamburger button of a resource and the available actions will appear in a couple of seconds. The user can click and confirm that action to execute it.
Currently, the Rollout action has two available custom actions in Argo CD: resume and restart.
- Resume unpauses a Rollout with a PauseCondition
- Restart: Sets the RestartAt and causes all the pods to be restarted.
"},{"location":"FAQ/#does-argo-rollout-require-a-service-mesh-like-istio","title":"Does Argo Rollout require a Service Mesh like Istio?","text":"Argo Rollouts does not require a service mesh or ingress controller to be used. In the absence of a traffic routing provider, Argo Rollouts manages the replica counts of the canary/stable ReplicaSets to achieve the desired canary weights. Normal Kubernetes Service routing (via kube-proxy) is used to split traffic between the ReplicaSets.
"},{"location":"FAQ/#does-argo-rollout-require-we-follow-gitops-in-my-organization","title":"Does Argo Rollout require we follow GitOps in my organization?","text":"Argo Rollouts is a Kubernetes controller that will react to any manifest change regardless of how the manifest was changed. The manifest can be changed by a Git commit, an API call, another controller or even a manual kubectl command. You can use Argo Rollouts with any traditional CI/CD solution that does not follow the GitOps approach.
"},{"location":"FAQ/#can-we-run-the-argo-rollouts-controller-in-ha-mode","title":"Can we run the Argo Rollouts controller in HA mode?","text":"Yes. A k8s cluster can run multiple replicas of Argo-rollouts controllers to achieve HA. To enable this feature, run the controller with --leader-elect flag and increase the number of replicas in the controller's deployment manifest. The implementation is based on the k8s client-go's leaderelection package. This implementation is tolerant to arbitrary clock skew among replicas. The level of tolerance to skew rate can be configured by setting --leader-election-lease-duration and --leader-election-renew-deadline appropriately. Please refer to the package documentation for details.
"},{"location":"FAQ/#can-we-install-argo-rollouts-centrally-in-a-cluster-and-manage-rollout-resources-in-external-clusters","title":"Can we install Argo Rollouts centrally in a cluster and manage Rollout resources in external clusters?","text":"No you cannot do that (even though Argo CD can work that way). This is by design because the Rollout is a custom resource unknown to vanilla Kubernetes. You need the Rollout CRD as well as the controller in the deployment cluster (every cluster that will use workloads with Rollouts).
"},{"location":"FAQ/#rollouts","title":"Rollouts","text":""},{"location":"FAQ/#which-deployment-strategies-does-argo-rollouts-support","title":"Which deployment strategies does Argo Rollouts support?","text":"Argo Rollouts supports BlueGreen, Canary, and Rolling Update. Additionally, Progressive Delivery features can be enabled on top of the blue-green/canary update, which further provides advanced deployment such as automated analysis and rollback.
"},{"location":"FAQ/#does-the-rollout-object-follow-the-provided-strategy-when-it-is-first-created","title":"Does the Rollout object follow the provided strategy when it is first created?","text":"As with Deployments, Rollouts does not follow the strategy parameters on the initial deploy. The controller tries to get the Rollout into a steady state as fast as possible by creating a fully scaled up ReplicaSet from the provided .spec.template. Once the Rollout has a stable ReplicaSet to transition from, the controller starts using the provided strategy to transition the previous ReplicaSet to the desired ReplicaSet.
"},{"location":"FAQ/#how-does-bluegreen-rollback-work","title":"How does BlueGreen rollback work?","text":"A BlueGreen Rollout keeps the old ReplicaSet up and running for 30 seconds or the value of the scaleDownDelaySeconds. The controller tracks the remaining time before scaling down by adding an annotation called argo-rollouts.argoproj.io/scale-down-deadline to the old ReplicaSet. If the user applies the old Rollout manifest before the old ReplicaSet scales down, the controller does something called a fast rollback. The controller immediately switches the active service\u2019s selector back to the old ReplicaSet\u2019s rollout-pod-template-hash and removes the scaled down annotation from that ReplicaSet. The controller does not do any of the normal operations when trying to introduce a new version since it is trying to revert as fast as possible. A non-fast-track rollback occurs when the scale down annotation has past and the old ReplicaSet has been scaled down. In this case, the Rollout treats the ReplicaSet like any other new ReplicaSet and follows the usual procedure for deploying a new ReplicaSet.
"},{"location":"FAQ/#what-is-the-argo-rolloutsargoprojiomanaged-by-rollouts-annotation","title":"What is the argo-rollouts.argoproj.io/managed-by-rollouts annotation?","text":"Argo Rollouts adds an argo-rollouts.argoproj.io/managed-by-rollouts annotation to Services and Ingresses that the controller modifies. They are used when the Rollout managing these resources is deleted and the controller tries to revert them back into their previous state.
"},{"location":"FAQ/#rollbacks","title":"Rollbacks","text":""},{"location":"FAQ/#does-argo-rollouts-write-back-in-git-when-a-rollback-takes-place","title":"Does Argo Rollouts write back in Git when a rollback takes place?","text":"No. Argo Rollouts doesn't read/write anything to Git. Actually Argo Rollouts knows nothing about Git repositories (only Argo CD has this information if it manages the Rollout). When a rollback takes place, Argo Rollouts marks the application as \"degraded\" and changes the version on the cluster back to the known stable one.
"},{"location":"FAQ/#if-i-use-both-argo-rollouts-and-argo-cd-wouldnt-i-have-an-endless-loop-in-the-case-of-a-rollback","title":"If I use both Argo Rollouts and Argo CD wouldn't I have an endless loop in the case of a Rollback?","text":"No there is no endless loop. As explained already in the previous question, Argo Rollouts doesn't tamper with Git in any way. If you use both Argo projects together, the sequence of events for a rollback is the following:
- Version N runs on the cluster as a Rollout (managed by Argo CD). The Git repository is updated with version N+1 in the Rollout/Deployment manifest
- Argo CD sees the changes in Git and updates the live state in the cluster with the new Rollout object
- Argo Rollouts takes over as it watches for all changes in Rollout Objects. Argo Rollouts is completely oblivious to what is happening in Git. It only cares about what is happening with Rollout objects that are live in the cluster.
- Argo Rollouts tries to apply version N+1 with the selected strategy (e.g. blue/green)
- Version N+1 fails to deploy for some reason
- Argo Rollouts scales back again (or switches traffic back) to version N in the cluster. No change in Git takes place from Argo Rollouts
- Cluster is running version N and is completely healthy
- The Rollout is marked as \"Degraded\" both in ArgoCD and Argo Rollouts.
- Argo CD syncs take no further action as the Rollout object in Git is exactly the same as in the cluster. They both mention version N+1
"},{"location":"FAQ/#so-how-can-i-make-argo-rollouts-write-back-in-git-when-a-rollback-takes-place","title":"So how can I make Argo Rollouts write back in Git when a rollback takes place?","text":"You don't need to do that if you simply want to go back to the previous version using Argo CD. When a deployment fails, Argo Rollouts automatically sets the cluster back to the stable/previous version as explained in the previous question. You don't need to write anything in Git to achieve this. The cluster is still healthy and you have avoided downtime. You are then expected to fix the issue and roll-forward (i.e. deploy the next version) if you want to follow GitOps in a pedantic manner. If you want Argo Rollouts to write back in Git after a failed deployment then you need to orchestrate this with an external system or write custom glue code. But this is normally not needed.
"},{"location":"FAQ/#what-is-the-relationship-between-rollbacks-with-argo-rollouts-and-rollbacks-with-argo-cd","title":"What is the relationship between Rollbacks with Argo Rollouts and Rollbacks with Argo CD?","text":"They are completely unrelated. Argo Rollouts \"rollbacks\" switch the cluster back to the previous version as explained in the previous question. They don't touch or affect Git in any way. Argo CD rollbacks simply point the cluster back a previous Git hash. Normally if you have Argo Rollouts, you don't need to use the Argo CD rollback command.
"},{"location":"FAQ/#how-can-i-deploy-multiple-services-in-a-single-step-and-roll-them-back-according-to-their-dependencies","title":"How can I deploy multiple services in a single step and roll them back according to their dependencies?","text":"The Rollout specification focuses on a single application/deployment. Argo Rollouts knows nothing about application dependencies. If you want to deploy multiple applications together in a smart way (e.g. automatically rollback a frontend if backend deployment fails) you need to write your own solution on top of Argo Rollouts. In most cases, you would need one Rollout resource for each application that you are deploying. Ideally you should also make your services backwards and forwards compatible (i.e. frontend should be able to work with both backend-preview and backend-active).
"},{"location":"FAQ/#how-can-i-run-my-own-custom-tests-eg-smoke-tests-to-decide-if-a-rollback-should-take-place-or-not","title":"How can I run my own custom tests (e.g. smoke tests) to decide if a Rollback should take place or not?","text":"Use a custom Job or Web Analysis. You can pack all your smoke tests in a single container and run them as a Job analysis. Argo Rollouts will use the results of the analysis to automatically rollback if the tests fail.
"},{"location":"FAQ/#experiments","title":"Experiments","text":""},{"location":"FAQ/#why-doesnt-my-experiment-end","title":"Why doesn't my Experiment end?","text":"An Experiment\u2019s duration is controlled by the .spec.duration field and the analyses created for the Experiment. The .spec.duration indicates how long the ReplicaSets created by the Experiment should run. Once the duration passes, the experiment scales down the ReplicaSets it created and marks the AnalysisRuns successful unless the requiredForCompletion field is used in the Experiment. If enabled, the ReplicaSets are still scaled-down, but the Experiment does not finish until the Analysis Run finishes.
Additionally, the .spec.duration is an optional field. If it\u2019s left unset, and the Experiment creates no AnalysisRuns, the ReplicaSets run indefinitely. The Experiment creates AnalysisRuns without the requiredForCompletion field, the Experiment fails only when the AnalysisRun created fails or errors out. If the requiredForCompletion field is set, the Experiment only marks itself as Successful and scales down the created ReplicaSets when the AnalysisRun finishes Successfully.
Additionally, an Experiment ends if the .spec.terminate field is set to true regardless of the state of the Experiment.
"},{"location":"FAQ/#analysis","title":"Analysis","text":""},{"location":"FAQ/#why-doesnt-my-analysisrun-end","title":"Why doesn't my AnalysisRun end?","text":"The AnalysisRun\u2019s duration is controlled by the metrics specified. Each Metric can specify an interval, count, and various limits (ConsecutiveErrorLimit, InconclusiveLimit, FailureLimit). If the interval is omitted, the AnalysisRun takes a single measurement. The count indicates how many measurements should be taken and causes the AnalysisRun to run indefinitely if omitted. The ConsecutiveErrorLimit, InconclusiveLimit, and FailureLimit define the thresholds allowed before putting the rollout into a completed state.
Additionally, an AnalysisRun ends if the .spec.terminate field is set to true regardless of the state of the AnalysisRun.
"},{"location":"FAQ/#what-is-the-difference-between-failures-and-errors","title":"What is the difference between failures and errors?","text":"Failures are when the failure condition evaluates to true or an AnalysisRun without a failure condition evaluates the success condition to false. Errors are when the controller has any kind of issue with taking a measurement (i.e. invalid Prometheus URL).
"},{"location":"architecture/","title":"Architecture","text":"Here is an overview of all the components that take part in a deployment managed by Argo Rollouts.
"},{"location":"architecture/#argo-rollouts-controller","title":"Argo Rollouts controller","text":"This is the main controller that monitors the cluster for events and reacts whenever a resource of type Rollout is changed. The controller will read all the details of the rollout and bring the cluster to the same state as described in the rollout definition.
Note that Argo Rollouts will not tamper with or respond to any changes that happen on normal Deployment Resources. This means that you can install Argo Rollouts in a cluster that is also deploying applications with alternative methods.
To install the controller in your cluster and get started with Progressive Delivery, see the Installation page.
"},{"location":"architecture/#rollout-resource","title":"Rollout resource","text":"The Rollout resource is a custom Kubernetes resource introduced and managed by Argo Rollouts. It is mostly compatible with the native Kubernetes Deployment resource but with extra fields that control the stages, thresholds and methods of advanced deployment methods such as canaries and blue/green deployments.
Note that the Argo Rollouts controller will only respond to those changes that happen in Rollout sources. It will do nothing for normal deployment resources. This means that you need to migrate your Deployments to Rollouts if you want to manage them with Argo Rollouts.
You can see all possible options of a Rollout in the full specification page.
"},{"location":"architecture/#replica-sets-for-old-and-new-version","title":"Replica sets for old and new version","text":"These are instances of the standard Kubernetes ReplicaSet resources. Argo Rollouts puts some extra metadata on them in order to keep track of the different versions that are part of an application.
Note also that the replica sets that take part in a Rollout are fully managed by the controller in an automatic way. You should not tamper with them with external tools.
"},{"location":"architecture/#ingressservice","title":"Ingress/Service","text":"This is the mechanism that traffic from live users enters your cluster and is redirected to the appropriate version. Argo Rollouts use the standard Kubernetes service resource, but with some extra metadata needed for management.
Argo Rollouts is very flexible on networking options. First of all you can have different services during a Rollout, that go only to the new version, only to the old version or both. Specifically for Canary deployments, Argo Rollouts supports several service mesh and ingress solutions for splitting traffic with specific percentages instead of simple balancing based on pod counts and it is possible to use multiple routing providers simultaneously.
"},{"location":"architecture/#analysistemplate-and-analysisrun","title":"AnalysisTemplate and AnalysisRun","text":"Analysis is the capability to connect a Rollout to your metrics provider and define specific thresholds for certain metrics that will decide if an update is successful or not. For each analysis you can define one or more metric queries along with their expected results. A Rollout will progress on its own if metric queries are good, rollback automatically if metrics show failure and pause the rollout if metrics cannot provide a success/failure answer.
For performing an analysis, Argo Rollouts includes two custom Kubernetes resources: AnalysisTemplate and AnalysisRun.
AnalysisTemplate contains instructions on what metrics to query. The actual result that is attached to a Rollout is the AnalysisRun custom resource. You can define an AnalysisTemplate on a specific Rollout or globally on the cluster to be shared by multiple rollouts as a ClusterAnalysisTemplate. The AnalysisRun resource is scoped on a specific rollout.
Note that using an analysis and metrics in a Rollout is completely optional. You can manually pause and promote a rollout or use other external methods (e.g. smoke tests) via the API or the CLI. You don't need a metric solution just to use Argo Rollouts. You can also mix both automated (i.e. analysis based) and manual steps in a Rollout.
Apart from metrics, you can also decide the success of a rollout by running a Kubernetes job or running a webhook.
"},{"location":"architecture/#metric-providers","title":"Metric providers","text":"Argo Rollouts includes native integration for several popular metrics providers that you can use in the Analysis resources to automatically promote or rollback a rollout. See the documentation of each provider for specific setup options.
"},{"location":"architecture/#cli-and-ui-not-shown-in-the-diagram","title":"CLI and UI (Not shown in the diagram)","text":"You can view and manage Rollouts with the Argo Rollouts CLI or the integrated UI. Both are optional.
"},{"location":"best-practices/","title":"Best Practices","text":"This document describes some best practices, tips and tricks when using Argo Rollouts. Be sure to read the FAQ page as well.
"},{"location":"best-practices/#check-application-compatibility","title":"Check application compatibility","text":"Argo Rollouts is a great solution for applications that your team is deploying in a continuous manner (and you have access to the source code). Before using Argo Rollouts you need to contact the developers of the application and verify that you can indeed run multiple versions of the same application at the same time.
Not all applications can work with Argo Rollouts. Applications that use shared resources (e.g. writing to a shared file) will have issues, and \"worker\" type applications (that load data from queues) will rarely work ok without source code modifications.
Note that using Argo Rollouts for \"infrastructure\" applications such as cert-manager, nginx, coredns, sealed-secrets etc is NOT recommended.
"},{"location":"best-practices/#understand-the-scope-of-argo-rollouts","title":"Understand the scope of Argo Rollouts","text":"Currently Argo Rollouts works with a single Kubernetes deployment/application and within a single cluster only. You also need to have the controller deployed on every cluster where a Rollout is running if have more than one clusters using Rollout workloads.
If you want to look at multiple-services on multiple clusters see discussion at issues 2737, 451 and 2088.
Note also that Argo Rollouts is a self-contained solution. It doesn't need Argo CD or any other Argo project to work.
"},{"location":"best-practices/#understand-your-use-case","title":"Understand your use case","text":"Argo Rollouts is perfect for all progressive delivery scenarios as explained in the concepts page.
You should NOT use Argo Rollouts for preview/ephemeral environments. For that use case check the Argo CD Pull Request generator.
The recommended way to use Argo Rollouts is for brief deployments that take 15-20 minutes or maximum 1-2 hours. If you want to run new versions for days or weeks before deciding to promote, then Argo Rollouts is probably not the best solution for you.
Also, if you want to run a wave of multiple versions at the same time (i.e. have 1.1 and 1.2 and 1.3 running at the same time), know that Argo Rollouts was not designed for this scenario.
A version that has just been promoted is assumed to be ready for production and has already passed all your tests (either manual or automated).
"},{"location":"best-practices/#prepare-your-metrics","title":"Prepare your metrics","text":"The end-goal for using Argo Rollouts is to have fully automated deployments that also include rollbacks when needed.
While Argo Rollouts supports manual promotions and other manual pauses, these are best used for experimentation and test reasons.
Ideally you should have proper metrics that tell you in 5-15 minutes if a deployment is successful or not. If you don't have those metrics, then you will miss a lot of value from Argo Rollouts.
Get your metrics in place first and test them with dry-runs before applying them to production deployments.
"},{"location":"best-practices/#there-is-no-argo-rollouts-api","title":"There is no \"Argo Rollouts API\"","text":"A lot of people want to find an official API for managing Rollouts. There isn't any separate Argo Rollouts API. You can always use the Kubernetes API and patching of resources if you want to control a rollout.
But as explained in the previous point the end goal should be fully automated deployments without you having to tell Argo Rollouts to promote or abort.
"},{"location":"best-practices/#integrating-with-other-systems-and-processes","title":"Integrating with other systems and processes","text":"There are two main ways to integrate other systems with Argo Rollouts.
The easiest way is to use Notifications. This means that when a rollout is finished/aborted you send a notification to another system that does other tasks that you want to happen.
Alternatively you can control Rollouts with the CLI or by patching manually the Kubernetes resources.
"},{"location":"best-practices/#use-the-kubernetes-downward-api","title":"Use the Kubernetes Downward API","text":"If you want your applications to know if they are part of a canary or not, you can use Ephemeral labels along with the Kubernetes downward api.
This means that your application will read from files its configuration in a dynamic manner and adapt according to the situation.
"},{"location":"best-practices/#ingress-desiredstable-host-routes","title":"Ingress desired/stable host routes","text":"For various reasons, it is often desired that external services are able to reach the desired pods (aka canary/preview) or stable pods specifically, without the possibility of traffic arbitrarily being split between the two versions. Some use cases include:
- The new version of the service is able to be reach internally/privately (e.g. for manual verification), before exposing it externally.
- An external CI/CD pipeline runs tests against the blue-green preview stack before it is promoted to production.
- Running tests which compare the behavior of old version against the new version.
If you are using an Ingress to route traffic to the service, additional host rules can be added to the ingress rules so that it is possible to specifically reach to the desired (canary/preview) pods or stable pods.
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n name: guestbook\nspec:\n rules:\n # host rule to only reach the desired pods (aka canary/preview)\n - host: guestbook-desired.argoproj.io\n http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n name: guestbook-desired\n port:\n number: 443\n\n # host rule to only reach the stable pods\n - host: guestbook-stable.argoproj.io\n http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n name: guestbook-stable\n port:\n number: 443\n\n # default rule which omits host, and will split traffic between desired vs. stable\n - http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n name: guestbook-root\n port:\n number: 443\n
The above technique has the a benefit in that it would not incur additional cost of allocating additional load balancers.
"},{"location":"best-practices/#reducing-operator-memory-usage","title":"Reducing operator memory usage","text":"On clusters with thousands of rollouts memory usage for the argo-rollouts controller can be reduced significantly by changing the RevisionHistoryLimit property from the default of 10 to a lower number.
One user of Argo Rollouts saw a 27% reduction in memory usage for a cluster with 1290 rollouts by changing RevisionHistoryLimit from 10 to 0.
"},{"location":"best-practices/#rollout-a-configmap-change","title":"Rollout a ConfigMap change","text":"Argo Rollouts is meant to work on a Kubernetes Deployment. When a ConfigMap is mounted inside one the Deployment container and a change occurs inside the ConfigMap, it won't trigger a new Rollout by default.
One technique to trigger the Rollout it to name dynamically the ConfigMap. For example, adding a hash of its content at the end of the name:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: my-config-7270e14e6\n
Each time a change occurs in the ConfigMap, its name will change in the Deployment reference as well, triggering a Rollout.
However, it's not enough to perform correctly progressive rollouts, as the old ConfigMap might get deleted as soon as the new one is created. This would prevent Experiments and rollbacks in case of rollout failure to work correctly.
While no magical solution exist to work aroud that, you can tweak your deployment tool to remove the ConfigMap only when the Rollout is completed successfully.
Example with Argo CD:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: my-config-7270e14e6\n annotations:\n argocd.argoproj.io/sync-options: PruneLast=true\n
"},{"location":"concepts/","title":"Concepts","text":""},{"location":"concepts/#rollout","title":"Rollout","text":"A Rollout is Kubernetes workload resource which is equivalent to a Kubernetes Deployment object. It is intended to replace a Deployment object in scenarios when more advanced deployment or progressive delivery functionality is needed. A Rollout provides the following features which a Kubernetes Deployment cannot:
- blue-green deployments
- canary deployments
- integration with ingress controllers and service meshes for advanced traffic routing
- integration with metric providers for blue-green & canary analysis
- automated promotion or rollback based on successful or failed metrics
"},{"location":"concepts/#progressive-delivery","title":"Progressive Delivery","text":"Progressive delivery is the process of releasing updates of a product in a controlled and gradual manner, thereby reducing the risk of the release, typically coupling automation and metric analysis to drive the automated promotion or rollback of the update.
Progressive delivery is often described as an evolution of continuous delivery, extending the speed benefits made in CI/CD to the deployment process. This is accomplished by limiting the exposure of the new version to a subset of users, observing and analyzing for correct behavior, then progressively increasing the exposure to a broader and wider audience while continuously verifying correctness.
"},{"location":"concepts/#deployment-strategies","title":"Deployment Strategies","text":"While the industry has used a consistent terminology to describe various deployment strategies, the implementations of these strategies tend to differ across tooling. To make it clear how the Argo Rollouts will behave, here are the descriptions of the various deployment strategies implementations offered by the Argo Rollouts.
"},{"location":"concepts/#rolling-update","title":"Rolling Update","text":"A RollingUpdate slowly replaces the old version with the new version. As the new version comes up, the old version is scaled down in order to maintain the overall count of the application. This is the default strategy of the Deployment object.
"},{"location":"concepts/#recreate","title":"Recreate","text":"A Recreate deployment deletes the old version of the application before bring up the new version. As a result, this ensures that two versions of the application never run at the same time, but there is downtime during the deployment.
"},{"location":"concepts/#blue-green","title":"Blue-Green","text":"A Blue-Green deployment (sometimes referred to as a Red-Black) has both the new and old version of the application deployed at the same time. During this time, only the old version of the application will receive production traffic. This allows the developers to run tests against the new version before switching the live traffic to the new version.
"},{"location":"concepts/#canary","title":"Canary","text":"A Canary deployment exposes a subset of users to the new version of the application while serving the rest of the traffic to the old version. Once the new version is verified to be correct, the new version can gradually replace the old version. Ingress controllers and service meshes such as NGINX and Istio, enable more sophisticated traffic shaping patterns for canarying than what is natively available (e.g. achieving very fine-grained traffic splitting, or splitting based on HTTP headers).
The picture above shows a canary with two stages (10% and 33% of traffic goes to new version) but this is just an example. With Argo Rollouts you can define the exact number of stages and percentages of traffic according to your use case.
"},{"location":"concepts/#which-strategy-to-choose","title":"Which strategy to choose","text":"In general Blue/Green is the easier strategy to start with, but also the more limited. We recommend you start with Blue/Green deployments first and as you gain confidence for your metrics and applications switch to Canaries.
You also need to examine if your application can handle canaries or not.
- Blue/Green always works because only one application is active at a time. Not all applications can have different versions running in parallel at the same time (which is what canaries are doing). This can be a showstopper for adopting canary deployments especially for legacy applications.
- Blue/Green is simpler because you can get their full value WITHOUT a traffic manager. While canaries can also work without a traffic manager, most of their advanced features assume a fine-grained way to control traffic. If you don't have a traffic manager, then you can easily get the full value of blue/green deployments but only the basic capabilities of canaries.
- Blue/Green also works with services that use queues and databases (workers that fetch tasks). Argo Rollouts doesn't control traffic flow for connections it doesn't understand (i.e. binary/queue channels).
Here is a summary table for the possible approaches.
Blue/Green Basic Canary Canary with Traffic manager Adoption Complexity Low Medium High Flexibility Low High Maximum Needs traffic provider No No Yes Works with queue workers Yes No No Works with shared/locked resources Yes No No Traffic switch All or nothing Gradual percentage Gradual percentage Traffic control 0% or 100% coarse grained fine grained Traffic depends on deployment state number of canary pods Any split option is possible Advanced routing scenarios No No Yes Failure Blast Radius Massive impact Low impact Low impact Note that that traffic manager can be any compatible Service Mesh or Ingress Controller or Gateway API implementation (via a plugin).
"},{"location":"dashboard/","title":"UI Dashboard","text":"The Argo Rollouts Kubectl plugin can serve a local UI Dashboard to visualize your Rollouts.
To start it, run kubectl argo rollouts dashboard in the namespace that contains your Rollouts. Then visit localhost:3100 to view the user interface.
"},{"location":"dashboard/#list-view","title":"List view","text":""},{"location":"dashboard/#individual-rollout-view","title":"Individual Rollout view","text":""},{"location":"getting-started/","title":"Getting Started","text":"This guide will demonstrate various concepts and features of Argo Rollouts by going through deployment, upgrade, promotion, and abortion of a Rollout.
"},{"location":"getting-started/#requirements","title":"Requirements","text":" - Kubernetes cluster with argo-rollouts controller installed (see install guide)
- kubectl with argo-rollouts plugin installed (see install guide)
"},{"location":"getting-started/#1-deploying-a-rollout","title":"1. Deploying a Rollout","text":"First we deploy a Rollout resource and a Kubernetes Service targeting that Rollout. The example Rollout in this guide utilizes a canary update strategy which sends 20% of traffic to the canary, followed by a manual promotion, and finally gradual automated traffic increases for the remainder of the upgrade. This behavior is described in the following portion of the Rollout spec:
spec:\n replicas: 5\n strategy:\n canary:\n steps:\n - setWeight: 20\n - pause: {}\n - setWeight: 40\n - pause: {duration: 10}\n - setWeight: 60\n - pause: {duration: 10}\n - setWeight: 80\n - pause: {duration: 10}\n
Run the following command to deploy the initial Rollout and Service:
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/basic/rollout.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/basic/service.yaml\n
Initial creations of any Rollout will immediately scale up the replicas to 100% (skipping any canary upgrade steps, analysis, etc...) since there was no upgrade that occurred.
The Argo Rollouts kubectl plugin allows you to visualize the Rollout, its related resources (ReplicaSets, Pods, AnalysisRuns), and presents live state changes as they occur. To watch the rollout as it deploys, run the get rollout --watch command from plugin:
kubectl argo rollouts get rollout rollouts-demo --watch\n
"},{"location":"getting-started/#2-updating-a-rollout","title":"2. Updating a Rollout","text":"Next it is time to perform an update. Just as with Deployments, any change to the Pod template field (spec.template) results in a new version (i.e. ReplicaSet) to be deployed. Updating a Rollout involves modifying the rollout spec, typically changing the container image field with a new version, and then running kubectl apply against the new manifest. As a convenience, the rollouts plugin provides a set image command, which performs these steps against the live rollout object in-place. Run the following command to update the rollouts-demo Rollout with the \"yellow\" version of the container:
kubectl argo rollouts set image rollouts-demo \\\n rollouts-demo=argoproj/rollouts-demo:yellow\n
During a rollout update, the controller will progress through the steps defined in the Rollout's update strategy. The example rollout sets a 20% traffic weight to the canary, and pauses the rollout indefinitely until user action is taken to unpause/promote the rollout. After updating the image, watch the rollout again until it reaches the paused state:
kubectl argo rollouts get rollout rollouts-demo --watch\n
When the demo rollout reaches the second step, we can see from the plugin that the Rollout is in a paused state, and now has 1 of 5 replicas running the new version of the pod template, and 4 of 5 replicas running the old version. This equates to the 20% canary weight as defined by the setWeight: 20 step.
"},{"location":"getting-started/#3-promoting-a-rollout","title":"3. Promoting a Rollout","text":"The rollout is now in a paused state. When a Rollout reaches a pause step with no duration, it will remain in a paused state indefinitely until it is resumed/promoted. To manually promote a rollout to the next step, run the promote command of the plugin:
kubectl argo rollouts promote rollouts-demo\n
After promotion, Rollout will proceed to execute the remaining steps. The remaining rollout steps in our example are fully automated, so the Rollout will eventually complete steps until it has has fully transitioned to the new version. Watch the rollout again until it has completed all steps:
kubectl argo rollouts get rollout rollouts-demo --watch\n
Tip
The promote command also supports the ability to skip all remaining steps and analysis with the --full flag.
Once all steps complete successfully, the new ReplicaSet is marked as the \"stable\" ReplicaSet. Whenever a rollout is aborted during an update, either automatically via a failed canary analysis, or manually by a user, the Rollout will fall back to the \"stable\" version.
"},{"location":"getting-started/#4-aborting-a-rollout","title":"4. Aborting a Rollout","text":"Next we will learn how to manually abort a rollout during an update. First, deploy a new \"red\" version of the container using the set image command, and wait for the rollout to reach the paused step again:
kubectl argo rollouts set image rollouts-demo \\\n rollouts-demo=argoproj/rollouts-demo:red\n
This time, instead of promoting the rollout to the next step, we will abort the update, so that it falls back to the \"stable\" version. The plugin provides an abort command as a way to manually abort a rollout at any time during an update:
kubectl argo rollouts abort rollouts-demo\n
When a rollout is aborted, it will scale up the \"stable\" version of the ReplicaSet (in this case the yellow image), and scale down any other versions. Although the stable version of the ReplicaSet may be running and is healthy, the overall rollout is still considered Degraded, since the desired version (the red image) is not the version which is actually running.
In order to make Rollout considered Healthy again and not Degraded, it is necessary to change the desired state back to the previous, stable version. This typically involves running kubectl apply against the previous Rollout spec. In our case, we can simply re-run the set image command using the previous, \"yellow\" image.
kubectl argo rollouts set image rollouts-demo \\\n rollouts-demo=argoproj/rollouts-demo:yellow\n
After running this command, you should notice that the Rollout immediately becomes Healthy, and there is no activity with regards to new ReplicaSets becoming created.
When a Rollout has not yet reached its desired state (e.g. it was aborted, or in the middle of an update), and the stable manifest were re-applied, the Rollout detects this as a rollback and not a update, and will fast-track the deployment of the stable ReplicaSet by skipping analysis, and the steps.
"},{"location":"getting-started/#summary","title":"Summary","text":"In this guide, we have learned basic capabilities of Argo Rollouts, including:
- Deploying a rollout
- Performing a canary update
- Manual promotion
- Manual abortion
The Rollout in this basic example did not utilize a ingress controller or service mesh provider to route traffic. Instead, it used normal Kubernetes Service networking (i.e. kube-proxy) to achieve an approximate canary weight, based on the closest ratio of new to old replica counts. As a result, this Rollout had a limitation in that it could only achieve a minimum canary weight of 20%, by scaling 1 of 5 pods to run the new version. In order to achieve much finer grained canaries, an ingress controller or service mesh is necessary.
Follow one of the traffic routing guides to see how Argo Rollouts can leverage a networking provider to achieve more advanced traffic shaping.
- ALB Guide
- App Mesh Guide
- Ambassador Guide
- Istio Guide
- Multiple Providers Guide
- NGINX Guide
- SMI Guide
"},{"location":"installation/","title":"Installation","text":""},{"location":"installation/#controller-installation","title":"Controller Installation","text":"Two types of installation:
- install.yaml - Standard installation method.
kubectl create namespace argo-rollouts\nkubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml\n
This will create a new namespace, argo-rollouts, where Argo Rollouts controller will run.
Tip
If you are using another namespace name, please update install.yaml clusterrolebinding's serviceaccount namespace name.
Tip
When installing Argo Rollouts on Kubernetes v1.14 or lower, the CRD manifests must be kubectl applied with the --validate=false option. This is caused by use of new CRD fields introduced in v1.15, which are rejected by default in lower API servers.
Tip
On GKE, you will need grant your account the ability to create new cluster roles:
kubectl create clusterrolebinding YOURNAME-cluster-admin-binding --clusterrole=cluster-admin --user=YOUREMAIL@gmail.com\n
- namespace-install.yaml - Installation of Argo Rollouts which requires only namespace level privileges. An example usage of this installation method would be to run several Argo Rollouts controller instances in different namespaces on the same cluster.
Note: Argo Rollouts CRDs are not included into namespace-install.yaml. and have to be installed separately. The CRD manifests are located in manifests/crds directory. Use the following command to install them:
kubectl apply -k https://github.com/argoproj/argo-rollouts/manifests/crds\\?ref\\=stable\n
You can find released container images of the controller at Quay.io. There are also old releases at Dockerhub, but since the introduction of rate limiting, the Argo project has moved to Quay.
"},{"location":"installation/#kubectl-plugin-installation","title":"Kubectl Plugin Installation","text":"The kubectl plugin is optional, but is convenient for managing and visualizing rollouts from the command line.
"},{"location":"installation/#brew","title":"Brew","text":"brew install argoproj/tap/kubectl-argo-rollouts\n
"},{"location":"installation/#manual","title":"Manual","text":" -
Install Argo Rollouts Kubectl plugin with curl.
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-darwin-amd64\n
For Linux dist, replace darwin with linux
-
Make the kubectl-argo-rollouts binary executable.
chmod +x ./kubectl-argo-rollouts-darwin-amd64\n
-
Move the binary into your PATH.
sudo mv ./kubectl-argo-rollouts-darwin-amd64 /usr/local/bin/kubectl-argo-rollouts\n
Test to ensure the version you installed is up-to-date:
kubectl argo rollouts version\n
"},{"location":"installation/#shell-auto-completion","title":"Shell auto completion","text":"To enable auto completion for the plugin when used with kubectl (version 1.26 or newer), you need to create a shell script on your PATH called kubectl_complete-argo-rollouts which will provide the completions.
cat <<EOF >kubectl_complete-argo-rollouts\n#!/usr/bin/env sh\n\n# Call the __complete command passing it all arguments\nkubectl argo rollouts __complete \"\\$@\"\nEOF\n\nchmod +x kubectl_complete-argo-rollouts\nsudo mv ./kubectl_complete-argo-rollouts /usr/local/bin/\n
To enable auto completion for the CLI run as a standalone binary, the CLI can export shell completion code for several shells.
For bash, ensure you have bash completions installed and enabled. To access completions in your current shell, run $ source <(kubectl-argo-rollouts completion bash). Alternatively, write it to a file and source in .bash_profile.
The completion command supports bash, zsh, fish and powershell.
See the completion command documentation for more details.
"},{"location":"installation/#using-the-cli-with-docker","title":"Using the CLI with Docker","text":"The CLI is also available as a container image at https://quay.io/repository/argoproj/kubectl-argo-rollouts
You can run it like any other Docker image or use it in any CI platform that supports Docker images.
docker run quay.io/argoproj/kubectl-argo-rollouts:master version\n
"},{"location":"installation/#supported-versions","title":"Supported versions","text":"Check e2e testing file to see what the Kubernetes version is being fully tested.
You can switch to different tags to see what relevant Kubernetes versions were being tested for the respective version.
"},{"location":"installation/#upgrading-argo-rollouts","title":"Upgrading Argo Rollouts","text":"Argo Rollouts is a Kubernetes controller that doesn't hold any external state. It is active only when deployments are actually happening.
To upgrade Argo Rollouts:
- Try to find a time period when no deployments are happening
- Delete the previous version of the controller and apply/install the new one
- When a new Rollout takes place the new controller will be activated.
If deployments are happening while you upgrade the controller, then you shouldn't have any downtime. Current Rollouts will be paused and as soon as the new controller becomes active it will resume all in-flight deployments.
"},{"location":"migrating/","title":"Migrating to Rollouts","text":"There are ways to migrate to Rollout:
- Convert an existing Deployment resource to a Rollout resource.
- Reference an existing Deployment from a Rollout using
workloadRef field.
"},{"location":"migrating/#convert-deployment-to-rollout","title":"Convert Deployment to Rollout","text":"When converting a Deployment to a Rollout, it involves changing three fields:
- Replacing the
apiVersion from apps/v1 to argoproj.io/v1alpha1 - Replacing the
kind from Deployment to Rollout - Replacing the deployment strategy with a blue-green or canary strategy
Below is an example of a Rollout resource using the canary strategy.
apiVersion: argoproj.io/v1alpha1 # Changed from apps/v1\nkind: Rollout # Changed from Deployment\nmetadata:\n name: rollouts-demo\nspec:\n selector:\n matchLabels:\n app: rollouts-demo\n template:\n metadata:\n labels:\n app: rollouts-demo\n spec:\n containers:\n - name: rollouts-demo\n image: argoproj/rollouts-demo:blue\n ports:\n - containerPort: 8080\n strategy:\n canary: # Changed from rollingUpdate or recreate\n steps:\n - setWeight: 20\n - pause: {}\n
Warning
When migrating a Deployment which is already serving live production traffic, a Rollout should run next to the Deployment before deleting the Deployment or scaling down the Deployment. Not following this approach might result in downtime. It also allows for the Rollout to be tested before deleting the original Deployment.
"},{"location":"migrating/#reference-deployment-from-rollout","title":"Reference Deployment From Rollout","text":"Instead of removing Deployment you can scale it down to zero and reference it from the Rollout resource:
- Create a Rollout resource.
- Reference an existing Deployment using
workloadRef field. - In the
workloadRef field set the scaleDown attribute, which specifies how the Deployment should be scaled down. There are three options available: never: the Deployment is not scaled downonsuccess: the Deployment is scaled down after the Rollout becomes healthyprogressively: as the Rollout is scaled up the Deployment is scaled down.
Alternatively, manually scale down an existing Deployment by changing replicas field of an existing Deployment to zero. 1. To perform an update, the change should be made to the Pod template field of the Deployment.
Below is an example of a Rollout resource referencing a Deployment.
apiVersion: argoproj.io/v1alpha1 # Create a rollout resource\nkind: Rollout\nmetadata:\n name: rollout-ref-deployment\nspec:\n replicas: 5\n selector:\n matchLabels:\n app: rollout-ref-deployment\n workloadRef: # Reference an existing Deployment using workloadRef field\n apiVersion: apps/v1\n kind: Deployment\n name: rollout-ref-deployment\n scaleDown: onsuccess\n strategy:\n canary:\n steps:\n - setWeight: 20\n - pause: {duration: 10s}\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n labels:\n app.kubernetes.io/instance: rollout-canary\n name: rollout-ref-deployment\nspec:\n replicas: 0 # Scale down existing deployment\n selector:\n matchLabels:\n app: rollout-ref-deployment\n template:\n metadata:\n labels:\n app: rollout-ref-deployment\n spec:\n containers:\n - name: rollouts-demo\n image: argoproj/rollouts-demo:blue\n imagePullPolicy: Always\n ports:\n - containerPort: 8080\n
Consider following if your Deployment runs in production:
"},{"location":"migrating/#running-rollout-and-deployment-side-by-side","title":"Running Rollout and Deployment side-by-side","text":"After creation Rollout will spinup required number of Pods side-by-side with the Deployment Pods. Rollout won't try to manage existing Deployment Pods. That means you can safely update add Rollout to the production environment without any interruption but you are going to run twice more Pods during migration.
Argo-rollouts controller patches the spec of rollout object with an annotation of rollout.argoproj.io/workload-generation, which equals the generation of referenced deployment. Users can detect if the rollout matches desired generation of deployment by checking the workloadObservedGeneration in the rollout status.
"},{"location":"migrating/#traffic-management-during-migration","title":"Traffic Management During Migration","text":"The Rollout offers traffic management functionality that manages routing rules and flows the traffic to different versions of an application. For example Blue-Green deployment strategy manipulates Kubernetes Service selector and direct production traffic to \"green\" instances only.
If you are using this feature then Rollout switches production\u00a0traffic to Pods that it manages. The switch happens only when the required number of Pod is running and healthy so it is safe in production as well. However, if you want to be extra careful then consider creating a temporal Service or Ingress object to validate Rollout behavior. Once testing is done delete temporal Service/Ingress and switch rollout to production one.
"},{"location":"migrating/#migrating-to-deployments","title":"Migrating to Deployments","text":"In case users want to rollback to the deployment kinds from rollouts, there are two scenarios aligned with those in Migrating to Rollouts.
- Convert a Rollout resource to a Deployment resource.
- Reference an existing Deployment from a Rollout using
workloadRef field.
"},{"location":"migrating/#convert-rollout-to-deployment","title":"Convert Rollout to Deployment","text":"When converting a Rollout to a Deployment, it involves changing three fields:
- Changing the apiVersion from argoproj.io/v1alpha1 to apps/v1
- Changing the kind from Rollout to Deployment
- Remove the rollout strategy in
spec.strategy.canary or spec.strategy.blueGreen
Warning
When migrating a Rollout which is already serving live production traffic, a Deployment should run next to the rollout before deleting the rollout or scaling down the rollout. Not following this approach might result in downtime. It also allows for the Deployment to be tested before deleting the original Rollout.
"},{"location":"migrating/#reference-deployment-from-rollout_1","title":"Reference Deployment From Rollout","text":"When a rollout is referencing to a deployment:
- Scale-up an existing Deployment by changing its
replicas field to a desired number of pods. - Wait for the Deployment pods to become Ready.
- Scale-down an existing Rollout by changing its
replicas field to zero.
Please refer to Running Rollout and Deployment side-by-side and Traffic Management During Migration for caveats.
"},{"location":"plugins/","title":"Creating an Argo Rollouts Plugin","text":""},{"location":"plugins/#high-level-overview","title":"High Level Overview","text":"Argo Rollouts plugins depend on hashicorp's go-plugin library. This library provides a way for a plugin to be compiled as a standalone executable and then loaded by the rollouts controller at runtime. This works by having the plugin executable act as a rpc server and the rollouts controller act as a client. The plugin executable is started by the rollouts controller and is a long-lived process and that the rollouts controller connects to over a unix socket.
Here is an overview of how plugins are loaded:
The communication protocol uses golang built in net/rpc library so plugins have to be written in golang.
"},{"location":"plugins/#plugin-repository","title":"Plugin Repository","text":"In order to get plugins listed in the main argo rollouts documentation we ask that the plugin repository be created under the argoproj-labs organization. Please open an issue under argo-rollouts requesting a repo which you would be granted admin access on.
There is also a standard naming convention for plugin names used for configmap registration, as well as what the plugin uses for locating its specific configuration on rollout or analysis resources. The name needs to be in the form of <namespace>/<name> and both and have a regular expression check that matches Github's requirements for username/org and repository name. This requirement is in place to help with allowing multiple creators of the same plugin types to exist such as <org1>/nginx and <org2>/nginx. These names could be based of the repo name such as argoproj-labs/rollouts-plugin-metric-sample-prometheus but it is not a requirement.
There will also be a standard for naming repositories under argoproj-labs in the form of rollouts-plugin-<type>-<tool> where <type> is say metric, or trafficrouter and <tool> is the software the plugin is for say nginx.
"},{"location":"plugins/#plugin-name","title":"Plugin Name","text":"So now that we have an idea on plugin naming and repository standards let's pick a name to use for the rest of this documentation and call our plugin argoproj-labs/nginx.
This name will be used in a few different spots the first is the config map that your plugin users will need to configure. It looks like this below.
kind: ConfigMap\nmetadata:\n name: argo-rollouts-config\ndata:\n metricProviderPlugins: |-\n - name: \"argoproj-labs/metrics\"\n location: \"file:///tmp/argo-rollouts/metric-plugin\"\n args:\n - \"--log-level\"\n - \"debug\"\n trafficRouterPlugins: |-\n - name: \"argoproj-labs/nginx\"\n location: \"file:///tmp/argo-rollouts/traffic-plugin\"\n args:\n - \"--log-level\"\n - \"debug\"\n
As you can see there is a field called name: under both metrics or trafficrouters this is the first place where your end users will need to configure the name of the plugin. The second location is either in the rollout object or the analysis template which you can see the examples below. The third args holds the command line arguments of the plugin.
"},{"location":"plugins/#analysistemplate-example","title":"AnalysisTemplate Example","text":"apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n metrics:\n - name: success-rate\n ...\n provider:\n plugin:\n argoproj-labs/metrics:\n address: http://prometheus.local\n
"},{"location":"plugins/#traffic-router-example","title":"Traffic Router Example","text":"apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: example-plugin-ro\nspec:\n strategy:\n canary:\n canaryService: example-plugin-ro-canary-analysis\n stableService: example-plugin-ro-stable-analysis\n trafficRouting:\n plugins:\n argoproj-labs/nginx:\n stableIngress: canary-demo\n
You can see that we use the plugin name under spec.metrics[].provider.plugin for analysis template and spec.strategy.canary.trafficRouting.plugins for traffic routers. You as a plugin author can then put any configuration you need under argoproj-labs/nginx and you will be able to look up that config in your plugin via the plugin name key. You will also want to document what configuration options your plugin supports.
"},{"location":"plugins/#plugin-interfaces","title":"Plugin Interfaces","text":"Argo Rollouts currently supports two plugin systems as a plugin author your end goal is to implement these interfaces as a hashicorp go-plugin. The two interfaces are MetricsPlugin and TrafficRouterPlugin for each of the respective plugins:
type MetricProviderPlugin interface {\n // InitPlugin initializes the traffic router plugin this gets called once when the plugin is loaded.\n InitPlugin() RpcError\n // Run start a new external system call for a measurement\n // Should be idempotent and do nothing if a call has already been started\n Run(*v1alpha1.AnalysisRun, v1alpha1.Metric) v1alpha1.Measurement\n // Resume Checks if the external system call is finished and returns the current measurement\n Resume(*v1alpha1.AnalysisRun, v1alpha1.Metric, v1alpha1.Measurement) v1alpha1.Measurement\n // Terminate will terminate an in-progress measurement\n Terminate(*v1alpha1.AnalysisRun, v1alpha1.Metric, v1alpha1.Measurement) v1alpha1.Measurement\n // GarbageCollect is used to garbage collect completed measurements to the specified limit\n GarbageCollect(*v1alpha1.AnalysisRun, v1alpha1.Metric, int) RpcError\n // Type gets the provider type\n Type() string\n // GetMetadata returns any additional metadata which providers need to store/display as part\n // of the metric result. For example, Prometheus uses is to store the final resolved queries.\n GetMetadata(metric v1alpha1.Metric) map[string]string\n}\n\ntype TrafficRouterPlugin interface {\n // InitPlugin initializes the traffic router plugin this gets called once when the plugin is loaded.\n InitPlugin() RpcError\n // UpdateHash informs a traffic routing reconciler about new canary, stable, and additionalDestination(s) pod hashes\n UpdateHash(rollout *v1alpha1.Rollout, canaryHash, stableHash string, additionalDestinations []v1alpha1.WeightDestination) RpcError\n // SetWeight sets the canary weight to the desired weight\n SetWeight(rollout *v1alpha1.Rollout, desiredWeight int32, additionalDestinations []v1alpha1.WeightDestination) RpcError\n // SetHeaderRoute sets the header routing step\n SetHeaderRoute(rollout *v1alpha1.Rollout, setHeaderRoute *v1alpha1.SetHeaderRoute) RpcError\n // SetMirrorRoute sets up the traffic router to mirror traffic to a service\n SetMirrorRoute(rollout *v1alpha1.Rollout, setMirrorRoute *v1alpha1.SetMirrorRoute) RpcError\n // VerifyWeight returns true if the canary is at the desired weight and additionalDestinations are at the weights specified\n // Returns nil if weight verification is not supported or not applicable\n VerifyWeight(rollout *v1alpha1.Rollout, desiredWeight int32, additionalDestinations []v1alpha1.WeightDestination) (RpcVerified, RpcError)\n // RemoveManagedRoutes Removes all routes that are managed by rollouts by looking at spec.strategy.canary.trafficRouting.managedRoutes\n RemoveManagedRoutes(ro *v1alpha1.Rollout) RpcError\n // Type returns the type of the traffic routing reconciler\n Type() string\n}\n
"},{"location":"plugins/#plugin-init-function","title":"Plugin Init Function","text":"Each plugin interface has a InitPlugin function, this function is called when the plugin is first started up and is only called once per startup. The InitPlugin function is used as a means to initialize the plugin it gives you the plugin author the ability to either set up a client for a specific metrics provider or in the case of a traffic router construct a client or informer for kubernetes api. The one thing to note about this though is because these calls happen over RPC the plugin author should not depend on state being stored in the plugin struct as it will not be persisted between calls.
"},{"location":"plugins/#kubernetes-rbac","title":"Kubernetes RBAC","text":"The plugin runs as a child process of the rollouts controller and as such it will inherit the same RBAC permissions as the controller. This means that the service account for the rollouts controller will need the correct permissions for the plugin to function. This might mean instructing users to create a role and role binding to the standard rollouts service account for the plugin to use. This will probably affect traffic router plugins more than metrics plugins.
"},{"location":"plugins/#sample-plugins","title":"Sample Plugins","text":"There are two sample plugins within the argo-rollouts repo that you can use as a reference for creating your own plugin.
- Metrics Plugin Sample
- Traffic Router Plugin Sample
"},{"location":"releasing/","title":"Releasing","text":" -
Ensure that the release branch already exist.
-
Checkout the release branch. Example: git fetch upstream && git checkout release-1.5
-
Run the script found at hack/trigger-release.sh as follows:
./hack/trigger-release.sh <version> <remote name>\n
Example:
./hack/trigger-release.sh v1.6.0-rc1 upstream\n
Tip
The tag must be in one of the following formats to trigger the GH workflow: * GA: v<MAJOR>.<MINOR>.<PATCH> * Pre-release: v<MAJOR>.<MINOR>.<PATCH>-rc<RC#>
Once the script is executed successfully, a GitHub workflow will start execution. You can follow its progress under the Actions tab, the name of the action is Release.
- When the action completes, visit the generated draft Github releases and enter the details about the release:
- Getting started (copy from previous release and new version)
- Changelog
"},{"location":"releasing/#update-brew-formula","title":"Update Brew formula","text":" -
Update Brew formula:
-
Fork the repo https://github.com/argoproj/homebrew-tap
- Run the following commands to update the brew formula:
cd homebrew-tap\n./update.sh kubectl-argo-rollouts $VERSION\n
- If there is a new minor version we want to update the versioned formula as well:
- Run the following commands to update the versioned brew formula:
./update.sh kubectl-argo-rollouts $VERSION @<version_without_patch_and_v>\n
- Example: If the new version is
v1.3.2, we want to update the formula for v1.3 as well. ./update.sh kubectl-argo-rollouts v1.3.2 @1.3\n
- Commit and push the changes to your fork
git commit -am \"Update kubectl-argo-rollouts to $VERSION\"\n
- Create a PR with the modified files pointing to upstream/master
- Once the PR is approved by a maintainer, it can be merged.
"},{"location":"releasing/#verify","title":"Verify","text":" -
Install locally using the command below and follow the Getting Started Guide:
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/download/${VERSION}/install.yaml\n
-
Check the Kubectl Argo Rollout plugin:
brew upgrade kubectl-argo-rollouts\nkubectl argo rollouts version\n
"},{"location":"roadmap/","title":"Roadmap","text":"The Argo Rollouts roadmap is maintained in Github Milestones on the Github repository.
"},{"location":"roadmap/#release-cycle","title":"Release Cycle","text":""},{"location":"roadmap/#schedule","title":"Schedule","text":"These are the upcoming releases date estimates:
Release Release Planning Meeting Release Candidate 1 General Availability v1.4 TBD Monday, Dec. 19, 2022 Tuesday, Jan. 9, 2023 v1.5 Monday, Mar. 6, 2023 Monday, Mar. 20, 2023 Monday, Apr. 10, 2023 v1.6 Monday, Jun. 5, 2023 Monday, Jun. 19, 2023 Wednesday, Jul. 12, 2023 v1.7 Monday, Sep. 4, 2023 Monday, Sep. 18, 2023 Monday, Oct. 9, 2023"},{"location":"roadmap/#release-process","title":"Release Process","text":""},{"location":"roadmap/#minor-releases-eg-1x0","title":"Minor Releases (e.g. 1.x.0)","text":"A minor Argo Rollouts release occurs four times a year, once every three months. Each General Availability (GA) release is preceded by several Release Candidates (RCs). The first RC is released three weeks before the scheduled GA date.
These are the approximate release dates:
- The first Monday of January
- The first Monday of April
- The first Monday of July
- The first Monday of October
Dates may be shifted slightly to accommodate holidays. Those shifts should be minimal.
"},{"location":"roadmap/#patch-releases-eg-14x","title":"Patch Releases (e.g. 1.4.x)","text":"Argo Rollouts patch releases occur on an as-needed basis. Only the three most recent minor versions are eligible for patch releases. Versions older than the three most recent minor versions are considered EOL and will not receive bug fixes or security updates.
"},{"location":"roadmap/#feature-acceptance-criteria","title":"Feature Acceptance Criteria","text":"To be eligible for inclusion in a minor release, a new feature must meet the following criteria before the release\u2019s RC date.
If it is a large feature that involves significant design decisions, that feature must be described in a Proposal.
The feature PR must include:
- Tests (passing)
- Documentation
- If necessary, a note in the Upgrading docs for the planned minor release
- The PR must be reviewed, approved, and merged by an Approver.
If these criteria are not met by the RC date, the feature will be ineligible for inclusion in the RC series or GA for that minor release. It will have to wait for the next minor release.
"},{"location":"analysis/cloudwatch/","title":"CloudWatch Metrics","text":"Important
Available since v1.1.0
A CloudWatch using GetMetricData can be used to obtain measurements for analysis.
"},{"location":"analysis/cloudwatch/#setup","title":"Setup","text":"You can use CloudWatch Metrics if you have used to EKS or not. This analysis is required IAM permission for cloudwatch:GetMetricData and you need to define AWS_REGION in Deployment for argo-rollouts.
"},{"location":"analysis/cloudwatch/#eks","title":"EKS","text":"If you create new cluster on EKS, you can attach cluster IAM role or attach IAM roles for service accounts. If you have already cluster on EKS, you can attach IAM roles for service accounts.
"},{"location":"analysis/cloudwatch/#not-eks","title":"not EKS","text":"You need to define access key and secret key.
apiVersion: v1\nkind: Secret\nmetadata:\n name: cloudwatch-secret\ntype: Opaque\nstringData:\n AWS_ACCESS_KEY_ID: <aws-access-key-id>\n AWS_SECRET_ACCESS_KEY: <aws-secret-access-key>\n AWS_REGION: <aws-region>\n
apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: argo-rollouts\nspec:\n template:\n spec:\n containers:\n - name: argo-rollouts\n env:\n - name: AWS_ACCESS_KEY_ID\n valueFrom:\n secretKeyRef:\n name: cloudwatch-secret\n key: AWS_ACCESS_KEY_ID\n - name: AWS_SECRET_ACCESS_KEY\n valueFrom:\n secretKeyRef:\n name: cloudwatch-secret\n key: AWS_SECRET_ACCESS_KEY\n - name: AWS_REGION\n valueFrom:\n secretKeyRef:\n name: cloudwatch-secret\n key: AWS_REGION\n
"},{"location":"analysis/cloudwatch/#configuration","title":"Configuration","text":" metricDataQueries - GetMetricData query: MetricDataQueryinterval - optional interval, e.g. 30m, default: 5m
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n metrics:\n - name: success-rate\n interval: 1m\n successCondition: \"len(result[0].Values) >= 5 and all(result[0].Values, {# <= 0.01})\"\n failureLimit: 3\n provider:\n cloudWatch:\n interval: 30m\n metricDataQueries:\n - {\n \"id\": \"rate\",\n \"expression\": \"errors / requests\"\n }\n - {\n \"id\": \"errors\",\n \"metricStat\": {\n \"metric\": {\n \"namespace\": \"app\",\n \"metricName\": \"errors\"\n },\n \"period\": 300,\n \"stat\": \"Sum\",\n \"unit\": \"Count\"\n },\n \"returnData\": false\n }\n - {\n \"id\": \"requests\",\n \"metricStat\": {\n \"metric\": {\n \"namespace\": \"app\",\n \"metricName\": \"requests\"\n },\n \"period\": 300,\n \"stat\": \"Sum\",\n \"unit\": \"Count\"\n },\n \"returnData\": false\n }\n
"},{"location":"analysis/cloudwatch/#debug","title":"debug","text":"You can confirm the results value in AnalysisRun.
$ kubectl get analysisrun/rollouts-name-xxxxxxxxxx-xx -o yaml\n(snip)\nstatus:\n metricResults:\n - count: 2\n failed: 1\n measurements:\n - finishedAt: \"2021-09-08T17:29:14Z\"\n phase: Failed\n startedAt: \"2021-09-08T17:29:13Z\"\n value: '[[0.0029476787030213707 0.006100422336931018 0.01020408163265306 0.007932573128408527\n 0.00589622641509434 0.006339144215530904]]'\n - finishedAt: \"2021-09-08T17:30:14Z\"\n phase: Successful\n startedAt: \"2021-09-08T17:30:14Z\"\n value: '[[0.004484304932735426 0.0058374494836102376 0.006736068585425597 0.008444444444444444\n 0.006859756097560976 0.0045385779122541605]]'\n name: success-rate\n phase: Running\n successful: 1\n phase: Running\n startedAt: \"2021-09-08T17:29:14Z\"\n
"},{"location":"analysis/datadog/","title":"Datadog Metrics","text":"A Datadog query can be used to obtain measurements for analysis.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: loq-error-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: error-rate\n interval: 5m\n successCondition: result <= 0.01\n failureLimit: 3\n provider:\n datadog:\n apiVersion: v2\n interval: 5m\n query: |\n sum:requests.error.rate{service:{{args.service-name}}}\n
The field apiVersion refers to the API version of Datadog (v1 or v2). Default value is v1 if this is omitted. See \"Working with Datadog API v2\" below for more information.
Datadog api and app tokens can be configured in a kubernetes secret in argo-rollouts namespace.
apiVersion: v1\nkind: Secret\nmetadata:\n name: datadog\ntype: Opaque\nstringData:\n address: https://api.datadoghq.com\n api-key: <datadog-api-key>\n app-key: <datadog-app-key>\n
apiVersion here is different from the apiVersion from the Datadog configuration above.
"},{"location":"analysis/datadog/#working-with-datadog-api-v2","title":"Working with Datadog API v2","text":"Important
While some basic v2 functionality is working in earlier versions, the new properties of formula and queries are only available as of v1.7
"},{"location":"analysis/datadog/#moving-to-v2","title":"Moving to v2","text":"If your old v1 was just a simple metric query - no formula as part of the query - then you can just move to v2 by updating the apiVersion in your existing Analysis Template, and everything should work.
If you have a formula, you will need to update how you configure your metric. Here is a before/after example of what your Analysis Template should look like:
Before:
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: log-error-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: error-rate\n interval: 30s\n successCondition: default(result, 0) < 10\n failureLimit: 3\n provider:\n datadog:\n apiVersion: v1\n interval: 5m\n query: \"moving_rollup(sum:requests.errors{service:{{args.service-name}}}.as_count(), 60, 'sum') / sum:requests{service:{{args.service-name}}}.as_count()\"\n
After:
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: loq-error-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: error-rate\n # Polling rate against the Datadog API\n interval: 30s\n successCondition: default(result, 0) < 10\n failureLimit: 3\n provider:\n datadog:\n apiVersion: v2\n # The window of time we are looking at in DD. Basically we will fetch data from (now-5m) to now.\n interval: 5m\n queries:\n a: sum:requests.errors{service:{{args.service-name}}}.as_count()\n b: sum:requests{service:{{args.service-name}}}.as_count()\n formula: \"moving_rollup(a, 60, 'sum') / b\"\n
"},{"location":"analysis/datadog/#examples","title":"Examples","text":"Simple v2 query with no formula
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: canary-container-restarts\nspec:\n args:\n # This is set in rollout using the valueFrom: podTemplateHashValue functionality\n - name: canary-hash\n - name: service-name\n - name: restarts.initial-delay\n value: \"60s\"\n - name: restarts.max-restarts\n value: \"4\"\n metrics:\n - name: kubernetes.containers.restarts\n initialDelay: \"{{ args.restarts.initial-delay }}\"\n interval: 15s\n failureCondition: default(result, 0) > {{ args.restarts.max-restarts }}\n failureLimit: 0\n provider:\n datadog:\n apiVersion: v2\n interval: 5m\n queries:\n # The key is arbitrary - you will use this key to refer to the query if you use a formula.\n q: \"max:kubernetes.containers.restarts{service-name:{{args.service-name}},rollouts_pod_template_hash:{{args.canary-hash}}}\"\n
"},{"location":"analysis/datadog/#tips","title":"Tips","text":""},{"location":"analysis/datadog/#datadog-results","title":"Datadog Results","text":"Datadog queries can return empty results if the query takes place during a time interval with no metrics. The Datadog provider will return a nil value yielding an error during the evaluation phase like:
invalid operation: < (mismatched types <nil> and float64)\n
However, empty query results yielding a nil value can be handled using the default() function. Here is a succeeding example using the default() function:
successCondition: default(result, 0) < 0.05\n
"},{"location":"analysis/datadog/#metric-aggregation-v2-only","title":"Metric aggregation (v2 only)","text":"By default, Datadog analysis run is configured to use last metric aggregator when querying Datadog v2 API. This value can be overriden by specifying a new aggregator value from a list of supported aggregators (avg,min,max,sum,last,percentile,mean,l2norm,area) for the V2 API (docs).
For example, using count-based distribution metric (count:metric{*}.as_count()) with values 1,9,3,7,5 in a given interval will make last aggregator return 5. To return a sum of all values (25), set aggregator: sum in Datadog provider block and use moving_rollup() function to aggregate values in the specified rollup interval. These functions can be combined in a formula to perform additional calculations:
...<snip>\n metrics:\n - name: error-percentage\n interval: 30s\n successCondition: default(result, 0) < 5\n failureLimit: 3\n provider:\n datadog:\n apiVersion: v2\n interval: 5m\n aggregator: sum # override default aggregator\n queries:\n a: count:requests.errors{service:my-service}.as_count()\n b: count:requests{service:my-service}.as_count()\n formula: \"moving_rollup(a, 300, 'sum') / moving_rollup(b, 300, 'sum') * 100\" # percentage of requests with errors\n
"},{"location":"analysis/datadog/#templates-and-helm","title":"Templates and Helm","text":"Helm and Argo Rollouts both try to parse things between {{ ... }} when rendering templates. If you use Helm to deliver your manifests, you will need to escape {{ args.whatever }}. Using the example above, here it is set up for Helm:
...<snip>\nmetrics:\n - name: kubernetes.containers.restarts\n initialDelay: \"{{ `{{ args.restarts.initial-delay }}` }}\"\n interval: 15s\n failureCondition: default(result, 0) > {{ `{{ args.restarts.max-restarts }}` }}\n failureLimit: 0\n provider:\n datadog:\n apiVersion: v2\n interval: 5m\n queries:\n q: \"{{ `max:kubernetes.containers.restarts{kube_app_name:{{args.kube_app_name}},rollouts_pod_template_hash:{{args.canary-hash}}}` }}\"\n
"},{"location":"analysis/datadog/#rate-limits","title":"Rate Limits","text":"For the v1 API, you ask for an increase on the api/v1/query route.
For the v2 API, the Ratelimit-Name you ask for an increase in is the query_scalar_public.
"},{"location":"analysis/graphite/","title":"Graphite Metrics","text":"A Graphite query can be used to obtain measurements for analysis.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: success-rate\n interval: 5m\n # Note that the Argo Rollouts Graphite metrics provider returns results as an array of float64s with 6 decimal places.\n successCondition: results[0] >= 90.000000\n failureLimit: 3\n provider:\n graphite:\n address: http://graphite.example.com:9090\n query: |\n target=summarize(\n asPercent(\n sumSeries(\n stats.timers.httpServerRequests.app.{{args.service-name}}.exception.*.method.*.outcome.{CLIENT_ERROR,INFORMATIONAL,REDIRECTION,SUCCESS}.status.*.uri.*.count\n ),\n sumSeries(\n stats.timers.httpServerRequests.app.{{args.service-name}}.exception.*.method.*.outcome.*.status.*.uri.*.count\n )\n ),\n '5min',\n 'avg'\n )\n
"},{"location":"analysis/influxdb/","title":"InfluxDB Metrics","text":"An InfluxDB query using Flux can be used to obtain measurements for analysis.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: error-rate\nspec:\n args:\n - name: application-name\n metrics:\n - name: error-rate\n # NOTE: To be consistent with the prometheus metrics provider InfluxDB query results are returned as an array.\n # In the example we're looking at index 0 of the returned array to obtain the value we're using for the success condition\n successCondition: result[0] <= 0.01\n provider:\n influxdb:\n profile: my-influxdb-secret # optional, defaults to 'influxdb'\n query: |\n from(bucket: \"app_istio\")\n |> range(start: -15m)\n |> filter(fn: (r) => r[\"destination_workload\"] == \"{{ args.application-name }}\")\n |> filter(fn: (r) => r[\"_measurement\"] == \"istio:istio_requests_errors_percentage:rate1m:5xx\")\n
An InfluxDB access profile can be configured using a Kubernetes secret in the argo-rollouts namespace. Alternate accounts can be used by creating more secrets of the same format and specifying which secret to use in the metric provider configuration using the profile field.
apiVersion: v1\nkind: Secret\nmetadata:\n name: influxdb\ntype: Opaque\nstringData:\n address: <infuxdb-url>\n authToken: <influxdb-auth-token>\n org: <influxdb-org>\n
"},{"location":"analysis/job/","title":"Job Metrics","text":"A Kubernetes Job can be used to run analysis. When a Job is used, the metric is considered successful if the Job completes and had an exit code of zero, otherwise it is failed.
metrics:\n - name: test\n provider:\n job:\n metadata:\n annotations:\n foo: bar # annotations defined here will be copied to the Job object\n labels:\n foo: bar # labels defined here will be copied to the Job object\n spec:\n backoffLimit: 1\n template:\n spec:\n containers:\n - name: test\n image: my-image:latest\n command: [my-test-script, my-service.default.svc.cluster.local]\n restartPolicy: Never\n
"},{"location":"analysis/kayenta/","title":"Kayenta","text":""},{"location":"analysis/kayenta/#kayenta-eg-mann-whitney-analysis","title":"Kayenta (e.g. Mann-Whitney Analysis)","text":"Analysis can also be done as part of an Experiment.
This example starts both a canary and baseline ReplicaSet. The ReplicaSets run for 1 hour, then scale down to zero. Call out to Kayenta to perform Mann-Whintney analysis against the two pods. Demonstrates ability to start a short-lived experiment and an asynchronous analysis.
This example demonstrates:
- The ability to start an Experiment as part of rollout steps, which launches multiple ReplicaSets (e.g. baseline & canary)
- The ability to reference and supply pod-template-hash to an AnalysisRun
- Kayenta metrics
Rollout apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\n labels:\n app: guestbook\nspec:\n strategy:\n canary:\n steps:\n - experiment:\n duration: 1h\n templates:\n - name: baseline\n specRef: stable\n - name: canary\n specRef: canary\n analyses:\n - templateName: mann-whitney\n args:\n - name: stable-hash\n valueFrom:\n podTemplateHashValue: Stable\n - name: canary-hash\n valueFrom:\n podTemplateHashValue: Latest\n
AnalysisTemplate apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: mann-whitney\nspec:\n args:\n - name: start-time\n - name: end-time\n - name: stable-hash\n - name: canary-hash\n metrics:\n - name: mann-whitney\n provider:\n kayenta:\n address: https://kayenta.example.com\n application: guestbook\n canaryConfigName: my-test\n thresholds:\n pass: 90\n marginal: 75\n scopes:\n - name: default\n controlScope:\n scope: app=guestbook and rollouts-pod-template-hash={{args.stable-hash}}\n step: 60\n start: \"{{args.start-time}}\"\n end: \"{{args.end-time}}\"\n experimentScope:\n scope: app=guestbook and rollouts-pod-template-hash={{args.canary-hash}}\n step: 60\n start: \"{{args.start-time}}\"\n end: \"{{args.end-time}}\"\n
Experiment # This is the resulting experiment that is produced by the step\napiVersion: argoproj.io/v1alpha1\nkind: Experiment\nname:\n name: guestbook-6c54544bf9-0\nspec:\n duration: 1h\n templates:\n - name: baseline\n replicas: 1\n spec:\n containers:\n - name: guestbook\n image: guestbook:v1\n - name: canary\n replicas: 1\n spec:\n containers:\n - name: guestbook\n image: guestbook:v2\n analysis:\n templateName: mann-whitney\n args:\n - name: start-time\n value: \"{{experiment.availableAt}}\"\n - name: end-time\n value: \"{{experiment.finishedAt}}\"\n
In order to perform multiple kayenta runs over some time duration, the interval and count fields can be supplied. When the start and end fields are omitted from the kayenta scopes, the values will be implicitly decided as:
start = if lookback: true start of analysis, otherwise current time - intervalend = current time
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: mann-whitney\nspec:\n args:\n - name: stable-hash\n - name: canary-hash\n metrics:\n - name: mann-whitney\n provider:\n kayenta:\n address: https://kayenta.intuit.com\n application: guestbook\n canaryConfigName: my-test\n interval: 3600\n count: 3\n # loopback will cause start time value to be equal to start of analysis\n # lookback: true\n thresholds:\n pass: 90\n marginal: 75\n scopes:\n - name: default\n controlScope:\n scope: app=guestbook and rollouts-pod-template-hash={{args.stable-hash}}\n step: 60\n experimentScope:\n scope: app=guestbook and rollouts-pod-template-hash={{args.canary-hash}}\n step: 60\n
"},{"location":"analysis/newrelic/","title":"NewRelic Metrics","text":"Important
Available since v0.10.0
A New Relic query using NRQL can be used to obtain measurements for analysis.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: application-name\n metrics:\n - name: success-rate\n successCondition: result.successRate >= 0.95\n provider:\n newRelic:\n profile: my-newrelic-secret # optional, defaults to 'newrelic'\n query: |\n FROM Transaction SELECT percentage(count(*), WHERE httpResponseCode != 500) as successRate where appName = '{{ args.application-name }}'\n
The result evaluated for the condition will always be map or list of maps. The name will follow the pattern of either function or function.field, e.g. SELECT average(duration) from Transaction will yield average.duration. In this case the field result cannot be accessed with dot notation and instead should be accessed like result['average.duration']. Query results can be renamed using the NRQL clause AS as seen above.
A New Relic access profile can be configured using a Kubernetes secret in the argo-rollouts namespace. Alternate accounts can be used by creating more secrets of the same format and specifying which secret to use in the metric provider configuration using the profile field.
apiVersion: v1\nkind: Secret\nmetadata:\n name: newrelic\ntype: Opaque\nstringData:\n personal-api-key: <newrelic-personal-api-key>\n account-id: <newrelic-account-id>\n region: \"us\" # optional, defaults to \"us\" if not set. Only set to \"eu\" if you use EU New Relic\n
To use the New Relic metric provider from behind a proxy, provide a base-url-rest key pointing to the base URL of the New Relic REST API for your proxy, and a base-url-nerdgraph key pointing to the base URL for NerdGraph for your proxy:
apiVersion: v1\nkind: Secret\nmetadata:\n name: newrelic\ntype: Opaque\nstringData:\n personal-api-key: <newrelic-personal-api-key>\n account-id: <newrelic-account-id>\n region: \"us\" # optional, defaults to \"us\" if not set. Only set to \"eu\" if you use EU New Relic\n base-url-rest: <your-base-url>\n base-url-nerdgraph: <your-base-url>\n
"},{"location":"analysis/plugins/","title":"Metric Plugins","text":"Important
Available since v1.5 - Status: Alpha
Argo Rollouts supports getting analysis metrics via 3rd party plugin system. This allows users to extend the capabilities of Rollouts to support metric providers that are not natively supported. Rollout's uses a plugin library called go-plugin to do this. You can find a sample plugin here: rollouts-plugin-metric-sample-prometheus
"},{"location":"analysis/plugins/#using-a-metric-plugin","title":"Using a Metric Plugin","text":"There are two methods of installing and using an argo rollouts plugin. The first method is to mount up the plugin executable into the rollouts controller container. The second method is to use a HTTP(S) server to host the plugin executable.
"},{"location":"analysis/plugins/#mounting-the-plugin-executable-into-the-rollouts-controller-container","title":"Mounting the plugin executable into the rollouts controller container","text":"There are a few ways to mount the plugin executable into the rollouts controller container. Some of these will depend on your particular infrastructure. Here are a few methods:
- Using an init container to download the plugin executable
- Using a Kubernetes volume mount with a shared volume such as NFS, EBS, etc.
- Building the plugin into the rollouts controller container
Then you can use the configmap to point to the plugin executable file location. Example:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-config\ndata:\n metricProviderPlugins: |-\n - name: \"argoproj-labs/sample-prometheus\" # name of the plugin, it must match the name required by the plugin so it can find it's configuration\n location: \"file://./my-custom-plugin\" # supports http(s):// urls and file://\n
"},{"location":"analysis/plugins/#using-a-https-server-to-host-the-plugin-executable","title":"Using a HTTP(S) server to host the plugin executable","text":"Argo Rollouts supports downloading the plugin executable from a HTTP(S) server. To use this method, you will need to configure the controller via the argo-rollouts-config configmap and set pluginLocation to a http(s) url. Example:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-config\ndata:\n metricProviderPlugins: |-\n - name: \"argoproj-labs/sample-prometheus\" # name of the plugin, it must match the name required by the plugin so it can find it's configuration\n location: \"https://github.com/argoproj-labs/rollouts-plugin-metric-sample-prometheus/releases/download/v0.0.4/metric-plugin-linux-amd64\" # supports http(s):// urls and file://\n sha256: \"dac10cbf57633c9832a17f8c27d2ca34aa97dd3d\" #optional sha256 checksum of the plugin executable\n
"},{"location":"analysis/plugins/#some-words-of-caution","title":"Some words of caution","text":"Depending on which method you use to install and the plugin, there are some things to be aware of. The rollouts controller will not start if it can not download or find the plugin executable. This means that if you are using a method of installation that requires a download of the plugin and the server hosting the plugin for some reason is not available and the rollouts controllers pod got deleted while the server was down or is coming up for the first time, it will not be able to start until the server hosting the plugin is available again.
Argo Rollouts will download the plugin at startup only once but if the pod is deleted it will need to download the plugin again on next startup. Running Argo Rollouts in HA mode can help a little with this situation because each pod will download the plugin at startup. So if a single pod gets deleted during a server outage, the other pods will still be able to take over because there will already be a plugin executable available to it. It is the responsibility of the Argo Rollouts administrator to define the plugin installation method considering the risks of each approach.
"},{"location":"analysis/plugins/#list-of-available-plugins-alphabetical-order","title":"List of Available Plugins (alphabetical order)","text":""},{"location":"analysis/plugins/#add-your-plugin-here","title":"Add Your Plugin Here","text":" - If you have created a plugin, please submit a PR to add it to this list.
"},{"location":"analysis/plugins/#rollouts-plugin-metric-sample-prometheus","title":"rollouts-plugin-metric-sample-prometheus","text":" - This is just a sample plugin that can be used as a starting point for creating your own plugin. It is not meant to be used in production. It is based on the built-in prometheus provider.
"},{"location":"analysis/prometheus/","title":"Prometheus Metrics","text":"A Prometheus query can be used to obtain measurements for analysis.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: success-rate\n interval: 5m\n # NOTE: prometheus queries return results in the form of a vector.\n # So it is common to access the index 0 of the returned array to obtain the value\n successCondition: result[0] >= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n # timeout is expressed in seconds\n timeout: 40\n headers:\n - key: X-Scope-OrgID\n value: tenant_a\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
The example shows Istio metrics, but you can use any kind of metric available to your prometheus instance. We suggest you validate your PromQL expression using the Prometheus GUI first.
See the Analysis Overview page for more details on the available options.
"},{"location":"analysis/prometheus/#authorization","title":"Authorization","text":""},{"location":"analysis/prometheus/#utilizing-amazon-managed-prometheus","title":"Utilizing Amazon Managed Prometheus","text":"Amazon Managed Prometheus can be used as the prometheus data source for analysis. In order to do this the namespace where your analysis is running will have to have the appropriate IRSA attached to allow for prometheus queries. Once you ensure the proper permissions are in place to access AMP, you can use an AMP workspace url in your provider block and add a SigV4 config for Sigv4 signing:
provider:\n prometheus:\n address: https://aps-workspaces.$REGION.amazonaws.com/workspaces/$WORKSPACEID\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n authentication:\n sigv4:\n region: $REGION\n profile: $PROFILE\n roleArn: $ROLEARN\n
"},{"location":"analysis/prometheus/#with-oauth2","title":"With OAuth2","text":"You can setup an OAuth2 client credential flow using the following values:
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n # from secret\n - name: oauthSecret # This is the OAuth2 shared secret\n valueFrom:\n secretKeyRef:\n name: oauth-secret\n key: secret\n metrics:\n - name: success-rate\n interval: 5m\n # NOTE: prometheus queries return results in the form of a vector.\n # So it is common to access the index 0 of the returned array to obtain the value\n successCondition: result[0] >= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n # timeout is expressed in seconds\n timeout: 40\n authentication:\n oauth2:\n tokenUrl: https://my-oauth2-provider/token\n clientId: my-cliend-id\n clientSecret: \"{{ args.oauthSecret }}\"\n scopes: [\n \"my-oauth2-scope\"\n ]\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
The AnalysisRun will first get an access token using that information, and provide it as an Authorization: Bearer header for the metric provider call.
"},{"location":"analysis/prometheus/#additional-metadata","title":"Additional Metadata","text":"Any additional metadata from the Prometheus controller, like the resolved queries after substituting the template's arguments, etc. will appear under the Metadata map in the MetricsResult object of AnalysisRun.
"},{"location":"analysis/prometheus/#skip-tls-verification","title":"Skip TLS verification","text":"You can skip the TLS verification of the prometheus host provided by setting the options insecure: true.
provider:\n prometheus:\n address: https://prometheus.example.com\n insecure: true\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
"},{"location":"analysis/skywalking/","title":"Apache SkyWalking Metrics","text":"Important
Available since v1.5.0
A SkyWalking query using GraphQL can be used to obtain measurements for analysis.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: apdex\nspec:\n args:\n - name: service-name\n metrics:\n - name: apdex\n interval: 5m\n successCondition: \"all(result.service_apdex.values.values, {asFloat(.value) >= 9900})\"\n failureLimit: 3\n provider:\n skywalking:\n interval: 3m\n address: http://skywalking-oap.istio-system:12800\n query: |\n query queryData($duration: Duration!) {\n service_apdex: readMetricsValues(\n condition: { name: \"service_apdex\", entity: { scope: Service, serviceName: \"{{ args.service-name }}\", normal: true } },\n duration: $duration) {\n label values { values { value } }\n }\n }\n
The result evaluated for the query depends on the specific GraphQL you give, you can try to run the GraphQL query first and inspect the output format, then compose the condition.
"},{"location":"analysis/wavefront/","title":"Wavefront Metrics","text":"A Wavefront query can be used to obtain measurements for analysis.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: success-rate\n interval: 5m\n successCondition: result >= 0.95\n failureLimit: 3\n provider:\n wavefront:\n address: example.wavefront.com\n query: |\n sum(rate(\n 5m, ts(\"istio.requestcount.count\", response_code!=500 and destination_service=\"{{args.service-name}}\"\n ))) /\n sum(rate(\n 5m, ts(\"istio.requestcount.count\", reporter=client and destination_service=\"{{args.service-name}}\"\n )))\n
Wavefront api tokens can be configured in a kubernetes secret in argo-rollouts namespace.
apiVersion: v1\nkind: Secret\nmetadata:\n name: wavefront-api-tokens\ntype: Opaque\nstringData:\n example1.wavefront.com: <token1>\n example2.wavefront.com: <token2>\n
"},{"location":"analysis/web/","title":"Web Metrics","text":"An HTTP request can be performed against some external service to obtain the measurement. This example makes a HTTP GET request to some URL. The webhook response must return JSON content. The result of the optional jsonPath expression will be assigned to the result variable that can be referenced in the successCondition and failureCondition expressions. If omitted, will use the entire body of the as the result variable.
metrics:\n - name: webmetric\n successCondition: result == true\n provider:\n web:\n url: \"http://my-server.com/api/v1/measurement?service={{ args.service-name }}\"\n timeoutSeconds: 20 # defaults to 10 seconds\n headers:\n - key: Authorization\n value: \"Bearer {{ args.api-token }}\"\n jsonPath: \"{$.data.ok}\"\n
In the following example, given the payload, the measurement will be Successful if the data.ok field was true, and the data.successPercent was greater than 0.90
{\n \"data\": {\n \"ok\": true,\n \"successPercent\": 0.95\n }\n}\n
metrics:\n - name: webmetric\n successCondition: \"result.ok && result.successPercent >= 0.90\"\n provider:\n web:\n url: \"http://my-server.com/api/v1/measurement?service={{ args.service-name }}\"\n headers:\n - key: Authorization\n value: \"Bearer {{ args.api-token }}\"\n jsonPath: \"{$.data}\"\n
NOTE: if the result is a string, two convenience functions asInt and asFloat are provided to convert a result value to a numeric type so that mathematical comparison operators can be used (e.g. >, <, >=, <=).
"},{"location":"analysis/web/#optional-web-methods","title":"Optional web methods","text":"It is possible to use a POST or PUT requests, by specifying the method and either body or jsonBody fields
metrics:\n - name: webmetric\n successCondition: result == true\n provider:\n web:\n method: POST # valid values are GET|POST|PUT, defaults to GET\n url: \"http://my-server.com/api/v1/measurement?service={{ args.service-name }}\"\n timeoutSeconds: 20 # defaults to 10 seconds\n headers:\n - key: Authorization\n value: \"Bearer {{ args.api-token }}\"\n - key: Content-Type # if body is a json, it is recommended to set the Content-Type\n value: \"application/json\"\n body: \"string value\"\n jsonPath: \"{$.data.ok}\"\n
!!! tip In order to send in JSON, you can use jsonBody and Content-Type will be automatically set as json. Setting a body or jsonBody field for a GET request will result in an error. Set either body or jsonBody and setting both will result in an error. metrics:\n - name: webmetric\n successCondition: result == true\n provider:\n web:\n method: POST # valid values are GET|POST|PUT, defaults to GET\n url: \"http://my-server.com/api/v1/measurement?service={{ args.service-name }}\"\n timeoutSeconds: 20 # defaults to 10 seconds\n headers:\n - key: Authorization\n value: \"Bearer {{ args.api-token }}\"\n - key: Content-Type # if body is a json, it is recommended to set the Content-Type\n value: \"application/json\"\n jsonBody: # If using jsonBody Content-Type header will be automatically set to json\n key1: value_1\n key2:\n nestedObj: nested value\n key3: \"{{ args.service-name }}\"\n jsonPath: \"{$.data.ok}\"\n
"},{"location":"analysis/web/#skip-tls-verification","title":"Skip TLS verification","text":"You can skip the TLS verification of the web host provided by setting the options insecure: true.
metrics:\n - name: webmetric\n successCondition: \"result.ok && result.successPercent >= 0.90\"\n provider:\n web:\n url: \"https://my-server.com/api/v1/measurement?service={{ args.service-name }}\"\n insecure: true\n headers:\n - key: Authorization\n value: \"Bearer {{ args.api-token }}\"\n jsonPath: \"{$.data}\"\n
"},{"location":"analysis/web/#authorization","title":"Authorization","text":""},{"location":"analysis/web/#with-oauth2","title":"With OAuth2","text":"You can setup an OAuth2 client credential flow using the following values:
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n # from secret\n - name: oauthSecret # This is the OAuth2 shared secret\n valueFrom:\n secretKeyRef:\n name: oauth-secret\n key: secret\n metrics:\n - name: webmetric\n successCondition: result == true\n provider:\n web:\n url: \"http://my-server.com/api/v1/measurement?service={{ args.service-name }}\"\n timeoutSeconds: 20 # defaults to 10 seconds\n authentication:\n oauth2:\n tokenUrl: https://my-oauth2-provider/token\n clientId: my-cliend-id\n clientSecret: \"{{ args.oauthSecret }}\"\n scopes: [\n \"my-oauth2-scope\"\n ]\n headers:\n - key: Content-Type # if body is a json, it is recommended to set the Content-Type\n value: \"application/json\"\n jsonPath: \"{$.data.ok}\"\n
In that case, no need to provide specifically the Authentication header. The AnalysisRun will first get an access token using that information, and provide it as an Authorization: Bearer header for the metric provider call.
"},{"location":"features/analysis/","title":"Analysis & Progressive Delivery","text":"Argo Rollouts provides several ways to perform analysis to drive progressive delivery. This document describes how to achieve various forms of progressive delivery, varying the point in time analysis is performed, its frequency, and occurrence.
"},{"location":"features/analysis/#custom-resource-definitions","title":"Custom Resource Definitions","text":"CRD Description Rollout A Rollout acts as a drop-in replacement for a Deployment resource. It provides additional blueGreen and canary update strategies. These strategies can create AnalysisRuns and Experiments during the update, which will progress the update, or abort it. AnalysisTemplate An AnalysisTemplate is a template spec which defines how to perform a canary analysis, such as the metrics which it should perform, its frequency, and the values which are considered successful or failed. AnalysisTemplates may be parameterized with inputs values. ClusterAnalysisTemplate A ClusterAnalysisTemplate is like an AnalysisTemplate, but it is not limited to its namespace. It can be used by any Rollout throughout the cluster. AnalysisRun An AnalysisRun is an instantiation of an AnalysisTemplate. AnalysisRuns are like Jobs in that they eventually complete. Completed runs are considered Successful, Failed, or Inconclusive, and the result of the run affect if the Rollout's update will continue, abort, or pause, respectively. Experiment An Experiment is limited run of one or more ReplicaSets for the purposes of analysis. Experiments typically run for a pre-determined duration, but can also run indefinitely until stopped. Experiments may reference an AnalysisTemplate to run during or after the experiment. The canonical use case for an Experiment is to start a baseline and canary deployment in parallel, and compare the metrics produced by the baseline and canary pods for an equal comparison."},{"location":"features/analysis/#background-analysis","title":"Background Analysis","text":"Analysis can be run in the background -- while the canary is progressing through its rollout steps.
The following example gradually increments the canary weight by 20% every 10 minutes until it reaches 100%. In the background, an AnalysisRun is started based on the AnalysisTemplate named success-rate. The success-rate template queries a prometheus server, measuring the HTTP success rates at 5 minute intervals/samples. It has no end time, and continues until stopped or failed. If the metric is measured to be less than 95%, and there are three such measurements, the analysis is considered Failed. The failed analysis causes the Rollout to abort, setting the canary weight back to zero, and the Rollout would be considered in a Degraded. Otherwise, if the rollout completes all of its canary steps, the rollout is considered successful and the analysis run is stopped by the controller.
This example highlights:
- Background analysis style of progressive delivery
- Using a Prometheus query to perform a measurement
- The ability to parameterize the analysis
- Delay starting the analysis run until step 3 (Set Weight 40%)
Rollout apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\nspec:\n...\n strategy:\n canary:\n analysis:\n templates:\n - templateName: success-rate\n startingStep: 2 # delay starting analysis run until setWeight: 40%\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n steps:\n - setWeight: 20\n - pause: {duration: 10m}\n - setWeight: 40\n - pause: {duration: 10m}\n - setWeight: 60\n - pause: {duration: 10m}\n - setWeight: 80\n - pause: {duration: 10m}\n
AnalysisTemplate apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: success-rate\n interval: 5m\n # NOTE: prometheus queries return results in the form of a vector.\n # So it is common to access the index 0 of the returned array to obtain the value\n successCondition: result[0] >= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
"},{"location":"features/analysis/#inline-analysis","title":"Inline Analysis","text":"Analysis can also be performed as a rollout step as an inline \"analysis\" step. When analysis is performed \"inlined,\" an AnalysisRun is started when the step is reached, and blocks the rollout until the run is completed. The success or failure of the analysis run decides if the rollout will proceed to the next step, or abort the rollout completely.
This example sets the canary weight to 20%, pauses for 5 minutes, then runs an analysis. If the analysis was successful, continues with rollout, otherwise aborts.
This example demonstrates:
- The ability to invoke an analysis in-line as part of steps
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\nspec:\n...\n strategy:\n canary:\n steps:\n - setWeight: 20\n - pause: {duration: 5m}\n - analysis:\n templates:\n - templateName: success-rate\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n
In this example, the AnalysisTemplate is identical to the background analysis example, but since no interval is specified, the analysis will perform a single measurement and complete.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n - name: prometheus-port\n value: 9090\n metrics:\n - name: success-rate\n successCondition: result[0] >= 0.95\n provider:\n prometheus:\n address: \"http://prometheus.example.com:{{args.prometheus-port}}\"\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
Multiple measurements can be performed over a longer duration period, by specifying the count and interval fields:
metrics:\n - name: success-rate\n successCondition: result[0] >= 0.95\n interval: 60s\n count: 5\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: ...\n
"},{"location":"features/analysis/#clusteranalysistemplates","title":"ClusterAnalysisTemplates","text":"Important
Available since v0.9.0
A Rollout can reference a Cluster scoped AnalysisTemplate called a ClusterAnalysisTemplate. This can be useful when you want to share an AnalysisTemplate across multiple Rollouts; in different namespaces, and avoid duplicating the same template in every namespace. Use the field clusterScope: true to reference a ClusterAnalysisTemplate instead of an AnalysisTemplate.
Rollout apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\nspec:\n...\n strategy:\n canary:\n steps:\n - setWeight: 20\n - pause: {duration: 5m}\n - analysis:\n templates:\n - templateName: success-rate\n clusterScope: true\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n
ClusterAnalysisTemplate apiVersion: argoproj.io/v1alpha1\nkind: ClusterAnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n - name: prometheus-port\n value: 9090\n metrics:\n - name: success-rate\n successCondition: result[0] >= 0.95\n provider:\n prometheus:\n address: \"http://prometheus.example.com:{{args.prometheus-port}}\"\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
Note
The resulting AnalysisRun will still run in the namespace of the Rollout
"},{"location":"features/analysis/#analysis-with-multiple-templates","title":"Analysis with Multiple Templates","text":"A Rollout can reference multiple AnalysisTemplates when constructing an AnalysisRun. This allows users to compose analysis from multiple AnalysisTemplates. If multiple templates are referenced, then the controller will merge the templates together. The controller combines the metrics and args fields of all the templates.
Rollout apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\nspec:\n...\n strategy:\n canary:\n analysis:\n templates:\n - templateName: success-rate\n - templateName: error-rate\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n
AnalysisTemplate apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: success-rate\n interval: 5m\n successCondition: result[0] >= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n---\napiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: error-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: error-rate\n interval: 5m\n successCondition: result[0] <= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code=~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
AnalysisRun # NOTE: Generated AnalysisRun from the multiple templates\napiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\nmetadata:\n name: guestbook-CurrentPodHash-multiple-templates\nspec:\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n metrics:\n - name: success-rate\n interval: 5m\n successCondition: result[0] >= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n - name: error-rate\n interval: 5m\n successCondition: result[0] <= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code=~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
Note
The controller will error when merging the templates if:
- Multiple metrics in the templates have the same name
- Two arguments with the same name have different default values no matter the argument value in Rollout
"},{"location":"features/analysis/#analysis-template-referencing-other-analysis-templates","title":"Analysis Template referencing other Analysis Templates","text":"AnalysisTemplates and ClusterAnalysisTemplates may reference other templates.
They can be combined with other metrics:
AnalysisTemplate apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: error-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: error-rate\n interval: 5m\n successCondition: result[0] <= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code=~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n---\napiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: rates\nspec:\n args:\n - name: service-name\n metrics:\n - name: success-rate\n interval: 5m\n successCondition: result[0] >= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n templates:\n - templateName: error-rate\n clusterScope: false\n
Or without additional metrics:
AnalysisTemplate apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: success-rate\n interval: 5m\n successCondition: result[0] >= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n---\napiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: error-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: error-rate\n interval: 5m\n successCondition: result[0] <= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code=~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n---\napiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: rates\nspec:\n args:\n - name: service-name\n templates:\n - templateName: success-rate\n clusterScope: false\n - templateName: error-rate\n clusterScope: false\n
The result in the AnalysisRun will have the aggregation of metrics of each template:
AnalysisRun # NOTE: Generated AnalysisRun from a single template referencing several templates\napiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\nmetadata:\n name: guestbook-CurrentPodHash-templates-in-template\nspec:\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n metrics:\n - name: success-rate\n interval: 5m\n successCondition: result[0] >= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n - name: error-rate\n interval: 5m\n successCondition: result[0] <= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code=~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
Note
The same limitations as for the multiple templates feature apply. The controller will error when merging the templates if:
- Multiple metrics in the templates have the same name
- Two arguments with the same name have different default values no matter the argument value in Rollout
However, if the same AnalysisTemplate is referenced several times along the chain of references, the controller will only keep it once and discard the other references.
"},{"location":"features/analysis/#analysis-template-arguments","title":"Analysis Template Arguments","text":"AnalysisTemplates may declare a set of arguments that can be passed by Rollouts. The args can then be used as in metrics configuration and are resolved at the time the AnalysisRun is created. Argument placeholders are defined as {{ args.<name> }}.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: args-example\nspec:\n args:\n # required in Rollout due to no default value\n - name: service-name\n - name: stable-hash\n - name: latest-hash\n # optional in Rollout given the default value\n - name: api-url\n value: http://example/measure\n # from secret\n - name: api-token\n valueFrom:\n secretKeyRef:\n name: token-secret\n key: apiToken\n metrics:\n - name: webmetric\n successCondition: result == 'true'\n provider:\n web:\n # placeholders are resolved when an AnalysisRun is created\n url: \"{{ args.api-url }}?service={{ args.service-name }}\"\n headers:\n - key: Authorization\n value: \"Bearer {{ args.api-token }}\"\n jsonPath: \"{$.results.ok}\"\n
Analysis arguments defined in a Rollout are merged with the args from the AnalysisTemplate when the AnalysisRun is created.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\nspec:\n...\n strategy:\n canary:\n analysis:\n templates:\n - templateName: args-example\n args:\n # required value\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n # override default value\n - name: api-url\n value: http://other-api\n # pod template hash from the stable ReplicaSet\n - name: stable-hash\n valueFrom:\n podTemplateHashValue: Stable\n # pod template hash from the latest ReplicaSet\n - name: latest-hash\n valueFrom:\n podTemplateHashValue: Latest\n
Analysis arguments also support valueFrom for reading metadata fields and passing them as arguments to AnalysisTemplate. An example would be to reference metadata labels like env and region and passing them along to AnalysisTemplate. apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\n labels:\n appType: demo-app\n buildType: nginx-app\n ...\n env: dev\n region: us-west-2\nspec:\n...\n strategy:\n canary:\n analysis:\n templates:\n - templateName: args-example\n args:\n ...\n - name: env\n valueFrom:\n fieldRef:\n fieldPath: metadata.labels['env']\n # region where this app is deployed\n - name: region\n valueFrom:\n fieldRef:\n fieldPath: metadata.labels['region']\n
Important
Available since v1.2
Analysis arguments also support valueFrom for reading any field from Rollout status and passing them as arguments to AnalysisTemplate. Following example references Rollout status field like aws canaryTargetGroup name and passing them along to AnalysisTemplate
from the Rollout status
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\n labels:\n appType: demo-app\n buildType: nginx-app\n ...\n env: dev\n region: us-west-2\nspec:\n...\n strategy:\n canary:\n analysis:\n templates:\n - templateName: args-example\n args:\n ...\n - name: canary-targetgroup-name\n valueFrom:\n fieldRef:\n fieldPath: status.alb.canaryTargetGroup.name\n
"},{"location":"features/analysis/#bluegreen-pre-promotion-analysis","title":"BlueGreen Pre Promotion Analysis","text":"A Rollout using the BlueGreen strategy can launch an AnalysisRun before it switches traffic to the new version using pre-promotion. This can be used to block the Service selector switch until the AnalysisRun finishes successfully. The success or failure of the AnalysisRun decides if the Rollout switches traffic, or abort the Rollout completely.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\nspec:\n...\n strategy:\n blueGreen:\n activeService: active-svc\n previewService: preview-svc\n prePromotionAnalysis:\n templates:\n - templateName: smoke-tests\n args:\n - name: service-name\n value: preview-svc.default.svc.cluster.local\n
In this example, the Rollout creates a pre-promotion AnalysisRun once the new ReplicaSet is fully available. The Rollout will not switch traffic to the new version until the analysis run finishes successfully.
Note: if theautoPromotionSeconds field is specified and the Rollout has waited auto promotion seconds amount of time, the Rollout marks the AnalysisRun successful and switches the traffic to a new version automatically. If the AnalysisRun completes before then, the Rollout will not create another AnalysisRun and wait out the rest of the autoPromotionSeconds.
"},{"location":"features/analysis/#bluegreen-post-promotion-analysis","title":"BlueGreen Post Promotion Analysis","text":"A Rollout using a BlueGreen strategy can launch an analysis run after the traffic switch to the new version using post-promotion analysis. If post-promotion Analysis fails or errors, the Rollout enters an aborted state and switches traffic back to the previous stable Replicaset. When post-analysis is Successful, the Rollout is considered fully promoted and the new ReplicaSet will be marked as stable. The old ReplicaSet will then be scaled down according to scaleDownDelaySeconds (default 30 seconds).
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\nspec:\n...\n strategy:\n blueGreen:\n activeService: active-svc\n previewService: preview-svc\n scaleDownDelaySeconds: 600 # 10 minutes\n postPromotionAnalysis:\n templates:\n - templateName: smoke-tests\n args:\n - name: service-name\n value: preview-svc.default.svc.cluster.local\n
"},{"location":"features/analysis/#failure-conditions-and-failure-limit","title":"Failure Conditions and Failure Limit","text":"failureCondition can be used to cause an analysis run to fail. failureLimit is the maximum number of failed run an analysis is allowed. The following example continually polls the defined Prometheus server to get the total number of errors(i.e., HTTP response code >= 500) every 5 minutes, causing the measurement to fail if ten or more errors are encountered. The entire analysis run is considered as Failed after three failed measurements.
metrics:\n - name: total-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code=~\"5.*\"}[5m]\n ))\n
"},{"location":"features/analysis/#dry-run-mode","title":"Dry-Run Mode","text":"Important
Available since v1.2
dryRun can be used on a metric to control whether or not to evaluate that metric in a dry-run mode. A metric running in the dry-run mode won't impact the final state of the rollout or experiment even if it fails or the evaluation comes out as inconclusive.
The following example queries prometheus every 5 minutes to get the total number of 4XX and 5XX errors, and even if the evaluation of the metric to monitor the 5XX error-rate fail, the analysis run will pass.
dryRun:\n - metricName: total-5xx-errors\n metrics:\n - name: total-5xx-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code~\"5.*\"}[5m]\n ))\n - name: total-4xx-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code~\"4.*\"}[5m]\n ))\n
RegEx matches are also supported. .* can be used to make all the metrics run in the dry-run mode. In the following example, even if one or both metrics fail, the analysis run will pass.
dryRun:\n - metricName: .*\n metrics:\n - name: total-5xx-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code~\"5.*\"}[5m]\n ))\n - name: total-4xx-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code~\"4.*\"}[5m]\n ))\n
"},{"location":"features/analysis/#dry-run-summary","title":"Dry-Run Summary","text":"If one or more metrics are running in the dry-run mode, the summary of the dry-run results gets appended to the analysis run message. Assuming that the total-4xx-errors metric fails in the above example but, the total-5xx-errors succeeds, the final dry-run summary will look like this.
Message: Run Terminated\nRun Summary:\n ...\nDry Run Summary: \n Count: 2\n Successful: 1\n Failed: 1\nMetric Results:\n...\n
"},{"location":"features/analysis/#dry-run-rollouts","title":"Dry-Run Rollouts","text":"If a rollout wants to dry run its analysis, it simply needs to specify the dryRun field to its analysis stanza. In the following example, all the metrics from random-fail and always-pass get merged and executed in the dry-run mode.
kind: Rollout\nspec:\n...\n steps:\n - analysis:\n templates:\n - templateName: random-fail\n - templateName: always-pass\n dryRun:\n - metricName: .*\n
"},{"location":"features/analysis/#dry-run-experiments","title":"Dry-Run Experiments","text":"If an experiment wants to dry run its analysis, it simply needs to specify the dryRun field under its specs. In the following example, all the metrics from analyze-job matching the RegEx rule test.* will be executed in the dry-run mode.
kind: Experiment\nspec:\n templates:\n - name: baseline\n selector:\n matchLabels:\n app: rollouts-demo\n template:\n metadata:\n labels:\n app: rollouts-demo\n spec:\n containers:\n - name: rollouts-demo\n image: argoproj/rollouts-demo:blue\n analyses:\n - name: analyze-job\n templateName: analyze-job\n dryRun:\n - metricName: test.*\n
"},{"location":"features/analysis/#measurements-retention","title":"Measurements Retention","text":"Important
Available since v1.2
measurementRetention can be used to retain other than the latest ten results for the metrics running in any mode (dry/non-dry). Setting this option to 0 would disable it and, the controller will revert to the existing behavior of retaining the latest ten measurements.
The following example queries Prometheus every 5 minutes to get the total number of 4XX and 5XX errors and retains the latest twenty measurements for the 5XX metric run results instead of the default ten.
measurementRetention:\n - metricName: total-5xx-errors\n limit: 20\n metrics:\n - name: total-5xx-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code~\"5.*\"}[5m]\n ))\n - name: total-4xx-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code~\"4.*\"}[5m]\n ))\n
RegEx matches are also supported. .* can be used to apply the same retention rule to all the metrics. In the following example, the controller will retain the latest twenty run results for all the metrics instead of the default ten results.
measurementRetention:\n - metricName: .*\n limit: 20\n metrics:\n - name: total-5xx-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code~\"5.*\"}[5m]\n ))\n - name: total-4xx-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code~\"4.*\"}[5m]\n ))\n
"},{"location":"features/analysis/#measurements-retention-for-rollouts-analysis","title":"Measurements Retention for Rollouts Analysis","text":"If a rollout wants to retain more results of its analysis metrics, it simply needs to specify the measurementRetention field to its analysis stanza. In the following example, all the metrics from random-fail and always-pass get merged, and their latest twenty measurements get retained instead of the default ten.
kind: Rollout\nspec:\n...\n steps:\n - analysis:\n templates:\n - templateName: random-fail\n - templateName: always-pass\n measurementRetention:\n - metricName: .*\n limit: 20\n
"},{"location":"features/analysis/#define-custom-labelsannotations-for-analysisrun","title":"Define custom Labels/Annotations for AnalysisRun","text":"If you would like to annotate/label the AnalysisRun with the custom labels your can do it by specifying analysisRunMetadata field.
kind: Rollout\nspec:\n...\n steps:\n - analysis:\n templates:\n - templateName: my-template\n analysisRunMetadata:\n labels:\n my-custom-label: label-value\n annotations:\n my-custom-annotation: annotation-value\n
"},{"location":"features/analysis/#measurements-retention-for-experiments","title":"Measurements Retention for Experiments","text":"If an experiment wants to retain more results of its analysis metrics, it simply needs to specify the measurementRetention field under its specs. In the following example, all the metrics from analyze-job matching the RegEx rule test.* will have their latest twenty measurements get retained instead of the default ten.
kind: Experiment\nspec:\n templates:\n - name: baseline\n selector:\n matchLabels:\n app: rollouts-demo\n template:\n metadata:\n labels:\n app: rollouts-demo\n spec:\n containers:\n - name: rollouts-demo\n image: argoproj/rollouts-demo:blue\n analyses:\n - name: analyze-job\n templateName: analyze-job\n measurementRetention:\n - metricName: test.*\n limit: 20\n
"},{"location":"features/analysis/#time-to-live-ttl-strategy","title":"Time-to-live (TTL) Strategy","text":"Important
Available since v1.7
ttlStrategy limits the lifetime of an analysis run that has finished execution depending on if it Succeeded or Failed. If this struct is set, once the run finishes, it will be deleted after the time to live expires. If this field is unset, the analysis controller will keep the completed runs, unless they are associated with rollouts using other garbage collection policies (e.g. successfulRunHistoryLimit and unsuccessfulRunHistoryLimit).
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\nspec:\n ...\n ttlStrategy:\n secondsAfterCompletion: 3600\n secondsAfterSuccess: 1800\n secondsAfterFailure: 1800\n
"},{"location":"features/analysis/#inconclusive-runs","title":"Inconclusive Runs","text":"Analysis runs can also be considered Inconclusive, which indicates the run was neither successful, nor failed. Inconclusive runs causes a rollout to become paused at its current step. Manual intervention is then needed to either resume the rollout, or abort. One example of how analysis runs could become Inconclusive, is when a metric defines no success or failure conditions.
metrics:\n - name: my-query\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: ...\n
Inconclusive analysis runs might also happen when both success and failure conditions are specified, but the measurement value did not meet either condition.
metrics:\n - name: success-rate\n successCondition: result[0] >= 0.90\n failureCondition: result[0] < 0.50\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: ...\n
A use case for having Inconclusive analysis runs are to enable Argo Rollouts to automate the execution of analysis runs, and collect the measurement, but still allow human judgement to decide whether or not measurement value is acceptable and decide to proceed or abort.
"},{"location":"features/analysis/#delay-analysis-runs","title":"Delay Analysis Runs","text":"If the analysis run does not need to start immediately (i.e give the metric provider time to collect metrics on the canary version), Analysis Runs can delay the specific metric analysis. Each metric can be configured to have a different delay. In additional to the metric specific delays, the rollouts with background analysis can delay creating an analysis run until a certain step is reached
Delaying a specific analysis metric:
metrics:\n - name: success-rate\n # Do not start this analysis until 5 minutes after the analysis run starts\n initialDelay: 5m\n successCondition: result[0] >= 0.90\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: ...\n
Delaying starting background analysis run until step 3 (Set Weight 40%):
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\nspec:\n strategy:\n canary:\n analysis:\n templates:\n - templateName: success-rate\n startingStep: 2\n steps:\n - setWeight: 20\n - pause: {duration: 10m}\n - setWeight: 40\n - pause: {duration: 10m}\n
"},{"location":"features/analysis/#referencing-secrets","title":"Referencing Secrets","text":"AnalysisTemplates and AnalysisRuns can reference secret objects in .spec.args. This allows users to securely pass authentication information to Metric Providers, like login credentials or API tokens.
An AnalysisRun can only reference secrets from the same namespace as it's running in. This is only relevant for AnalysisRuns, since AnalysisTemplates do not resolve the secret.
In the following example, an AnalysisTemplate references an API token and passes it to a Web metric provider.
This example demonstrates:
- The ability to reference a secret in the AnalysisTemplate
.spec.args - The ability to pass secret arguments to Metric Providers
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nspec:\n args:\n - name: api-token\n valueFrom:\n secretKeyRef:\n name: token-secret\n key: apiToken\n metrics:\n - name: webmetric\n provider:\n web:\n headers:\n - key: Authorization\n value: \"Bearer {{ args.api-token }}\"\n
"},{"location":"features/analysis/#handling-metric-results","title":"Handling Metric Results","text":""},{"location":"features/analysis/#nan-and-infinity","title":"NaN and Infinity","text":"Metric providers can sometimes return values of NaN (not a number) and infinity. Users can edit the successCondition and failureCondition fields to handle these cases accordingly.
Here are three examples where a metric result of NaN is considered successful, inconclusive and failed respectively.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\n ...\n successCondition: isNaN(result) || result >= 0.95\nstatus:\n metricResults:\n - count: 1\n measurements:\n - finishedAt: \"2021-02-10T00:15:26Z\"\n phase: Successful\n startedAt: \"2021-02-10T00:15:26Z\"\n value: NaN\n name: success-rate\n phase: Successful\n successful: 1\n phase: Successful\n startedAt: \"2021-02-10T00:15:26Z\"\n
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\n ...\n successCondition: result >= 0.95\n failureCondition: result < 0.95\nstatus:\n metricResults:\n - count: 1\n measurements:\n - finishedAt: \"2021-02-10T00:15:26Z\"\n phase: Inconclusive\n startedAt: \"2021-02-10T00:15:26Z\"\n value: NaN\n name: success-rate\n phase: Inconclusive\n successful: 1\n phase: Inconclusive\n startedAt: \"2021-02-10T00:15:26Z\"\n
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\n ...\n successCondition: result >= 0.95\nstatus:\n metricResults:\n - count: 1\n measurements:\n - finishedAt: \"2021-02-10T00:15:26Z\"\n phase: Failed\n startedAt: \"2021-02-10T00:15:26Z\"\n value: NaN\n name: success-rate\n phase: Failed\n successful: 1\n phase: Failed\n startedAt: \"2021-02-10T00:15:26Z\"\n
Here are two examples where a metric result of infinity is considered successful and failed respectively.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\n ...\n successCondition: result >= 0.95\nstatus:\n metricResults:\n - count: 1\n measurements:\n - finishedAt: \"2021-02-10T00:15:26Z\"\n phase: Successful\n startedAt: \"2021-02-10T00:15:26Z\"\n value: +Inf\n name: success-rate\n phase: Successful\n successful: 1\n phase: Successful\n startedAt: \"2021-02-10T00:15:26Z\"\n
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\n ...\n failureCondition: isInf(result)\nstatus:\n metricResults:\n - count: 1\n measurements:\n - finishedAt: \"2021-02-10T00:15:26Z\"\n phase: Failed\n startedAt: \"2021-02-10T00:15:26Z\"\n value: +Inf\n name: success-rate\n phase: Failed\n successful: 1\n phase: Failed\n startedAt: \"2021-02-10T00:15:26Z\"\n
"},{"location":"features/analysis/#empty-array","title":"Empty array","text":""},{"location":"features/analysis/#prometheus","title":"Prometheus","text":"Metric providers can sometimes return empty array, e.g., no data returned from prometheus query.
Here are two examples where a metric result of empty array is considered successful and failed respectively.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\n ...\n successCondition: len(result) == 0 || result[0] >= 0.95\nstatus:\n metricResults:\n - count: 1\n measurements:\n - finishedAt: \"2021-09-08T19:15:49Z\"\n phase: Successful\n startedAt: \"2021-09-08T19:15:49Z\"\n value: '[]'\n name: success-rate\n phase: Successful\n successful: 1\n phase: Successful\n startedAt: \"2021-09-08T19:15:49Z\"\n
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\n ...\n successCondition: len(result) > 0 && result[0] >= 0.95\nstatus:\n metricResults:\n - count: 1\n measurements:\n - finishedAt: \"2021-09-08T19:19:44Z\"\n phase: Failed\n startedAt: \"2021-09-08T19:19:44Z\"\n value: '[]'\n name: success-rate\n phase: Failed\n successful: 1\n phase: Failed\n startedAt: \"2021-09-08T19:19:44Z\"\n
"},{"location":"features/analysis/#datadog","title":"Datadog","text":"Datadog queries can return empty results if the query takes place during a time interval with no metrics. The Datadog provider will return a nil value yielding an error during the evaluation phase like:
invalid operation: < (mismatched types <nil> and float64)\n
However, empty query results yielding a nil value can be handled using the default() function. Here is a succeeding example using the default() function:
successCondition: default(result, 0) < 0.05\n
"},{"location":"features/bluegreen/","title":"BlueGreen Deployment Strategy","text":"A Blue Green Deployment allows users to reduce the amount of time multiple versions running at the same time.
"},{"location":"features/bluegreen/#overview","title":"Overview","text":"In addition to managing ReplicaSets, the rollout controller will modify a Service resource during the BlueGreenUpdate strategy. The Rollout spec has users specify a reference to active service and optionally a preview service in the same namespace. The active Service is used to send regular application traffic to the old version, while the preview Service is used as funnel traffic to the new version. The rollout controller ensures proper traffic routing by injecting a unique hash of the ReplicaSet to these services' selectors. This allows the rollout to define an active and preview stack and a process to migrate replica sets from the preview to the active.
When there is a change to the .spec.template field of a rollout, the controller will create the new ReplicaSet. If the active service is not sending traffic to a ReplicaSet, the controller will immediately start sending traffic to the ReplicaSet. Otherwise, the active service will point at the old ReplicaSet while the ReplicaSet becomes available. Once the new ReplicaSet becomes available, the controller will modify the active service to point at the new ReplicaSet. After waiting some time configured by the .spec.strategy.blueGreen.scaleDownDelaySeconds, the controller will scale down the old ReplicaSet.
Important
When the rollout changes the selector on a service, there is a propagation delay before all the nodes update their IP tables to send traffic to the new pods instead of the old. During this delay, traffic will be directed to the old pods if the nodes have not been updated yet. In order to prevent the packets from being sent to a node that killed the old pod, the rollout uses the scaleDownDelaySeconds field to give nodes enough time to broadcast the IP table changes.
"},{"location":"features/bluegreen/#example","title":"Example","text":"apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollout-bluegreen\nspec:\n replicas: 2\n revisionHistoryLimit: 2\n selector:\n matchLabels:\n app: rollout-bluegreen\n template:\n metadata:\n labels:\n app: rollout-bluegreen\n spec:\n containers:\n - name: rollouts-demo\n image: argoproj/rollouts-demo:blue\n imagePullPolicy: Always\n ports:\n - containerPort: 8080\n strategy:\n blueGreen: \n # activeService specifies the service to update with the new template hash at time of promotion.\n # This field is mandatory for the blueGreen update strategy.\n activeService: rollout-bluegreen-active\n # previewService specifies the service to update with the new template hash before promotion.\n # This allows the preview stack to be reachable without serving production traffic.\n # This field is optional.\n previewService: rollout-bluegreen-preview\n # autoPromotionEnabled disables automated promotion of the new stack by pausing the rollout\n # immediately before the promotion. If omitted, the default behavior is to promote the new\n # stack as soon as the ReplicaSet are completely ready/available.\n # Rollouts can be resumed using: `kubectl argo rollouts promote ROLLOUT`\n autoPromotionEnabled: false\n
"},{"location":"features/bluegreen/#configurable-features","title":"Configurable Features","text":"Here are the optional fields that will change the behavior of BlueGreen deployment:
spec:\n strategy:\n blueGreen:\n autoPromotionEnabled: boolean\n autoPromotionSeconds: *int32\n antiAffinity: object\n previewService: string\n prePromotionAnalysis: object\n postPromotionAnalysis: object\n previewReplicaCount: *int32\n scaleDownDelaySeconds: *int32\n scaleDownDelayRevisionLimit: *int32\n
"},{"location":"features/bluegreen/#sequence-of-events","title":"Sequence of Events","text":"The following describes the sequence of events that happen during a blue-green update.
- Beginning at a fully promoted, steady-state, a revision 1 ReplicaSet is pointed to by both the
activeService and previewService. - A user initiates an update by modifying the pod template (
spec.template.spec). - The revision 2 ReplicaSet is created with size 0.
- The
previewService is modified to point to the revision 2 ReplicaSet. The activeService remains pointing to revision 1. - The revision 2 ReplicaSet is scaled to either
spec.replicas or previewReplicaCount if set. - Once revision 2 ReplicaSet Pods are fully available,
prePromotionAnalysis begins. - Upon success of
prePromotionAnalysis, the blue/green pauses if autoPromotionEnabled is false, or autoPromotionSeconds is non-zero. - The rollout is resumed either manually by a user, or automatically by surpassing
autoPromotionSeconds. - The revision 2 ReplicaSet is scaled to the
spec.replicas, if the previewReplicaCount feature was used. - The rollout \"promotes\" the revision 2 ReplicaSet by updating the
activeService to point to it. At this point, there are no services pointing to revision 1 postPromotionAnalysis analysis begins- Once
postPromotionAnalysis completes successfully, the update is successful and the revision 2 ReplicaSet is marked as stable. The rollout is considered fully-promoted. - After waiting
scaleDownDelaySeconds (default 30 seconds), the revision 1 ReplicaSet is scaled down
"},{"location":"features/bluegreen/#autopromotionenabled","title":"autoPromotionEnabled","text":"The AutoPromotionEnabled will make the rollout automatically promote the new ReplicaSet to the active service once the new ReplicaSet is healthy. This field is defaulted to true if it is not specified.
Defaults to true
"},{"location":"features/bluegreen/#autopromotionseconds","title":"autoPromotionSeconds","text":"The AutoPromotionSeconds will make the rollout automatically promote the new ReplicaSet to active Service after the AutoPromotionSeconds time has passed since the rollout has entered a paused state. If the AutoPromotionEnabled field is set to false, this field will be ignored
Defaults to nil
"},{"location":"features/bluegreen/#antiaffinity","title":"antiAffinity","text":"Check out the Anti Affinity document document for more information.
Defaults to nil
"},{"location":"features/bluegreen/#maxunavailable","title":"maxUnavailable","text":"The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). This can not be 0 if MaxSurge is 0.
Defaults to 0
"},{"location":"features/bluegreen/#prepromotionanalysis","title":"prePromotionAnalysis","text":"Configures the Analysis before it switches traffic to the new version. The AnalysisRun can be used to block the Service selector switch until the AnalysisRun finishes successful. The success or failure of the analysis run decides if the Rollout will switch traffic, or abort the Rollout completely.
Defaults to nil
"},{"location":"features/bluegreen/#postpromotionanalysis","title":"postPromotionAnalysis","text":"Configures the Analysis after the traffic switch to new version. If the analysis run fails or errors out, the Rollout enters an aborted state and switch traffic back to the previous stable Replicaset. If scaleDownDelaySeconds is specified, the controller will cancel any AnalysisRuns at time of scaleDownDelay to scale down the ReplicaSet. If it is omitted, and post analysis is specified, it will scale down the ReplicaSet only after the AnalysisRun completes (with a minimum of 30 seconds).
Defaults to nil
"},{"location":"features/bluegreen/#previewservice","title":"previewService","text":"The PreviewService field references a Service that will be modified to send traffic to the new ReplicaSet before the new one is promoted to receiving traffic from the active service. Once the new ReplicaSet starts receiving traffic from the active service, the preview service will also be modified to send traffic to the new ReplicaSet as well. The Rollout always makes sure that the preview service is sending traffic to the newest ReplicaSet. As a result, if a new version is introduced before the old version is promoted to the active service, the controller will immediately switch over to that brand new version.
This feature is used to provide an endpoint that can be used to test a new version of an application.
Defaults to an empty string
"},{"location":"features/bluegreen/#previewreplicacount","title":"previewReplicaCount","text":"The PreviewReplicaCount field will indicate the number of replicas that the new version of an application should run. Once the application is ready to promote to the active service, the controller will scale the new ReplicaSet to the value of the spec.replicas. The rollout will not switch over the active service to the new ReplicaSet until it matches the spec.replicas count.
This feature is mainly used to save resources during the testing phase. If the application does not need a fully scaled up application for the tests, this feature can help save some resources.
If omitted, the preview ReplicaSet stack will be scaled to 100% of the replicas.
"},{"location":"features/bluegreen/#scaledowndelayseconds","title":"scaleDownDelaySeconds","text":"The ScaleDownDelaySeconds is used to delay scaling down the old ReplicaSet after the active Service is switched to the new ReplicaSet.
Defaults to 30
"},{"location":"features/bluegreen/#scaledowndelayrevisionlimit","title":"scaleDownDelayRevisionLimit","text":"The ScaleDownDelayRevisionLimit limits the number of old active ReplicaSets to keep scaled up while they wait for the scaleDownDelay to pass after being removed from the active service.
If omitted, all ReplicaSets will be retained for the specified scaleDownDelay
"},{"location":"features/canary/","title":"Canary Deployment Strategy","text":"A canary rollout is a deployment strategy where the operator releases a new version of their application to a small percentage of the production traffic.
"},{"location":"features/canary/#overview","title":"Overview","text":"Since there is no agreed upon standard for a canary deployment, the rollouts controller allows users to outline how they want to run their canary deployment. Users can define a list of steps the controller uses to manipulate the ReplicaSets when there is a change to the .spec.template. Each step will be evaluated before the new ReplicaSet is promoted to the stable version, and the old version is completely scaled down.
Each step can have one of two fields. The setWeight field dictates the percentage of traffic that should be sent to the canary, and the pause struct instructs the rollout to pause. When the controller reaches a pause step for a rollout, it will add a PauseCondition struct to the .status.PauseConditions field. If the duration field within the pause struct is set, the rollout will not progress to the next step until it has waited for the value of the duration field. Otherwise, the rollout will wait indefinitely until that Pause condition is removed. By using the setWeight and the pause fields, a user can declaratively describe how they want to progress to the new version. Below is an example of a canary strategy.
Important
If the canary Rollout does not use traffic management, the Rollout makes a best effort attempt to achieve the percentage listed in the last setWeight step between the new and old version. For example, if a Rollout has 10 Replicas and 10% for the first setWeight step, the controller will scale the new desired ReplicaSet to 1 replicas and the old stable ReplicaSet to 9. In the case where the setWeight is 41%, the Rollout attempts to get there by finding the whole number with the smallest delta, rounding up the calculation if the deltas are equals (i.e. the new ReplicaSet has 4 pods since 41% of 10 is closer to 4/10 than 5/10, and the old ReplicaSet has 6 pods). If a user wants to have more fine-grained control of the percentages without a large number of Replicas, that user should use the traffic management functionality.
"},{"location":"features/canary/#example","title":"Example","text":"apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: example-rollout\nspec:\n replicas: 10\n selector:\n matchLabels:\n app: nginx\n template:\n metadata:\n labels:\n app: nginx\n spec:\n containers:\n - name: nginx\n image: nginx:1.15.4\n ports:\n - containerPort: 80\n minReadySeconds: 30\n revisionHistoryLimit: 3\n strategy:\n canary: #Indicates that the rollout should use the Canary strategy\n maxSurge: \"25%\"\n maxUnavailable: 0\n steps:\n - setWeight: 10\n - pause:\n duration: 1h # 1 hour\n - setWeight: 20\n - pause: {} # pause indefinitely\n
"},{"location":"features/canary/#pause-duration","title":"Pause Duration","text":"Pause duration can be specified with an optional time unit suffix. Valid time units are \"s\", \"m\", \"h\". Defaults to \"s\" if not specified.
spec:\n strategy:\n canary:\n steps:\n - pause: { duration: 10 } # 10 seconds\n - pause: { duration: 10s } # 10 seconds\n - pause: { duration: 10m } # 10 minutes\n - pause: { duration: 10h } # 10 hours\n - pause: {} # pause indefinitely\n
If no duration is specified for a pause step, the rollout will be paused indefinitely. To unpause, use the argo kubectl plugin promote command.
# promote to the next step\nkubectl argo rollouts promote <rollout>\n
"},{"location":"features/canary/#dynamic-canary-scale-with-traffic-routing","title":"Dynamic Canary Scale (with Traffic Routing)","text":"By default, the rollout controller will scale the canary to match the current trafficWeight of the current step. For example, if the current weight is 25%, and there are four replicas, then the canary will be scaled to 1, to match the traffic weight.
It is possible to control the canary replica's scale during the steps such that it does not necessary match the traffic weight. Some use cases for this:
- The new version should not yet be exposed to the public (setWeight: 0), but you would like to scale the canary up for testing purposes.
- You wish to scale the canary stack up minimally, and use some header based traffic shaping to the canary, while setWeight is still set to 0.
- You wish to scale the canary up to 100%, in order to facilitate traffic shadowing.
Important
Setting canary scale is only available when using the canary strategy with a traffic router, since the basic canary needs to control canary scale in order to approximate canary weight.
To control canary scales and weights during steps, use the setCanaryScale step and indicate which scale the the canary should use:
- explicit replica count without changing traffic weight (
replicas) - explicit weight percentage of total spec.replicas without changing traffic weight(
weight) - to or not to match current canary's
setWeight step (matchTrafficWeight: true or false)
spec:\n strategy:\n canary:\n steps:\n # explicit count\n - setCanaryScale:\n replicas: 3\n # a percentage of spec.replicas\n - setCanaryScale:\n weight: 25\n # matchTrafficWeight returns to the default behavior of matching the canary traffic weight\n - setCanaryScale:\n matchTrafficWeight: true\n
When using setCanaryScale with explicit values for either replicas or weight, one must be careful if used in conjunction with the setWeight step. If done incorrectly, an imbalanced amount of traffic may be directed to the canary (in proportion to the Rollout's scale). For example, the following set of steps would cause 90% of traffic to only be served by 10% of pods:
spec:\n replicas: 10\n strategy:\n canary:\n steps:\n # 1 canary pod (10% of spec.replicas)\n - setCanaryScale:\n weight: 10\n # 90% of traffic to the 1 canary pod\n - setWeight: 90\n - pause: {}\n
The above situation is caused by the changed behvaior of setWeight after setCanaryScale. To reset, set matchTrafficWeight: true and the setWeight behavior will be restored, i.e., subsequent setWeight will create canary replicas matching the traffic weight.
"},{"location":"features/canary/#dynamic-stable-scale-with-traffic-routing","title":"Dynamic Stable Scale (with Traffic Routing)","text":"Important
Available since v1.1
When using traffic routing, by default the stable ReplicaSet is left scaled to 100% during the update. This has the advantage that if an abort occurs, traffic can be immediately shifted back to the stable ReplicaSet without delay. However, it has the disadvantage that during the update, there will eventually exist double the number of replica pods running (similar to in a blue-green deployment), since the stable ReplicaSet is left scaled up for the full duration of the update.
It is possible to dynamically reduce the scale of the stable ReplicaSet during an update such that it scales down as the traffic weight increases to canary. This would be desirable in scenarios where the Rollout has a high replica count and resource cost is a concern, or in bare-metal situations where it is not possible to create additional node capacity to accommodate double the replicas.
The ability to dynamically scale the stable ReplicaSet can be enabled by setting the canary.dynamicStableScale flag to true:
spec:\n strategy:\n canary:\n dynamicStableScale: true\n
NOTE: that if dynamicStableScale is set, and the rollout is aborted, the canary ReplicaSet will dynamically scale down as traffic shifts back to stable. If you wish to leave the canary ReplicaSet scaled up while aborting, an explicit value for abortScaleDownDelaySeconds can be set:
spec:\n strategy:\n canary:\n dynamicStableScale: true\n abortScaleDownDelaySeconds: 600\n
"},{"location":"features/canary/#mimicking-rolling-update","title":"Mimicking Rolling Update","text":"If the steps field is omitted, the canary strategy will mimic the rolling update behavior. Similar to the deployment, the canary strategy has the maxSurge and maxUnavailable fields to configure how the Rollout should progress to the new version.
"},{"location":"features/canary/#other-configurable-features","title":"Other Configurable Features","text":"Here are the optional fields that will modify the behavior of canary strategy:
spec:\n strategy:\n canary:\n analysis: object\n antiAffinity: object\n canaryService: string\n stableService: string\n maxSurge: stringOrInt\n maxUnavailable: stringOrInt\n trafficRouting: object\n
"},{"location":"features/canary/#analysis","title":"analysis","text":"Configure the background Analysis to execute during the rollout. If the analysis is unsuccessful the rollout will be aborted.
Defaults to nil
"},{"location":"features/canary/#antiaffinity","title":"antiAffinity","text":"Check out the Anti Affinity document document for more information.
Defaults to nil
"},{"location":"features/canary/#canaryservice","title":"canaryService","text":"canaryService references a Service that will be modified to send traffic to only the canary ReplicaSet. This allows users to only hit the canary ReplicaSet.
Defaults to an empty string
"},{"location":"features/canary/#stableservice","title":"stableService","text":"stableService the name of a Service which selects pods with stable version and doesn't select any pods with canary version. This allows users to only hit the stable ReplicaSet.
Defaults to an empty string
"},{"location":"features/canary/#maxsurge","title":"maxSurge","text":"maxSurge defines the maximum number of replicas the rollout can create to move to the correct ratio set by the last setWeight. Max Surge can either be an integer or percentage as a string (i.e. \"20%\")
Defaults to \"25%\".
"},{"location":"features/canary/#maxunavailable","title":"maxUnavailable","text":"The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). This can not be 0 if MaxSurge is 0.
Defaults to 25%
"},{"location":"features/canary/#trafficrouting","title":"trafficRouting","text":"The traffic management rules to apply to control the flow of traffic between the active and canary versions. If not set, the default weighted pod replica based routing will be used.
Defaults to nil
"},{"location":"features/controller-metrics/","title":"Controller Metrics","text":"The Argo Rollouts controller is already instrumented with Prometheus metrics available at /metrics in port 8090. You can use these metrics to look at the health of the controller either via dashboards or via other Prometheus integrations.
"},{"location":"features/controller-metrics/#installing-and-configuring-prometheus","title":"Installing and configuring Prometheus","text":"To take advantage of the metrics you need to have Prometheus installed in your Kubernetes cluster. If you don't have an existing installation of Prometheus you can use any of the common methods to install it in your cluster. Popular options include the Prometheus Helm chart or the Prometheus Operator.
Once Prometheus is running in your cluster you need to make sure that it scrapes the Argo Rollouts endpoint. Prometheus already contains a service discovery mechanism for Kubernetes, but you need to configure it first. Depending on your installation method you might need to take additional actions to scrape the Argo Rollouts endpoint.
For example, if you used the Helm chart of Prometheus you need to annotate your Argo Rollouts Controller with the following:
spec:\n template:\n metadata:\n annotations:\n prometheus.io/scrape: \"true\"\n prometheus.io/path: /metrics\n prometheus.io/port: \"8090\"\n
You can always see if the controller is reached successfully in the Prometheus \"Targets\" screen:
Once the controller metrics are read by your Prometheus instance, you can use them like any other Prometheus data source.
"},{"location":"features/controller-metrics/#creating-grafana-dashboards","title":"Creating Grafana Dashboards","text":"You can easily visualize the metrics from the controller using Grafana dashboards. Install Grafana in your cluster and connect it your Prometheus instance. Then you can create any dashboard by using the available metrics (described in detail in the next sections).
As a starting point you can find an existing dashboard at https://github.com/argoproj/argo-rollouts/blob/master/examples/dashboard.json
You can import this Dashboard in your Grafana installation as a JSON file.
"},{"location":"features/controller-metrics/#available-metrics-for-rollout-objects","title":"Available metrics for Rollout Objects","text":"The Argo Rollouts controller publishes the following prometheus metrics about Argo Rollout objects.
Name Description rollout_info Information about rollout. rollout_info_replicas_available The number of available replicas per rollout. rollout_info_replicas_unavailable The number of unavailable replicas per rollout. rollout_info_replicas_desired The number of desired replicas per rollout. rollout_info_replicas_updated The number of updated replicas per rollout. rollout_phase [DEPRECATED - use rollout_info] Information on the state of the rollout. rollout_reconcile Rollout reconciliation performance. rollout_reconcile_error Error occurring during the rollout. experiment_info Information about Experiment. experiment_phase Information on the state of the experiment. experiment_reconcile Experiments reconciliation performance. experiment_reconcile_error Error occurring during the experiment. analysis_run_info Information about analysis run. analysis_run_metric_phase Information on the duration of a specific metric in the Analysis Run. analysis_run_metric_type Information on the type of a specific metric in the Analysis Runs. analysis_run_phase Information on the state of the Analysis Run. analysis_run_reconcile Analysis Run reconciliation performance. analysis_run_reconcile_error Error occurring during the analysis run."},{"location":"features/controller-metrics/#available-metrics-for-the-controller-itself","title":"Available metrics for the controller itself","text":"The controller also publishes the following Prometheus metrics to describe the controller health.
Name Description controller_clientset_k8s_request_total Number of kubernetes requests executed during application reconciliation. workqueue_adds_total Total number of adds handled by workqueue workqueue_depth Current depth of workqueue workqueue_queue_duration_seconds How long in seconds an item stays in workqueue before being requested. workqueue_work_duration_seconds How long in seconds processing an item from workqueue takes. workqueue_unfinished_work_seconds How many seconds of work has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases. workqueue_longest_running_processor_seconds How many seconds has the longest running processor for workqueue been running workqueue_retries_total Total number of retries handled by workqueue In addition, the Argo-rollouts offers metrics on CPU, memory and file descriptor usage as well as the process start time and memory stats of current Go processes.
"},{"location":"features/ephemeral-metadata/","title":"Ephemeral Metadata","text":"Important
Available for canary rollouts since v0.10.0
Important
Available for blue-green rollouts since v1.0
One use case is for a Rollout to label or annotate the desired/stable pods with user-defined labels/annotations, for only the duration which they are the desired or stable set, and for the labels to be updated/removed as soon as the ReplicaSet switches roles (e.g. from desired to stable). The use case which this enables, is to allow prometheus, wavefront, datadog queries and dashboards to be built, which can rely on a consistent labels, rather than the rollouts-pod-template-hash which is unpredictable and changing from revision to revision.
A Rollout using the canary strategy has the ability to attach ephemeral metadata to the stable or canary Pods using the stableMetadata and canaryMetadata fields respectively.
spec:\n strategy:\n canary:\n stableMetadata:\n labels:\n role: stable\n canaryMetadata:\n labels:\n role: canary\n
A Rollout using the blue-green strategy has the ability to attach ephemeral metadata to the active or preview Pods using the activeMetadata and previewMetadata fields respectively.
spec:\n strategy:\n blueGreen:\n activeMetadata:\n labels:\n role: active\n previewMetadata:\n labels:\n role: preview\n
During an update, the Rollout will create the desired ReplicaSet while also merging the metadata defined in canaryMetadata/previewMetadata to the desired ReplicaSet's spec.template.metadata. This results in all Pods of the ReplicaSet being created with the desired metadata. When the rollout becomes fully promoted, the desired ReplicaSet becomes the stable, and is updated to use the labels and annotations under stableMetadata/activeMetadata. The Pods of the ReplicaSet will then be updated in place to use the stable metadata (without recreating the pods).
Important
In order for tooling to take advantage of this feature, they would need to recognize the change in labels and/or annotations that happen after the Pod has already started. Not all tools may detect this.
"},{"location":"features/experiment/","title":"Experiment CRD","text":""},{"location":"features/experiment/#what-is-the-experiment-crd","title":"What is the Experiment CRD?","text":"The Experiment CRD allows users to have ephemeral runs of one or more ReplicaSets. In addition to running ephemeral ReplicaSets, the Experiment CRD can launch AnalysisRuns alongside the ReplicaSets. Generally, those AnalysisRun is used to confirm that new ReplicaSets are running as expected.
A Service routing traffic to the Experiment ReplicaSet is also generated if a weight (which requires traffic routing) OR the Service attribute for that experiment is set.
"},{"location":"features/experiment/#use-cases-of-experiments","title":"Use cases of Experiments","text":" -
A user wants to run two versions of an application for a specific duration to enable Kayenta-style analysis of their application. The Experiment CRD creates 2 ReplicaSets (a baseline and a canary) based on the spec.templates field of the Experiment and waits until both are healthy. After the duration passes, the Experiment scales down the ReplicaSets, and the user can start the Kayenta analysis run.
-
A user can use experiments to enable A/B/C testing by launching multiple experiments with a different version of their application for a long duration. Each Experiment has one PodSpec template that defines a specific version a user would want to run. The Experiment allows users to launch multiple experiments at once and keep each Experiment self-contained.
-
Launching a new version of an existing application with different labels to avoid receiving traffic from a Kubernetes service. The user can run tests against the new version before continuing the Rollout.
"},{"location":"features/experiment/#experiment-spec","title":"Experiment Spec","text":"Below is an example of an experiment that creates two ReplicaSets with 1 replica each and runs them for 20 minutes once they both become available. Additionally, several AnalysisRuns are run to perform analysis against the pods of the Experiment
apiVersion: argoproj.io/v1alpha1\nkind: Experiment\nmetadata:\n name: example-experiment\nspec:\n # Duration of the experiment, beginning from when all ReplicaSets became healthy (optional)\n # If omitted, will run indefinitely until terminated, or until all analyses which were marked\n # `requiredForCompletion` have completed.\n duration: 20m\n\n # Deadline in seconds in which a ReplicaSet should make progress towards becoming available.\n # If exceeded, the Experiment will fail.\n progressDeadlineSeconds: 30\n\n # List of pod template specs to run in the experiment as ReplicaSets\n templates:\n - name: purple\n # Number of replicas to run (optional). If omitted, will run a single replica\n replicas: 1\n # Flag to create Service for this Experiment (optional)\n # If omitted, a Service won't be created.\n service:\n # Name of the Service (optional). If omitted, service: {} would also be acceptable.\n name: service-name\n selector:\n matchLabels:\n app: canary-demo\n color: purple\n template:\n metadata:\n labels:\n app: canary-demo\n color: purple\n spec:\n containers:\n - name: rollouts-demo\n image: argoproj/rollouts-demo:purple\n imagePullPolicy: Always\n ports:\n - name: http\n containerPort: 8080\n protocol: TCP\n - name: orange\n replicas: 1\n minReadySeconds: 10\n selector:\n matchLabels:\n app: canary-demo\n color: orange\n template:\n metadata:\n labels:\n app: canary-demo\n color: orange\n spec:\n containers:\n - name: rollouts-demo\n image: argoproj/rollouts-demo:orange\n imagePullPolicy: Always\n ports:\n - name: http\n containerPort: 8080\n protocol: TCP\n\n # List of AnalysisTemplate references to perform during the experiment\n analyses:\n - name: purple\n templateName: http-benchmark\n args:\n - name: host\n value: purple\n - name: orange\n templateName: http-benchmark\n args:\n - name: host\n value: orange\n - name: compare-results\n templateName: compare\n # If requiredForCompletion is true for an analysis reference, the Experiment will not complete\n # until this analysis has completed.\n requiredForCompletion: true\n args:\n - name: host\n value: purple\n
"},{"location":"features/experiment/#experiment-lifecycle","title":"Experiment Lifecycle","text":"An Experiment is intended to temporarily run one or more templates. The lifecycle of an Experiment is as follows:
- Create and scale a ReplicaSet for each pod template specified under
spec.templates. If service is specified under a pod template, a Service will also be created for that pod. - Wait for all ReplicaSets reach full availability. If a ReplicaSet does not become available within
spec.progressDeadlineSeconds, the Experiment will fail. Once available, the Experiment will transition from the Pending state to a Running state. - Once an Experiment is considered
Running, it will begin an AnalysisRun for every AnalysisTemplate referenced under spec.analyses. - If a duration is specified under
spec.duration, the Experiment will wait until the duration has elapsed before completing the Experiment. - If an AnalysisRun fails or errors, the Experiment will end prematurely, with a status equal to the unsuccessful AnalysisRun (i.e.
Failed or Error) - If one or more of the referenced AnalysisTemplates is marked with
requiredForCompletion: true, the Experiment will not complete until those AnalysisRuns have completed, even if it exceeds the Experiment duration. - If neither a
spec.duration or requiredForCompletion: true is specified, the Experiment will run indefinitely, until explicitly terminated (by setting spec.terminate: true). - Once an Experiment is complete, the ReplicaSets will be scaled to zero, and any incomplete AnalysisRuns will be terminated.
Note
ReplicaSet names are generated by combining the Experiment name with the template name.
"},{"location":"features/experiment/#integration-with-rollouts","title":"Integration With Rollouts","text":"A rollout using the Canary strategy can create an experiment using an experiment step. The experiment step serves as a blocking step for the Rollout, and a Rollout will not continue until the Experiment succeeds. The Rollout creates an Experiment using the configuration in the experiment step of the Rollout. If the Experiment fails or errors, the Rollout will abort.
Note
Experiment names are generated by combining the Rollout's name, the PodHash of the new ReplicaSet, the current revision of the Rollout, and the current step-index.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\n labels:\n app: guestbook\nspec:\n...\n strategy:\n canary: \n steps:\n - experiment:\n duration: 1h\n templates:\n - name: baseline\n specRef: stable\n - name: canary\n specRef: canary\n analyses:\n - name : mann-whitney\n templateName: mann-whitney\n args:\n - name: baseline-hash\n value: \"{{templates.baseline.podTemplateHash}}\"\n - name: canary-hash\n value: \"{{templates.canary.podTemplateHash}}\"\n
In the example above, during an update of a Rollout, the Rollout will launch an Experiment. The Experiment will create two ReplicaSets: baseline and canary, with one replica each, and will run for one hour. The baseline template uses the PodSpec from the stable ReplicaSet, and the canary template uses the PodSpec from the canary ReplicaSet.
Additionally, the Experiment will perform analysis using the AnalysisTemplate named mann-whitney. The AnalysisRun is supplied with the pod-hash details of the baseline and canary to perform the necessary metrics queries, using the {{templates.baseline.podTemplateHash}} and {{templates.canary.podTemplateHash}} variables respectively.
Note
The pod-hashes of the baseline/canary ReplicaSets created by the Experiment, will have different values than the pod-hashes of the stable/canary ReplicaSets created by the Rollout. This is despite the fact that the PodSpec are the same. This is intentional behavior, in order to allow the metrics of the Experiment's pods to be delineated and queried separately from the metrics of the Rollout pods.
"},{"location":"features/experiment/#weighted-experiment-step-with-traffic-routing","title":"Weighted Experiment Step with Traffic Routing","text":"Important
Available since v1.1
A Rollout using the Canary strategy along with Traffic Routing can split traffic to an experiment stack in a fine-grained manner. When Traffic Routing is enabled, the Rollout Experiment step allows traffic to be shifted to experiment pods.
Note
This feature is currently available only for the SMI, ALB, and Istio Traffic Routers.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\n labels:\n app: guestbook\nspec:\n...\nstrategy:\n canary:\n trafficRouting:\n alb:\n ingress: ingress\n ...\n steps:\n - experiment:\n duration: 1h\n templates:\n - name: experiment-baseline\n specRef: stable\n weight: 5\n - name: experiment-canary\n specRef: canary\n weight: 5\n
In the above example, during an update, the first step would start a baseline vs. canary experiment. When pods are ready (Experiment enters Running phase), the rollout would direct 5% of traffic to experiment-canary and 5% to experiment-baseline, leaving the remaining 90% of traffic to the old stack.
Note
When a weighted experiment step with traffic routing is used, a service is auto-created for each experiment template. The traffic routers use this service to send traffic to the experiment pods.
By default, the generated Service has the name of the ReplicaSet and inherits ports and selector from the specRef definition. It can be accessed in using the {{templates.baseline.replicaset.name}} or {{templates.canary.replicaset.name}} variables respectively.
"},{"location":"features/experiment/#experiment-service-creation-without-weight","title":"Experiment Service Creation without Weight","text":"If you don't want to use traffic routing for your Experiments but still want to create a Service for them, you can set a Service object which takes an optional Name, without having to set a Weight for them.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\n labels:\n app: guestbook\nspec:\n...\nstrategy:\n canary:\n steps:\n - experiment:\n duration: 1h\n templates:\n - name: experiment-baseline\n specRef: stable\n service:\n name: test-service\n - name: experiment-canary\n specRef: canary\n
In the above example, during an update, the first step would start a baseline vs. canary experiment. This time, a service would be created for experiment-baseline even without setting a weight for it or traffic routing for the rollout.
"},{"location":"features/helm/","title":"Using Argo Rollouts with Helm","text":"Argo Rollouts will always respond to changes in Rollouts resources regardless of how the change was made. This means that Argo Rollouts is compatible with all templating solutions that you might use to manage your deployments.
Argo Rollouts manifests can be managed with the Helm package manager. If your Helm chart contains Rollout Resources, then as soon as you install/upgrade your chart, Argo Rollouts will take over and initiate the progressive delivery process.
Here is an example Rollout that is managed with Helm:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: {{ template \"helm-guestbook.fullname\" . }}\n labels:\n app: {{ template \"helm-guestbook.name\" . }}\n chart: {{ template \"helm-guestbook.chart\" . }}\n release: {{ .Release.Name }}\n heritage: {{ .Release.Service }}\nspec:\n replicas: {{ .Values.replicaCount }}\n revisionHistoryLimit: 3\n selector:\n matchLabels:\n app: {{ template \"helm-guestbook.name\" . }}\n release: {{ .Release.Name }}\n strategy:\n blueGreen:\n activeService: {{ template \"helm-guestbook.fullname\" . }}\n previewService: {{ template \"helm-guestbook.fullname\" . }}-preview\n template:\n metadata:\n labels:\n app: {{ template \"helm-guestbook.name\" . }}\n release: {{ .Release.Name }}\n spec:\n containers:\n - name: {{ .Chart.Name }}\n image: \"{{ .Values.image.repository }}:{{ .Values.image.tag }}\"\n imagePullPolicy: {{ .Values.image.pullPolicy }}\n ports:\n - name: http\n containerPort: 80\n protocol: TCP\n livenessProbe:\n httpGet:\n path: /\n port: http\n readinessProbe:\n httpGet:\n path: /\n port: http\n resources:\n{{ toYaml .Values.resources | indent 12 }}\n {{- with .Values.nodeSelector }}\n nodeSelector:\n{{ toYaml . | indent 8 }}\n {{- end }}\n {{- with .Values.affinity }}\n affinity:\n{{ toYaml . | indent 8 }}\n {{- end }}\n {{- with .Values.tolerations }}\n tolerations:\n{{ toYaml . | indent 8 }}\n {{- end }}\n
You can find the full example at https://github.com/argoproj/argo-rollouts/tree/master/examples/helm-blue-green.
"},{"location":"features/hpa-support/","title":"Horizontal Pod Autoscaling","text":"Horizontal Pod Autoscaling (HPA) automatically scales the number of pods in owned by a Kubernetes resource based on observed CPU utilization or user-configured metrics. In order to accomplish this behavior, HPA only supports resources with the scale endpoint enabled with a couple of required fields. The scale endpoint allows the HPA to understand the current state of a resource and modify the resource to scale it appropriately. Argo Rollouts added support for the scale endpoint in the 0.3.0 release. After being modified by the HPA, the Argo Rollouts controller is responsible for reconciling that change in replicas. Since the strategies within a Rollout are very different, the Argo Rollouts controller handles the scale endpoint differently for various strategies. Below is the behavior for the different strategies:
"},{"location":"features/hpa-support/#blue-green","title":"Blue Green","text":"The HPA will scale rollouts using the BlueGreen strategy using the metrics from the ReplicaSet receiving traffic from the active service. When the HPA changes the replicas count, the Argo Rollouts controller will first scale up the ReplicaSet receiving traffic from the active service before ReplicaSet receiving traffic from the preview service. The controller will scale up the ReplicaSet receiving traffic from the preview service to prepare it for when the rollout switches the preview to active. If there are no ReplicaSets receiving from the active service, the controller will use all the pods that match the base selector to determine scaling events. In that case, the controller will scale up the latest ReplicaSet to the new count and scale down the older ReplicaSets.
"},{"location":"features/hpa-support/#canary-replicaset-based","title":"Canary (ReplicaSet based)","text":"The HPA will scale rollouts using the Canary Strategy using the metrics of all the ReplicasSets within the rollout. Since the Argo Rollouts controller does not control the service that sends traffic to those ReplicaSets, it assumes that all the ReplicaSets in the rollout are receiving traffic.
"},{"location":"features/hpa-support/#example","title":"Example","text":"Below is an example of a Horizontal Pod Autoscaler that scales a rollout based on CPU metrics:
apiVersion: autoscaling/v1\nkind: HorizontalPodAutoscaler\nmetadata:\n name: hpa-rollout-example\nspec:\n maxReplicas: 6\n minReplicas: 2\n scaleTargetRef:\n apiVersion: argoproj.io/v1alpha1\n kind: Rollout\n name: example-rollout\n targetCPUUtilizationPercentage: 80\n
"},{"location":"features/hpa-support/#requirements","title":"Requirements","text":"In order for the HPA to manipulate the rollout, the Kubernetes cluster hosting the rollout CRD needs the subresources support for CRDs. This feature was introduced as alpha in Kubernetes version 1.10 and transitioned to beta in Kubernetes version 1.11. If a user wants to use HPA on v1.10, the Kubernetes Cluster operator will need to add a custom feature flag to the API server. After 1.10, the flag is turned on by default. Check out the following link for more information on setting the custom feature flag.
"},{"location":"features/kubectl-plugin/","title":"Kubectl Plugin","text":"Kubectl plugins are a way to extend the kubectl command to provide additional behavior. Generally, they are used to add new functionality to kubectl and automate scriptable workflows against a cluster. The official documentation on them is here.
Argo Rollouts offers a Kubectl plugin to enrich the experience with Rollouts, Experiments, and Analysis from the command line. It offers the ability to visualize the Argo Rollouts resources and run routine operations like promote or retry on those resources from the command.
"},{"location":"features/kubectl-plugin/#installation","title":"Installation","text":"See the installation guide for instructions on installing the plugin.
"},{"location":"features/kubectl-plugin/#usage","title":"Usage","text":"The best way to get information on the available Argo Rollouts kubectl plugin commands is by run kubectl argo rollouts. The plugin lists all the available commands that the tool can execute along with a description of each commend. All the plugin's commands interact with the Kubernetes API server and use KubeConfig credentials for authentication. Since the plugin leverages the KubeConfig of the user running the command, the plugin has the permissions of those configs.
Similar to kubectl, the plugin uses many of the same flags as the kubectl. For example, the kubectl argo rollouts get rollout canary-demo -w command starts a watch on the canary-demo rollout object similar to how the kubectl get deployment canary-demo -w command starts a watch on a deployment.
"},{"location":"features/kubectl-plugin/#visualizing-rollouts-and-experiments","title":"Visualizing Rollouts and Experiments","text":"In addition to encapsulating many routine commands, the Argo Rollouts kubectl plugin supports visualizing rollouts and experiments with the get command. The get command provides a clean representation of either the rollouts or the experiments running in a cluster. It returns a bunch of metadata on a resource and a tree view of the child resources created by the parent. As an example, here is a rollout retrieved with a get command:
Here is a table to explain some of the icons on the tree view:
Icon Kind \u27f3 Rollout \u03a3 Experiment \u03b1 AnalysisRun # Revision \u29c9 ReplicaSet \u25a1 Pod \u229e Job If the get command includes the watch flag (-w or --watch), the terminal updates as the rollouts or experiment progress highlighting the progress.
"},{"location":"features/kustomize/","title":"Kustomize Integration","text":"Kustomize can be extended to understand CRD objects through the use of transformer configs. Using transformer configs, kustomize can be \"taught\" about the structure of a Rollout object and leverage kustomize features such as ConfigMap/Secret generators, variable references, and common labels & annotations. To use Rollouts with kustomize:
-
Download rollout-transform.yaml into your kustomize directory.
-
Include rollout-transform.yaml in your kustomize configurations section:
kind: Kustomization\napiVersion: kustomize.config.k8s.io/v1beta1\n\nconfigurations:\n- rollout-transform.yaml\n
An example kustomize app demonstrating the ability to use transformers with Rollouts can be seen here.
- With Kustomize 3.6.1 it is possible to reference the configuration directly from a remote resource:
configurations:\n - https://argoproj.github.io/argo-rollouts/features/kustomize/rollout-transform.yaml\n
- With Kustomize 4.5.5 kustomize can use kubernetes OpenAPI data to get merge key and patch strategy information about resource types. For example, given the following rollout:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollout-canary\nspec:\n strategy:\n canary:\n steps:\n # detail of the canary steps is omitted\n template:\n metadata:\n labels:\n app: rollout-canary\n spec:\n containers:\n - name: rollouts-demo\n image: argoproj/rollouts-demo:blue\n imagePullPolicy: Always\n ports:\n - containerPort: 8080\n
user can update the Rollout via a patch in a kustomization file, to change the image to nginx
apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n- rollout-canary.yaml\n\nopenapi:\n path: https://raw.githubusercontent.com/argoproj/argo-schema-generator/main/schema/argo_all_k8s_kustomize_schema.json\n\npatches:\n- patch: |-\n apiVersion: argoproj.io/v1alpha1\n kind: Rollout\n metadata:\n name: rollout-canary\n spec:\n template:\n spec:\n containers:\n - name: rollouts-demo\n image: nginx\n
The OpenAPI data is auto-generated and defined in this file.
An example kustomize app demonstrating the ability to use OpenAPI data with Rollouts can be seen here.
"},{"location":"features/notifications/","title":"Notifications","text":"Important
Available since v1.1
Argo Rollouts provides notifications powered by the Notifications Engine. Controller administrators can leverage flexible systems of triggers and templates to configure notifications requested by the end users. The end-users can subscribe to the configured triggers by adding an annotation to the Rollout objects.
"},{"location":"features/notifications/#configuration","title":"Configuration","text":"The trigger defines the condition when the notification should be sent as well as the notification content template. Default Argo Rollouts comes with a list of built-in triggers that cover the most important events of Argo Rollout live-cycle. Both triggers and templates are configured in the argo-rollouts-notification-configmap ConfigMap. In order to get started quickly, you can use pre-configured notification templates defined in notifications-install.yaml.
If you are leveraging Kustomize it is recommended to include notifications-install.yaml as a remote resource into your kustomization.yaml file:
apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n- https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml\n- https://github.com/argoproj/argo-rollouts/releases/latest/download/notifications-install.yaml\n
After including the argo-rollouts-notification-configmap ConfigMap the administrator needs to configure integration with the required notifications service such as Slack or MS Teams. An example below demonstrates Slack integration:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n # detail of the templates is omitted\n # detail of the triggers is omitted\n service.slack: |\n token: $slack-token\n---\napiVersion: v1\nkind: Secret\nmetadata:\n name: argo-rollouts-notification-secret\nstringData:\n slack-token: <my-slack-token>\n
Learn more about supported services and configuration settings in services documentation.
"},{"location":"features/notifications/#namespace-based-configuration","title":"Namespace based configuration","text":"Important
Available since v1.6
A common installation method for Argo Rollouts is to install it in a dedicated namespace to manage a whole cluster. In this case, the administrator is the only person who can configure notifications in that namespace generally. However, in some cases, it is required to allow end-users to configure notifications for their Rollout resources. For example, the end-user can configure notifications for their Rollouts in the namespace where they have access to and their rollout is running in.
To use this feature all you need to do is create the same configmap named argo-rollouts-notification-configmap and possibly a secret argo-rollouts-notification-secret in the namespace where the rollout object lives. When it is configured this way the controller will send notifications using both the controller level configuration (the configmap located in the same namespaces as the controller) as well as the configmap located in the same namespaces where the rollout object is at.
To enable you need to add a flag to the controller --self-service-notification-enabled
"},{"location":"features/notifications/#default-trigger-templates","title":"Default Trigger templates","text":"Currently the following triggers have built-in templates.
on-rollout-completed when a rollout is finished and all its steps are completedon-rollout-step-completed when an individual step inside a rollout definition is completedon-rollout-updated when a rollout definition is changedon-scaling-replica-set when the number of replicas in a rollout is changed
"},{"location":"features/notifications/#subscriptions","title":"Subscriptions","text":"The end-users can start leveraging notifications using notifications.argoproj.io/subscribe.<trigger>.<service>: <recipient> annotation. For example, the following annotation subscribes two Slack channels to notifications about canary rollout step completion:
---\napiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollout-canary\n annotations:\n notifications.argoproj.io/subscribe.on-rollout-step-completed.slack: my-channel1;my-channel2\n
Annotation key consists of following parts:
on-rollout-step-completed - trigger nameslack - notification service namemy-channel1;my-channel2 - a semicolon separated list of recipients
"},{"location":"features/notifications/#customization","title":"Customization","text":"The Rollout administrator can customize the notifications by configuring notification templates and custom triggers in argo-rollouts-notification-configmap ConfigMap.
"},{"location":"features/notifications/#templates","title":"Templates","text":"The notification template is a stateless function that generates the notification content. The template is leveraging html/template golang package. It is meant to be reusable and can be referenced by multiple triggers.
An example below demonstrates a sample template:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n template.my-purple-template: |\n message: |\n Rollout {{.rollout.metadata.name}} has purple image\n slack:\n attachments: |\n [{\n \"title\": \"{{ .rollout.metadata.name}}\",\n \"color\": \"#800080\"\n }]\n
Each template has access to the following fields:
rollout holds the rollout object.recipient holds the recipient name.
The message field of the template definition allows creating a basic notification for any notification service. You can leverage notification service-specific fields to create complex notifications. For example using service-specific you can add blocks and attachments for Slack, subject for Email or URL path, and body for Webhook. See corresponding service documentation for more information.
"},{"location":"features/notifications/#custom-triggers","title":"Custom Triggers","text":"In addition to custom notification template administrator and configure custom triggers. Custom trigger defines the condition when the notification should be sent. The definition includes name, condition and notification templates reference. The condition is a predicate expression that returns true if the notification should be sent. The trigger condition evaluation is powered by antonmedv/expr. The condition language syntax is described at Language-Definition.md.
The trigger is configured in argo-rollouts-notification-configmap ConfigMap. For example the following trigger sends a notification when rollout pod spec uses argoproj/rollouts-demo:purple image:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n trigger.on-purple: |\n - send: [my-purple-template]\n when: rollout.spec.template.spec.containers[0].image == 'argoproj/rollouts-demo:purple'\n
Each condition might use several templates. Typically each template is responsible for generating a service-specific notification part.
"},{"location":"features/notifications/#notification-metrics","title":"Notification Metrics","text":"The following prometheus metrics are emitted when notifications are enabled in argo-rollouts. - notification_send_success is a counter that measures how many times the notification is sent successfully. - notification_send_error is a counter that measures how many times the notification failed to send. - notification_send is a histogram that measures performance of sending notification.
"},{"location":"features/restart/","title":"Restarting Rollout Pods","text":"For various reasons, applications often need to be restarted, e.g. for hygiene purposes or to force startup logic to occur again such as reloading of a modified Secret. In these scenarios, it is undesirable to go through an entire blue-green or canary update process. Argo Rollouts supports the ability to restart all of its Pods by performing a rolling recreate of all the Pods in a Rollout while skipping the regular BlueGreen or Canary update strategy.
"},{"location":"features/restart/#how-it-works","title":"How it works","text":"A rollout can be restarted via the kubectl plugin, using the restart command:
kubectl-argo-rollouts restart ROLLOUT\n
Alternatively, if Rollouts is used with Argo CD, the there is a bundled \"restart\" action which can be performed via the Argo CD UI or CLI:
argocd app actions run my-app restart --kind Rollout --resource-name my-rollout\n
Both of these mechanisms updates the Rollout's .spec.restartAt to the current time in the form of a RFC 3339 formatted UTC string (e.g. 2020-03-30T21:19:35Z), which indicates to the Rollout controller that all of a Rollout's Pods should have been created after this timestamp.
During a restart, the controller iterates through each ReplicaSet to see if all the Pods have a creation timestamp which is newer than the restartAt time. For every pod older than the restartAt timestamp, the Pod will be evicted, allowing the ReplicaSet to replace the pod with a recreated one.
To prevent too many Pods from restarting at once, the controller limits itself to deleting up to maxUnavailable Pods at a time. Secondly, since pods are evicted and not deleted, the restart process will honor any PodDisruptionBudgets which are in place. The controller restarts ReplicaSets in the following order: 1. stable ReplicaSet 2. current ReplicaSet 3. all other ReplicaSets beginning with the oldest
If a Rollout's pod template spec (spec.template) is modified in the middle of a restart, the restart is canceled, and the normal blue-green or canary update will occur.
Note: Unlike deployments, where a \"restart\" is nothing but a normal rolling upgrade that happened to be triggered by a timestamp in the pod spec annotation, Argo Rollouts facilitates restarts by terminating pods and allowing the existing ReplicaSet to replace the terminated pods. This design choice was made in order to allow a restart to occur even when a Rollout was in the middle of a long-running blue-green/canary update (e.g. a paused canary). However, some consequences of this are:
- Restarting a Rollout which has a single replica will cause downtime since Argo Rollouts needs to terminate the pod in order to replace it.
- Restarting a rollout will be slower than a deployment's rolling update, since maxSurge is not used to bring up newer pods faster.
- maxUnavailable will be used to restart multiple pods at a time (starting in v0.10). But if maxUnavailable pods is 0, the controller will still restart pods one at a time.
"},{"location":"features/restart/#scheduled-restarts","title":"Scheduled Restarts","text":"Users can schedule a restart on their Rollout by setting the .spec.restartAt field to a time in the future. The controller only starts the restart after the current time is after the restartAt time.
"},{"location":"features/rollback/","title":"Rollback Windows","text":"Important
Available for blue-green and canary rollouts since v1.4
By default, when an older Rollout manifest is re-applied, the controller treats it the same as a spec change, and will execute the full list of steps, and perform analysis too. There are two exceptions to this rule: 1. the controller detects if it is moving back to a blue-green ReplicaSet which exists and is still scaled up (within its scaleDownDelay) 2. the controller detects it is moving back to the canary's \"stable\" ReplicaSet, and the upgrade had not yet completed.
It is often undesirable to re-run analysis and steps for a rollout, when the desired behavior is to rollback as soon as possible. To help with this, a rollback window feature allows users to indicate that the promotion to the ReplicaSet within the window will skip all steps.
Example:
spec:\n rollbackWindow:\n revisions: 3\n\n revisionHistoryLimit: 5\n
Assume a linear revision history: 1, 2, 3, 4, 5 (current). A rollback from revision 5 back to 4 or 3 will fall within the window, so it will be fast tracked.
"},{"location":"features/scaledown-aborted-rs/","title":"Scaledown New Replicaset on Aborted Rollout","text":"Upon an aborted update, we may scale down the new replicaset for all strategies. Users can then choose to leave the new replicaset scaled up indefinitely by setting abortScaleDownDelaySeconds to 0, or adjust the value to something larger (or smaller).
The following table summarizes the behavior under combinations of rollout strategy and abortScaleDownDelaySeconds. Note that abortScaleDownDelaySeconds is not applicable to argo-rollouts v1.0. abortScaleDownDelaySeconds = nil is the default, which means in v1.1 across all rollout strategies, the new replicaset is scaled down in 30 seconds on abort by default.
strategy v1.0 behavior abortScaleDownDelaySeconds v1.1 behavior blue-green does not scale down nil scales down after 30 seconds blue-green does not scale down 0 does not scale down blue-green does not scale down N scales down after N seconds basic canary rolling update back to stable N/A rolling update back to stable canary w/ traffic routing scales down immediately nil scales down after 30 seconds canary w/ traffic routing scales down immediately 0 does not scale down canary w/ traffic routing scales down immediately N scales down after N seconds canary w/ traffic routing + setCanaryScale does not scale down (bug) * should behave like canary w/ traffic routing"},{"location":"features/specification/","title":"Rollout Specification","text":"The following describes all the available fields of a rollout:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: example-rollout-canary\nspec:\n # Number of desired pods.\n # Defaults to 1.\n replicas: 5\n analysis:\n # limits the number of successful analysis runs and experiments to be stored in a history\n # Defaults to 5.\n successfulRunHistoryLimit: 10\n # limits the number of unsuccessful analysis runs and experiments to be stored in a history. \n # Stages for unsuccessful: \"Error\", \"Failed\", \"Inconclusive\"\n # Defaults to 5.\n unsuccessfulRunHistoryLimit: 10\n\n # Label selector for pods. Existing ReplicaSets whose pods are selected by\n # this will be the ones affected by this rollout. It must match the pod\n # template's labels.\n selector:\n matchLabels:\n app: guestbook\n\n # WorkloadRef holds a references to a workload that provides Pod template \n # (e.g. Deployment). If used, then do not use Rollout template property.\n workloadRef: \n apiVersion: apps/v1\n kind: Deployment\n name: rollout-ref-deployment\n # Specifies if the workload (Deployment) is scaled down after migrating to Rollout.\n # The possible options are:\n # \"never\": the Deployment is not scaled down\n # \"onsuccess\": the Deployment is scaled down after the Rollout becomes healthy\n # \"progressively\": as the Rollout is scaled up the Deployment is scaled down\n # If the Rollout fails the Deployment will be scaled back up.\n scaleDown: never|onsuccess|progressively\n\n # Template describes the pods that will be created. Same as deployment.\n # If used, then do not use Rollout workloadRef property. \n template:\n spec:\n containers:\n - name: guestbook\n image: argoproj/rollouts-demo:blue\n\n # Minimum number of seconds for which a newly created pod should be ready\n # without any of its container crashing, for it to be considered available.\n # Defaults to 0 (pod will be considered available as soon as it is ready)\n minReadySeconds: 30\n\n # The number of old ReplicaSets to retain.\n # Defaults to 10\n revisionHistoryLimit: 3\n\n # Pause allows a user to manually pause a rollout at any time. A rollout\n # will not advance through its steps while it is manually paused, but HPA\n # auto-scaling will still occur. Typically not explicitly set the manifest,\n # but controlled via tools (e.g. kubectl argo rollouts pause). If true at\n # initial creation of Rollout, replicas are not scaled up automatically\n # from zero unless manually promoted.\n paused: true\n\n # The maximum time in seconds in which a rollout must make progress during\n # an update, before it is considered to be failed. Argo Rollouts will\n # continue to process failed rollouts and a condition with a\n # ProgressDeadlineExceeded reason will be surfaced in the rollout status.\n # Note that progress will not be estimated during the time a rollout is\n # paused.\n # Defaults to 600s\n progressDeadlineSeconds: 600\n\n # Whether to abort the update when ProgressDeadlineSeconds is exceeded.\n # Optional and default is false.\n progressDeadlineAbort: false\n\n # UTC timestamp in which a Rollout should sequentially restart all of\n # its pods. Used by the `kubectl argo rollouts restart ROLLOUT` command.\n # The controller will ensure all pods have a creationTimestamp greater\n # than or equal to this value.\n restartAt: \"2020-03-30T21:19:35Z\"\n\n # The rollback window provides a way to fast track deployments to\n # previously deployed versions.\n # Optional, and by default is not set.\n rollbackWindow:\n revisions: 3\n\n strategy:\n\n # Blue-green update strategy\n blueGreen:\n\n # Reference to service that the rollout modifies as the active service.\n # Required.\n activeService: active-service\n\n # Pre-promotion analysis run which performs analysis before the service\n # cutover. +optional\n prePromotionAnalysis:\n templates:\n - templateName: success-rate\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n\n # Post-promotion analysis run which performs analysis after the service\n # cutover. +optional\n postPromotionAnalysis:\n templates:\n - templateName: success-rate\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n\n # Name of the service that the rollout modifies as the preview service.\n # +optional\n previewService: preview-service\n\n # The number of replicas to run under the preview service before the\n # switchover. Once the rollout is resumed the new ReplicaSet will be fully\n # scaled up before the switch occurs +optional\n previewReplicaCount: 1\n\n # Indicates if the rollout should automatically promote the new ReplicaSet\n # to the active service or enter a paused state. If not specified, the\n # default value is true. +optional\n autoPromotionEnabled: false\n\n # Automatically promotes the current ReplicaSet to active after the\n # specified pause delay in seconds after the ReplicaSet becomes ready.\n # If omitted, the Rollout enters and remains in a paused state until\n # manually resumed by resetting spec.Paused to false. +optional\n autoPromotionSeconds: 30\n\n # Adds a delay before scaling down the previous ReplicaSet. If omitted,\n # the Rollout waits 30 seconds before scaling down the previous ReplicaSet.\n # A minimum of 30 seconds is recommended to ensure IP table propagation\n # across the nodes in a cluster.\n scaleDownDelaySeconds: 30\n\n # Limits the number of old RS that can run at once before getting scaled\n # down. Defaults to nil\n scaleDownDelayRevisionLimit: 2\n\n # Add a delay in second before scaling down the preview replicaset\n # if update is aborted. 0 means not to scale down. Default is 30 second\n abortScaleDownDelaySeconds: 30\n\n # Anti Affinity configuration between desired and previous ReplicaSet.\n # Only one must be specified\n antiAffinity:\n requiredDuringSchedulingIgnoredDuringExecution: {}\n preferredDuringSchedulingIgnoredDuringExecution:\n weight: 1 # Between 1 - 100\n\n # activeMetadata will be merged and updated in-place into the ReplicaSet's spec.template.metadata\n # of the active pods. +optional\n activeMetadata:\n labels:\n role: active\n\n # Metadata which will be attached to the preview pods only during their preview phase.\n # +optional\n previewMetadata:\n labels:\n role: preview\n\n # Canary update strategy\n canary:\n\n # Reference to a service which the controller will update to select\n # canary pods. Required for traffic routing.\n canaryService: canary-service\n\n # Reference to a service which the controller will update to select\n # stable pods. Required for traffic routing.\n stableService: stable-service\n\n # Metadata which will be attached to the canary pods. This metadata will\n # only exist during an update, since there are no canary pods in a fully\n # promoted rollout.\n canaryMetadata:\n annotations:\n role: canary\n labels:\n role: canary\n\n # metadata which will be attached to the stable pods\n stableMetadata:\n annotations:\n role: stable\n labels:\n role: stable\n\n # The maximum number of pods that can be unavailable during the update.\n # Value can be an absolute number (ex: 5) or a percentage of total pods\n # at the start of update (ex: 10%). Absolute number is calculated from\n # percentage by rounding down. This can not be 0 if MaxSurge is 0. By\n # default, a fixed value of 1 is used. Example: when this is set to 30%,\n # the old RC can be scaled down by 30% immediately when the rolling\n # update starts. Once new pods are ready, old RC can be scaled down\n # further, followed by scaling up the new RC, ensuring that at least 70%\n # of original number of pods are available at all times during the\n # update. +optional\n maxUnavailable: 1\n\n # The maximum number of pods that can be scheduled above the original\n # number of pods. Value can be an absolute number (ex: 5) or a\n # percentage of total pods at the start of the update (ex: 10%). This\n # can not be 0 if MaxUnavailable is 0. Absolute number is calculated\n # from percentage by rounding up. By default, a value of 1 is used.\n # Example: when this is set to 30%, the new RC can be scaled up by 30%\n # immediately when the rolling update starts. Once old pods have been\n # killed, new RC can be scaled up further, ensuring that total number\n # of pods running at any time during the update is at most 130% of\n # original pods. +optional\n maxSurge: \"20%\"\n\n # Adds a delay before scaling down the previous ReplicaSet when the\n # canary strategy is used with traffic routing (default 30 seconds).\n # A delay in scaling down the previous ReplicaSet is needed after\n # switching the stable service selector to point to the new ReplicaSet,\n # in order to give time for traffic providers to re-target the new pods.\n # This value is ignored with basic, replica-weighted canary without\n # traffic routing.\n scaleDownDelaySeconds: 30\n\n # The minimum number of pods that will be requested for each ReplicaSet\n # when using traffic routed canary. This is to ensure high availability\n # of each ReplicaSet. Defaults to 1. +optional\n minPodsPerReplicaSet: 2\n\n # Limits the number of old RS that can run at one time before getting\n # scaled down. Defaults to nil\n scaleDownDelayRevisionLimit: 2\n\n # Background analysis to run during a rollout update. Skipped upon\n # initial deploy of a rollout. +optional\n analysis:\n templates:\n - templateName: success-rate\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n\n # valueFrom.podTemplateHashValue is a convenience to supply the\n # rollouts-pod-template-hash value of either the Stable ReplicaSet\n # or the Latest ReplicaSet\n - name: stable-hash\n valueFrom:\n podTemplateHashValue: Stable\n - name: latest-hash\n valueFrom:\n podTemplateHashValue: Latest\n\n # valueFrom.fieldRef allows metadata about the rollout to be\n # supplied as arguments to analysis.\n - name: region\n valueFrom:\n fieldRef:\n fieldPath: metadata.labels['region']\n\n # Steps define sequence of steps to take during an update of the\n # canary. Skipped upon initial deploy of a rollout. +optional\n steps:\n\n # Sets the ratio of canary ReplicaSet to 20%\n - setWeight: 20\n\n # Pauses the rollout for an hour. Supported units: s, m, h\n - pause:\n duration: 1h\n\n # Pauses indefinitely until manually resumed\n - pause: {}\n\n # set canary scale to a explicit count without changing traffic weight\n # (supported only with trafficRouting)\n - setCanaryScale:\n replicas: 3\n\n # set canary scale to a percentage of spec.replicas without changing traffic weight\n # (supported only with trafficRouting)\n - setCanaryScale:\n weight: 25\n\n # set canary scale to match the canary traffic weight (default behavior)\n - setCanaryScale:\n matchTrafficWeight: true\n\n # Sets header based route with specified header values\n # Setting header based route will send all traffic to the canary for the requests \n # with a specified header, in this case request header \"version\":\"2\"\n # (supported only with trafficRouting, for Istio only at the moment)\n - setHeaderRoute:\n # Name of the route that will be created by argo rollouts this must also be configured\n # in spec.strategy.canary.trafficRouting.managedRoutes\n name: \"header-route-1\"\n # The matching rules for the header route, if this is missing it acts as a removal of the route.\n match:\n # headerName The name of the header to apply the match rules to.\n - headerName: \"version\"\n # headerValue must contain exactly one field of exact, regex, or prefix. Not all traffic routers support \n # all types\n headerValue:\n # Exact will only match if the header value is exactly the same\n exact: \"2\"\n # Will match the rule if the regular expression matches\n regex: \"2.0.(.*)\"\n # prefix will be a prefix match of the header value\n prefix: \"2.0\"\n\n # Sets up a mirror/shadow based route with the specified match rules\n # The traffic will be mirrored at the configured percentage to the canary service\n # during the rollout\n # (supported only with trafficRouting, for Istio only at the moment)\n - setMirrorRoute:\n # Name of the route that will be created by argo rollouts this must also be configured\n # in spec.strategy.canary.trafficRouting.managedRoutes\n name: \"header-route-1\"\n # The percentage of the matched traffic to mirror to the canary\n percentage: 100\n # The matching rules for the header route, if this is missing it acts as a removal of the route.\n # All conditions inside a single match block have AND semantics, while the list of match blocks have OR semantics.\n # Each type within a match (method, path, headers) must have one and only one match type (exact, regex, prefix)\n # Not all match types (exact, regex, prefix) will be supported by all traffic routers.\n match:\n - method: # What HTTP method to match\n exact: \"GET\"\n regex: \"P.*\"\n prefix: \"POST\"\n path: # What HTTP url paths to match.\n exact: \"/test\"\n regex: \"/test/.*\"\n prefix: \"/\"\n headers:\n agent-1b: # What HTTP header name to use in the match.\n exact: \"firefox\"\n regex: \"firefox2(.*)\"\n prefix: \"firefox\"\n\n # an inline analysis step\n - analysis:\n templates:\n - templateName: success-rate\n\n # an inline experiment step\n - experiment:\n duration: 1h\n templates:\n - name: baseline\n specRef: stable\n # optional, creates a service for the experiment if set\n service:\n # optional, service: {} is also acceptable if name is not included\n name: test-service\n - name: canary\n specRef: canary\n # optional, set the weight of traffic routed to this version\n weight: 10\n analyses:\n - name : mann-whitney\n templateName: mann-whitney\n # Metadata which will be attached to the AnalysisRun.\n analysisRunMetadata:\n labels:\n app.service.io/analysisType: smoke-test\n annotations:\n link.argocd.argoproj.io/external-link: http://my-loggin-platform.com/pre-generated-link\n\n # Anti-affinity configuration between desired and previous ReplicaSet.\n # Only one must be specified.\n antiAffinity:\n requiredDuringSchedulingIgnoredDuringExecution: {}\n preferredDuringSchedulingIgnoredDuringExecution:\n weight: 1 # Between 1 - 100\n\n # Traffic routing specifies the ingress controller or service mesh\n # configuration to achieve advanced traffic splitting. If omitted,\n # will achieve traffic split via a weighted replica counts between\n # the canary and stable ReplicaSet.\n trafficRouting:\n # Supports nginx and plugins only: This lets you control the denominator or total weight of traffic.\n # The total weight of traffic. If unspecified, it defaults to 100\n maxTrafficWeight: 1000\n # This is a list of routes that Argo Rollouts has the rights to manage it is currently only required for\n # setMirrorRoute and setHeaderRoute. The order of managedRoutes array also sets the precedence of the route\n # in the traffic router. Argo Rollouts will place these routes in the order specified above any routes already\n # defined in the used traffic router if something exists. The names here must match the names from the \n # setHeaderRoute and setMirrorRoute steps.\n managedRoutes:\n - name: set-header\n - name: mirror-route\n # Istio traffic routing configuration\n istio:\n # Either virtualService or virtualServices can be configured.\n virtualService: \n name: rollout-vsvc # required\n routes:\n - primary # optional if there is a single route in VirtualService, required otherwise\n virtualServices:\n # One or more virtualServices can be configured\n - name: rollouts-vsvc1 # required\n routes:\n - primary # optional if there is a single route in VirtualService, required otherwise\n - name: rollouts-vsvc2 # required\n routes:\n - secondary # optional if there is a single route in VirtualService, required otherwise\n\n # NGINX Ingress Controller routing configuration\n nginx:\n # Either stableIngress or stableIngresses must be configured, but not both.\n stableIngress: primary-ingress\n stableIngresses:\n - primary-ingress\n - secondary-ingress\n - tertiary-ingress\n annotationPrefix: customingress.nginx.ingress.kubernetes.io # optional\n additionalIngressAnnotations: # optional\n canary-by-header: X-Canary\n canary-by-header-value: iwantsit\n\n # ALB Ingress Controller routing configuration\n alb:\n ingress: ingress # required\n servicePort: 443 # required\n annotationPrefix: custom.alb.ingress.kubernetes.io # optional\n\n # Service Mesh Interface routing configuration\n smi:\n rootService: root-svc # optional\n trafficSplitName: rollout-example-traffic-split # optional\n\n # Add a delay in second before scaling down the canary pods when update\n # is aborted for canary strategy with traffic routing (not applicable for basic canary).\n # 0 means canary pods are not scaled down. Default is 30 seconds.\n abortScaleDownDelaySeconds: 30\n\nstatus:\n pauseConditions:\n - reason: StepPause\n startTime: 2019-10-00T1234\n - reason: BlueGreenPause\n startTime: 2019-10-00T1234\n - reason: AnalysisRunInconclusive\n startTime: 2019-10-00T1234 \n
"},{"location":"features/specification/#examples","title":"Examples","text":"You can find examples of Rollouts at:
- The example directory
- The Argo Rollouts Demo application
"},{"location":"features/vpa-support/","title":"Vertical Pod Autoscaling","text":"Vertical Pod Autoscaling (VPA) reduces the maintenance cost and improve utilization of cluster resources by automating configuration of resource requirements.
"},{"location":"features/vpa-support/#vpa-modes","title":"VPA modes","text":"There are four modes in which VPAs operate
-
\"Auto\": VPA assigns resource requests on pod creation as well as updates them on existing pods using the preferred update mechanism. Currently this is equivalent to \"Recreate\" (see below). Once restart free (\"in-place\") update of pod requests is available, it may be used as the preferred update mechanism by the \"Auto\" mode. NOTE: This feature of VPA is experimental and may cause downtime for your applications.
-
\"Recreate\": VPA assigns resource requests on pod creation as well as updates them on existing pods by evicting them when the requested resources differ significantly from the new recommendation (respecting the Pod Disruption Budget, if defined). This mode should be used rarely, only if you need to ensure that the pods are restarted whenever the resource request changes. Otherwise prefer the \"Auto\" mode which may take advantage of restart free updates once they are available. NOTE: This feature of VPA is experimental and may cause downtime for your applications.
-
\"Initial\": VPA only assigns resource requests on pod creation and never changes them later.
-
\"Off\": VPA does not automatically change resource requirements of the pods. The recommendations are calculated and can be inspected in the VPA object.
"},{"location":"features/vpa-support/#example","title":"Example","text":"Below is an example of a Vertical Pod Autoscaler with Argo-Rollouts.
Rollout sample app:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: vpa-demo-rollout\n namespace: test-vpa\nspec:\n replicas: 5\n strategy:\n canary:\n steps:\n - setWeight: 20\n - pause: {duration: 10}\n - setWeight: 40\n - pause: {duration: 10}\n - setWeight: 60\n - pause: {duration: 10}\n - setWeight: 80\n - pause: {duration: 10}\n revisionHistoryLimit: 10\n selector:\n matchLabels:\n app: vpa-demo-rollout\n template:\n metadata:\n labels:\n app: vpa-demo-rollout\n spec:\n containers:\n - name: vpa-demo-rollout\n image: ravihari/nginx:v1\n ports:\n - containerPort: 80\n resources:\n requests:\n cpu: \"5m\" \n memory: \"5Mi\" \n
VPA configuration for Rollout sample app:
apiVersion: \"autoscaling.k8s.io/v1beta2\"\nkind: VerticalPodAutoscaler \nmetadata: \n name: vpa-rollout-example \n namespace: test-vpa \nspec: \n targetRef: \n apiVersion: \"argoproj.io/v1alpha1\" \n kind: Rollout \n name: vpa-demo-rollout \n updatePolicy: \n updateMode: \"Auto\" \n resourcePolicy: \n containerPolicies: \n - containerName: '*' \n minAllowed: \n cpu: 5m \n memory: 5Mi \n maxAllowed: \n cpu: 1 \n memory: 500Mi \n controlledResources: [\"cpu\", \"memory\"] \n
Describe VPA when initially deployed we donot see recommendations as it will take few mins.
Name: kubengix-vpa\nNamespace: test-vpa\nLabels: <none>\nAnnotations: <none>\nAPI Version: autoscaling.k8s.io/v1\nKind: VerticalPodAutoscaler\nMetadata:\n Creation Timestamp: 2022-03-14T12:54:06Z\n Generation: 1\n Managed Fields:\n API Version: autoscaling.k8s.io/v1beta2\n Fields Type: FieldsV1\n fieldsV1:\n f:metadata:\n f:annotations:\n .:\n f:kubectl.kubernetes.io/last-applied-configuration:\n f:spec:\n .:\n f:resourcePolicy:\n .:\n f:containerPolicies:\n f:targetRef:\n .:\n f:apiVersion:\n f:kind:\n f:name:\n f:updatePolicy:\n .:\n f:updateMode:\n Manager: kubectl-client-side-apply\n Operation: Update\n Time: 2022-03-14T12:54:06Z\n Resource Version: 3886\n UID: 4ac64e4c-c84b-478e-92e4-5f072f985971\nSpec:\n Resource Policy:\n Container Policies:\n Container Name: *\n Controlled Resources:\n cpu\n memory\n Max Allowed:\n Cpu: 1\n Memory: 500Mi\n Min Allowed:\n Cpu: 5m\n Memory: 5Mi\n Target Ref:\n API Version: argoproj.io/v1alpha1\n Kind: Rollout\n Name: vpa-demo-rollout\n Update Policy:\n Update Mode: Auto\nEvents: <none>\n
After few minutes when VPA starts to process and provide recommendation:
Name: kubengix-vpa\nNamespace: test-vpa\nLabels: <none>\nAnnotations: <none>\nAPI Version: autoscaling.k8s.io/v1\nKind: VerticalPodAutoscaler\nMetadata:\n Creation Timestamp: 2022-03-14T12:54:06Z\n Generation: 2\n Managed Fields:\n API Version: autoscaling.k8s.io/v1beta2\n Fields Type: FieldsV1\n fieldsV1:\n f:metadata:\n f:annotations:\n .:\n f:kubectl.kubernetes.io/last-applied-configuration:\n f:spec:\n .:\n f:resourcePolicy:\n .:\n f:containerPolicies:\n f:targetRef:\n .:\n f:apiVersion:\n f:kind:\n f:name:\n f:updatePolicy:\n .:\n f:updateMode:\n Manager: kubectl-client-side-apply\n Operation: Update\n Time: 2022-03-14T12:54:06Z\n API Version: autoscaling.k8s.io/v1\n Fields Type: FieldsV1\n fieldsV1:\n f:status:\n .:\n f:conditions:\n f:recommendation:\n .:\n f:containerRecommendations:\n Manager: recommender\n Operation: Update\n Time: 2022-03-14T12:54:52Z\n Resource Version: 3950\n UID: 4ac64e4c-c84b-478e-92e4-5f072f985971\nSpec:\n Resource Policy:\n Container Policies:\n Container Name: *\n Controlled Resources:\n cpu\n memory\n Max Allowed:\n Cpu: 1\n Memory: 500Mi\n Min Allowed:\n Cpu: 5m\n Memory: 5Mi\n Target Ref:\n API Version: argoproj.io/v1alpha1\n Kind: Rollout\n Name: vpa-demo-rollout\n Update Policy:\n Update Mode: Auto\nStatus:\n Conditions:\n Last Transition Time: 2022-03-14T12:54:52Z\n Status: True\n Type: RecommendationProvided\n Recommendation:\n Container Recommendations:\n Container Name: vpa-demo-rollout\n Lower Bound:\n Cpu: 25m\n Memory: 262144k\n Target:\n Cpu: 25m\n Memory: 262144k\n Uncapped Target:\n Cpu: 25m\n Memory: 262144k\n Upper Bound:\n Cpu: 1\n Memory: 500Mi\nEvents: <none>\n
Here we see the recommendation for cpu, memory with lowerbound, upper bound, Target etc., are provided. If we check the status of the pods.. the older pods with initial configuration would get terminated and newer pods get created.
# kubectl get po -n test-vpa -w \nNAME READY STATUS RESTARTS AGE\nvpa-demo-rollout-f5df6d577-65f26 1/1 Running 0 17m\nvpa-demo-rollout-f5df6d577-d55cx 1/1 Running 0 17m\nvpa-demo-rollout-f5df6d577-fdpn2 1/1 Running 0 17m\nvpa-demo-rollout-f5df6d577-jg2pw 1/1 Running 0 17m\nvpa-demo-rollout-f5df6d577-vlx5x 1/1 Running 0 17m\n...\n\nvpa-demo-rollout-f5df6d577-jg2pw 1/1 Terminating 0 17m\nvpa-demo-rollout-f5df6d577-vlx5x 1/1 Terminating 0 17m\nvpa-demo-rollout-f5df6d577-jg2pw 0/1 Terminating 0 18m\nvpa-demo-rollout-f5df6d577-vlx5x 0/1 Terminating 0 18m\nvpa-demo-rollout-f5df6d577-w7tx4 0/1 Pending 0 0s\nvpa-demo-rollout-f5df6d577-w7tx4 0/1 Pending 0 0s\nvpa-demo-rollout-f5df6d577-w7tx4 0/1 ContainerCreating 0 0s\nvpa-demo-rollout-f5df6d577-vdlqq 0/1 Pending 0 0s\nvpa-demo-rollout-f5df6d577-vdlqq 0/1 Pending 0 1s\nvpa-demo-rollout-f5df6d577-jg2pw 0/1 Terminating 0 18m\nvpa-demo-rollout-f5df6d577-jg2pw 0/1 Terminating 0 18m\nvpa-demo-rollout-f5df6d577-vdlqq 0/1 ContainerCreating 0 1s\nvpa-demo-rollout-f5df6d577-w7tx4 1/1 Running 0 6s\nvpa-demo-rollout-f5df6d577-vdlqq 1/1 Running 0 7s\nvpa-demo-rollout-f5df6d577-vlx5x 0/1 Terminating 0 18m\nvpa-demo-rollout-f5df6d577-vlx5x 0/1 Terminating 0 18m\n
If we check the new pod cpu and memory they would be updated as per VPA recommendation:
# kubectl describe po vpa-demo-rollout-f5df6d577-vdlqq -n test-vpa\nName: vpa-demo-rollout-f5df6d577-vdlqq\nNamespace: test-vpa\nPriority: 0\nNode: argo-rollouts-control-plane/172.18.0.2\nStart Time: Mon, 14 Mar 2022 12:55:06 +0000\nLabels: app=vpa-demo-rollout\n rollouts-pod-template-hash=f5df6d577\nAnnotations: vpaObservedContainers: vpa-demo-rollout\n vpaUpdates: Pod resources updated by kubengix-vpa: container 0: cpu request, memory request\nStatus: Running\nIP: 10.244.0.17\nIPs:\n IP: 10.244.0.17\nControlled By: ReplicaSet/vpa-demo-rollout-f5df6d577\nContainers:\n vpa-demo-rollout:\n Container ID: containerd://b79bd88851fe0622d33bc90a1560ca54ef2c27405a3bc9a4fc3a333eef5f9733\n Image: ravihari/nginx:v1\n Image ID: docker.io/ravihari/nginx@sha256:205961b09a80476af4c2379841bf6abec0022101a7e6c5585a88316f7115d17a\n Port: 80/TCP\n Host Port: 0/TCP\n State: Running\n Started: Mon, 14 Mar 2022 12:55:11 +0000\n Ready: True\n Restart Count: 0\n Requests:\n cpu: 25m\n memory: 262144k\n Environment: <none>\n Mounts:\n /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mk4fz (ro)\nConditions:\n Type Status\n Initialized True \n Ready True \n ContainersReady True \n PodScheduled True \nVolumes:\n kube-api-access-mk4fz:\n Type: Projected (a volume that contains injected data from multiple sources)\n TokenExpirationSeconds: 3607\n ConfigMapName: kube-root-ca.crt\n ConfigMapOptional: <nil>\n DownwardAPI: true\nQoS Class: Burstable\nNode-Selectors: <none>\nTolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s\n node.kubernetes.io/unreachable:NoExecute op=Exists for 300s\nEvents:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal Scheduled 38s default-scheduler Successfully assigned test-vpa/vpa-demo-rollout-f5df6d577-vdlqq to argo-rollouts-control-plane\n Normal Pulled 35s kubelet Container image \"ravihari/nginx:v1\" already present on machine\n Normal Created 35s kubelet Created container vpa-demo-rollout\n Normal Started 33s kubelet Started container vpa-demo-rollout\n
"},{"location":"features/vpa-support/#requirements","title":"Requirements","text":"In order for the VPA to manipulate the rollout, the Kubernetes cluster hosting the rollout CRD needs the subresources support for CRDs. This feature was introduced as alpha in Kubernetes version 1.10 and transitioned to beta in Kubernetes version 1.11. If a user wants to use VPA on v1.10, the Kubernetes Cluster operator will need to add a custom feature flag to the API server. After 1.10, the flag is turned on by default. Check out the following link for more information on setting the custom feature flag.
When installing VPA you may need to add the following in RBAC configurations for system:vpa-target-reader cluster role as by default VPA maynot support rollouts in all the versions.
- apiGroups:\n - argoproj.io\n resources:\n - rollouts\n - rollouts/scale\n - rollouts/status\n - replicasets\n verbs:\n - get\n - list\n - watch\n
Makes sure Metrics-Server is installed in the cluster and openssl is upto date for VPA latest version to apply recommendations to the pods properly.
"},{"location":"features/anti-affinity/anti-affinity/","title":"Anti Affinity","text":""},{"location":"features/anti-affinity/anti-affinity/#background","title":"Background","text":"Depending on a cluster's configuration, a Blue Green Rollout (or a Canary rollout that uses traffic management) can cause newly created pods to restart after deploying a new version. This can be problematic, especially for applications that cannot startup quickly or do not gracefully exit.
This behavior occurs because cluster auto-scaler wants to scale down the extra capacity which was created to support a Rollout running in double capacity. When a node is scaled down, the pods it owns are deleted and recreated. This usually happens if a Rollout has its own dedicated instance group since a Rollout has a greater effect on cluster auto-scaling. Therefore, clusters with a large pool of shared nodes experience the behavior less often.
For example, here is a Rollout is running with 8 pods spread across 2 nodes. Each node can hold 6 pods:
When the spec.template of the Rollout changes, the controller creates a new ReplicaSet with the spec update and the total number of pods doubles. In this case, the number of pods increases to 16.
Since each node can only hold 6 pods, the cluster autoscaler must increase the node count to 3 to accommodate all 16 pods. The resulting distribution of pods across nodes is shown here:
Once the Rollout finishes progressing, the old version is scaled down. This leaves the cluster with more nodes than necessary, thus wasting resources (as shown below).
The cluster auto-scaler terminates the extra node and the pods are rescheduled on the remaining 2 nodes.
To reduce the chance of this behavior, a rollout can inject anti-affinity into the ReplicaSet. This prevents new pods from running on nodes which have the previous version's pods.
You can learn more about anti-affinity here.
Repeating the above example with anti-affinity enabled, here is what happens when the .spec.template of the Rollout changes. Due to anti-affinity, the new pods cannot be scheduled on nodes which run the old ReplicaSet's pods. As a result, the cluster auto-scaler must create 2 nodes to host the new ReplicaSet's pods. In this case, pods won't be started since the scaled-down nodes are guaranteed to not have the new pods.
"},{"location":"features/anti-affinity/anti-affinity/#enabling-anti-affinity-in-rollouts","title":"Enabling Anti-Affinity in Rollouts","text":"Anti-affinity is enabled by adding the anti-affinity struct to the Blue-Green or Canary strategy. When the anti-affinity struct is set, controller injects a PodAntiAffinity struct into the ReplicaSet's Affinity. This feature will not modify any of the ReplicaSet's pre-existing affinity rules.
Users have a choice between these scheduling rules: RequiredDuringSchedulingIgnoredDuringExecution and PreferredDuringSchedulingIgnoredDuringExecution.
RequiredDuringSchedulingIgnoredDuringExecution requires a new version's pods to be on a separate node than the previous versions. If this is not possible, the the new version's pods will not be scheduled.
strategy:\n bluegreen:\n antiAffinity:\n requiredDuringSchedulingIgnoredDuringExecution: {}\n
Unlike the Required strategy, PreferredDuringSchedulingIgnoredDuringExecution does not force a new version's pods to be on a separate node than the previous versions. The scheduler attempts to place the new version's pods on separate node(s). If that's not possible, the new version's pods will still be scheduled. The Weight is used to create a priority order for preferred anti-affinity rules.
strategy:\n canary:\n antiAffinity:\n preferredDuringSchedulingIgnoredDuringExecution:\n weight: 1 # Between 1 - 100\n
Important
The main downside to this approach is that deployments can take longer because new nodes are more likely to be created in order to schedule pods with respect to anti-affinity rules. This delay most frequently occurs when a rollout has its own dedicated instance group, since new nodes are more likely to be created to honor anti-affinity rules.
"},{"location":"features/traffic-management/","title":"Traffic management","text":"Traffic management is controlling the data plane to have intelligent routing rules for an application. These routing rules can manipulate the flow of traffic to different versions of an application enabling Progressive Delivery. These controls limit the blast radius of a new release by ensuring a small percentage of users receive a new version while it is verified.
There are various techniques to achieve traffic management:
- Raw percentages (i.e., 5% of traffic should go to the new version while the rest goes to the stable version)
- Header-based routing (i.e., send requests with a specific header to the new version)
- Mirrored traffic where all the traffic is copied and send to the new version in parallel (but the response is ignored)
"},{"location":"features/traffic-management/#traffic-management-tools-in-kubernetes","title":"Traffic Management tools in Kubernetes","text":"The core Kubernetes objects do not have fine-grained tools needed to fulfill all the requirements of traffic management. At most, Kubernetes offers native load balancing capabilities through the Service object by offering an endpoint that routes traffic to a grouping of pods based on that Service's selector. Functionality like traffic mirroring or routing by headers is not possible with the default core Service object, and the only way to control the percentage of traffic to different versions of an application is by manipulating replica counts of those versions.
Service Meshes fill this missing functionality in Kubernetes. They introduce new concepts and functionality to control the data plane through the use of CRDs and other core Kubernetes resources.
"},{"location":"features/traffic-management/#how-does-argo-rollouts-enable-traffic-management","title":"How does Argo Rollouts enable traffic management?","text":"Argo Rollouts enables traffic management by manipulating the Service Mesh resources to match the intent of the Rollout. Argo Rollouts currently supports the following traffic providers:
- AWS ALB Ingress Controller
- Ambassador Edge Stack
- Apache APISIX
- Google Cloud
- Gateway API
- Istio
- Kong Ingress
- Nginx Ingress Controller
- Service Mesh Interface (SMI)
- Traefik Proxy
- Multiple Providers
- File a ticket here if you would like another implementation (or thumbs up it if that issue already exists)
Regardless of the Service Mesh used, the Rollout object has to set a canary Service and a stable Service in its spec. Here is an example with those fields set:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nspec:\n ...\n strategy:\n canary:\n canaryService: canary-service\n stableService: stable-service\n trafficRouting:\n ...\n
The controller modifies these Services to route traffic to the appropriate canary and stable ReplicaSets as the Rollout progresses. These Services are used by the Service Mesh to define what group of pods should receive the canary and stable traffic.
Additionally, the Argo Rollouts controller needs to treat the Rollout object differently when using traffic management. In particular, the Stable ReplicaSet owned by the Rollout remains fully scaled up as the Rollout progresses through the Canary steps.
Since the traffic is controlled independently by the Service Mesh resources, the controller needs to make a best effort to ensure that the Stable and New ReplicaSets are not overwhelmed by the traffic sent to them. By leaving the Stable ReplicaSet scaled up, the controller is ensuring that the Stable ReplicaSet can handle 100% of the traffic at any time1. The New ReplicaSet follows the same behavior as without traffic management. The new ReplicaSet's replica count is equal to the latest SetWeight step percentage multiple by the total replica count of the Rollout. This calculation ensures that the canary version does not receive more traffic than it can handle.
"},{"location":"features/traffic-management/#traffic-routing-with-managed-routes-and-route-precedence","title":"Traffic routing with managed routes and route precedence","text":""},{"location":"features/traffic-management/#traffic-router-support-istio","title":"Traffic router support: (Istio)","text":"When traffic routing is enabled, you have the ability to also let argo rollouts add and manage other routes besides just controlling the traffic weight to the canary. Two such routing rules are header and mirror based routes. When using these routes we also have to set a route precedence with the upstream traffic router. We do this using the spec.strategy.canary.trafficRouting.managedRoutes field which is an array the order of the items in the array determine the precedence. This set of routes will also be placed in the order specified on top of any other routes defined manually.
Warning
All routes listed in managed routes will be removed at the end of a rollout or on an abort. Do not put any manually created routes in the list.
Here is an example:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nspec:\n ...\n strategy:\n canary:\n ...\n trafficRouting:\n managedRoutes:\n - name: priority-route-1\n - name: priority-route-2\n - name: priority-route-3\n
"},{"location":"features/traffic-management/#traffic-routing-based-on-a-header-values-for-canary","title":"Traffic routing based on a header values for Canary","text":""},{"location":"features/traffic-management/#traffic-router-support-istio_1","title":"Traffic router support: (Istio)","text":"Argo Rollouts has ability to send all traffic to the canary-service based on a http request header value. The step for the header based traffic routing is setHeaderRoute and has a list of matchers for the header.
name - name of the header route.
match - header matching rules is an array of headerName, headerValue pairs.
headerName - name of the header to match.
headerValue- contains exactly one of exact - specify the exact header value, regex - value in a regex format, prefix - the prefix of the value could be provided. Not all traffic routers will support all match types.
To disable header based traffic routing just need to specify empty setHeaderRoute with only the name of the route.
Example:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nspec:\n ...\n strategy:\n canary:\n canaryService: canary-service\n stableService: stable-service\n trafficRouting:\n managedRoutes:\n - name: set-header-1\n istio:\n virtualService:\n name: rollouts-demo-vsvc\n steps:\n - setWeight: 20\n - setHeaderRoute: # enable header based traffic routing where\n name: \"set-header-1\"\n match:\n - headerName: Custom-Header1 # Custom-Header1=Mozilla\n headerValue:\n exact: Mozilla\n - headerName: Custom-Header2 # or Custom-Header2 has a prefix Mozilla\n headerValue:\n prefix: Mozilla\n - headerName: Custom-Header3 # or Custom-Header3 value match regex: Mozilla(.*)\n headerValue:\n regex: Mozilla(.*)\n - pause: {}\n - setHeaderRoute:\n name: \"set-header-1\" # disable header based traffic routing\n
"},{"location":"features/traffic-management/#traffic-routing-mirroring-traffic-to-canary","title":"Traffic routing mirroring traffic to canary","text":""},{"location":"features/traffic-management/#traffic-router-support-istio_2","title":"Traffic router support: (Istio)","text":"Argo Rollouts has ability to mirror traffic to the canary-service based on a various matching rules. The step for the mirror based traffic routing is setMirrorRoute and has a list of matchers for the header.
name - name of the mirror route.
percentage - what percentage of the matched traffic to mirror
match - The matching rules for the header route, if this is missing it acts as a removal of the route. All conditions inside a single match block have AND semantics, while the list of match blocks have OR semantics. Each type within a match (method, path, headers) must have one and only one match type (exact, regex, prefix) Not all match types (exact, regex, prefix) will be supported by all traffic routers.
To disable mirror based traffic route you just need to specify a setMirrorRoute with only the name of the route.
This example will mirror 35% of HTTP traffic that matches a GET requests and with the url prefix of /
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nspec:\n ...\n strategy:\n canary:\n canaryService: canary-service\n stableService: stable-service\n trafficRouting:\n managedRoutes:\n - name: mirror-route\n istio:\n virtualService:\n name: rollouts-demo-vsvc\n steps:\n - setCanaryScale:\n weight: 25\n - setMirrorRoute:\n name: mirror-route\n percentage: 35\n match:\n - method:\n exact: GET\n path:\n prefix: /\n - pause:\n duration: 10m\n - setMirrorRoute:\n name: \"mirror-route\" # removes mirror based traffic route\n
-
The Rollout has to assume that the application can handle 100% of traffic if it is fully scaled up. It should outsource to the HPA to detect if the Rollout needs to more replicas if 100% isn't enough.\u00a0\u21a9
"},{"location":"features/traffic-management/alb/","title":"AWS Load Balancer Controller (ALB)","text":""},{"location":"features/traffic-management/alb/#requirements","title":"Requirements","text":" - AWS Load Balancer Controller v1.1.5 or greater
"},{"location":"features/traffic-management/alb/#overview","title":"Overview","text":"AWS Load Balancer Controller (also known as AWS ALB Ingress Controller) enables traffic management through an Ingress object, which configures an AWS Application Load Balancer (ALB) to route traffic to one or more Kubernetes services. ALBs provides advanced traffic splitting capability through the concept of weighted target groups. This feature is supported by the AWS Load Balancer Controller through annotations made to the Ingress object to configure \"actions.\"
"},{"location":"features/traffic-management/alb/#how-it-works","title":"How it works","text":"ALBs are configured via Listeners, and Rules which contain Actions. Listeners define how traffic from a client comes in, and Rules define how to handle those requests with various Actions. One type of Action allows users to forward traffic to multiple TargetGroups (with each being defined as a Kubernetes service). You can read more about ALB concepts here.
An Ingress which is managed by the AWS Load Balancer Controller, controls an ALB's Listener and Rules through the Ingress' annotations and spec. In order to split traffic among multiple target groups (e.g. different Kubernetes services), the AWS Load Balancer controller looks to a specific \"action\" annotation on the Ingress, alb.ingress.kubernetes.io/actions.<service-name>. This annotation is injected and updated automatically by a Rollout during an update according to the desired traffic weights.
"},{"location":"features/traffic-management/alb/#usage","title":"Usage","text":"To configure a Rollout to use the ALB integration and split traffic between the canary and stable services during updates, the Rollout should be configured with the following fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\n...\nspec:\n strategy:\n canary:\n # canaryService and stableService are references to Services which the Rollout will modify\n # to target the canary ReplicaSet and stable ReplicaSet respectively (required).\n canaryService: canary-service\n stableService: stable-service\n trafficRouting:\n alb:\n # The referenced ingress will be injected with a custom action annotation, directing\n # the AWS Load Balancer Controller to split traffic between the canary and stable\n # Service, according to the desired traffic weight (required).\n ingress: ingress\n # If you want to controll multiple ingress resources you can use the ingresses field, if ingresses is specified\n # the ingress field will need to be omitted.\n ingresses:\n - ingress-1\n - ingress-2\n # Reference to a Service that the Ingress must target in one of the rules (optional).\n # If omitted, uses canary.stableService.\n rootService: root-service\n # Service port is the port which the Service listens on (required).\n servicePort: 443\n
The referenced Ingress should be deployed with an ingress rule that matches the Rollout service:
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n name: ingress\n annotations:\n kubernetes.io/ingress.class: alb\nspec:\n rules:\n - http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n # serviceName must match either: canary.trafficRouting.alb.rootService (if specified),\n # or canary.stableService (if rootService is omitted)\n name: root-service\n # servicePort must be the value: use-annotation\n # This instructs AWS Load Balancer Controller to look to annotations on how to direct traffic\n port:\n name: use-annotation\n
During an update, the rollout controller injects the alb.ingress.kubernetes.io/actions.<SERVICE-NAME> annotation, containing a JSON payload understood by the AWS Load Balancer Controller, directing it to split traffic between the canaryService and stableService according to the current canary weight.
The following is an example of our example Ingress after the rollout has injected the custom action annotation that splits traffic between the canary-service and stable-service, with a traffic weight of 10 and 90 respectively:
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n name: ingress\n annotations:\n kubernetes.io/ingress.class: alb\n alb.ingress.kubernetes.io/actions.root-service: |\n {\n \"Type\":\"forward\",\n \"ForwardConfig\":{\n \"TargetGroups\":[\n {\n \"Weight\":10,\n \"ServiceName\":\"canary-service\",\n \"ServicePort\":\"80\"\n },\n {\n \"Weight\":90,\n \"ServiceName\":\"stable-service\",\n \"ServicePort\":\"80\"\n }\n ]\n }\n }\nspec:\n rules:\n - http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n name: root-service\n port:\n name: use-annotation\n
Note
Argo rollouts additionally injects an annotation, rollouts.argoproj.io/managed-alb-actions, to the Ingress for bookkeeping purposes. The annotation indicates which actions are being managed by the Rollout object (since multiple Rollouts can reference one Ingress). Upon a rollout deletion, the rollout controller looks to this annotation to understand that this action is no longer managed, and is reset to point only the stable service with 100 weight.
"},{"location":"features/traffic-management/alb/#rootservice","title":"rootService","text":"By default, a rollout will inject the alb.ingress.kubernetes.io/actions.<SERVICE-NAME> annotation using the service/action name specified under spec.strategy.canary.stableService. However, it may be desirable to specify an explicit service/action name different from the stableService. For example, one pattern is to use a single Ingress containing three different rules to reach the canary, stable, and root service separately (e.g. for testing purposes). In this case, you may want to specify a \"root\" service as the service/action name instead of stable. To do so, reference a service under rootService under the alb specification:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nspec:\n strategy:\n canary:\n canaryService: guestbook-canary\n stableService: guestbook-stable\n trafficRouting:\n alb:\n rootService: guestbook-root\n...\n
"},{"location":"features/traffic-management/alb/#sticky-session","title":"Sticky session","text":"Because at least two target groups (canary and stable) are used, target group stickiness requires additional configuration: Sticky session must be activated on the target group via
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nspec:\n strategy:\n canary:\n...\n trafficRouting:\n alb:\n stickinessConfig:\n enabled: true\n durationSeconds: 3600\n...\n
More information can be found in the AWS ALB API
"},{"location":"features/traffic-management/alb/#zero-downtime-updates-with-aws-targetgroup-verification","title":"Zero-Downtime Updates with AWS TargetGroup Verification","text":"Argo Rollouts contains two features to help ensure zero-downtime updates when used with the AWS LoadBalancer controller: TargetGroup IP verification and TargetGroup weight verification. Both features involve the Rollout controller performing additional safety checks to AWS, to verify the changes made to the Ingress object are reflected in the underlying AWS TargetGroup.
"},{"location":"features/traffic-management/alb/#targetgroup-ip-verification","title":"TargetGroup IP Verification","text":"Note
Target Group IP verification available since Argo Rollouts v1.1
The AWS LoadBalancer controller can run in one of two modes:
- Instance mode
- IP mode
TargetGroup IP Verification is only applicable when the AWS LoadBalancer controller in IP mode. When using the AWS LoadBalancer controller in IP mode (e.g. using the AWS CNI), the ALB LoadBalancer targets individual Pod IPs, as opposed to K8s node instances. Targeting Pod IPs comes with an increased risk of downtime during an update, because the Pod IPs behind the underlying AWS TargetGroup can more easily become outdated from the actual availability and status of pods, causing HTTP 502 errors when the TargetGroup points to pods which have already been scaled down.
To mitigate this risk, AWS recommends the use of pod readiness gate injection when running the AWS LoadBalancer in IP mode. Readiness gates allow for the AWS LoadBalancer controller to verify that TargetGroups are accurate before marking newly created Pods as \"ready\", preventing premature scale down of the older ReplicaSet.
Pod readiness gate injection uses a mutating webhook which decides to inject readiness gates when a pod is created based on the following conditions:
- There exists a service matching the pod labels in the same namespace
- There exists at least one target group binding that refers to the matching service
Another way to describe this is: the AWS LoadBalancer controller injects readiness gates onto Pods only if they are \"reachable\" from an ALB Ingress at the time of pod creation. A pod is considered reachable if an (ALB) Ingress references a Service which matches the pod labels. It ignores all other Pods.
One challenge with this manner of pod readiness gate injection, is that modifications to the Service selector labels (spec.selector) do not allow for the AWS LoadBalancer controller to inject the readiness gates, because by that time the Pod was already created (and readiness gates are immutable). Note that this is an issue when you change Service selectors of any ALB Service, not just ones involved in Argo Rollouts.
Because Argo Rollout's blue-green strategy works by modifying the activeService selector to the new ReplicaSet labels during promotion, it suffers from the issue where readiness gates for the spec.strategy.blueGreen.activeService fail to be injected. This means there is a possibility of downtime in the following problematic scenario during an update from V1 to V2:
- Update is triggered and V2 ReplicaSet stack is scaled up
- V2 ReplicaSet pods become fully available and ready to be promoted
- Rollout promotes V2 by updating the label selectors of the active service to point to the V2 stack (from V1)
- Due to unknown issues (e.g. AWS load balancer controller downtime, AWS rate limiting), registration of the V2 Pod IPs to the TargetGroup does not happen or is delayed.
- V1 ReplicaSet is scaled down to complete the update
After step 5, when the V1 ReplicaSet is scaled down, the outdated TargetGroup would still be pointing to the V1 Pods IPs which no longer exist, causing downtime.
To allow for zero-downtime updates, Argo Rollouts has the ability to perform TargetGroup IP verification as an additional safety measure during an update. When this feature is enabled, whenever a service selector modification is made, the Rollout controller blocks progression of the update until it can verify the TargetGroup is accurately targeting the new Pod IPs of the bluegreen.activeService. Verification is achieved by querying AWS APIs to describe the underlying TargetGroup, iterating its registered IPs, and ensuring all Pod IPs of the activeService's Endpoints list are registered in the TargetGroup. Verification must succeed before running postPromotionAnalysis or scaling down the old ReplicaSet.
Similarly for the canary strategy, after updating the canary.stableService selector labels to point to the new ReplicaSet, the TargetGroup IP verification feature allows the controller to block the scale down of the old ReplicaSet until it verifies the Pods IP behind the stableService TargetGroup are accurate.
"},{"location":"features/traffic-management/alb/#targetgroup-weight-verification","title":"TargetGroup Weight Verification","text":"Note
TargetGroup weight verification available since Argo Rollouts v1.0
TargetGroup weight verification addresses a similar problem to TargetGroup IP verification, but instead of verifying that the Pod IPs of a service are reflected accurately in the TargetGroup, the controller verifies that the traffic weights are accurate from what was set in the ingress annotations. Weight verification is applicable to AWS LoadBalancer controllers which are running either in IP mode or Instance mode.
After Argo Rollouts adjusts a canary weight by updating the Ingress annotation, it moves on to the next step. However, due to external factors (e.g. AWS rate limiting, AWS load balancer controller downtime) it is possible that the weight modifications made to the Ingress, did not take effect in the underlying TargetGroup. This is potentially dangerous as the controller will believe it is safe to scale down the old stable stack when in reality, the outdated TargetGroup may still be pointing to it.
Using the TargetGroup weight verification feature, the rollout controller will additionally verify the canary weight after a setWeight canary step. It accomplishes this by querying AWS LoadBalancer APIs directly, to confirm that the Rules, Actions, and TargetGroups reflect the desire of Ingress annotation.
"},{"location":"features/traffic-management/alb/#usage_1","title":"Usage","text":"To enable AWS target group verification, add --aws-verify-target-group flag to the rollout-controller flags:
apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: argo-rollouts\nspec:\n template:\n spec:\n containers:\n - name: argo-rollouts\n args: [--aws-verify-target-group]\n # NOTE: in v1.0, the --alb-verify-weight flag should be used instead\n
For this feature to work, the argo-rollouts deployment requires the following AWS API permissions under the Elastic Load Balancing API:
{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Action\": [\n \"elasticloadbalancing:DescribeTargetGroups\",\n \"elasticloadbalancing:DescribeLoadBalancers\",\n \"elasticloadbalancing:DescribeListeners\",\n \"elasticloadbalancing:DescribeRules\",\n \"elasticloadbalancing:DescribeTags\",\n \"elasticloadbalancing:DescribeTargetHealth\"\n ],\n \"Resource\": \"*\",\n \"Effect\": \"Allow\"\n }\n ]\n}\n
There are various ways of granting AWS privileges to the argo-rollouts pods, which is highly dependent to your cluster's AWS environment, and out-of-scope of this documentation. Some solutions include:
- AWS access and secret keys
- kiam
- kube2iam
- EKS ServiceAccount IAM Roles
"},{"location":"features/traffic-management/alb/#zero-downtime-updates-with-ping-pong-feature","title":"Zero-Downtime Updates with Ping-Pong feature","text":"Above there was described the recommended way by AWS to solve zero-downtime issue. Is a use a pod readiness gate injection when running the AWS LoadBalancer in IP mode. There is a challenge with that approach, modifications of the Service selector labels (spec.selector) not allowed the AWS LoadBalancer controller to mutate the readiness gates. And Ping-Pong feature helps to deal with that challenge. At some particular moment one of the services (e.g. ping) is \"wearing a hat\" of stable service another one (e.g. pong) is \"wearing a hat\" of canary. At the end of the promotion step all 100% of traffic sending to the \"canary\" (e.g. pong). And then the Rollout swapped the hats of ping and pong services so the pong became a stable one. The Rollout status object holds the value of who is currently the stable ping or pong (status.canary.currentPingPong). And this way allows the rollout to use pod readiness gate injection as the services are not changing their labels at the end of the rollout progress.
Important
Ping-Pong feature available since Argo Rollouts v1.2
"},{"location":"features/traffic-management/alb/#example","title":"Example","text":"apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: example-rollout\nspec:\n selector:\n matchLabels:\n app: nginx\n template:\n metadata:\n labels:\n app: nginx\n spec:\n containers:\n - name: nginx\n image: nginx:1.15.4\n ports:\n - containerPort: 80\n strategy:\n canary:\n pingPong: #Indicates that the ping-pong services enabled\n pingService: ping-service\n pongService: pong-service\n trafficRouting:\n alb:\n ingress: alb-ingress\n servicePort: 80\n steps:\n - setWeight: 20\n - pause: {}\n
"},{"location":"features/traffic-management/alb/#custom-annotations-prefix","title":"Custom annotations-prefix","text":"The AWS Load Balancer Controller allows users to customize the annotation prefix used by the Ingress controller using a flag to the controller, --annotations-prefix (from the default of alb.ingress.kubernetes.io). If your AWS Load Balancer Controller is customized to use a different annotation prefix, annotationPrefix field should be specified such that the Ingress object will be annotated in a manner understood by the cluster's aws load balancer controller.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nspec:\n strategy:\n canary:\n trafficRouting:\n alb:\n annotationPrefix: custom.alb.ingress.kubernetes.io\n
"},{"location":"features/traffic-management/alb/#custom-ingress-class","title":"Custom Ingress Class","text":"By default, Argo Rollout will operate on Ingresses with the annotation:
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n annotations:\n kubernetes.io/ingress.class: alb\n
Or with the ingressClassName:
apiVersion: networking.k8s.io/v1\nkind: Ingress\nspec:\n ingressClassName: alb\n
To configure the controller to operate on Ingresses with a different class name, you can specify a different value through the --alb-ingress-classes flag in the controller command line arguments.
Note that the --alb-ingress-classes flag can be specified multiple times if the Argo Rollouts controller should operate on multiple values. This may be desired when a cluster has multiple Ingress controllers that operate on different kubernetes.io/ingress.class or spec.ingressClassName values.
If the controller needs to operate on any Ingress without the kubernetes.io/ingress.class annotation or spec.ingressClassName, the flag can be specified with an empty string (e.g. --alb-ingress-classes '').
"},{"location":"features/traffic-management/ambassador/","title":"Ambassador Edge Stack","text":"Ambassador Edge Stack provides the functionality you need at the edge your Kubernetes cluster (hence, an \"edge stack\"). This includes an API gateway, ingress controller, load balancer, developer portal, canary traffic routing and more. It provides a group of CRDs that users can configure to enable different functionalities.
Argo-Rollouts provides an integration that leverages Ambassador's canary routing capability. This allows the traffic to your application to be gradually incremented while new versions are being deployed.
"},{"location":"features/traffic-management/ambassador/#how-it-works","title":"How it works","text":"Ambassador Edge Stack provides a resource called Mapping that is used to configure how to route traffic to services. Ambassador canary deployment is achieved by creating 2 mappings with the same URL prefix pointing to different services. Consider the following example:
apiVersion: getambassador.io/v2\nkind: Mapping\nmetadata:\n name: stable-mapping\nspec:\n prefix: /someapp\n rewrite: /\n service: someapp-stable:80\n---\napiVersion: getambassador.io/v2\nkind: Mapping\nmetadata:\n name: canary-mapping\nspec:\n prefix: /someapp\n rewrite: /\n service: someapp-canary:80\n weight: 30\n
In the example above we are configuring Ambassador to route 30% of the traffic coming from <public ingress>/someapp to the service someapp-canary and the rest of the traffic will go to the service someapp-stable. If users want to gradually increase the traffic to the canary service, they have to update the canary-mapping setting the weight to the desired value either manually or automating it somehow.
With Argo-Rollouts there is no need to create the canary-mapping. The process of creating it and gradually updating its weight is fully automated by the Argo-Rollouts controller. The following example shows how to configure the Rollout resource to use Ambassador as a traffic router for canary deployments:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\n...\nspec:\n strategy:\n canary:\n stableService: someapp-stable\n canaryService: someapp-canary\n trafficRouting:\n ambassador:\n mappings:\n - stable-mapping\n steps:\n - setWeight: 30\n - pause: {duration: 60s}\n - setWeight: 60\n - pause: {duration: 60s}\n
Under spec.strategy.canary.trafficRouting.ambassador there are 2 possible attributes:
mappings: Required. At least one Ambassador mapping must be provided for Argo-Rollouts to be able to manage the canary deployment. Multiple mappings are also supported in case there are multiple routes to the service (e.g., your service has multiple ports, or can be accessed via different URLs). If no mapping is provided Argo-Rollouts will send an error event and the rollout will be aborted.
When Ambassador is configured in the trafficRouting attribute of the manifest, the Rollout controller will: 1. Create one canary mapping for each stable mapping provided in the Rollout manifest 1. Proceed with the steps according to the configuration updating the canary mapping weight 1. At the end of the process Argo-Rollout will delete all the canary mappings created
"},{"location":"features/traffic-management/ambassador/#endpoint-resolver","title":"Endpoint Resolver","text":"By default, Ambassador uses kube-proxy to route traffic to Pods. However we should configure it to bypass kube-proxy and route traffic directly to pods. This will provide true L7 load balancing which is desirable in a canary workflow. This approach is called endpoint routing and can be achieve by configuring endpoint resolvers.
To configure Ambassador to use endpoint resolver it is necessary to apply the following resource in the cluster:
apiVersion: getambassador.io/v2\nkind: KubernetesEndpointResolver\nmetadata:\n name: endpoint\n
And then configure the mapping to use it setting the resolver attribute:
apiVersion: getambassador.io/v2\nkind: Mapping\nmetadata:\n name: stable-mapping\nspec:\n resolver: endpoint\n prefix: /someapp\n rewrite: /\n service: someapp-stable:80\n
For more details about the Ambassador and Argo-Rollouts integration, see the Ambassador Argo documentation.
"},{"location":"features/traffic-management/apisix/","title":"Apache APISIX","text":"You can use the Apache APISIX and Apache APISIX Ingress Controller for traffic management with Argo Rollouts.
The ApisixRoute is the object that supports the ability for weighted round robin load balancing when using Apache APISIX Ingress Controller as ingress.
This guide shows you how to integrate ApisixRoute with Argo Rollouts using it as weighted round robin load balancer
"},{"location":"features/traffic-management/apisix/#prerequisites","title":"Prerequisites","text":"Argo Rollouts requires Apache APISIX v2.15 or newer and Apache APISIX Ingress Controller v1.5.0 or newer.
Install Apache APISIX and Apache APISIX Ingress Controller with Helm v3:
helm repo add apisix https://charts.apiseven.com\nkubectl create ns apisix\n\nhelm upgrade -i apisix apisix/apisix --version=0.11.3 \\\n--namespace apisix \\\n--set ingress-controller.enabled=true \\\n--set ingress-controller.config.apisix.serviceNamespace=apisix\n
"},{"location":"features/traffic-management/apisix/#bootstrap","title":"Bootstrap","text":"First, we need to create the ApisixRoute object using its ability for weighted round robin load balancing.
apiVersion: apisix.apache.org/v2\nkind: ApisixRoute\nmetadata:\n name: rollouts-apisix-route\nspec:\n http:\n - name: rollouts-apisix\n match:\n paths:\n - /*\n hosts:\n - rollouts-demo.apisix.local\n backends:\n - serviceName: rollout-apisix-canary-stable\n servicePort: 80\n - serviceName: rollout-apisix-canary-canary\n servicePort: 80\n
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/examples/apisix/route.yaml\n
Notice, we don't specify the weight field. It is necessary to be synced with ArgoCD. If we specify this field and Argo Rollouts controller changes it, then the ArgoCD controller will notice it and will show that this resource is out of sync (if you are using Argo CD to manage your Rollout).
Secondly, we need to create the Argo Rollouts object.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollout-apisix-canary\nspec:\n replicas: 5\n strategy:\n canary:\n canaryService: rollout-apisix-canary-canary\n stableService: rollout-apisix-canary-stable\n trafficRouting:\n managedRoutes:\n - name: set-header\n apisix:\n route:\n name: rollouts-apisix-route\n rules:\n - rollouts-apisix\n steps:\n - setCanaryScale:\n replicas: 1\n setHeaderRoute:\n match:\n - headerName: trace\n headerValue:\n exact: debug\n name: set-header\n - setWeight: 20\n - pause: {}\n - setWeight: 40\n - pause:\n duration: 15\n - setWeight: 60\n - pause:\n duration: 15\n - setWeight: 80\n - pause:\n duration: 15\n revisionHistoryLimit: 2\n selector:\n matchLabels:\n app: rollout-apisix-canary\n template:\n metadata:\n labels:\n app: rollout-apisix-canary\n spec:\n containers:\n - name: rollout-apisix-canary\n image: argoproj/rollouts-demo:blue\n ports:\n - name: http\n containerPort: 8080\n protocol: TCP\n resources:\n requests:\n memory: 32Mi\n cpu: 5m\n
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/examples/apisix/rollout.yaml\n
Finally, we need to create the services for the Argo Rollouts object.
apiVersion: v1\nkind: Service\nmetadata:\n name: rollout-apisix-canary-canary\nspec:\n ports:\n - port: 80\n targetPort: http\n protocol: TCP\n name: http\n selector:\n app: rollout-apisix-canary\n # This selector will be updated with the pod-template-hash of the canary ReplicaSet. e.g.:\n # rollouts-pod-template-hash: 7bf84f9696\n---\napiVersion: v1\nkind: Service\nmetadata:\n name: rollout-apisix-canary-stable\nspec:\n ports:\n - port: 80\n targetPort: http\n protocol: TCP\n name: http\n selector:\n app: rollout-apisix-canary\n # This selector will be updated with the pod-template-hash of the stable ReplicaSet. e.g.:\n # rollouts-pod-template-hash: 789746c88d\n
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/examples/apisix/services.yaml\n
Initial creations of any Rollout will immediately scale up the replicas to 100% (skipping any canary upgrade steps, analysis, etc...) since there was no upgrade that occurred.
The Argo Rollouts kubectl plugin allows you to visualize the Rollout, its related resources (ReplicaSets, Pods, AnalysisRuns), and presents live state changes as they occur. To watch the rollout as it deploys, run the get rollout --watch command from plugin:
kubectl argo rollouts get rollout rollout-apisix-canary --watch\n
"},{"location":"features/traffic-management/apisix/#updating-a-rollout","title":"Updating a Rollout","text":"Next it is time to perform an update. Just as with Deployments, any change to the Pod template field (spec.template) results in a new version (i.e. ReplicaSet) to be deployed. Updating a Rollout involves modifying the rollout spec, typically changing the container image field with a new version, and then running kubectl apply against the new manifest. As a convenience, the rollouts plugin provides a set image command, which performs these steps against the live rollout object in-place. Run the following command to update the rollout-apisix-canary Rollout with the \"yellow\" version of the container:
kubectl argo rollouts set image rollout-apisix-canary \\\n rollouts-demo=argoproj/rollouts-demo:yellow\n
During a rollout update, the controller will progress through the steps defined in the Rollout's update strategy. The example rollout sets a 20% traffic weight to the canary, and pauses the rollout indefinitely until user action is taken to unpause/promote the rollout.
You can check ApisixRoute's backend weights by the following command
kubectl describe apisixroute rollouts-apisix-route\n\n......\nSpec:\n Http:\n Backends:\n Service Name: rollout-apisix-canary-stable\n Service Port: 80\n Weight: 80\n Service Name: rollout-apisix-canary-canary\n Service Port: 80\n Weight: 20\n......\n
The rollout-apisix-canary-canary service gets 20% traffic through the Apache APISIX. You can check SetHeader ApisixRoute's match by the following command
kubectl describe apisixroute set-header\n\n......\nSpec:\n Http:\n Backends:\n Service Name: rollout-apisix-canary-canary\n Service Port: 80\n Weight: 100\n Match:\n Exprs:\n Op: Equal\n Subject:\n Name: trace\n Scope: Header\n Value: debug\n......\n
"},{"location":"features/traffic-management/google-cloud/","title":"Google Cloud","text":"With the introduction of the Kubernetes Gateway API it is now possible to use Argo Rollouts with all compliant implementations that support it. The integration is available with the Argo Rollouts Gateway API plugin currently hosted in Argo Labs.
Useful resources:
- The Gateway API specification
- Support of the Gateway API in Google Cloud
- Argo Rollouts Plugin capabilities
- Plugin for the Gateway API
The process involves the following steps:
- Creating a Kubernetes cluster with support for the Gateway API in Google Cloud
- Creating a Load balancer that is managed by the Gateway API in Google Cloud
- Installing Argo Rollouts + gateway API plugin in the cluster
- Defining a Rollout that takes advantage of the plugin
For a full application that includes all manifests see the plugin example.
"},{"location":"features/traffic-management/istio/","title":"Istio","text":"Istio is a service mesh that offers a rich feature-set to control the flow of traffic to a web service. Istio offers this functionality through a set of CRDs, and Argo Rollouts automates the management of these resources to provide advanced traffic shaping capabilities to the different versions of the Rollout during an update.
"},{"location":"features/traffic-management/istio/#how-it-works","title":"How it works","text":"Traffic splitting is accomplished in Istio by adjusting traffic weights defined in an Istio VirtualService. When using Argo Rollouts with Istio, a user deploys a VirtualService containing at least one HTTP route containing two HTTP route destinations: a route destination targeting the pods of canary ReplicaSet, and a route destination targeting the pods stable ReplicaSet. Istio provides two approaches for weighted traffic splitting, both approaches are available as options in Argo Rollouts:
- Host-level traffic splitting
- Subset-level traffic splitting
"},{"location":"features/traffic-management/istio/#host-level-traffic-splitting","title":"Host-level Traffic Splitting","text":"The first approach to traffic splitting using Argo Rollouts and Istio, is splitting between two hostnames, or Kubernetes Services: a canary Service and a stable Service. This approach is similar to the way all other Argo Rollouts mesh/ingress-controller integrations work (e.g. ALB, SMI, Nginx). Using this approach, the user is required to deploy the following resources:
- Rollout
- Service (canary)
- Service (stable)
- VirtualService
The Rollout should define the following fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollout-example\nspec:\n ...\n strategy:\n canary:\n canaryService: canary-svc # required\n stableService: stable-svc # required\n trafficRouting:\n istio:\n virtualService:\n name: rollout-vsvc # required\n routes:\n - primary # optional if there is a single route in VirtualService, required otherwise\n steps:\n - setWeight: 5\n - pause:\n duration: 10m\n
The VirtualService must contain an HTTP route with a name referenced in the Rollout, containing two route destinations with host values that match the canaryService and stableService referenced in the Rollout. If the VirtualService is defined in a different namespace than the rollout, its name should be rollout-vsvc.<vsvc namespace name>. Note that Istio requires that all weights add to 100, so the initial weights can be 100% to stable, and 0% to canary.
apiVersion: networking.istio.io/v1alpha3\nkind: VirtualService\nmetadata:\n name: rollout-vsvc\nspec:\n gateways:\n - istio-rollout-gateway\n hosts:\n - istio-rollout.dev.argoproj.io\n http:\n - name: primary # referenced in canary.trafficRouting.istio.virtualService.routes\n route:\n - destination:\n host: stable-svc # referenced in canary.stableService\n weight: 100\n - destination:\n host: canary-svc # referenced in canary.canaryService\n weight: 0\n
Finally, a canary and stable Service should be deployed. The selector of these Services will be modified by the Rollout during an update to target the canary and stable ReplicaSet pods. Note that if the VirtualService and destination host resides in different namespaces (e.g., VirtualService and Rollout are not in the same namespace), the namespace should be included in the destination host (e.g. stable-svc.<namespace>).
apiVersion: v1\nkind: Service\nmetadata:\n name: canary-svc\nspec:\n ports:\n - port: 80\n targetPort: http\n protocol: TCP\n name: http\n selector:\n app: rollouts-demo\n # This selector will be updated with the pod-template-hash of the canary ReplicaSet. e.g.:\n # rollouts-pod-template-hash: 7bf84f9696\n---\napiVersion: v1\nkind: Service\nmetadata:\n name: stable-svc\nspec:\n ports:\n - port: 80\n targetPort: http\n protocol: TCP\n name: http\n selector:\n app: rollouts-demo\n # This selector will be updated with the pod-template-hash of the stable ReplicaSet. e.g.:\n # rollouts-pod-template-hash: 123746c88d\n
During the lifecycle of a Rollout update, Argo Rollouts will continuously:
- modify the canary Service
spec.selector to contain the rollouts-pod-template-hash label of the canary ReplicaSet - modify the stable Service
spec.selector to contain the rollouts-pod-template-hash label of the stable ReplicaSet - modify the VirtualService
spec.http[].route[].weight to match the current desired canary weight
Note
Rollout does not make any other assumptions about the fields within the VirtualService or the Istio mesh. The user could specify additional configurations for the VirtualService like URI rewrite rules on the primary route or any other route if desired. The user can also create specific DestinationRules for each of the services.
"},{"location":"features/traffic-management/istio/#subset-level-traffic-splitting","title":"Subset-level Traffic Splitting","text":"Important
Available since v1.0
The second approach to traffic splitting using Argo Rollouts and Istio, is splitting between two Istio DestinationRule Subsets: a canary subset and a stable subset. When splitting by DestinationRule subsets, the user is required to deploy the following resources:
- Rollout
- Service
- VirtualService
- DestinationRule
The Rollout should define the following fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollout-example\nspec:\n ...\n strategy:\n canary:\n trafficRouting:\n istio:\n virtualService:\n name: rollout-vsvc # required\n routes:\n - primary # optional if there is a single route in VirtualService, required otherwise\n destinationRule:\n name: rollout-destrule # required\n canarySubsetName: canary # required\n stableSubsetName: stable # required\n steps:\n - setWeight: 5\n - pause:\n duration: 10m\n
A single service should be defined, which targets the Rollout pods. Note that unlike the first approach, where traffic splitting is against multiple Services which are modified to contain the rollout-pod-template-hash of the canary/stable ReplicaSets, this Service is not modified by the rollout controller.
apiVersion: v1\nkind: Service\nmetadata:\n name: rollout-example\nspec:\n ports:\n - port: 80\n targetPort: http\n protocol: TCP\n name: http\n selector:\n app: rollout-example\n
The VirtualService must contain an HTTP route with a name referenced in the Rollout, containing two route destinations with subset values that match the canarySubsetName and stableSubsetName referenced in the Rollout. Note that Istio requires that all weights add to 100, so the initial weights can be 100% to stable, and 0% to canary.
apiVersion: networking.istio.io/v1alpha3\nkind: VirtualService\nmetadata:\n name: rollout-vsvc\nspec:\n gateways:\n - istio-rollout-gateway\n hosts:\n - istio-rollout.dev.argoproj.io\n http:\n - name: primary # referenced in canary.trafficRouting.istio.virtualService.routes\n route:\n - destination:\n host: rollout-example\n subset: stable # referenced in canary.trafficRouting.istio.destinationRule.stableSubsetName\n weight: 100\n - destination:\n host: rollout-example\n subset: canary # referenced in canary.trafficRouting.istio.destinationRule.canarySubsetName\n weight: 0\n
Finally, the DestinationRule containing the canary and stable subsets referenced in the Rollout.
apiVersion: networking.istio.io/v1alpha3\nkind: DestinationRule\nmetadata:\n name: rollout-destrule\nspec:\n host: rollout-example\n subsets:\n - name: canary # referenced in canary.trafficRouting.istio.destinationRule.canarySubsetName\n labels: # labels will be injected with canary rollouts-pod-template-hash value\n app: rollout-example\n - name: stable # referenced in canary.trafficRouting.istio.destinationRule.stableSubsetName\n labels: # labels will be injected with stable rollouts-pod-template-hash value\n app: rollout-example\n
During the lifecycle of a Rollout using Istio DestinationRule, Argo Rollouts will continuously:
- modify the VirtualService
spec.http[].route[].weight to match the current desired canary weight - modify the DestinationRule
spec.subsets[].labels to contain the rollouts-pod-template-hash label of the canary and stable ReplicaSets
"},{"location":"features/traffic-management/istio/#tcp-traffic-splitting","title":"TCP Traffic Splitting","text":"Important
Available since v1.2.2
Support for splitting TCP traffic was introduced and requires the Rollout to define the following fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollout-example\nspec:\n ...\n strategy:\n canary:\n canaryService: canary-svc # required\n stableService: stable-svc # required\n trafficRouting:\n istio:\n virtualService:\n name: rollout-vsvc # required\n tcpRoutes:\n # Below fields are optional but if defined, they should match exactly with at least one of the TCP route match rules in your VirtualService\n - port: 3000 # Only required if you want to match any rule in your VirtualService which contains this port\n steps:\n - setWeight: 5\n - pause:\n duration: 10m\n
The VirtualService must contain a TCP route with a matching port referenced in the Rollout
apiVersion: networking.istio.io/v1alpha3\nkind: VirtualService\nmetadata:\n name: rollout-vsvc\nspec:\n gateways:\n - istio-rollout-gateway\n hosts:\n - istio-rollout.dev.argoproj.io\n tcp:\n - match:\n - port: 3000\n route:\n - destination:\n host: stable-svc # referenced in canary.stableService\n weight: 100\n - destination:\n host: canary-svc # referenced in canary.canaryService\n weight: 0\n
"},{"location":"features/traffic-management/istio/#multicluster-setup","title":"Multicluster Setup","text":"If you have Istio multicluster setup where the primary Istio cluster is different than the cluster where the Argo Rollout controller is running, then you need to do the following setup:
- Create a
ServiceAccount in the Istio primary cluster. apiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: argo-rollouts-istio-primary\n namespace: <any-namespace-preferrably-config-namespace>\n
- Create a
ClusterRole that provides access to Rollout controller in the Istio primary cluster. apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRole\nmetadata:\n name: argo-rollouts-istio-primary\nrules:\n- apiGroups:\n - networking.istio.io\n resources:\n - virtualservices\n - destinationrules\n verbs:\n - get\n - list\n - watch\n - update\n - patch\n
Note: If Argo Rollout controller is also installed in the Istio primary cluster, then you can reuse the argo-rollouts-clusterrole ClusterRole instead of creating a new one. - Link the
ClusterRole with the ServiceAccount in the Istio primary cluster. apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n name: argo-rollouts-istio-primary\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: argo-rollouts-istio-primary\nsubjects:\n- kind: ServiceAccount\n name: argo-rollouts-istio-primary\n namespace: <namespace-of-the-service-account>\n
- Now, use the following command to generate a secret for Rollout controller to access the Istio primary cluster. This secret will be applied to the cluster where Argo Rollout is running (i.e, Istio remote cluster), but will be generated from the Istio primary cluster. This secret can be generated right after Step 1, it only requires
ServiceAccount to exist. Reference to the command. istioctl x create-remote-secret --type remote --name <cluster-name> \\\n --namespace <namespace-of-the-service-account> \\\n --service-account <service-account-created-in-step1> \\\n --context=\"<ISTIO_PRIMARY_CLUSTER>\" | \\\n kubectl apply -f - --context=\"<ARGO_ROLLOUT_CLUSTER/ISTIO_REMOTE_CLUSTER>\"\n
- Label the secret.
kubectl label secret <istio-remote-secret> istio.argoproj.io/primary-cluster=\"true\" -n <namespace-of-the-secret>\n
"},{"location":"features/traffic-management/istio/#comparison-between-approaches","title":"Comparison Between Approaches","text":"There are some advantages and disadvantages of host-level traffic splitting vs. subset-level traffic splitting.
"},{"location":"features/traffic-management/istio/#dns-requirements","title":"DNS requirements","text":"With host-level splitting, the VirtualService requires different host values to split among the two destinations. However, using two host values implies the use of different DNS names (one for the canary, the other for the stable). For north-south traffic, which reaches the Service through the Istio Gateway, having multiple DNS names to reach the canary vs. stable pods may not matter. However, for east-west or intra-cluster traffic, it forces microservice-to-microservice communication to choose whether to hit the stable or the canary DNS name, go through the gateway, or add DNS entries for the VirtualServices. In this situation, the DestinationRule subset traffic splitting would be a better option for intra-cluster canarying.
"},{"location":"features/traffic-management/istio/#metrics","title":"Metrics","text":"Depending on the choice of host-level splitting vs. subset-level splitting, there will be different styles of prometheus metrics available. For example, if using host-level splitting, the metrics of the canary vs. stable would appear in the Istio Service metrics dashboard:
On the other hand, when splitting via subsets, it would be necessary to query prometheus using different parameters, such as the workload name:
"},{"location":"features/traffic-management/istio/#integrating-with-gitops","title":"Integrating with GitOps","text":"Earlier it was explained that VirtualServices should be deployed with an initial canary and stable weight of 0 and 100, respectively, such as in the following example:
http:\n - name: primary\n route:\n - destination:\n host: stable-svc\n weight: 100\n - destination:\n host: canary-svc\n weight: 0\n
This introduces a problem for users practicing GitOps. Since a Rollout will modify these VirtualService weights as the Rollout progresses through its steps, it unfortunately causes the VirtualService to become OutOfSync with the version in git. Additionally, if the VirtualService in git were to be applied while the Rollout is in this state (splitting traffic between the services), the apply would revert the weights back to the values in git (i.e. 100 to stable, 0 to canary).
One protection which is implemented in Argo Rollouts, is that it continually watches for changes to managed VirtualServices. In the event that a kubectl apply were to happen using the VirtualService in git, the change would be detected immediately by the rollout controller, and the controller will instantly set the VirtualService weights back to the canary weight appropriate for the given step of the Rollout. But since there is momentary flapping of weights, this behavior should be understood.
Some best practices to follow when using Argo CD with Argo Rollouts to prevent this behavior, is to leverage the following Argo CD features:
-
Configure the application to ignore differences in the VirtualService. e.g.:
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n name: guestbook\nspec:\n ignoreDifferences:\n - group: networking.istio.io\n kind: VirtualService\n jsonPointers:\n - /spec/http/0\n
Ignoring the differences in the VirtualServices HTTP route, prevents gitops differences in the VirtualService HTTP routes to contribute to the overall sync status of the Argo CD application. This adds the additional benefit of prevent auto-sync operations from being triggered.
-
Configure the Application to only apply OutOfSync resources:
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n name: guestbook\nspec:\n syncPolicy:\n syncOptions:\n - ApplyOutOfSyncOnly=true\n
By default, when Argo CD syncs an application, it runs kubectl apply against all resources in git which are part of the application. The ApplyOutOfSyncOnly=true sync option indicates to Argo CD to skip applying resources which it already considers Synced, and only apply the ones which are OutOfSync. This option, when used in conjunction with the ignoreDifferences feature, provides a way to manage the conflict in the desired state of a VirtualService between Argo CD and Argo Rollouts.
Argo CD also has an open issue here which would help address this problem. The proposed solution is to introduce an annotation to resources, which indicates to Argo CD to respect and preserve the differences at a specified path, in order to allow other controllers (e.g. Argo Rollouts) controller manage them instead.
"},{"location":"features/traffic-management/istio/#ping-pong","title":"Ping Pong","text":"Important
Available since v1.7
Argo Rollouts also supports ping pong when using Istio this was added to support configuring both ALB and Istio traffic routers at the same time. When using an ALB, ping-pong is generally a best practice especially with ALB readiness gates enabled. However, when we change the service selectors when a rollout is aborted back to stable pod hash it causes a blip of traffic outage because the ALB controller will set the pod readiness gates to false for a short while due to the label changes. If we configure both ALB and Istio with ping-pong this selector change does not happen and hence we do not see any outages.
"},{"location":"features/traffic-management/istio/#alternatives-considered","title":"Alternatives Considered","text":""},{"location":"features/traffic-management/istio/#rollout-ownership-over-the-virtual-service","title":"Rollout ownership over the Virtual Service","text":"An early design alternative was that instead of the controller modifying a referenced VirtualService, the Rollout controller would create, manage, and own a Virtual Service. While this approach is GitOps friendly, it introduces other issues:
- To provide the same flexibility as referencing VirtualService within a Rollout, the Rollout needs to inline a large portion of the Istio spec. However, networking is outside the responsibility of the Rollout and makes the Rollout spec unnecessarily complicated.
- If Istio introduces a feature, that feature will not be available in Argo Rollouts until implemented within Argo Rollouts.
Both of these issues adds more complexity to the users and Argo Rollouts developers compared to referencing a Virtual Service.
"},{"location":"features/traffic-management/istio/#istio-support-through-the-smi-adapter-for-istio","title":"Istio support through the SMI Adapter for Istio","text":"SMI is the Service Mesh Interface, which serves as a standard interface for all common features of a service mesh. This feature is GitOps friendly, but native Istio has extra functionality that SMI does not currently provide.
"},{"location":"features/traffic-management/kong/","title":"Kong Ingress","text":"With the introduction of the Kubernetes Gateway API it is now possible to use Argo Rollouts with all compliant implementations that support it. The integration is available with the Argo Rollouts Gateway API plugin currently hosted in Argo Labs.
Useful resources:
- The Gateway API specification
- Support of the Gateway API in Kong
- Argo Rollouts Plugin capabilities
- Plugin for the Gateway API
The process involves the following steps:
- Installing the Gateway API CRDs in your cluster
- Installing Kong and enabling the Gateway API support feature
- Creating a GatewayClass and Gateway resources
- Installing Argo Rollouts + gateway API plugin in the cluster
- Defining a Rollout that takes advantage of the plugin
For a full application that includes all manifests see the plugin example.
"},{"location":"features/traffic-management/mixed/","title":"Multiple Providers","text":"Note
Multiple trafficRouting is available since Argo Rollouts v1.2
The usage of multiple providers tries to cover scenarios where, for some reason, we have to use different providers on North-South and West-East traffic routing or any other hybrid architecture that requires the use of multiple providers.
"},{"location":"features/traffic-management/mixed/#examples-of-when-you-can-use-multiple-providers","title":"Examples of when you can use multiple providers","text":""},{"location":"features/traffic-management/mixed/#avoid-injecting-sidecars-on-your-ingress-controller","title":"Avoid injecting sidecars on your Ingress controller","text":"This is a common requirement of the service mesh and with multiple trafficRoutings you can leverage North-South traffic shifting to NGiNX and West-East traffic shifting to SMI, avoiding the need of adding the Ingress controller inside the mesh.
"},{"location":"features/traffic-management/mixed/#avoid-manipulation-of-the-host-header-at-the-ingress","title":"Avoid manipulation of the host header at the Ingress","text":"Another common side effect of adding some of the Ingress controllers into the mesh, and is caused by the usage of those mesh host headers to be pointing into a mesh hostname in order to be routed.
"},{"location":"features/traffic-management/mixed/#avoid-big-bang","title":"Avoid Big-Bang","text":"This takes place on existing fleets where downtime is very reduced or nearly impossible. To avoid big-bang-adoption the use of multiple providers can ease how teams can implement gradually new technologies. An example, where an existing fleet that is using a provider such as Ambassador and is already performing canary in a North-South fashion as part of their rollouts can gradually implement more providers such as Istio, SMI, etc.
"},{"location":"features/traffic-management/mixed/#hybrid-scenarios","title":"Hybrid Scenarios","text":"In this case, its very similar to avoiding the Big-Bang, either if it is part of the platform roadmap or a new redesign of the architecture, there are multiple scenarios where having the capacity of using multiple trafficRoutings is very much in need: gradual implementation, eased rollback of architecture or even for a fallback.
"},{"location":"features/traffic-management/mixed/#requirements","title":"Requirements","text":"The use of multiple providers requires that both providers comply with its minimum requirements independently. By example, if you want to use NGiNX and SMI you would need to have both SMI and NGiNX in place and produce the rollout configuration for both.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n strategy:\n canary:\n # Reference to a Service which the controller will update to point to the canary ReplicaSet\n canaryService: rollouts-demo-canary\n # Reference to a Service which the controller will update to point to the stable ReplicaSet\n stableService: rollouts-demo-stable\n trafficRouting:\n nginx:\n # Reference to an Ingress which has a rule pointing to the stable service (e.g. rollouts-demo-stable)\n # This ingress will be cloned with a new name, in order to achieve NGINX traffic splitting.\n stableIngress: rollouts-demo-stable\n smi: {}\n
"},{"location":"features/traffic-management/nginx/","title":"Nginx","text":"The Nginx Ingress Controller enables traffic management through one or more Ingress objects to configure an Nginx deployment that routes traffic directly to pods. Each Nginx Ingress contains multiple annotations that modify the behavior of the Nginx Deployment. For traffic management between different versions of an application, the Nginx Ingress controller provides the capability to split traffic by introducing a second Ingress object (referred to as the canary Ingress) with some special annotations. You can read more about these canary annotations on the official canary annotations documentation page. The canary Ingress ignores any other non-canary nginx annotations. Instead, it leverages the annotation settings from the primary Ingress.
The Rollout controller will always set the following two annotations on the canary Ingress (using your configured or the default nginx.ingress.kubernetes.io prefix):
canary: true to indicate that this is the canary Ingresscanary-weight: <num> to indicate what percentage of traffic to send to the canary. If all traffic is routed to the stable Service, this is set to 0
You can provide additional annotations to add to the canary Ingress via the additionalIngressAnnotations field to enable features like routing by header or cookie.
"},{"location":"features/traffic-management/nginx/#integration-with-argo-rollouts","title":"Integration with Argo Rollouts","text":"There are a couple of required fields in a Rollout to send split traffic between versions using Nginx. Below is an example of a Rollout with those fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nspec:\n ...\n strategy:\n canary:\n canaryService: canary-service # required\n stableService: stable-service # required\n trafficRouting:\n nginx:\n # Either stableIngress or stableIngresses must be configured, but not both.\n stableIngress: primary-ingress\n stableIngresses:\n - primary-ingress\n - secondary-ingress\n - tertiary-ingress\n annotationPrefix: customingress.nginx.ingress.kubernetes.io # optional\n additionalIngressAnnotations: # optional\n canary-by-header: X-Canary\n canary-by-header-value: iwantsit\n
The stable Ingress field is a reference to an Ingress in the same namespace of the Rollout. The Rollout requires the primary Ingress routes traffic to the stable Service. The Rollout checks that condition by confirming the Ingress has a backend that matches the Rollout's stableService.
The controller routes traffic to the canary Service by creating a second Ingress with the canary annotations. As the Rollout progresses through the Canary steps, the controller updates the canary Ingress's canary annotations to reflect the desired state of the Rollout enabling traffic splitting between two different versions.
Since the Nginx Ingress controller allows users to configure the annotation prefix used by the Ingress controller, Rollouts can specify the optional annotationPrefix field. The canary Ingress uses that prefix instead of the default nginx.ingress.kubernetes.io if the field set.
"},{"location":"features/traffic-management/nginx/#using-argo-rollouts-with-multiple-nginx-ingress-controllers-per-service","title":"Using Argo Rollouts with multiple NGINX ingress controllers per service","text":"Starting with v1.5, argo rollouts supports multiple Nginx ingress controllers pointing at one service with canary deployments. If only one ingress controller is needed, utilize the existing key stableIngress. If multiple ingress controllers are needed (e.g., separating internal vs external traffic), use the key stableIngresses instead. It takes an array of string values that are the names of the ingress controllers. Canary steps are applied identically across all ingress controllers.
"},{"location":"features/traffic-management/nginx/#using-argo-rollouts-with-custom-nginx-ingress-controller-names","title":"Using Argo Rollouts with custom NGINX ingress controller names","text":"As a default, the Argo Rollouts controller only operates on ingresses with the kubernetes.io/ingress.class annotation or spec.ingressClassName set to nginx. A user can configure the controller to operate on Ingresses with different class name by specifying the --nginx-ingress-classes flag. A user can list the --nginx-ingress-classes flag multiple times if the Argo Rollouts controller should operate on multiple values. This solves the case where a cluster has multiple Ingress controllers operating on different class values.
If the user would like the controller to operate on any Ingress without the kubernetes.io/ingress.class annotation or spec.ingressClassName, a user should add the following --nginx-ingress-classes ''.
"},{"location":"features/traffic-management/plugins/","title":"Traffic Router Plugins","text":"Important
Available since v1.5 - Status: Alpha
Argo Rollouts supports getting analysis metrics via 3rd party plugin system. This allows users to extend the capabilities of Rollouts to support metric providers that are not natively supported. Rollout's uses a plugin library called go-plugin to do this. You can find a sample plugin here: rollouts-plugin-trafficrouter-sample-nginx
"},{"location":"features/traffic-management/plugins/#using-a-traffic-router-plugin","title":"Using a Traffic Router Plugin","text":"There are two methods of installing and using an argo rollouts plugin. The first method is to mount up the plugin executable into the rollouts controller container. The second method is to use a HTTP(S) server to host the plugin executable.
"},{"location":"features/traffic-management/plugins/#mounting-the-plugin-executable-into-the-rollouts-controller-container","title":"Mounting the plugin executable into the rollouts controller container","text":"There are a few ways to mount the plugin executable into the rollouts controller container. Some of these will depend on your particular infrastructure. Here are a few methods:
- Using an init container to download the plugin executable
- Using a Kubernetes volume mount with a shared volume such as NFS, EBS, etc.
- Building the plugin into the rollouts controller container
Then you can use the configmap to point to the plugin executable file location. Example:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-config\ndata:\n trafficRouterPlugins: |-\n - name: \"argoproj-labs/sample-nginx\" # name of the plugin, it must match the name required by the plugin so it can find it's configuration\n location: \"file://./my-custom-plugin\" # supports http(s):// urls and file://\n
"},{"location":"features/traffic-management/plugins/#using-a-https-server-to-host-the-plugin-executable","title":"Using a HTTP(S) server to host the plugin executable","text":"Argo Rollouts supports downloading the plugin executable from a HTTP(S) server. To use this method, you will need to configure the controller via the argo-rollouts-config configmap and set pluginLocation to a http(s) url. Example:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-config\ndata:\n trafficRouterPlugins: |-\n - name: \"argoproj-labs/sample-nginx\" # name of the plugin, it must match the name required by the plugin so it can find it's configuration\n location: \"https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-sample-nginx/releases/download/v0.0.1/metric-plugin-linux-amd64\" # supports http(s):// urls and file://\n sha256: \"08f588b1c799a37bbe8d0fc74cc1b1492dd70b2c\" #optional sha256 checksum of the plugin executable\n
"},{"location":"features/traffic-management/plugins/#some-words-of-caution","title":"Some words of caution","text":"Depending on which method you use to install and the plugin, there are some things to be aware of. The rollouts controller will not start if it can not download or find the plugin executable. This means that if you are using a method of installation that requires a download of the plugin and the server hosting the plugin for some reason is not available and the rollouts controllers pod got deleted while the server was down or is coming up for the first time, it will not be able to start until the server hosting the plugin is available again.
Argo Rollouts will download the plugin at startup only once but if the pod is deleted it will need to download the plugin again on next startup. Running Argo Rollouts in HA mode can help a little with this situation because each pod will download the plugin at startup. So if a single pod gets deleted during a server outage, the other pods will still be able to take over because there will already be a plugin executable available to it. It is the responsibility of the Argo Rollouts administrator to define the plugin installation method considering the risks of each approach.
"},{"location":"features/traffic-management/plugins/#list-of-available-plugins-alphabetical-order","title":"List of Available Plugins (alphabetical order)","text":""},{"location":"features/traffic-management/plugins/#add-your-plugin-here","title":"Add Your Plugin Here","text":" - If you have created a plugin, please submit a PR to add it to this list.
"},{"location":"features/traffic-management/plugins/#rollouts-plugin-trafficrouter-sample-nginx","title":"rollouts-plugin-trafficrouter-sample-nginx","text":" - This is just a sample plugin that can be used as a starting point for creating your own plugin. It is not meant to be used in production. It is based on the built-in prometheus provider.
"},{"location":"features/traffic-management/plugins/#consul","title":"Consul","text":" - This is a plugin that allows argo-rollouts to work with Consul's service mesh for traffic shaping patterns.
"},{"location":"features/traffic-management/plugins/#contour","title":"Contour","text":" - This is a plugin that allows argo-rollouts to work with contour's resource: HTTPProxy. It enables traffic shaping patterns such as canary releases and more.
"},{"location":"features/traffic-management/plugins/#gateway-api","title":"Gateway API","text":" - Provide support for Gateway API, which includes Kuma, Traefix, cilium, Contour, GloodMesh, HAProxy, and many others.
"},{"location":"features/traffic-management/smi/","title":"Service Mesh Interface (SMI)","text":"Important
Available since v0.9.0
Warning
The Cloud Native Computing Foundation has archived the SMI Spec. The recommended way forward is to look at the Gateway API, Project Gamma and the Argo Rollouts Gateway API Plugin.
Service Mesh Interface (SMI) is a standard interface for service meshes on Kubernetes leveraged by many Service Mesh implementations (like Linkerd). SMI offers this functionality through a set of CRDs, and the Argo Rollouts controller creates these resources to manipulate the traffic routing into the desired state.
The Argo Rollout controller achieves traffic shaping by creating and manipulating the TrafficSplit CR. A TrafficSplit describes the desired traffic routing for an application and relies on the underlying Service Meshes implement that desired state. Instead of worrying about the details of a specific service mesh, a user needs to specify a root Service that clients use to communicate and a list of backends consisting of a Service and weight. The Service Mesh implementing SMI uses this spec to route traffic to the backends Services based on the weights of the backends. For Rollout users, the Argo Rollout controller creates and manipulates the TrafficSplit using the following information:
- Canary Service: Name of the service that sends traffic only to the canary pods
- Stable Service: Name of the service that sends traffic only to the stable pods
- Root Service: Name of the service that clients use to communicate. If a request comes to this root service not through a proxy, the standard Kubernetes service routing will be used.
Below is an example of a Rollout with all the required fields configured:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollout-example\nspec:\n ...\n strategy:\n canary:\n steps:\n - setWeight: 5\n - pause:\n duration: 600\n canaryService: canary-svc # required\n stableService: stable-svc # required\n trafficRouting:\n smi:\n rootService: root-svc # optional\n trafficSplitName: rollout-example-traffic-split # optional\n
With the above configuration, the controller can automate dynamic traffic splitting. First, the controller manipulates the canary and stable Service listed in the Rollout to make them only receive traffic from the respective canary and stable ReplicaSets. The controller achieves this by adding the ReplicaSet's unique pod template hash to that Service's selector. With the stable and canary Services configured, the controller creates a TrafficSplit using these Services in the backend, and the weights of the backend are dynamically configured from the current desired weight of the Rollout's canary steps. The controller sets the TrafficSplit's root service to the stableService unless the Rollout has the rootService field specified. This configured TrafficSplit along with the Service and Rollout resources enable fine-grained percentages of traffic between two versions of an application. Optionally, the user can specify a name for the traffic split. If there is no name listed in the Rollout, the controller uses the Rollout's name for the TrafficSplit. If a TrafficSplit with that name already exists and isn't owned by that Rollout, the controller marks the Rollout as an error state.
Here is the TrafficSplit created from the above Rollout:
apiVersion: split.smi-spec.io/v1alpha1\nkind: TrafficSplit\nmetadata:\n name: rollout-example-traffic-split\nspec:\n service: root-svc # controller uses the stableService if Rollout does not specify the rootService field\n backends:\n - service: stable-svc\n weight: 95\n - service: canary-svc\n weight: 5\n
As a Rollout progresses through all its steps, the controller updates the TrafficSplit's backend weights to reflect the current weight of the Rollout. When the Rollout has successfully finished executing all the steps, the controller modifies the stable Service's selector to point at the desired ReplicaSet and TrafficSplit's weight to send 100% of traffic to the stable Service.
Note
The controller defaults to using the v1alpha1 version of the TrafficSplit. The Argo Rollouts operator can change the api version used by specifying a --traffic-split-api-version flag in the controller args.
"},{"location":"features/traffic-management/traefik/","title":"Traefik","text":"You can use the Traefik Proxy for traffic management with Argo Rollouts.
The TraefikService is the object that supports the ability for weighted round robin load balancing and traffic mirroring when using Traefik as ingress.
Note
Traefik is also supported via the Argo Rollouts Gateway API plugin.
"},{"location":"features/traffic-management/traefik/#how-to-integrate-traefikservice-with-argo-rollouts-using-it-as-weighted-round-robin-load-balancer","title":"How to integrate TraefikService with Argo Rollouts using it as weighted round robin load balancer","text":"First, we need to create the TraefikService object using its ability for weighted round robin load balancing.
apiVersion: traefik.containo.us/v1alpha1\nkind: TraefikService\nmetadata:\n name: traefik-service\nspec:\n weighted:\n services:\n - name: stable-rollout # k8s service name that you need to create for stable application version\n port: 80\n - name: canary-rollout # k8s service name that you need to create for new application version\n port: 80\n
Notice, we don't specify the weight field. It is necessary to be synced with ArgoCD. If we specify this field and Argo Rollouts controller changes it, then the ArgoCD controller will notice it and will show that this resource is out of sync (if you are using Argo CD to manage your Rollout).
Secondly, we need to create the Argo Rollouts object.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n replicas: 5\n strategy:\n canary:\n canaryService: canary-rollout\n stableService: stable-rollout\n trafficRouting:\n traefik:\n weightedTraefikServiceName: traefik-service # specify traefikService resource name that we have created before\n steps:\n - setWeight: 30\n - pause: {}\n - setWeight: 40\n - pause: {duration: 10}\n - setWeight: 60\n - pause: {duration: 10}\n - setWeight: 80\n - pause: {duration: 10}\n ...\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts/","title":"Rollouts","text":"Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts/#synopsis","title":"Synopsis","text":"This command consists of multiple subcommands which can be used to manage Argo Rollouts.
kubectl argo rollouts COMMAND [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts/#examples","title":"Examples","text":"# Get guestbook rollout and watch progress\nkubectl argo rollouts get rollout guestbook -w\n\n# Pause the guestbook rollout\nkubectl argo rollouts pause guestbook\n\n# Promote the guestbook rollout\nkubectl argo rollouts promote guestbook\n\n# Abort the guestbook rollout\nkubectl argo rollouts abort guestbook\n\n# Retry the guestbook rollout\nkubectl argo rollouts retry guestbook\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts/#options","title":"Options","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n -h, --help help for kubectl-argo-rollouts\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts/#available-commands","title":"Available Commands","text":" - rollouts abort - Abort a rollout
- rollouts completion - Generate completion script
- rollouts create - Create a Rollout, Experiment, AnalysisTemplate, ClusterAnalysisTemplate, or AnalysisRun resource
- rollouts dashboard - Start UI dashboard
- rollouts get - Get details about rollouts and experiments
- rollouts lint - Lint and validate a Rollout
- rollouts list - List rollouts or experiments
- rollouts notifications - Set of CLI commands that helps manage notifications settings
- rollouts pause - Pause a rollout
- rollouts promote - Promote a rollout
- rollouts restart - Restart the pods of a rollout
- rollouts retry - Retry a rollout or experiment
- rollouts set - Update various values on resources
- rollouts status - Show the status of a rollout
- rollouts terminate - Terminate an AnalysisRun or Experiment
- rollouts undo - Undo a rollout
- rollouts version - Print version
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_abort/","title":"Rollouts Abort","text":"Abort a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_abort/#synopsis","title":"Synopsis","text":"This command stops progressing the current rollout and reverts all steps. The previous ReplicaSet will be active.
Note the 'spec.template' still represents the new rollout version. If the Rollout leaves the aborted state, it will try to go to the new version. Updating the 'spec.template' back to the previous version will fully revert the rollout.
kubectl argo rollouts abort ROLLOUT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_abort/#examples","title":"Examples","text":"# Abort a rollout\nkubectl argo rollouts abort guestbook\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_abort/#options","title":"Options","text":" -h, --help help for abort\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_abort/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_abort/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_completion/","title":"Rollouts Completion","text":"Generate completion script
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_completion/#synopsis","title":"Synopsis","text":"To load completions:
Bash:\n\n $ source <(yourprogram completion bash)\n\n # To load completions for each session, execute once:\n # Linux:\n $ yourprogram completion bash > /etc/bash_completion.d/yourprogram\n # macOS:\n $ yourprogram completion bash > /usr/local/etc/bash_completion.d/yourprogram\n\nZsh:\n\n # If shell completion is not already enabled in your environment,\n # you will need to enable it. You can execute the following once:\n\n $ echo \"autoload -U compinit; compinit\" >> ~/.zshrc\n\n # To load completions for each session, execute once:\n $ yourprogram completion zsh > \"${fpath[1]}/_yourprogram\"\n\n # You will need to start a new shell for this setup to take effect.\n\nfish:\n\n $ yourprogram completion fish | source\n\n # To load completions for each session, execute once:\n $ yourprogram completion fish > ~/.config/fish/completions/yourprogram.fish\n\nPowerShell:\n\n PS> yourprogram completion powershell | Out-String | Invoke-Expression\n\n # To load completions for every new session, run:\n PS> yourprogram completion powershell > yourprogram.ps1\n # and source this file from your PowerShell profile.\n
kubectl argo rollouts completion [bash|zsh|fish|powershell]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_completion/#options","title":"Options","text":" -h, --help help for completion\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_completion/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_completion/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create/","title":"Rollouts Create","text":"Create a Rollout, Experiment, AnalysisTemplate, ClusterAnalysisTemplate, or AnalysisRun resource
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create/#synopsis","title":"Synopsis","text":"This command creates a new Rollout, Experiment, AnalysisTemplate, ClusterAnalysisTemplate, or AnalysisRun resource from a file.
kubectl argo rollouts create [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create/#examples","title":"Examples","text":"# Create an experiment and watch it\nkubectl argo rollouts create -f my-experiment.yaml -w\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create/#options","title":"Options","text":" -f, --filename stringArray Files to use to create the resource\n -h, --help help for create\n --no-color Do not colorize output\n -w, --watch Watch live updates to the resource after creating\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create/#available-commands","title":"Available Commands","text":" - rollouts create analysisrun - Create an AnalysisRun from an AnalysisTemplate or a ClusterAnalysisTemplate
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create_analysisrun/","title":"Rollouts Create Analysisrun","text":"Create an AnalysisRun from an AnalysisTemplate or a ClusterAnalysisTemplate
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create_analysisrun/#synopsis","title":"Synopsis","text":"This command creates a new AnalysisRun from an existing AnalysisTemplate resources or from an AnalysisTemplate file.
kubectl argo rollouts create analysisrun [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create_analysisrun/#examples","title":"Examples","text":"# Create an AnalysisRun from a local AnalysisTemplate file\nkubectl argo rollouts create analysisrun --from-file my-analysis-template.yaml\n\n# Create an AnalysisRun from a AnalysisTemplate in the cluster\nkubectl argo rollouts create analysisrun --from my-analysis-template\n\n# Create an AnalysisRun from a local ClusterAnalysisTemplate file\nkubectl argo rollouts create analysisrun --global --from my-analysis-cluster-template.yaml\n\n# Create an AnalysisRun from a ClusterAnalysisTemplate in the cluster\nkubectl argo rollouts create analysisrun --global --from my-analysis-cluster-template\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create_analysisrun/#options","title":"Options","text":" -a, --argument stringArray Arguments to the parameter template\n --from string Create an AnalysisRun from an AnalysisTemplate or ClusterAnalysisTemplate in the cluster\n --from-file string Create an AnalysisRun from an AnalysisTemplate or ClusterAnalysisTemplate in a local file\n --generate-name string Use the specified generateName for the run\n --global Use a ClusterAnalysisTemplate instead of a AnalysisTemplate\n -h, --help help for analysisrun\n --instance-id string Instance-ID for the AnalysisRun\n --name string Use the specified name for the run\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create_analysisrun/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create_analysisrun/#see-also","title":"See Also","text":" - rollouts create - Create a Rollout, Experiment, AnalysisTemplate, ClusterAnalysisTemplate, or AnalysisRun resource
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_dashboard/","title":"Rollouts Dashboard","text":"Start UI dashboard
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_dashboard/#synopsis","title":"Synopsis","text":"Start UI dashboard
kubectl argo rollouts dashboard [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_dashboard/#examples","title":"Examples","text":"# Start UI dashboard\nkubectl argo rollouts dashboard\n\n# Start UI dashboard on a specific port\nkubectl argo rollouts dashboard --port 8080\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_dashboard/#options","title":"Options","text":" -h, --help help for dashboard\n -p, --port int port to listen on (default 3100)\n --root-path string changes the root path of the dashboard (default \"rollouts\")\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_dashboard/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_dashboard/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get/","title":"Rollouts Get","text":"Get details about rollouts and experiments
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get/#synopsis","title":"Synopsis","text":"This command consists of multiple subcommands which can be used to get extended information about a rollout or experiment.
kubectl argo rollouts get <rollout|experiment> RESOURCE_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get/#examples","title":"Examples","text":"# Get a rollout\nkubectl argo rollouts get rollout guestbook\n\n# Watch a rollouts progress\nkubectl argo rollouts get rollout guestbook -w\n\n# Get an experiment\nkubectl argo rollouts get experiment my-experiment\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get/#options","title":"Options","text":" -h, --help help for get\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get/#available-commands","title":"Available Commands","text":" - rollouts get experiment - Get details about an Experiment
- rollouts get rollout - Get details about a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_experiment/","title":"Rollouts Get Experiment","text":"Get details about an Experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_experiment/#synopsis","title":"Synopsis","text":"Get details about and visual representation of a experiment. It returns a bunch of metadata on a resource and a tree view of the child resources created by the parent.
Tree view icons
Icon Kind \u27f3 Rollout \u03a3 Experiment \u03b1 AnalysisRun # Revision \u29c9 ReplicaSet \u25a1 Pod \u229e Job kubectl argo rollouts get experiment EXPERIMENT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_experiment/#examples","title":"Examples","text":"# Get an experiment\nkubectl argo rollouts get experiment my-experiment\n\n# Watch experiment progress\nkubectl argo rollouts get experiment my-experiment -w\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_experiment/#options","title":"Options","text":" -h, --help help for experiment\n --no-color Do not colorize output\n -w, --watch Watch live updates to the rollout\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_experiment/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_experiment/#see-also","title":"See Also","text":" - rollouts get - Get details about rollouts and experiments
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_rollout/","title":"Rollouts Get Rollout","text":"Get details about a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_rollout/#synopsis","title":"Synopsis","text":"Get details about and visual representation of a rollout. It returns a bunch of metadata on a resource and a tree view of the child resources created by the parent.
Tree view icons
Icon Kind \u27f3 Rollout \u03a3 Experiment \u03b1 AnalysisRun # Revision \u29c9 ReplicaSet \u25a1 Pod \u229e Job kubectl argo rollouts get rollout ROLLOUT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_rollout/#examples","title":"Examples","text":"# Get a rollout\nkubectl argo rollouts get rollout guestbook\n\n# Watch progress of a rollout\nkubectl argo rollouts get rollout guestbook -w\n\n# Watch the rollout, fail if it takes more than 60 seconds\nkubectl argo rollouts get rollout guestbook -w --timeout-seconds 60\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_rollout/#options","title":"Options","text":" -h, --help help for rollout\n --no-color Do not colorize output\n --timeout-seconds int Timeout after specified seconds\n -w, --watch Watch live updates to the rollout\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_rollout/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_rollout/#see-also","title":"See Also","text":" - rollouts get - Get details about rollouts and experiments
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_lint/","title":"Rollouts Lint","text":"Lint and validate a Rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_lint/#synopsis","title":"Synopsis","text":"This command lints and validates a new Rollout resource from a file.
kubectl argo rollouts lint [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_lint/#examples","title":"Examples","text":"# Lint a rollout\nkubectl argo rollouts lint -f my-rollout.yaml\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_lint/#options","title":"Options","text":" -f, --filename string File to lint\n -h, --help help for lint\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_lint/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_lint/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list/","title":"Rollouts List","text":"List rollouts or experiments
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list/#synopsis","title":"Synopsis","text":"This command consists of multiple subcommands which can be used to lists all of the rollouts or experiments for a specified namespace (uses current namespace context if namespace not specified).
kubectl argo rollouts list <rollout|experiment> [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list/#examples","title":"Examples","text":"# List rollouts\nkubectl argo rollouts list rollouts\n\n# List experiments\nkubectl argo rollouts list experiments\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list/#options","title":"Options","text":" -h, --help help for list\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list/#available-commands","title":"Available Commands","text":" - rollouts list experiments - List experiments
- rollouts list rollouts - List rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_experiments/","title":"Rollouts List Experiments","text":"List experiments
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_experiments/#synopsis","title":"Synopsis","text":"This command lists all of the experiments for a specified namespace (uses current namespace context if namespace not specified).
kubectl argo rollouts list experiments [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_experiments/#examples","title":"Examples","text":"# List rollouts\nkubectl argo rollouts list experiments\n\n# List rollouts from all namespaces\nkubectl argo rollouts list experiments --all-namespaces\n\n# List rollouts and watch for changes\nkubectl argo rollouts list experiments --watch\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_experiments/#options","title":"Options","text":" -A, --all-namespaces Include all namespaces\n -h, --help help for experiments\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_experiments/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_experiments/#see-also","title":"See Also","text":" - rollouts list - List rollouts or experiments
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_rollouts/","title":"Rollouts List Rollouts","text":"List rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_rollouts/#synopsis","title":"Synopsis","text":"This command lists all of the rollouts for a specified namespace (uses current namespace context if namespace not specified).
kubectl argo rollouts list rollouts [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_rollouts/#examples","title":"Examples","text":"# List rollouts\nkubectl argo rollouts list rollouts\n\n# List rollouts with a specific name\nkubectl argo rollouts list rollouts --name my-rollout\n\n# List rollouts from all namespaces\nkubectl argo rollouts list rollouts --all-namespaces\n\n# List rollouts and watch for changes\nkubectl argo rollouts list rollouts --watch\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_rollouts/#options","title":"Options","text":" -A, --all-namespaces Include all namespaces\n -h, --help help for rollouts\n --name string Only show rollout with specified name\n --timestamps Print timestamps on updates\n -w, --watch Watch for changes\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_rollouts/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_rollouts/#see-also","title":"See Also","text":" - rollouts list - List rollouts or experiments
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications/","title":"Rollouts Notifications","text":"Set of CLI commands that helps manage notifications settings
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications/#synopsis","title":"Synopsis","text":"Set of CLI commands that helps manage notifications settings
kubectl argo rollouts notifications [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications/#options","title":"Options","text":" --as string Username to impersonate for the operation\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --config-map string argo-rollouts-notification-configmap.yaml file path\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n -h, --help help for notifications\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n --kubeconfig string Path to a kube config. Only required if out-of-cluster\n -n, --namespace string If present, the namespace scope for this CLI request\n --password string Password for basic authentication to the API server\n --proxy-url string If provided, this URL will be used to connect via proxy\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --secret string argo-rollouts-notification-secret.yaml file path. Use empty secret if provided value is ':empty'\n --server string The address and port of the Kubernetes API server\n --tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n --username string Username for basic authentication to the API server\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n -v, --kloglevel int Log level for kubernetes client library\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications/#available-commands","title":"Available Commands","text":" - rollouts notifications template - Notification templates related commands
- rollouts notifications trigger - Notification triggers related commands
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template/","title":"Rollouts Notifications Template","text":"Notification templates related commands
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template/#synopsis","title":"Synopsis","text":"Notification templates related commands
kubectl argo rollouts notifications template [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template/#options","title":"Options","text":" -h, --help help for template\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --config-map string argo-rollouts-notification-configmap.yaml file path\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to a kube config. Only required if out-of-cluster\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --password string Password for basic authentication to the API server\n --proxy-url string If provided, this URL will be used to connect via proxy\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --secret string argo-rollouts-notification-secret.yaml file path. Use empty secret if provided value is ':empty'\n --server string The address and port of the Kubernetes API server\n --tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n --username string Username for basic authentication to the API server\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template/#available-commands","title":"Available Commands","text":" - rollouts notifications template get - Prints information about configured templates
- rollouts notifications template notify - Generates notification using the specified template and send it to specified recipients
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template/#see-also","title":"See Also","text":" - rollouts notifications - Set of CLI commands that helps manage notifications settings
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_get/","title":"Rollouts Notifications Template Get","text":"Prints information about configured templates
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_get/#synopsis","title":"Synopsis","text":"Prints information about configured templates
kubectl argo rollouts notifications template get [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_get/#examples","title":"Examples","text":"# prints all templates\nkubectl argo rollouts notifications template get\n# print YAML formatted app-sync-succeeded template definition\nkubectl argo rollouts notifications template get app-sync-succeeded -o=yaml\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_get/#options","title":"Options","text":" -h, --help help for get\n -o, --output string Output format. One of:json|yaml|wide|name (default \"wide\")\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_get/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --config-map string argo-rollouts-notification-configmap.yaml file path\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to a kube config. Only required if out-of-cluster\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --password string Password for basic authentication to the API server\n --proxy-url string If provided, this URL will be used to connect via proxy\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --secret string argo-rollouts-notification-secret.yaml file path. Use empty secret if provided value is ':empty'\n --server string The address and port of the Kubernetes API server\n --tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n --username string Username for basic authentication to the API server\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_get/#see-also","title":"See Also","text":" - rollouts notifications template - Notification templates related commands
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_notify/","title":"Rollouts Notifications Template Notify","text":"Generates notification using the specified template and send it to specified recipients
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_notify/#synopsis","title":"Synopsis","text":"Generates notification using the specified template and send it to specified recipients
kubectl argo rollouts notifications template notify NAME RESOURCE_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_notify/#examples","title":"Examples","text":"# Trigger notification using in-cluster config map and secret\nkubectl argo rollouts notifications template notify app-sync-succeeded guestbook --recipient slack:my-slack-channel\n\n# Render notification render generated notification in console\nkubectl argo rollouts notifications template notify app-sync-succeeded guestbook\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_notify/#options","title":"Options","text":" -h, --help help for notify\n --recipient stringArray List of recipients (default [console:stdout])\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_notify/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --config-map string argo-rollouts-notification-configmap.yaml file path\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to a kube config. Only required if out-of-cluster\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --password string Password for basic authentication to the API server\n --proxy-url string If provided, this URL will be used to connect via proxy\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --secret string argo-rollouts-notification-secret.yaml file path. Use empty secret if provided value is ':empty'\n --server string The address and port of the Kubernetes API server\n --tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n --username string Username for basic authentication to the API server\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_notify/#see-also","title":"See Also","text":" - rollouts notifications template - Notification templates related commands
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger/","title":"Rollouts Notifications Trigger","text":"Notification triggers related commands
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger/#synopsis","title":"Synopsis","text":"Notification triggers related commands
kubectl argo rollouts notifications trigger [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger/#options","title":"Options","text":" -h, --help help for trigger\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --config-map string argo-rollouts-notification-configmap.yaml file path\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to a kube config. Only required if out-of-cluster\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --password string Password for basic authentication to the API server\n --proxy-url string If provided, this URL will be used to connect via proxy\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --secret string argo-rollouts-notification-secret.yaml file path. Use empty secret if provided value is ':empty'\n --server string The address and port of the Kubernetes API server\n --tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n --username string Username for basic authentication to the API server\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger/#available-commands","title":"Available Commands","text":" - rollouts notifications trigger get - Prints information about configured triggers
- rollouts notifications trigger run - Evaluates specified trigger condition and prints the result
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger/#see-also","title":"See Also","text":" - rollouts notifications - Set of CLI commands that helps manage notifications settings
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_get/","title":"Rollouts Notifications Trigger Get","text":"Prints information about configured triggers
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_get/#synopsis","title":"Synopsis","text":"Prints information about configured triggers
kubectl argo rollouts notifications trigger get [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_get/#examples","title":"Examples","text":"# prints all triggers\nkubectl argo rollouts notifications trigger get\n# print YAML formatted on-sync-failed trigger definition\nkubectl argo rollouts notifications trigger get on-sync-failed -o=yaml\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_get/#options","title":"Options","text":" -h, --help help for get\n -o, --output string Output format. One of:json|yaml|wide|name (default \"wide\")\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_get/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --config-map string argo-rollouts-notification-configmap.yaml file path\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to a kube config. Only required if out-of-cluster\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --password string Password for basic authentication to the API server\n --proxy-url string If provided, this URL will be used to connect via proxy\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --secret string argo-rollouts-notification-secret.yaml file path. Use empty secret if provided value is ':empty'\n --server string The address and port of the Kubernetes API server\n --tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n --username string Username for basic authentication to the API server\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_get/#see-also","title":"See Also","text":" - rollouts notifications trigger - Notification triggers related commands
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_run/","title":"Rollouts Notifications Trigger Run","text":"Evaluates specified trigger condition and prints the result
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_run/#synopsis","title":"Synopsis","text":"Evaluates specified trigger condition and prints the result
kubectl argo rollouts notifications trigger run NAME RESOURCE_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_run/#examples","title":"Examples","text":"# Execute trigger configured in 'argocd-notification-cm' ConfigMap\nkubectl argo rollouts notifications trigger run on-sync-status-unknown ./sample-app.yaml\n\n# Execute trigger using my-config-map.yaml instead of 'argo-rollouts-notification-configmap' ConfigMap\nkubectl argo rollouts notifications trigger run on-sync-status-unknown ./sample-app.yaml \\\n--config-map ./my-config-map.yaml\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_run/#options","title":"Options","text":" -h, --help help for run\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_run/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --config-map string argo-rollouts-notification-configmap.yaml file path\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to a kube config. Only required if out-of-cluster\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --password string Password for basic authentication to the API server\n --proxy-url string If provided, this URL will be used to connect via proxy\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --secret string argo-rollouts-notification-secret.yaml file path. Use empty secret if provided value is ':empty'\n --server string The address and port of the Kubernetes API server\n --tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n --username string Username for basic authentication to the API server\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_run/#see-also","title":"See Also","text":" - rollouts notifications trigger - Notification triggers related commands
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_pause/","title":"Rollouts Pause","text":"Pause a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_pause/#synopsis","title":"Synopsis","text":"Set the rollout paused state to 'true'
kubectl argo rollouts pause ROLLOUT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_pause/#examples","title":"Examples","text":"# Pause a rollout\nkubectl argo rollouts pause guestbook\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_pause/#options","title":"Options","text":" -h, --help help for pause\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_pause/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_pause/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_promote/","title":"Rollouts Promote","text":"Promote a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_promote/#synopsis","title":"Synopsis","text":"Promote a rollout
Promotes a rollout paused at a canary step, or a paused blue-green pre-promotion. To skip analysis, pauses and steps entirely, use '--full' to fully promote the rollout
kubectl argo rollouts promote ROLLOUT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_promote/#examples","title":"Examples","text":"# Promote a paused rollout\nkubectl argo rollouts promote guestbook\n\n# Fully promote a rollout to desired version, skipping analysis, pauses, and steps\nkubectl argo rollouts promote guestbook --full\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_promote/#options","title":"Options","text":" --full Perform a full promotion, skipping analysis, pauses, and steps\n -h, --help help for promote\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_promote/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_promote/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_restart/","title":"Rollouts Restart","text":"Restart the pods of a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_restart/#synopsis","title":"Synopsis","text":"Restart the pods of a rollout
kubectl argo rollouts restart ROLLOUT [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_restart/#examples","title":"Examples","text":"# Restart the pods of a rollout in now\nkubectl argo rollouts restart ROLLOUT_NAME\n\n# Restart the pods of a rollout in ten seconds\nkubectl argo rollouts restart ROLLOUT_NAME --in 10s\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_restart/#options","title":"Options","text":" -h, --help help for restart\n -i, --in string Amount of time before a restart. (e.g. 30s, 5m, 1h)\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_restart/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_restart/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry/","title":"Rollouts Retry","text":"Retry a rollout or experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry/#synopsis","title":"Synopsis","text":"This command consists of multiple subcommands which can be used to restart an aborted rollout or a failed experiment.
kubectl argo rollouts retry <rollout|experiment> RESOURCE_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry/#examples","title":"Examples","text":"# Retry an aborted rollout\nkubectl argo rollouts retry rollout guestbook\n\n# Retry a failed experiment\nkubectl argo rollouts retry experiment my-experiment\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry/#options","title":"Options","text":" -h, --help help for retry\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry/#available-commands","title":"Available Commands","text":" - rollouts retry experiment - Retry an experiment
- rollouts retry rollout - Retry an aborted rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_experiment/","title":"Rollouts Retry Experiment","text":"Retry an experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_experiment/#synopsis","title":"Synopsis","text":"Retry a failed experiment.
kubectl argo rollouts retry experiment EXPERIMENT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_experiment/#examples","title":"Examples","text":"# Retry an experiment\nkubectl argo rollouts retry experiment my-experiment\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_experiment/#options","title":"Options","text":" -h, --help help for experiment\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_experiment/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_experiment/#see-also","title":"See Also","text":" - rollouts retry - Retry a rollout or experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_rollout/","title":"Rollouts Retry Rollout","text":"Retry an aborted rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_rollout/#synopsis","title":"Synopsis","text":"Retry an aborted rollout
kubectl argo rollouts retry rollout ROLLOUT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_rollout/#examples","title":"Examples","text":"# Retry an aborted rollout\nkubectl argo rollouts retry rollout guestbook\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_rollout/#options","title":"Options","text":" -h, --help help for rollout\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_rollout/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_rollout/#see-also","title":"See Also","text":" - rollouts retry - Retry a rollout or experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set/","title":"Rollouts Set","text":"Update various values on resources
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set/#synopsis","title":"Synopsis","text":"This command consists of multiple subcommands which can be used to update rollout resources.
kubectl argo rollouts set COMMAND [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set/#examples","title":"Examples","text":"# Set rollout image\nkubectl argo rollouts set image my-rollout demo=argoproj/rollouts-demo:yellow\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set/#options","title":"Options","text":" -h, --help help for set\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set/#available-commands","title":"Available Commands","text":" - rollouts set image - Update the image of a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set_image/","title":"Rollouts Set Image","text":"Update the image of a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set_image/#synopsis","title":"Synopsis","text":"Update the image of a rollout
kubectl argo rollouts set image ROLLOUT_NAME CONTAINER=IMAGE [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set_image/#examples","title":"Examples","text":"# Set rollout image (containers contains 'initContainer', 'container', 'ephemeralContainer')\nkubectl argo rollouts set image my-rollout containerName=imageName\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set_image/#options","title":"Options","text":" -h, --help help for image\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set_image/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set_image/#see-also","title":"See Also","text":" - rollouts set - Update various values on resources
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_status/","title":"Rollouts Status","text":"Show the status of a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_status/#synopsis","title":"Synopsis","text":"Watch rollout until it finishes or the timeout is exceeded. Returns success if the rollout is healthy upon completion and an error otherwise.
kubectl argo rollouts status ROLLOUT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_status/#examples","title":"Examples","text":"# Watch the rollout until it succeeds\nkubectl argo rollouts status guestbook\n\n# Show the rollout status\nkubectl argo rollouts status guestbook --watch false\n\n# Watch the rollout until it succeeds, fail if it takes more than 60 seconds\nkubectl argo rollouts status --timeout 60s guestbook\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_status/#options","title":"Options","text":" -h, --help help for status\n -t, --timeout duration The length of time to watch before giving up. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). Zero means wait forever\n -w, --watch Watch the status of the rollout until it's done (default true)\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_status/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_status/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate/","title":"Rollouts Terminate","text":"Terminate an AnalysisRun or Experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate/#synopsis","title":"Synopsis","text":"This command consists of multiple subcommands which can be used to terminate an AnalysisRun or Experiment that is in progress.
kubectl argo rollouts terminate <analysisrun|experiment> RESOURCE_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate/#examples","title":"Examples","text":"# Terminate an AnalysisRun\nkubectl argo rollouts terminate analysisrun guestbook-877894d5b-4-success-rate.1\n\n# Terminate a failed experiment\nkubectl argo rollouts terminate experiment my-experiment\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate/#options","title":"Options","text":" -h, --help help for terminate\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate/#available-commands","title":"Available Commands","text":" - rollouts terminate analysisrun - Terminate an AnalysisRun
- rollouts terminate experiment - Terminate an experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_analysisrun/","title":"Rollouts Terminate Analysisrun","text":"Terminate an AnalysisRun
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_analysisrun/#synopsis","title":"Synopsis","text":"This command terminates an AnalysisRun.
kubectl argo rollouts terminate analysisrun ANALYSISRUN_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_analysisrun/#examples","title":"Examples","text":"# Terminate an AnalysisRun\nkubectl argo rollouts terminate analysisrun guestbook-877894d5b-4-success-rate.1\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_analysisrun/#options","title":"Options","text":" -h, --help help for analysisrun\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_analysisrun/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_analysisrun/#see-also","title":"See Also","text":" - rollouts terminate - Terminate an AnalysisRun or Experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_experiment/","title":"Rollouts Terminate Experiment","text":"Terminate an experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_experiment/#synopsis","title":"Synopsis","text":"This command terminates an Experiment.
kubectl argo rollouts terminate experiment EXPERIMENT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_experiment/#examples","title":"Examples","text":"# Terminate an experiment\nkubectl argo rollouts terminate experiment my-experiment\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_experiment/#options","title":"Options","text":" -h, --help help for experiment\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_experiment/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_experiment/#see-also","title":"See Also","text":" - rollouts terminate - Terminate an AnalysisRun or Experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_undo/","title":"Rollouts Undo","text":"Undo a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_undo/#synopsis","title":"Synopsis","text":"Rollback to the previous rollout.
kubectl argo rollouts undo ROLLOUT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_undo/#examples","title":"Examples","text":"# Undo a rollout\nkubectl argo rollouts undo guestbook\n\n# Undo a rollout to revision 3\nkubectl argo rollouts undo guestbook --to-revision=3\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_undo/#options","title":"Options","text":" -h, --help help for undo\n --to-revision int The revision to rollback to. Default to 0 (last revision).\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_undo/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_undo/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_version/","title":"Rollouts Version","text":"Print version
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_version/#synopsis","title":"Synopsis","text":"Show the version and build information of the Argo Rollouts plugin.
kubectl argo rollouts version [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_version/#examples","title":"Examples","text":"# Get full version info\nkubectl argo rollouts version\n\n# Get just plugin version number\nkubectl argo rollouts version --short\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_version/#options","title":"Options","text":" -h, --help help for version\n --short print just the version number\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_version/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_version/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/notification-services/alertmanager/","title":"Alertmanager","text":""},{"location":"generated/notification-services/alertmanager/#parameters","title":"Parameters","text":"The notification service is used to push events to Alertmanager, and the following settings need to be specified:
targets - the alertmanager service address, array typescheme - optional, default is \"http\", e.g. http or httpsapiPath - optional, default is \"/api/v2/alerts\"insecureSkipVerify - optional, default is \"false\", when scheme is https whether to skip the verification of cabasicAuth - optional, server authbearerToken - optional, server authtimeout - optional, the timeout in seconds used when sending alerts, default is \"3 seconds\"
basicAuth or bearerToken is used for authentication, you can choose one. If the two are set at the same time, basicAuth takes precedence over bearerToken.
"},{"location":"generated/notification-services/alertmanager/#example","title":"Example","text":""},{"location":"generated/notification-services/alertmanager/#prometheus-alertmanager-config","title":"Prometheus Alertmanager config","text":"global:\n resolve_timeout: 5m\n\nroute:\n group_by: ['alertname']\n group_wait: 10s\n group_interval: 10s\n repeat_interval: 1h\n receiver: 'default'\nreceivers:\n- name: 'default'\n webhook_configs:\n - send_resolved: false\n url: 'http://10.5.39.39:10080/api/alerts/webhook'\n
You should turn off \"send_resolved\" or you will receive unnecessary recovery notifications after \"resolve_timeout\".
"},{"location":"generated/notification-services/alertmanager/#send-one-alertmanager-without-auth","title":"Send one alertmanager without auth","text":"apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.alertmanager: |\n targets:\n - 10.5.39.39:9093\n
"},{"location":"generated/notification-services/alertmanager/#send-alertmanager-cluster-with-custom-api-path","title":"Send alertmanager cluster with custom api path","text":"If your alertmanager has changed the default api, you can customize \"apiPath\".
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.alertmanager: |\n targets:\n - 10.5.39.39:443\n scheme: https\n apiPath: /api/events\n insecureSkipVerify: true\n
"},{"location":"generated/notification-services/alertmanager/#send-high-availability-alertmanager-with-auth","title":"Send high availability alertmanager with auth","text":"Store auth token in argo-rollouts-notification-secret Secret and use configure in argo-rollouts-notification-configmap ConfigMap.
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n alertmanager-username: <username>\n alertmanager-password: <password>\n alertmanager-bearer-token: <token>\n
- with basicAuth
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.alertmanager: |\n targets:\n - 10.5.39.39:19093\n - 10.5.39.39:29093\n - 10.5.39.39:39093\n scheme: https\n apiPath: /api/v2/alerts\n insecureSkipVerify: true\n basicAuth:\n username: $alertmanager-username\n password: $alertmanager-password \n
- with bearerToken
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.alertmanager: |\n targets:\n - 10.5.39.39:19093\n - 10.5.39.39:29093\n - 10.5.39.39:39093\n scheme: https\n apiPath: /api/v2/alerts\n insecureSkipVerify: true\n bearerToken: $alertmanager-bearer-token\n
"},{"location":"generated/notification-services/alertmanager/#templates","title":"Templates","text":" labels - at least one label pair required, implement different notification strategies according to alertmanager routingannotations - optional, specifies a set of information labels, which can be used to store longer additional information, but only for displaygeneratorURL - optional, default is '{{.app.spec.source.repoURL}}', backlink used to identify the entity that caused this alert in the client
the label or annotations or generatorURL values can be templated.
context: |\n argocdUrl: https://example.com/argocd\n\ntemplate.app-deployed: |\n message: Application {{.app.metadata.name}} has been healthy.\n alertmanager:\n labels:\n fault_priority: \"P5\"\n event_bucket: \"deploy\"\n event_status: \"succeed\"\n recipient: \"{{.recipient}}\"\n annotations:\n application: '<a href=\"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}\">{{.app.metadata.name}}</a>'\n author: \"{{(call .repo.GetCommitMetadata .app.status.sync.revision).Author}}\"\n message: \"{{(call .repo.GetCommitMetadata .app.status.sync.revision).Message}}\"\n
You can do targeted push on Alertmanager according to labels.
template.app-deployed: |\n message: Application {{.app.metadata.name}} has been healthy.\n alertmanager:\n labels:\n alertname: app-deployed\n fault_priority: \"P5\"\n event_bucket: \"deploy\"\n
There is a special label alertname. If you don\u2019t set its value, it will be equal to the template name by default.
"},{"location":"generated/notification-services/awssqs/","title":"AWS SQS","text":""},{"location":"generated/notification-services/awssqs/#parameters","title":"Parameters","text":"This notification service is capable of sending simple messages to AWS SQS queue.
queue - name of the queue you are intending to send messages to. Can be overridden with target destination annotation.region - region of the sqs queue can be provided via env variable AWS_DEFAULT_REGIONkey - optional, aws access key must be either referenced from a secret via variable or via env variable AWS_ACCESS_KEY_IDsecret - optional, aws access secret must be either referenced from a secret via variable or via env variable AWS_SECRET_ACCESS_KEYaccount optional, external accountId of the queueendpointUrl optional, useful for development with localstack
"},{"location":"generated/notification-services/awssqs/#example","title":"Example","text":""},{"location":"generated/notification-services/awssqs/#using-secret-for-credential-retrieval","title":"Using Secret for credential retrieval:","text":"Resource Annotation:
apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: nginx-deployment\n annotations:\n notifications.argoproj.io/subscribe.on-deployment-ready.awssqs: \"overwrite-myqueue\"\n
- ConfigMap
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.awssqs: |\n region: \"us-east-2\"\n queue: \"myqueue\"\n account: \"1234567\"\n key: \"$awsaccess_key\"\n secret: \"$awsaccess_secret\"\n\n template.deployment-ready: |\n message: |\n Deployment {{.obj.metadata.name}} is ready!\n\n trigger.on-deployment-ready: |\n - when: any(obj.status.conditions, {.type == 'Available' && .status == 'True'})\n send: [deployment-ready]\n - oncePer: obj.metadata.annotations[\"generation\"]\n
Secret apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n awsaccess_key: test\n awsaccess_secret: test\n
"},{"location":"generated/notification-services/awssqs/#minimal-configuration-using-aws-env-variables","title":"Minimal configuration using AWS Env variables","text":"Ensure the following list of environment variables are injected via OIDC, or another method. And assuming SQS is local to the account. You may skip usage of secret for sensitive data and omit other parameters. (Setting parameters via ConfigMap takes precedent.)
Variables:
export AWS_ACCESS_KEY_ID=\"test\"\nexport AWS_SECRET_ACCESS_KEY=\"test\"\nexport AWS_DEFAULT_REGION=\"us-east-1\"\n
Resource Annotation:
apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: nginx-deployment\n annotations:\n notifications.argoproj.io/subscribe.on-deployment-ready.awssqs: \"\"\n
- ConfigMap
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.awssqs: |\n queue: \"myqueue\"\n\n template.deployment-ready: |\n message: |\n Deployment {{.obj.metadata.name}} is ready!\n\n trigger.on-deployment-ready: |\n - when: any(obj.status.conditions, {.type == 'Available' && .status == 'True'})\n send: [deployment-ready]\n - oncePer: obj.metadata.annotations[\"generation\"]\n
"},{"location":"generated/notification-services/awssqs/#fifo-sqs-queues","title":"FIFO SQS Queues","text":"FIFO queues require a MessageGroupId to be sent along with every message, every message with a matching MessageGroupId will be processed one by one in order.
To send to a FIFO SQS Queue you must include a messageGroupId in the template such as in the example below:
template.deployment-ready: |\n message: |\n Deployment {{.obj.metadata.name}} is ready!\n messageGroupId: {{.obj.metadata.name}}-deployment\n
"},{"location":"generated/notification-services/email/","title":"Email","text":""},{"location":"generated/notification-services/email/#parameters","title":"Parameters","text":"The Email notification service sends email notifications using SMTP protocol and requires specifying the following settings:
host - the SMTP server host nameport - the SMTP server portusername - usernamepassword - passwordfrom - from email addresshtml - optional bool, true or falseinsecure_skip_verify - optional bool, true or false
"},{"location":"generated/notification-services/email/#example","title":"Example","text":"The following snippet contains sample Gmail service configuration:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.email.gmail: |\n username: $email-username\n password: $email-password\n host: smtp.gmail.com\n port: 465\n from: $email-username\n
Without authentication:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.email.example: |\n host: smtp.example.com\n port: 587\n from: $email-username\n
"},{"location":"generated/notification-services/email/#template","title":"Template","text":"Notification templates support specifying subject for email notifications:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n template.app-sync-succeeded: |\n email:\n subject: Application {{.app.metadata.name}} has been successfully synced.\n message: |\n {{if eq .serviceType \"slack\"}}:white_check_mark:{{end}} Application {{.app.metadata.name}} has been successfully synced at {{.app.status.operationState.finishedAt}}.\n Sync operation details are available at: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true .\n
"},{"location":"generated/notification-services/github/","title":"GitHub","text":""},{"location":"generated/notification-services/github/#parameters","title":"Parameters","text":"The GitHub notification service changes commit status using GitHub Apps and requires specifying the following settings:
appID - the app idinstallationID - the app installation idprivateKey - the app private keyenterpriseBaseURL - optional URL, e.g. https://git.example.com/
"},{"location":"generated/notification-services/github/#configuration","title":"Configuration","text":" - Create a GitHub Apps using https://github.com/settings/apps/new
- Change repository permissions to enable write commit statuses and/or deployments and/or pull requests comments
- Generate a private key, and download it automatically
- Install app to account
- Store privateKey in
argo-rollouts-notification-secret Secret and configure GitHub integration in argo-rollouts-notification-configmap ConfigMap
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.github: |\n appID: <app-id>\n installationID: <installation-id>\n privateKey: $github-privateKey\n
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n github-privateKey: |\n -----BEGIN RSA PRIVATE KEY-----\n (snip)\n -----END RSA PRIVATE KEY-----\n
- Create subscription for your GitHub integration
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.<trigger-name>.github: \"\"\n
"},{"location":"generated/notification-services/github/#templates","title":"Templates","text":"template.app-deployed: |\n message: |\n Application {{.app.metadata.name}} is now running new version of deployments manifests.\n github:\n repoURLPath: \"{{.app.spec.source.repoURL}}\"\n revisionPath: \"{{.app.status.operationState.syncResult.revision}}\"\n status:\n state: success\n label: \"continuous-delivery/{{.app.metadata.name}}\"\n targetURL: \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true\"\n deployment:\n state: success\n environment: production\n environmentURL: \"https://{{.app.metadata.name}}.example.com\"\n logURL: \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true\"\n requiredContexts: []\n autoMerge: true\n transientEnvironment: false\n pullRequestComment:\n content: |\n Application {{.app.metadata.name}} is now running new version of deployments manifests.\n See more here: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true\n
Notes: - If the message is set to 140 characters or more, it will be truncated. - If github.repoURLPath and github.revisionPath are same as above, they can be omitted. - Automerge is optional and true by default for github deployments to ensure the requested ref is up to date with the default branch. Setting this option to false is required if you would like to deploy older refs in your default branch. For more information see the GitHub Deployment API Docs. - If github.pullRequestComment.content is set to 65536 characters or more, it will be truncated.
"},{"location":"generated/notification-services/googlechat/","title":"Google Chat","text":""},{"location":"generated/notification-services/googlechat/#parameters","title":"Parameters","text":"The Google Chat notification service send message notifications to a google chat webhook. This service uses the following settings:
webhooks - a map of the form webhookName: webhookUrl
"},{"location":"generated/notification-services/googlechat/#configuration","title":"Configuration","text":" - Open
Google chat and go to the space to which you want to send messages - From the menu at the top of the page, select Configure Webhooks
- Under Incoming Webhooks, click Add Webhook
- Give a name to the webhook, optionally add an image and click Save
- Copy the URL next to your webhook
- Store the URL in
argocd-notification-secret and declare it in argo-rollouts-notification-configmap
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.googlechat: |\n webhooks:\n spaceName: $space-webhook-url\n
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n space-webhook-url: https://chat.googleapis.com/v1/spaces/<space_id>/messages?key=<key>&token=<token> \n
- Create a subscription for your space
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.googlechat: spaceName\n
"},{"location":"generated/notification-services/googlechat/#templates","title":"Templates","text":"You can send simple text or card messages to a Google Chat space. A simple text message template can be defined as follows:
template.app-sync-succeeded: |\n message: The app {{ .app.metadata.name }} has successfully synced!\n
A card message can be defined as follows:
template.app-sync-succeeded: |\n googlechat:\n cardsV2: |\n - header:\n title: ArgoCD Bot Notification\n sections:\n - widgets:\n - decoratedText:\n text: The app {{ .app.metadata.name }} has successfully synced!\n - widgets:\n - decoratedText:\n topLabel: Repository\n text: {{ call .repo.RepoURLToHTTPS .app.spec.source.repoURL }}\n - decoratedText:\n topLabel: Revision\n text: {{ .app.spec.source.targetRevision }}\n - decoratedText:\n topLabel: Author\n text: {{ (call .repo.GetCommitMetadata .app.status.sync.revision).Author }}\n
All Card fields are supported and can be used in notifications. It is also possible to use the previous (now deprecated) cards key to use the legacy card fields, but this is not recommended as Google has deprecated this field and recommends using the newer cardsV2. The card message can be written in JSON too.
"},{"location":"generated/notification-services/googlechat/#chat-threads","title":"Chat Threads","text":"It is possible send both simple text and card messages in a chat thread by specifying a unique key for the thread. The thread key can be defined as follows:
template.app-sync-succeeded: |\n message: The app {{ .app.metadata.name }} has successfully synced!\n googlechat:\n threadKey: {{ .app.metadata.name }}\n
"},{"location":"generated/notification-services/grafana/","title":"Grafana","text":"To be able to create Grafana annotation with argocd-notifications you have to create an API Key inside your Grafana.
Available parameters :
apiURL - the server url, e.g. https://grafana.example.comapiKey - the API key for the serviceaccount-
insecureSkipVerify - optional bool, true or false
-
Login to your Grafana instance as admin
- On the left menu, go to Configuration / API Keys
- Click \"Add API Key\"
- Fill the Key with name
ArgoCD Notification, role Editor and Time to Live 10y (for example) - Click on Add button
- Store apiKey in
argo-rollouts-notification-secret Secret and Copy your API Key and define it in argo-rollouts-notification-configmap ConfigMap
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.grafana: |\n apiUrl: https://grafana.example.com/api\n apiKey: $grafana-api-key\n
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n grafana-api-key: api-key\n
- Create subscription for your Grafana integration
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.<trigger-name>.grafana: tag1|tag2 # list of tags separated with |\n
- Change the annotations settings
"},{"location":"generated/notification-services/mattermost/","title":"Mattermost","text":""},{"location":"generated/notification-services/mattermost/#parameters","title":"Parameters","text":" apiURL - the server url, e.g. https://mattermost.example.comtoken - the bot tokeninsecureSkipVerify - optional bool, true or false
"},{"location":"generated/notification-services/mattermost/#configuration","title":"Configuration","text":" - Create a bot account and copy token after creating it
- Invite team
- Store token in
argo-rollouts-notification-secret Secret and configure Mattermost integration in argo-rollouts-notification-configmap ConfigMap
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.mattermost: |\n apiURL: <api-url>\n token: $mattermost-token\n
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n mattermost-token: token\n
-
Copy channel id
-
Create subscription for your Mattermost integration
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.<trigger-name>.mattermost: <channel-id>\n
"},{"location":"generated/notification-services/mattermost/#templates","title":"Templates","text":"You can reuse the template of slack. Mattermost is compatible with attachments of Slack. See Mattermost Integration Guide.
template.app-deployed: |\n message: |\n Application {{.app.metadata.name}} is now running new version of deployments manifests.\n mattermost:\n attachments: |\n [{\n \"title\": \"{{.app.metadata.name}}\",\n \"title_link\": \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}\",\n \"color\": \"#18be52\",\n \"fields\": [{\n \"title\": \"Sync Status\",\n \"value\": \"{{.app.status.sync.status}}\",\n \"short\": true\n }, {\n \"title\": \"Repository\",\n \"value\": \"{{.app.spec.source.repoURL}}\",\n \"short\": true\n }]\n }]\n
"},{"location":"generated/notification-services/newrelic/","title":"NewRelic","text":""},{"location":"generated/notification-services/newrelic/#parameters","title":"Parameters","text":" apiURL - the api server url, e.g. https://api.newrelic.comapiKey - a NewRelic ApiKey
"},{"location":"generated/notification-services/newrelic/#configuration","title":"Configuration","text":" - Create a NewRelic Api Key
- Store apiKey in
argo-rollouts-notification-secret Secret and configure NewRelic integration in argo-rollouts-notification-configmap ConfigMap
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.newrelic: |\n apiURL: <api-url>\n apiKey: $newrelic-apiKey\n
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n newrelic-apiKey: apiKey\n
- Copy Application ID
- Create subscription for your NewRelic integration
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.<trigger-name>.newrelic: <app-id>\n
"},{"location":"generated/notification-services/newrelic/#templates","title":"Templates","text":" description - optional, high-level description of this deployment, visible in the Summary page and on the Deployments page when you select an individual deployment.- Defaults to
message changelog - optional, A summary of what changed in this deployment, visible in the Deployments page when you select (selected deployment) > Change log.- Defaults to
{{(call .repo.GetCommitMetadata .app.status.sync.revision).Message}} user - optional, A username to associate with the deployment, visible in the Summary and on the Deployments.- Defaults to
{{(call .repo.GetCommitMetadata .app.status.sync.revision).Author}}
context: |\n argocdUrl: https://example.com/argocd\n\ntemplate.app-deployed: |\n message: Application {{.app.metadata.name}} has successfully deployed.\n newrelic:\n description: Application {{.app.metadata.name}} has successfully deployed\n
"},{"location":"generated/notification-services/opsgenie/","title":"Opsgenie","text":"To be able to send notifications with argocd-notifications you have to create an API Integration inside your Opsgenie Team.
- Login to Opsgenie at https://app.opsgenie.com or https://app.eu.opsgenie.com (if you have an account in the european union)
- Make sure you already have a team, if not follow this guide https://docs.opsgenie.com/docs/teams
- Click \"Teams\" in the Menu on the left
- Select the team that you want to notify
- In the teams configuration menu select \"Integrations\"
- Click \"Add Integration\" in the top right corner
- Select \"API\" integration
- Give your integration a name, copy the \"API key\" and safe it somewhere for later
- Click \"Edit\" in the integration settings
- Make sure the checkbox for \"Create and Update Access\" is selected, disable the other checkboxes to remove unnecessary permissions
- Click \"Save\" at the bottom
- Click \"Turn on integration\" in the top right corner
- Check your browser for the correct server apiURL. If it is \"app.opsgenie.com\" then use the US/international api url
api.opsgenie.com in the next step, otherwise use api.eu.opsgenie.com (European API). - You are finished with configuring Opsgenie. Now you need to configure argocd-notifications. Use the apiUrl, the team name and the apiKey to configure the Opsgenie integration in the
argo-rollouts-notification-secret secret. - You can find the example
argo-rollouts-notification-configmap configuration at the below.
Option Required Type Description Example description True string Description field of the alert that is generally used to provide a detailed information about the alert. Hello from Argo CD! priority False string Priority level of the alert. Possible values are P1, P2, P3, P4 and P5. Default value is P3. P1 alias False string Client-defined identifier of the alert, that is also the key element of Alert De-Duplication. Life is too short for no alias note False string Additional note that will be added while creating the alert. Error from Argo CD! apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.opsgenie: |\n apiUrl: <api-url>\n apiKeys:\n <your-team>: <integration-api-key>\n template.opsgenie: |\n message: |\n [Argo CD] Application {{.app.metadata.name}} has a problem.\n opsgenie:\n description: |\n Application: {{.app.metadata.name}}\n Health Status: {{.app.status.health.status}}\n Operation State Phase: {{.app.status.operationState.phase}}\n Sync Status: {{.app.status.sync.status}}\n priority: P1\n alias: {{.app.metadata.name}}\n note: Error from Argo CD!\n trigger.on-a-problem: |\n - description: Application has a problem.\n send:\n - opsgenie\n when: app.status.health.status == 'Degraded' or app.status.operationState.phase in ['Error', 'Failed'] or app.status.sync.status == 'Unknown'\n
- Add annotation in application yaml file to enable notifications for specific Argo CD app.
apiVersion: argoproj.io/v1alpha1\n kind: Application\n metadata:\n annotations:\n notifications.argoproj.io/subscribe.on-a-problem.opsgenie: <your-team>\n
"},{"location":"generated/notification-services/overview/","title":"Overview","text":"The notification services represent integration with services such as slack, email or custom webhook. Services are configured in argo-rollouts-notification-configmap ConfigMap using service.<type>.(<custom-name>) keys and might reference sensitive data from argo-rollouts-notification-secret Secret. Following example demonstrates slack service configuration:
service.slack: |\n token: $slack-token\n
The slack indicates that service sends slack notification; name is missing and defaults to slack.
"},{"location":"generated/notification-services/overview/#sensitive-data","title":"Sensitive Data","text":"Sensitive data like authentication tokens should be stored in <secret-name> Secret and can be referenced in service configuration using $<secret-key> format. For example $slack-token referencing value of key slack-token in <secret-name> Secret.
"},{"location":"generated/notification-services/overview/#custom-names","title":"Custom Names","text":"Service custom names allow configuring two instances of the same service type.
service.slack.workspace1: |\n token: $slack-token-workspace1\n service.slack.workspace2: |\n token: $slack-token-workspace2\n
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.workspace1: my-channel\n notifications.argoproj.io/subscribe.on-sync-succeeded.workspace2: my-channel\n
"},{"location":"generated/notification-services/overview/#service-types","title":"Service Types","text":" - AwsSqs
- Email
- GitHub
- Slack
- Mattermost
- Opsgenie
- Grafana
- Webhook
- Telegram
- Teams
- Google Chat
- Rocket.Chat
- Pushover
- Alertmanager
"},{"location":"generated/notification-services/pagerduty/","title":"PagerDuty","text":""},{"location":"generated/notification-services/pagerduty/#parameters","title":"Parameters","text":"The PagerDuty notification service is used to create PagerDuty incidents and requires specifying the following settings:
pagerdutyToken - the PagerDuty auth tokenfrom - email address of a valid user associated with the account making the request.serviceID - The ID of the resource.
"},{"location":"generated/notification-services/pagerduty/#example","title":"Example","text":"The following snippet contains sample PagerDuty service configuration:
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n pagerdutyToken: <pd-api-token>\n
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.pagerduty: |\n token: $pagerdutyToken\n from: <emailid>\n
"},{"location":"generated/notification-services/pagerduty/#template","title":"Template","text":"Notification templates support specifying subject for PagerDuty notifications:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n template.rollout-aborted: |\n message: Rollout {{.rollout.metadata.name}} is aborted.\n pagerduty:\n title: \"Rollout {{.rollout.metadata.name}}\"\n urgency: \"high\"\n body: \"Rollout {{.rollout.metadata.name}} aborted \"\n priorityID: \"<priorityID of incident>\"\n
NOTE: A Priority is a label representing the importance and impact of an incident. This is only available on Standard and Enterprise plans of pagerduty.
"},{"location":"generated/notification-services/pagerduty/#annotation","title":"Annotation","text":"Annotation sample for pagerduty notifications:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-rollout-aborted.pagerduty: \"<serviceID for PagerDuty>\"\n
"},{"location":"generated/notification-services/pagerduty_v2/","title":"PagerDuty V2","text":""},{"location":"generated/notification-services/pagerduty_v2/#parameters","title":"Parameters","text":"The PagerDuty notification service is used to trigger PagerDuty events and requires specifying the following settings:
serviceKeys - a dictionary with the following structure:service-name: $pagerduty-key-service-name where service-name is the name you want to use for the service to make events for, and $pagerduty-key-service-name is a reference to the secret that contains the actual PagerDuty integration key (Events API v2 integration)
If you want multiple Argo apps to trigger events to their respective PagerDuty services, create an integration key in each service you want to setup alerts for.
To create a PagerDuty integration key, follow these instructions to add an Events API v2 integration to the service of your choice.
"},{"location":"generated/notification-services/pagerduty_v2/#configuration","title":"Configuration","text":"The following snippet contains sample PagerDuty service configuration. It assumes the service you want to alert on is called my-service.
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n pagerduty-key-my-service: <pd-integration-key>\n
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.pagerdutyv2: |\n serviceKeys:\n my-service: $pagerduty-key-my-service\n
"},{"location":"generated/notification-services/pagerduty_v2/#template","title":"Template","text":"Notification templates support specifying subject for PagerDuty notifications:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n template.rollout-aborted: |\n message: Rollout {{.rollout.metadata.name}} is aborted.\n pagerdutyv2:\n summary: \"Rollout {{.rollout.metadata.name}} is aborted.\"\n severity: \"critical\"\n source: \"{{.rollout.metadata.name}}\"\n
The parameters for the PagerDuty configuration in the template generally match with the payload for the Events API v2 endpoint. All parameters are strings.
summary - (required) A brief text summary of the event, used to generate the summaries/titles of any associated alerts.severity - (required) The perceived severity of the status the event is describing with respect to the affected system. Allowed values: critical, warning, error, infosource - (required) The unique location of the affected system, preferably a hostname or FQDN.component - Component of the source machine that is responsible for the event.group - Logical grouping of components of a service.class - The class/type of the event.url - The URL that should be used for the link \"View in ArgoCD\" in PagerDuty.
The timestamp and custom_details parameters are not currently supported.
"},{"location":"generated/notification-services/pagerduty_v2/#annotation","title":"Annotation","text":"Annotation sample for PagerDuty notifications:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-rollout-aborted.pagerdutyv2: \"<serviceID for PagerDuty>\"\n
"},{"location":"generated/notification-services/pushover/","title":"Pushover","text":" - Create an app at pushover.net.
- Store the API key in
<secret-name> Secret and define the secret name in argo-rollouts-notification-configmap ConfigMap:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.pushover: |\n token: $pushover-token\n
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n pushover-token: avtc41pn13asmra6zaiyf7dh6cgx97\n
- Add your user key to your Application resource:
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.pushover: uumy8u4owy7bgkapp6mc5mvhfsvpcd\n
"},{"location":"generated/notification-services/rocketchat/","title":"Rocket.Chat","text":""},{"location":"generated/notification-services/rocketchat/#parameters","title":"Parameters","text":"The Rocket.Chat notification service configuration includes following settings:
email - the Rocker.Chat user's emailpassword - the Rocker.Chat user's passwordalias - optional alias that should be used to post messageicon - optional message iconavatar - optional message avatarserverUrl - optional Rocket.Chat server url
"},{"location":"generated/notification-services/rocketchat/#configuration","title":"Configuration","text":" - Login to your RocketChat instance
- Go to user management
- Add new user with
bot role. Also note that Require password change checkbox mus be not checked
- Copy username and password that you was created for bot user
- Create a public or private channel, or a team, for this example
my_channel - Add your bot to this channel otherwise it won't work
- Store email and password in argocd_notifications-secret Secret
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n rocketchat-email: <email>\n rocketchat-password: <password>\n
- Finally, use these credentials to configure the RocketChat integration in the
argocd-configmap config map:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.rocketchat: |\n email: $rocketchat-email\n password: $rocketchat-password\n
- Create a subscription for your Rocket.Chat integration:
Note: channel, team or user must be prefixed with # or @ elsewhere we will be interpretative destination as a room ID
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.rocketchat: #my_channel\n
"},{"location":"generated/notification-services/rocketchat/#templates","title":"Templates","text":"Notification templates can be customized with RocketChat attachments.
Note: Attachments structure in Rocketchat is same with Slack attachments feature.
The message attachments can be specified in attachments string fields under rocketchat field:
template.app-sync-status: |\n message: |\n Application {{.app.metadata.name}} sync is {{.app.status.sync.status}}.\n Application details: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}.\n rocketchat:\n attachments: |\n [{\n \"title\": \"{{.app.metadata.name}}\",\n \"title_link\": \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}\",\n \"color\": \"#18be52\",\n \"fields\": [{\n \"title\": \"Sync Status\",\n \"value\": \"{{.app.status.sync.status}}\",\n \"short\": true\n }, {\n \"title\": \"Repository\",\n \"value\": \"{{.app.spec.source.repoURL}}\",\n \"short\": true\n }]\n }]\n
"},{"location":"generated/notification-services/slack/","title":"Slack","text":"If you want to send message using incoming webhook, you can use webhook.
"},{"location":"generated/notification-services/slack/#parameters","title":"Parameters","text":"The Slack notification service configuration includes following settings:
Option Required Type Description Example apiURL False string The server URL. https://example.com/api channels False list[string] [\"my-channel-1\", \"my-channel-2\"] icon False string The app icon. :robot_face: or https://example.com/image.png insecureSkipVerify False bool true signingSecret False string 8f742231b10e8888abcd99yyyzzz85a5 token True string The app's OAuth access token. xoxb-1234567890-1234567890123-5n38u5ed63fgzqlvuyxvxcx6 username False string The app username. argocd disableUnfurl False bool Disable slack unfurling links in messages true"},{"location":"generated/notification-services/slack/#configuration","title":"Configuration","text":" - Create Slack Application using https://api.slack.com/apps?new_app=1
- Once application is created navigate to
Enter OAuth & Permissions - Click
Permissions under Add features and functionality section and add chat:write scope. To use the optional username and icon overrides in the Slack notification service also add the chat:write.customize scope. - Scroll back to the top, click 'Install App to Workspace' button and confirm the installation.
-
Once installation is completed copy the OAuth token.
-
Create a public or private channel, for this example my_channel
- Invite your slack bot to this channel otherwise slack bot won't be able to deliver notifications to this channel
-
Store Oauth access token in argo-rollouts-notification-secret secret
apiVersion: v1\n kind: Secret\n metadata:\n name: <secret-name>\n stringData:\n slack-token: <Oauth-access-token>\n
-
Define service type slack in data section of argo-rollouts-notification-configmap configmap:
apiVersion: v1\n kind: ConfigMap\n metadata:\n name: argo-rollouts-notification-configmap\n data:\n service.slack: |\n token: $slack-token\n
-
Add annotation in application yaml file to enable notifications for specific argocd app. The following example uses the on-sync-succeeded trigger:
apiVersion: argoproj.io/v1alpha1\n kind: Application\n metadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.slack: my_channel\n
-
Annotation with more than one trigger, with multiple destinations and recipients
apiVersion: argoproj.io/v1alpha1\n kind: Application\n metadata:\n annotations:\n notifications.argoproj.io/subscriptions: |\n - trigger: [on-scaling-replica-set, on-rollout-updated, on-rollout-step-completed]\n destinations:\n - service: slack\n recipients: [my-channel-1, my-channel-2]\n - service: email\n recipients: [recipient-1, recipient-2, recipient-3 ]\n - trigger: [on-rollout-aborted, on-analysis-run-failed, on-analysis-run-error]\n destinations:\n - service: slack\n recipients: [my-channel-21, my-channel-22]\n
"},{"location":"generated/notification-services/slack/#templates","title":"Templates","text":"Notification templates can be customized to leverage slack message blocks and attachments feature.
The message blocks and attachments can be specified in blocks and attachments string fields under slack field:
template.app-sync-status: |\n message: |\n Application {{.app.metadata.name}} sync is {{.app.status.sync.status}}.\n Application details: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}.\n slack:\n attachments: |\n [{\n \"title\": \"{{.app.metadata.name}}\",\n \"title_link\": \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}\",\n \"color\": \"#18be52\",\n \"fields\": [{\n \"title\": \"Sync Status\",\n \"value\": \"{{.app.status.sync.status}}\",\n \"short\": true\n }, {\n \"title\": \"Repository\",\n \"value\": \"{{.app.spec.source.repoURL}}\",\n \"short\": true\n }]\n }]\n
The messages can be aggregated to the slack threads by grouping key which can be specified in a groupingKey string field under slack field. groupingKey is used across each template and works independently on each slack channel. When multiple applications will be updated at the same time or frequently, the messages in slack channel can be easily read by aggregating with git commit hash, application name, etc. Furthermore, the messages can be broadcast to the channel at the specific template by notifyBroadcast field.
template.app-sync-status: |\n message: |\n Application {{.app.metadata.name}} sync is {{.app.status.sync.status}}.\n Application details: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}.\n slack:\n attachments: |\n [{\n \"title\": \"{{.app.metadata.name}}\",\n \"title_link\": \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}\",\n \"color\": \"#18be52\",\n \"fields\": [{\n \"title\": \"Sync Status\",\n \"value\": \"{{.app.status.sync.status}}\",\n \"short\": true\n }, {\n \"title\": \"Repository\",\n \"value\": \"{{.app.spec.source.repoURL}}\",\n \"short\": true\n }]\n }]\n # Aggregate the messages to the thread by git commit hash\n groupingKey: \"{{.app.status.sync.revision}}\"\n notifyBroadcast: false\ntemplate.app-sync-failed: |\n message: |\n Application {{.app.metadata.name}} sync is {{.app.status.sync.status}}.\n Application details: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}.\n slack:\n attachments: |\n [{\n \"title\": \"{{.app.metadata.name}}\",\n \"title_link\": \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}\",\n \"color\": \"#ff0000\",\n \"fields\": [{\n \"title\": \"Sync Status\",\n \"value\": \"{{.app.status.sync.status}}\",\n \"short\": true\n }, {\n \"title\": \"Repository\",\n \"value\": \"{{.app.spec.source.repoURL}}\",\n \"short\": true\n }]\n }]\n # Aggregate the messages to the thread by git commit hash\n groupingKey: \"{{.app.status.sync.revision}}\"\n notifyBroadcast: true\n
The message is sent according to the deliveryPolicy string field under the slack field. The available modes are Post (default), PostAndUpdate, and Update. The PostAndUpdate and Update settings require groupingKey to be set.
"},{"location":"generated/notification-services/teams/","title":"Teams","text":""},{"location":"generated/notification-services/teams/#parameters","title":"Parameters","text":"The Teams notification service send message notifications using Teams bot and requires specifying the following settings:
recipientUrls - the webhook url map, e.g. channelName: https://example.com
"},{"location":"generated/notification-services/teams/#configuration","title":"Configuration","text":" - Open
Teams and goto Apps - Find
Incoming Webhook microsoft app and click on it - Press
Add to a team -> select team and channel -> press Set up a connector - Enter webhook name and upload image (optional)
- Press
Create then copy webhook url and store it in argo-rollouts-notification-secret and define it in argo-rollouts-notification-configmap
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.teams: |\n recipientUrls:\n channelName: $channel-teams-url\n
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n channel-teams-url: https://example.com\n
- Create subscription for your Teams integration:
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.teams: channelName\n
"},{"location":"generated/notification-services/teams/#templates","title":"Templates","text":"Notification templates can be customized to leverage teams message sections, facts, themeColor, summary and potentialAction feature.
template.app-sync-succeeded: |\n teams:\n themeColor: \"#000080\"\n sections: |\n [{\n \"facts\": [\n {\n \"name\": \"Sync Status\",\n \"value\": \"{{.app.status.sync.status}}\"\n },\n {\n \"name\": \"Repository\",\n \"value\": \"{{.app.spec.source.repoURL}}\"\n }\n ]\n }]\n potentialAction: |-\n [{\n \"@type\":\"OpenUri\",\n \"name\":\"Operation Details\",\n \"targets\":[{\n \"os\":\"default\",\n \"uri\":\"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true\"\n }]\n }]\n title: Application {{.app.metadata.name}} has been successfully synced\n text: Application {{.app.metadata.name}} has been successfully synced at {{.app.status.operationState.finishedAt}}.\n summary: \"{{.app.metadata.name}} sync succeeded\"\n
"},{"location":"generated/notification-services/teams/#facts-field","title":"facts field","text":"You can use facts field instead of sections field.
template.app-sync-succeeded: |\n teams:\n facts: |\n [{\n \"name\": \"Sync Status\",\n \"value\": \"{{.app.status.sync.status}}\"\n },\n {\n \"name\": \"Repository\",\n \"value\": \"{{.app.spec.source.repoURL}}\"\n }]\n
"},{"location":"generated/notification-services/teams/#theme-color-field","title":"theme color field","text":"You can set theme color as hex string for the message.
template.app-sync-succeeded: |\n teams:\n themeColor: \"#000080\"\n
"},{"location":"generated/notification-services/teams/#summary-field","title":"summary field","text":"You can set a summary of the message that will be shown on Notification & Activity Feed
template.app-sync-succeeded: |\n teams:\n summary: \"Sync Succeeded\"\n
"},{"location":"generated/notification-services/telegram/","title":"Telegram","text":" - Get an API token using @Botfather.
- Store token in
<secret-name> Secret and configure telegram integration in argo-rollouts-notification-configmap ConfigMap:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.telegram: |\n token: $telegram-token\n
- Create new Telegram channel.
- Add your bot as an administrator.
- Use this channel
username (public channel) or chatID (private channel) in the subscription for your Telegram integration:
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.telegram: username\n
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.telegram: -1000000000000\n
If your private chat contains threads, you can optionally specify a thread id by seperating it with a |:
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.telegram: -1000000000000|2\n
"},{"location":"generated/notification-services/webex/","title":"Webex Teams","text":""},{"location":"generated/notification-services/webex/#parameters","title":"Parameters","text":"The Webex Teams notification service configuration includes following settings:
token - the app token
"},{"location":"generated/notification-services/webex/#configuration","title":"Configuration","text":" - Create a Webex Bot
-
Copy the bot access token and store it in the argo-rollouts-notification-secret Secret and configure Webex Teams integration in argo-rollouts-notification-configmap ConfigMap
apiVersion: v1\nkind: Secret\nmetadata:\nname: <secret-name>\nstringData:\nwebex-token: <bot access token>\n
apiVersion: v1\nkind: ConfigMap\nmetadata:\nname: argo-rollouts-notification-configmap\ndata:\nservice.webex: |\n token: $webex-token\n
-
Create subscription for your Webex Teams integration
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\nannotations:\n notifications.argoproj.io/subscribe.<trigger-name>.webex: <personal email or room id>\n
"},{"location":"generated/notification-services/webhook/","title":"Webhook","text":"The webhook notification service allows sending a generic HTTP request using the templatized request body and URL. Using Webhook you might trigger a Jenkins job, update GitHub commit status.
"},{"location":"generated/notification-services/webhook/#parameters","title":"Parameters","text":"The Webhook notification service configuration includes following settings:
url - the url to send the webhook toheaders - optional, the headers to pass along with the webhookbasicAuth - optional, the basic authentication to pass along with the webhookinsecureSkipVerify - optional bool, true or falseretryWaitMin - Optional, the minimum wait time between retries. Default value: 1s.retryWaitMax - Optional, the maximum wait time between retries. Default value: 5s.retryMax - Optional, the maximum number of retries. Default value: 3.
"},{"location":"generated/notification-services/webhook/#retry-behavior","title":"Retry Behavior","text":"The webhook service will automatically retry the request if it fails due to network errors or if the server returns a 5xx status code. The number of retries and the wait time between retries can be configured using the retryMax, retryWaitMin, and retryWaitMax parameters.
The wait time between retries is between retryWaitMin and retryWaitMax. If all retries fail, the Send method will return an error.
"},{"location":"generated/notification-services/webhook/#configuration","title":"Configuration","text":"Use the following steps to configure webhook:
1 Register webhook in argo-rollouts-notification-configmap ConfigMap:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.webhook.<webhook-name>: |\n url: https://<hostname>/<optional-path>\n headers: #optional headers\n - name: <header-name>\n value: <header-value>\n basicAuth: #optional username password\n username: <username>\n password: <api-key>\n insecureSkipVerify: true #optional bool\n
2 Define template that customizes webhook request method, path and body:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n template.github-commit-status: |\n webhook:\n <webhook-name>:\n method: POST # one of: GET, POST, PUT, PATCH. Default value: GET \n path: <optional-path-template>\n body: |\n <optional-body-template>\n trigger.<trigger-name>: |\n - when: app.status.operationState.phase in ['Succeeded']\n send: [github-commit-status]\n
3 Create subscription for webhook integration:
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.<trigger-name>.<webhook-name>: \"\"\n
"},{"location":"generated/notification-services/webhook/#examples","title":"Examples","text":""},{"location":"generated/notification-services/webhook/#set-github-commit-status","title":"Set GitHub commit status","text":"apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.webhook.github: |\n url: https://api.github.com\n headers: #optional headers\n - name: Authorization\n value: token $github-token\n
2 Define template that customizes webhook request method, path and body:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.webhook.github: |\n url: https://api.github.com\n headers: #optional headers\n - name: Authorization\n value: token $github-token\n\n template.github-commit-status: |\n webhook:\n github:\n method: POST\n path: /repos/{{call .repo.FullNameByRepoURL .app.spec.source.repoURL}}/statuses/{{.app.status.operationState.operation.sync.revision}}\n body: |\n {\n {{if eq .app.status.operationState.phase \"Running\"}} \"state\": \"pending\"{{end}}\n {{if eq .app.status.operationState.phase \"Succeeded\"}} \"state\": \"success\"{{end}}\n {{if eq .app.status.operationState.phase \"Error\"}} \"state\": \"error\"{{end}}\n {{if eq .app.status.operationState.phase \"Failed\"}} \"state\": \"error\"{{end}},\n \"description\": \"ArgoCD\",\n \"target_url\": \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}\",\n \"context\": \"continuous-delivery/{{.app.metadata.name}}\"\n }\n
"},{"location":"generated/notification-services/webhook/#start-jenkins-job","title":"Start Jenkins Job","text":"apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.webhook.jenkins: |\n url: http://<jenkins-host>/job/<job-name>/build?token=<job-secret>\n basicAuth:\n username: <username>\n password: <api-key>\n\ntype: Opaque\n
"},{"location":"generated/notification-services/webhook/#send-form-data","title":"Send form-data","text":"apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.webhook.form: |\n url: https://form.example.com\n headers:\n - name: Content-Type\n value: application/x-www-form-urlencoded\n\n template.form-data: |\n webhook:\n form:\n method: POST\n body: key1=value1&key2=value2\n
"},{"location":"generated/notification-services/webhook/#send-slack","title":"Send Slack","text":"apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.webhook.slack_webhook: |\n url: https://hooks.slack.com/services/xxxxx\n headers:\n - name: Content-Type\n value: application/json\n\n template.send-slack: |\n webhook:\n slack_webhook:\n method: POST\n body: |\n {\n \"attachments\": [{\n \"title\": \"{{.app.metadata.name}}\",\n \"title_link\": \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}\",\n \"color\": \"#18be52\",\n \"fields\": [{\n \"title\": \"Sync Status\",\n \"value\": \"{{.app.status.sync.status}}\",\n \"short\": true\n }, {\n \"title\": \"Repository\",\n \"value\": \"{{.app.spec.source.repoURL}}\",\n \"short\": true\n }]\n }]\n }\n
"},{"location":"getting-started/alb/","title":"Getting Started - AWS Load Balancer Controller","text":"This guide covers how Argo Rollouts integrates with the AWS Load Balancer Controller for traffic shaping. This guide builds upon the concepts of the basic getting started guide.
"},{"location":"getting-started/alb/#requirements","title":"Requirements","text":" - Kubernetes cluster with AWS ALB Ingress Controller installed
Tip
See the Load Balancer Controller Installation instructions on how to install the AWS Load Balancer Controller
"},{"location":"getting-started/alb/#1-deploy-the-rollout-services-and-ingress","title":"1. Deploy the Rollout, Services, and Ingress","text":"When an AWS ALB Ingress is used as the traffic router, the Rollout canary strategy must define the following fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n strategy:\n canary:\n # canaryService and stableService are references to Services which the Rollout will modify\n # to target the canary ReplicaSet and stable ReplicaSet respectively (required).\n canaryService: rollouts-demo-canary\n stableService: rollouts-demo-stable\n trafficRouting:\n alb:\n # The referenced ingress will be injected with a custom action annotation, directing\n # the AWS Load Balancer Controller to split traffic between the canary and stable\n # Service, according to the desired traffic weight (required).\n ingress: rollouts-demo-ingress\n # Reference to a Service that the Ingress must target in one of the rules (optional).\n # If omitted, uses canary.stableService.\n rootService: rollouts-demo-root\n # Service port is the port which the Service listens on (required).\n servicePort: 443\n...\n
The Ingress referenced by the Rollout must have a rule which matches one of Rollout services. This should be canary.trafficRouting.alb.rootService (if specified), otherwise the rollout will use canary.stableService.
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n name: rollouts-demo-ingress\n annotations:\n kubernetes.io/ingress.class: alb\nspec:\n rules:\n - http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n # serviceName must match either: canary.trafficRouting.alb.rootService (if specified),\n # or canary.stableService (if rootService is omitted)\n name: rollouts-demo-root\n # servicePort must be the value: use-annotation\n # This instructs AWS Load Balancer Controller to look to annotations on how to direct traffic\n port:\n name: use-annotation\n
During an update, the Ingress will be injected with a custom action annotation, which directs the ALB to splits traffic between the stable and canary Services referenced by the Rollout. In this example, those Services are named: rollouts-demo-stable and rollouts-demo-canary respectively.
Run the following commands to deploy:
- A Rollout
- Three Services (root, stable, canary)
- An Ingress
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/alb/rollout.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/alb/services.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/alb/ingress.yaml\n
After applying the manifests you should see the following rollout, services, and ingress resources in the cluster:
$ kubectl get ro\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE\nrollouts-demo 1 1 1 1\n\n$ kubectl get svc\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nrollouts-demo-root NodePort 10.100.16.123 <none> 80:30225/TCP 2m43s\nrollouts-demo-canary NodePort 10.100.16.64 <none> 80:30224/TCP 2m43s\nrollouts-demo-stable NodePort 10.100.146.232 <none> 80:31135/TCP 2m43s\n\n$ kubectl get ingress\nNAME HOSTS ADDRESS PORTS AGE\nrollouts-demo-ingress * b0548428-default-rolloutsd-6951-1972570952.ap-northeast-1.elb.amazonaws.com 80 6m36s\n
kubectl argo rollouts get rollout rollouts-demo\n
"},{"location":"getting-started/alb/#2-perform-an-update","title":"2. Perform an update","text":"Update the rollout by changing the image, and wait for it to reach the paused state.
kubectl argo rollouts set image rollouts-demo rollouts-demo=argoproj/rollouts-demo:yellow\nkubectl argo rollouts get rollout rollouts-demo\n
At this point, both the canary and stable version of the Rollout are running, with 5% of the traffic directed to the canary. To understand how this works, inspect the listener rules for the ALB. When looking at the listener rules, we see that the forward action weights have been modified by the controller to reflect the current weight of the canary.
The controller has added rollouts-pod-template-hash selector to the Services and attached the same label to the Pods. Therefore, you can split the traffic by simply forwarding the requests to the Services according to the weights.
As the Rollout progresses through steps, the forward action weights will be adjusted to match the current setWeight of the steps.
"},{"location":"getting-started/ambassador/","title":"Argo Rollouts and Ambassador Quick Start","text":"This tutorial will walk you through the process of configuring Argo Rollouts to work with Ambassador to facilitate canary releases. All files used in this guide are available in the examples directory of this repository.
"},{"location":"getting-started/ambassador/#requirements","title":"Requirements","text":" - Kubernetes cluster
- Argo-Rollouts installed in the cluster
Note If using Ambassador Edge Stack or Emissary-ingress 2.0+, you will need to install Argo-Rollouts version v1.1+, and you will need to supply --ambassador-api-version getambassador.io/v3alpha1 to your argo-rollouts deployment.
"},{"location":"getting-started/ambassador/#1-install-and-configure-ambassador-edge-stack","title":"1. Install and configure Ambassador Edge Stack","text":"If you don't have Ambassador in your cluster you can install it following the Edge Stack documentation.
By default, Edge Stack routes via Kubernetes services. For best performance with canaries, we recommend you use endpoint routing. Enable endpoint routing on your cluster by saving the following configuration in a file called resolver.yaml:
apiVersion: getambassador.io/v2\nkind: KubernetesEndpointResolver\nmetadata:\n name: endpoint\n
Apply this configuration to your cluster: kubectl apply -f resolver.yaml.
"},{"location":"getting-started/ambassador/#2-create-the-kubernetes-services","title":"2. Create the Kubernetes Services","text":"We'll create two Kubernetes services, named echo-stable and echo-canary. Save this configuration to the file echo-service.yaml.
apiVersion: v1\nkind: Service\nmetadata:\n labels:\n app: echo\n name: echo-stable\nspec:\n type: ClusterIP\n ports:\n - name: http\n port: 80\n protocol: TCP\n targetPort: 8080\n selector:\n app: echo\n---\napiVersion: v1\nkind: Service\nmetadata:\n labels:\n app: echo\n name: echo-canary\nspec:\n type: ClusterIP\n ports:\n - name: http\n port: 80\n protocol: TCP\n targetPort: 8080\n selector:\n app: echo \n
We'll also create an Edge Stack route to the services. Save the following configuration to a file called echo-mapping.yaml.
apiVersion: getambassador.io/v2\nkind: Mapping\nmetadata:\n name: echo\nspec:\n prefix: /echo\n rewrite: /echo\n service: echo-stable:80\n resolver: endpoint\n
Apply both of these configurations to the Kubernetes cluster:
kubectl apply -f echo-service.yaml\nkubectl apply -f echo-mapping.yaml\n
"},{"location":"getting-started/ambassador/#3-deploy-the-echo-service","title":"3. Deploy the Echo Service","text":"Create a Rollout resource and save it to a file called rollout.yaml. Note the trafficRouting attribute, which tells Argo to use Ambassador Edge Stack for routing.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: echo-rollout\nspec:\n selector:\n matchLabels:\n app: echo\n template:\n metadata:\n labels:\n app: echo\n spec:\n containers:\n - image: hashicorp/http-echo\n args:\n - \"-text=VERSION 1\"\n - -listen=:8080\n imagePullPolicy: Always\n name: echo-v1\n ports:\n - containerPort: 8080\n strategy:\n canary:\n stableService: echo-stable\n canaryService: echo-canary\n trafficRouting:\n ambassador:\n mappings:\n - echo\n steps:\n - setWeight: 30\n - pause: {duration: 30s}\n - setWeight: 60\n - pause: {duration: 30s}\n - setWeight: 100\n - pause: {duration: 10}\n
Apply the rollout to your cluster kubectl apply -f rollout.yaml. Note that no canary rollout will occur, as this is the first version of the service being deployed.
"},{"location":"getting-started/ambassador/#4-test-the-service","title":"4. Test the service","text":"We'll now test that this rollout works as expected. Open a new terminal window. We'll use it to send requests to the cluster. Get the external IP address for Edge Stack:
export AMBASSADOR_LB_ENDPOINT=$(kubectl -n ambassador get svc ambassador -o \"go-template={{range .status.loadBalancer.ingress}}{{or .ip .hostname}}{{end}}\")\n
Send a request to the echo service:
curl -Lk \"https://$AMBASSADOR_LB_ENDPOINT/echo/\"\n
You should get a response of \"VERSION 1\".
"},{"location":"getting-started/ambassador/#5-rollout-a-new-version","title":"5. Rollout a new version","text":"It's time to rollout a new version of the service. Update the echo container in the rollout.yaml to display \"VERSION 2\":
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: echo-rollout\nspec:\n selector:\n matchLabels:\n app: echo\n template:\n metadata:\n labels:\n app: echo\n spec:\n containers:\n - image: hashicorp/http-echo\n args:\n - \"-text=VERSION 2\"\n - -listen=:8080\n imagePullPolicy: Always\n name: echo-v1\n ports:\n - containerPort: 8080\n strategy:\n canary:\n stableService: echo-stable\n canaryService: echo-canary\n trafficRouting:\n ambassador:\n mappings:\n - echo\n steps:\n - setWeight: 30\n - pause: {duration: 30s}\n - setWeight: 60\n - pause: {duration: 30s}\n - setWeight: 100\n - pause: {duration: 10}\n
Apply the rollout to the cluster by typing kubectl apply -f rollout.yaml. This will rollout a version 2 of the service by routing 30% of traffic to the service for 30 seconds, followed by 60% of traffic for another 30 seconds.
You can monitor the status of your rollout at the command line:
kubectl argo rollouts get rollout echo-rollout --watch\n
Will display an output similar to the following:
Name: echo-rollout\nNamespace: default\nStatus: \u0965 Paused\nMessage: CanaryPauseStep\nStrategy: Canary\n Step: 1/6\n SetWeight: 30\n ActualWeight: 30\nImages: hashicorp/http-echo (canary, stable)\nReplicas:\n Desired: 1\n Current: 2\n Updated: 1\n Ready: 2\n Available: 2\n\nNAME KIND STATUS AGE INFO\n\u27f3 echo-rollout Rollout \u0965 Paused 2d21h\n\u251c\u2500\u2500# revision:3\n\u2502 \u2514\u2500\u2500\u29c9 echo-rollout-64fb847897 ReplicaSet \u2714 Healthy 2s canary\n\u2502 \u2514\u2500\u2500\u25a1 echo-rollout-64fb847897-49sg6 Pod \u2714 Running 2s ready:1/1\n\u251c\u2500\u2500# revision:2\n\u2502 \u2514\u2500\u2500\u29c9 echo-rollout-578bfdb4b8 ReplicaSet \u2714 Healthy 3h5m stable\n\u2502 \u2514\u2500\u2500\u25a1 echo-rollout-578bfdb4b8-86z6n Pod \u2714 Running 3h5m ready:1/1\n\u2514\u2500\u2500# revision:1\n \u2514\u2500\u2500\u29c9 echo-rollout-948d9c9f9 ReplicaSet \u2022 ScaledDown 2d21h\n
In your other terminal window, you can verify that the canary is progressing appropriately by sending requests in a loop:
while true; do curl -k https://$AMBASSADOR_LB_ENDPOINT/echo/; sleep 0.2; done\n
This will display a running list of responses from the service that will gradually transition from VERSION 1 strings to VERSION 2 strings.
For more details about the Ambassador and Argo-Rollouts integration, see the Ambassador Argo documentation.
"},{"location":"getting-started/appmesh/","title":"Getting Started - App Mesh","text":"This guide covers how Argo Rollouts integrates with service-meshes managed by AWS App Mesh. This guide builds upon the concepts of the basic getting started guide.
"},{"location":"getting-started/appmesh/#requirements","title":"Requirements","text":" - Kubernetes cluster with AWS App Mesh Controller for K8s installed
Tip
See the App Mesh Controler Installation instructions on how to get started using App Mesh with Kubernetes.
"},{"location":"getting-started/appmesh/#1-deploy-the-rollout-services-app-mesh-crd","title":"1. Deploy the Rollout, Services, App Mesh CRD","text":"When App Mesh is used as the traffic router, the Rollout canary strategy must define the following mandatory fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: my-rollout\nspec:\n strategy:\n canary:\n # canaryService and stableService are references to Services which the Rollout will modify\n # to target the canary ReplicaSet and stable ReplicaSet respectively (required).\n canaryService: my-svc-canary\n stableService: my-svc-stable\n trafficRouting:\n appMesh:\n # The referenced virtual-service will be used to determine the virtual-router that is\n # manipulated to update canary weights.\n virtualService:\n # name of the virtual-service App Mesh CR\n name: my-svc\n # Optional set of routes to update. If empty, all routes associated with the virtual-service are updated.\n routes:\n - http-primary\n # virtualNodeGroup is a structure to refer App Mesh virtual-node CR corresponding to Canary and Stable versions\n virtualNodeGroup:\n # canaryVirtualNodeRef refers to virtual-node corresponding to canary version. Rollouts controller will\n # update the podSelector of this virtual-node to latest canary pod-hash generated by controller.\n canaryVirtualNodeRef:\n name: my-vn-canary\n # stableVirtualNodeRef refers to virtual-node corresponding to stable version. Rollouts controller will\n # update the podSelector of this virtual-node to latest stable pod-hash generated by controller.\n stableVirtualNodeRef:\n name: my-vn-stable\n steps:\n - setWeight: 25\n - pause: {}\n ...\n
In this guide, the two services are: my-svc-canary and my-svc-stable respectively. There are two virtual-node CRs corresponding to these services named my-vn-canary and my-vn-stable respectively. In addition, there is a virtual-service named rollout-demo-vsvc that is provided by a virtual-router CR named rollout-demo-vrouter. This virtual-router need have at least one route with action to forward traffic to the canary and stable virtual-nodes. Initially weight for canary is set to 0% while for stable it is 100%. During rollout, controller will modify the weights on route(s) based on the configuraiton defined in steps[N].setWeight.
The canary and stable services are configured to be headless. This is necessary to allow App Mesh to properly handle conneciton pooling as pods are reassigned from canary to stable.
To summarize, run the following commands to deploy a service:
- Two services (stable and canary)
- One service (for VIP and DNS lookup)
- Two App Mesh virtual-nodes (stable and canary)
- One App Mesh virtual-router with routes to virtual-nodes
- One App Mesh virtual-service corresponding to VIP service
- A rollout
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/examples/appmesh/canary-service.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/examples/appmesh/canary-rollout.yaml\n
"},{"location":"getting-started/appmesh/#2-verify-service","title":"2. Verify service","text":"First make sure that rollout is stable.
kubectl argo rollouts get rollout my-rollout -n argo-examples -w\n
Then make sure the service is functional.
kubectl -n argo-examples port-forward svc/my-svc 8181:80\n
"},{"location":"getting-started/appmesh/#3-rollout-new-version","title":"3. Rollout new version","text":"Now its time to deploy new version. Update the rollout with new image.
kubectl argo rollouts set image my-rollout demo=argoproj/rollouts-demo:green -n argo-examples\n
Rollout should deploy a new canary revision and update the weights under virtual-router.
kubectl get -n argo-examples virtualrouter my-vrouter -o json | jq \".spec.routes[0].httpRoute.action.weightedTargets\"\n[\n {\n \"virtualNodeRef\": {\n \"name\": \"my-vn-canary\"\n },\n \"weight\": 25\n },\n {\n \"virtualNodeRef\": {\n \"name\": \"my-vn-stable\"\n },\n \"weight\": 75\n }\n]\n
Now manually approve the rollout that is paused indefinitely, and continue watching the routes get updated
kubectl argo rollouts promote my-rollout -n argo-examples\n\nwatch -d 'kubectl get -n argo-examples virtualrouter my-vrouter -o json | jq \".spec.routes[0].httpRoute.action.weightedTargets\"'\n
"},{"location":"getting-started/istio/","title":"Getting Started - Istio","text":"This guide covers how Argo Rollouts integrates with the Istio Service Mesh for traffic shaping. This guide builds upon the concepts of the basic getting started guide.
"},{"location":"getting-started/istio/#requirements","title":"Requirements","text":" - Kubernetes cluster with Istio installed
Tip
See the environment setup guide for Istio on how to setup a local minikube environment with Istio
"},{"location":"getting-started/istio/#1-deploy-the-rollout-services-istio-virtualservice-and-istio-gateway","title":"1. Deploy the Rollout, Services, Istio VirtualService, and Istio Gateway","text":"When Istio is used as the traffic router, the Rollout canary strategy must define the following mandatory fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n strategy:\n canary:\n # Reference to a Service which the controller updates to point to the canary ReplicaSet\n canaryService: rollouts-demo-canary\n # Reference to a Service which the controller updates to point to the stable ReplicaSet\n stableService: rollouts-demo-stable\n trafficRouting:\n istio:\n virtualServices:\n # One or more virtualServices can be configured\n # Reference to a VirtualService which the controller updates with canary weights\n - name: rollouts-demo-vsvc1\n # Optional if there is a single HTTP route in the VirtualService, otherwise required\n routes:\n - http-primary\n # Optional if there is a single HTTPS/TLS route in the VirtualService, otherwise required\n tlsRoutes:\n # Below fields are optional but if defined, they should match exactly with at least one of the TLS route match rules in your VirtualService\n - port: 443 # Only required if you want to match any rule in your VirtualService which contains this port\n # Only required if you want to match any rule in your VirtualService which contain all these SNI hosts\n sniHosts:\n - reviews.bookinfo.com\n - localhost\n - name: rollouts-demo-vsvc2\n # Optional if there is a single HTTP route in the VirtualService, otherwise required\n routes:\n - http-secondary\n # Optional if there is a single HTTPS/TLS route in the VirtualService, otherwise required\n tlsRoutes:\n # Below fields are optional but if defined, they should match exactly with at least one of the TLS route match rules in your VirtualService\n - port: 443 # Only required if you want to match any rule in your VirtualService which contains this port\n # Only required if you want to match any rule in your VirtualService which contain all these SNI hosts\n sniHosts:\n - reviews.bookinfo.com\n - localhost\n tcpRoutes:\n # Below fields are optional but if defined, they should match exactly with at least one of the TCP route match rules in your VirtualService\n - port: 8020 # Only required if you want to match any rule in your VirtualService which contains this port\n
The VirtualService and route referenced in either trafficRouting.istio.virtualService or trafficRouting.istio.virtualServices. trafficRouting.istio.virtualServices helps in adding one or more virtualServices unlike trafficRouting.istio.virtualService where only single virtualService can be added. This is required to have either HTTP, TLS, TCP or a mixed route specs that splits between the stable and the canary services referenced in the rollout. If the route is HTTPS/TLS, we can match it based on the given port number and/or SNI hosts. Note that both of them are optional and only needed if you want to match any rule in your VirtualService which contains these.
In this guide, the two services are: rollouts-demo-stable and rollouts-demo-canary respectively. The weights for these two services should initially be set to 100% on the stable service and 0% on the canary service. During an update, these values will get modified by the controller. If there are multiple VirtualService then weight values for stable and canary service of each VirtualService will be modified by the controller simultaneously.
Note that since we have both the HTTP and HTTPS routes in our rollout spec and they match the VirtualService specs, weights will get modified for both these routes.
apiVersion: networking.istio.io/v1alpha3\nkind: VirtualService\nmetadata:\n name: rollouts-demo-vsvc1\nspec:\n gateways:\n - rollouts-demo-gateway\n hosts:\n - rollouts-demo-vsvc1.local\n http:\n - name: http-primary # Should match rollout.spec.strategy.canary.trafficRouting.istio.virtualServices.routes\n route:\n - destination:\n host: rollouts-demo-stable # Should match rollout.spec.strategy.canary.stableService\n port:\n number: 15372\n weight: 100\n - destination:\n host: rollouts-demo-canary # Should match rollout.spec.strategy.canary.canaryService\n port:\n number: 15372\n weight: 0\n tls:\n - match:\n - port: 443 # Should match the port number of the route defined in rollout.spec.strategy.canary.trafficRouting.istio.virtualServices.tlsRoutes\n sniHosts: # Should match all the SNI hosts of the route defined in rollout.spec.strategy.canary.trafficRouting.istio.virtualServices.tlsRoutes\n - reviews.bookinfo.com\n - localhost\n route:\n - destination:\n host: rollouts-demo-stable # Should match rollout.spec.strategy.canary.stableService\n weight: 100\n - destination:\n host: rollouts-demo-canary # Should match rollout.spec.strategy.canary.canaryService\n weight: 0\n tcp:\n - match:\n - port: 8020 # Should match the port number of the route defined in rollout.spec.strategy.canary.trafficRouting.istio.virtualServices.tcpRoutes\n route:\n - destination:\n host: rollouts-demo-stable # Should match rollout.spec.strategy.canary.stableService\n weight: 100\n - destination:\n host: rollouts-demo-canary # Should match rollout.spec.strategy.canary.canaryService\n weight: 0\n
apiVersion: networking.istio.io/v1alpha3\nkind: VirtualService\nmetadata:\n name: rollouts-demo-vsvc2\nspec:\n gateways:\n - rollouts-demo-gateway\n hosts:\n - rollouts-demo-vsvc2.local\n http:\n - name: http-secondary # Should match rollout.spec.strategy.canary.trafficRouting.istio.virtualServices.routes\n route:\n - destination:\n host: rollouts-demo-stable # Should match rollout.spec.strategy.canary.stableService\n port:\n number: 15373\n weight: 100\n - destination:\n host: rollouts-demo-canary # Should match rollout.spec.strategy.canary.canaryService\n port:\n number: 15373\n weight: 0\n tls:\n - match:\n - port: 443 # Should match the port number of the route defined in rollout.spec.strategy.canary.trafficRouting.istio.virtualServices.tlsRoutes\n sniHosts: # Should match all the SNI hosts of the route defined in rollout.spec.strategy.canary.trafficRouting.istio.virtualServices.tlsRoutes\n - reviews.bookinfo.com\n route:\n - destination:\n host: rollouts-demo-stable # Should match rollout.spec.strategy.canary.stableService\n weight: 100\n - destination:\n host: rollouts-demo-canary # Should match rollout.spec.strategy.canary.canaryService\n weight: 0\n tcp:\n - match:\n - port: 8020 # Should match the port number of the route defined in rollout.spec.strategy.canary.trafficRouting.istio.virtualServices.tcpRoutes\n route:\n - destination:\n host: rollouts-demo-stable # Should match rollout.spec.strategy.canary.stableService\n weight: 100\n - destination:\n host: rollouts-demo-canary # Should match rollout.spec.strategy.canary.canaryService\n weight: 0\n
Run the following commands to deploy:
- A Rollout
- Two Services (stable and canary)
- One or more Istio VirtualServices
- An Istio Gateway
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/istio/rollout.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/istio/services.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/istio/multipleVirtualsvc.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/istio/gateway.yaml\n
After applying the manifests you should see the following rollout, services, virtualservices, and gateway resources in the cluster:
$ kubectl get ro\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE\nrollouts-demo 1 1 1 1\n\n$ kubectl get svc\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nrollouts-demo-canary ClusterIP 10.103.146.137 <none> 80/TCP 37s\nrollouts-demo-stable ClusterIP 10.101.158.227 <none> 80/TCP 37s\n\n$ kubectl get virtualservice\nNAME GATEWAYS HOSTS AGE\nrollouts-demo-vsvc1 [rollouts-demo-gateway] [rollouts-demo-vsvc1.local] 54s\nrollouts-demo-vsvc2 [rollouts-demo-gateway] [rollouts-demo-vsvc2.local] 54s\n\n$ kubectl get gateway\nNAME AGE\nrollouts-demo-gateway 71s\n
kubectl argo rollouts get rollout rollouts-demo\n
"},{"location":"getting-started/istio/#2-perform-an-update","title":"2. Perform an update","text":"Update the rollout by changing the image, and wait for it to reached the paused state.
kubectl argo rollouts set image rollouts-demo rollouts-demo=argoproj/rollouts-demo:yellow\nkubectl argo rollouts get rollout rollouts-demo\n
At this point, both the canary and stable version of the Rollout are running, with 5% of the traffic directed to the canary. To understand how this works, inspect the VirtualService which the Rollout was referencing. When looking at both the VirtualService, we see that the route destination weights have been modified by the controller to reflect the current weight of the canary.
apiVersion: networking.istio.io/v1beta1\nkind: VirtualService\nmetadata:\n name: rollouts-demo-vsvc1\n namespace: default\nspec:\n gateways:\n - rollouts-demo-gateway\n hosts:\n - rollouts-demo-vsvc1.local\n http:\n - name: http-primary\n route:\n - destination:\n host: rollouts-demo-stable\n port:\n number: 15372\n weight: 95\n - destination:\n host: rollouts-demo-canary\n port:\n number: 15372\n weight: 5\n tls:\n - match:\n - port: 443\n sniHosts:\n - reviews.bookinfo.com\n - localhost\n route:\n - destination:\n host: rollouts-demo-stable\n weight: 95\n - destination:\n host: rollouts-demo-canary\n weight: 5\n tcp:\n - match:\n - port: 8020\n route:\n - destination:\n host: rollouts-demo-stable\n weight: 95\n - destination:\n host: rollouts-demo-canary\n weight: 5\n
apiVersion: networking.istio.io/v1beta1\nkind: VirtualService\nmetadata:\n name: rollouts-demo-vsvc2\n namespace: default\nspec:\n gateways:\n - rollouts-demo-gateway\n hosts:\n - rollouts-demo-vsvc2.local\n http:\n - name: http-primary\n route:\n - destination:\n host: rollouts-demo-stable\n port:\n number: 15373\n weight: 95\n - destination:\n host: rollouts-demo-canary\n port:\n number: 15373\n weight: 5\n tls:\n - match:\n - port: 443\n sniHosts:\n - reviews.bookinfo.com\n route:\n - destination:\n host: rollouts-demo-stable\n weight: 95\n - destination:\n host: rollouts-demo-canary\n weight: 5\n tcp:\n - match:\n - port: 8020\n route:\n - destination:\n host: rollouts-demo-stable\n weight: 95\n - destination:\n host: rollouts-demo-canary\n weight: 5\n
As the Rollout progresses through steps, the HTTP, TLS, and/or TCP route(s) destination weights will be adjusted to match the current setWeight of the steps.
"},{"location":"getting-started/mixed/","title":"Getting Started - Multiple Providers (Service Mesh Interface and NGiNX Ingress)","text":"Important
Available since v1.2
This guide covers how Argo Rollouts integrates with multiple TrafficRoutings, using Linkerd and NGINX Ingress Controller for traffic shaping, but you should be able to produce any other combination between the existing trafficRouting options.
This guide builds upon the concepts of the basic getting started guide, NGINX Guide, and SMI Guide.
"},{"location":"getting-started/mixed/#requirements","title":"Requirements","text":" - Kubernetes cluster with Linkerd installed
- Kubernetes cluster with NGINX ingress controller installed and part of the mesh
Tip
See the environment setup guide for linkerd on how to setup a local minikube environment with linkerd and nginx.
"},{"location":"getting-started/mixed/#1-deploy-the-rollout-services-and-ingress","title":"1. Deploy the Rollout, Services, and Ingress","text":"When SMI is used as one the traffic routers, the Rollout canary strategy must define the following mandatory fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n strategy:\n canary:\n # Reference to a Service which the controller will update to point to the canary ReplicaSet\n canaryService: rollouts-demo-canary\n # Reference to a Service which the controller will update to point to the stable ReplicaSet\n stableService: rollouts-demo-stable\n trafficRouting:\n smi: {}\n
When NGINX Ingress is used as the traffic router, the Rollout canary strategy must define the following mandatory fields: apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n strategy:\n canary:\n # Reference to a Service which the controller will update to point to the canary ReplicaSet\n canaryService: rollouts-demo-canary\n # Reference to a Service which the controller will update to point to the stable ReplicaSet\n stableService: rollouts-demo-stable\n trafficRouting:\n nginx:\n # Reference to an Ingress which has a rule pointing to the stable service (e.g. rollouts-demo-stable)\n # This ingress will be cloned with a new name, in order to achieve NGINX traffic splitting.\n stableIngress: rollouts-demo-stable\n...\n
A combination of both should have comply with each TrafficRouting requirements, in this case:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n strategy:\n canary:\n # Reference to a Service which the controller will update to point to the canary ReplicaSet\n canaryService: rollouts-demo-canary\n # Reference to a Service which the controller will update to point to the stable ReplicaSet\n stableService: rollouts-demo-stable\n trafficRouting:\n nginx:\n # Reference to an Ingress which has a rule pointing to the stable service (e.g. rollouts-demo-stable)\n # This ingress will be cloned with a new name, in order to achieve NGINX traffic splitting.\n stableIngress: rollouts-demo-stable\n smi: {}\n
The Ingress referenced in canary.trafficRouting.nginx.stableIngress is required to have a host rule which has a backend targeting the Service referenced under canary.stableService. In our example, that stable Service is named: rollouts-demo-stable:
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n name: rollouts-demo-stable\n annotations:\n kubernetes.io/ingress.class: nginx\nspec:\n rules:\n - host: rollouts-demo.local\n http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n # Reference to a Service name, also specified in the Rollout spec.strategy.canary.stableService field\n name: rollouts-demo-stable\n port:\n number: 80\n
Run the following commands to deploy:
- A Rollout with the Linkerd
linkerd.io/inject: enabled annotation - Two Services (stable and canary)
- An Ingress
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/mixed/rollout.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/mixed/services.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/mixed/ingress.yaml\n
After applying the manifests you should see the following rollout, services, and ingress resources in the cluster:
$ kubectl get ro\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE\nrollouts-demo 1 2 1 2\n\n$ kubectl get svc\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nrollouts-demo-canary ClusterIP 10.111.69.188 <none> 80/TCP 23m\nrollouts-demo-stable ClusterIP 10.109.175.248 <none> 80/TCP 23m\n\n$ kubectl get ing\nNAME CLASS HOSTS ADDRESS PORTS AGE\nrollouts-demo-stable <none> rollouts-demo.local 192.168.64.2 80 23m\n
You should also see a TrafficSplit resource which is created automatically and owned by the rollout:
$ kubectl get trafficsplit\nNAME SERVICE\nrollouts-demo rollouts-demo-stable\n
When inspecting the generated TrafficSplit resource, the weights are automatically configured to send 100% traffic to the rollouts-demo-stable service, and 0% traffic to the rollouts-demo-canary. These values will be updated during an update.
apiVersion: split.smi-spec.io/v1alpha1\nkind: TrafficSplit\nmetadata:\n name: rollouts-demo\n namespace: default\nspec:\n backends:\n - service: rollouts-demo-canary\n weight: \"0\"\n - service: rollouts-demo-stable\n weight: \"100\"\n service: rollouts-demo-stable\n
You should also notice a second ingress created by the rollouts controller, rollouts-demo-rollouts-demo-stable-canary. This ingress is the \"canary ingress\", which is a clone of the user-managed Ingress referenced under nginx.stableIngress. It is used by nginx ingress controller to achieve canary traffic splitting. The name of the generated ingress is formulated using <ROLLOUT-NAME>-<INGRESS-NAME>-canary. More details on the second Ingress are discussed in the following section.
"},{"location":"getting-started/mixed/#2-perform-an-update","title":"2. Perform an update","text":"Now perform an update the rollout by changing the image, and wait for it to reached the paused state.
kubectl argo rollouts set image rollouts-demo rollouts-demo=argoproj/rollouts-demo:yellow\nkubectl argo rollouts get rollout rollouts-demo\n
At this point, both the canary and stable version of the Rollout are running, with 5% of the traffic directed to the canary and 95% to the stable. When inspecting the TrafficSplit generated by the controller, we see that the weight has been updated to reflect the current setWeight: 5 step of the canary deploy.
apiVersion: split.smi-spec.io/v1alpha1\nkind: TrafficSplit\nmetadata:\n name: rollouts-demo\n namespace: default\nspec:\n backends:\n - service: rollouts-demo-canary\n weight: \"5\"\n - service: rollouts-demo-stable\n weight: \"95\"\n service: rollouts-demo-stable\n
When inspecting the rollout controller generated Ingress copy, we see that it has the following changes over the original ingress: - Two additional NGINX specific canary annotations are added to the annotations.
- The Ingress rules will have an rule which points the backend to the canary service.
apiVersion: extensions/v1beta1\nkind: Ingress\nmetadata:\n name: rollouts-demo-rollouts-demo-stable-canary\n annotations:\n kubernetes.io/ingress.class: nginx\n nginx.ingress.kubernetes.io/canary: \"true\"\n nginx.ingress.kubernetes.io/canary-weight: \"5\"\nspec:\n rules:\n - host: rollouts-demo.local\n http:\n paths:\n - backend:\n serviceName: rollouts-demo-canary\n servicePort: 80\n
As the Rollout progresses through steps, the weights in the TrafficSplit and Ingress resource will be adjusted to match the current setWeight of the steps.
"},{"location":"getting-started/nginx/","title":"Getting Started - NGINX Ingress","text":"This guide covers how Argo Rollouts integrates with the NGINX Ingress Controller for traffic shaping. This guide builds upon the concepts of the basic getting started guide.
"},{"location":"getting-started/nginx/#requirements","title":"Requirements","text":" - Kubernetes cluster with NGINX ingress controller installed
Tip
See the environment setup guide for NGINX on how to setup a local minikube environment with nginx.
"},{"location":"getting-started/nginx/#1-deploy-the-rollout-services-and-ingress","title":"1. Deploy the Rollout, Services, and Ingress","text":"When NGINX Ingress is used as the traffic router, the Rollout canary strategy must define the following mandatory fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n strategy:\n canary:\n # Reference to a Service which the controller will update to point to the canary ReplicaSet\n canaryService: rollouts-demo-canary\n # Reference to a Service which the controller will update to point to the stable ReplicaSet\n stableService: rollouts-demo-stable\n trafficRouting:\n nginx:\n # Reference to an Ingress which has a rule pointing to the stable service (e.g. rollouts-demo-stable)\n # This ingress will be cloned with a new name, in order to achieve NGINX traffic splitting.\n stableIngress: rollouts-demo-stable\n...\n
The Ingress referenced in canary.trafficRouting.nginx.stableIngress is required to have a host rule which has a backend targeting the Service referenced under canary.stableService. In our example, that stable Service is named: rollouts-demo-stable:
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n name: rollouts-demo-stable\n annotations:\n kubernetes.io/ingress.class: nginx\nspec:\n rules:\n - host: rollouts-demo.local\n http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n # Reference to a Service name, also specified in the Rollout spec.strategy.canary.stableService field\n name: rollouts-demo-stable\n port:\n number: 80\n
Run the following commands to deploy:
- A Rollout
- Two Services (stable and canary)
- An Ingress
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/nginx/rollout.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/nginx/services.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/nginx/ingress.yaml\n
After applying the manifests you should see the following rollout, services, and ingress resources in the cluster:
$ kubectl get ro\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE\nrollouts-demo 1 1 1 1\n\n$ kubectl get svc\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nrollouts-demo-canary ClusterIP 10.96.6.241 <none> 80/TCP 33s\nrollouts-demo-stable ClusterIP 10.102.229.83 <none> 80/TCP 33s\n\n$ kubectl get ing\nNAME CLASS HOSTS ADDRESS PORTS AGE\nrollouts-demo-stable <none> rollouts-demo.local 192.168.64.2 80 36s\nrollouts-demo-rollouts-demo-stable-canary <none> rollouts-demo.local 192.168.64.2 80 35s\n
You should also notice a second ingress created by the rollouts controller, rollouts-demo-rollouts-demo-stable-canary. This ingress is the \"canary ingress\", which is a clone of the user-managed Ingress referenced under nginx.stableIngress. It is used by nginx ingress controller to achieve canary traffic splitting. The name of the generated ingress is formulated using <ROLLOUT-NAME>-<INGRESS-NAME>-canary. More details on the second Ingress are discussed in the following section.
kubectl argo rollouts get rollout rollouts-demo\n
"},{"location":"getting-started/nginx/#2-perform-an-update","title":"2. Perform an update","text":"Update the rollout by changing the image, and wait for it to reached the paused state.
kubectl argo rollouts set image rollouts-demo rollouts-demo=argoproj/rollouts-demo:yellow\nkubectl argo rollouts get rollout rollouts-demo\n
At this point, both the canary and stable version of the Rollout are running, with 5% of the traffic directed to the canary. One thing to note, is that the rollout is able to achieve a 5% canary weight despite only running two pods. This is able to be achieved since the traffic split happens at the ingress controller (as opposed to weighted replica counts and kube-proxy in the basic guide).
When inspecting the rollout controller generated Ingress copy, we see that it has the following changes over the original ingress:
- Two additional NGINX specific canary annotations are added to the annotations.
- The Ingress rules will have an rule which points the backend to the canary service.
apiVersion: extensions/v1beta1\nkind: Ingress\nmetadata:\n name: rollouts-demo-rollouts-demo-stable-canary\n annotations:\n kubernetes.io/ingress.class: nginx\n nginx.ingress.kubernetes.io/canary: \"true\"\n nginx.ingress.kubernetes.io/canary-weight: \"5\"\nspec:\n rules:\n - host: rollouts-demo.local\n http:\n paths:\n - backend:\n serviceName: rollouts-demo-canary\n servicePort: 80\n
As the Rollout progresses through steps, the canary-weight annotation will be adjusted to match the current setWeight of the steps. The NGINX ingress controller examines the original Ingress, the canary Ingress, and the canary-weight annotation to determine what percentage of traffic to split between the two Ingresses.
"},{"location":"getting-started/setup/","title":"Environment Set Up","text":"This guide shows how to set up a local environment for development, testing, learning, or demoing purposes.
"},{"location":"getting-started/setup/#helm","title":"Helm","text":"Some dependencies are installable via the Helm stable repository:
helm repo add stable https://charts.helm.sh/stable\nhelm repo add grafana https://grafana.github.io/helm-charts\nhelm repo add prometheus-community https://prometheus-community.github.io/helm-charts\nhelm repo update\n
"},{"location":"getting-started/setup/#minikube","title":"Minikube","text":""},{"location":"getting-started/setup/#nginx-ingress-controller-setup","title":"NGINX Ingress Controller Setup","text":"The following instructions describe how to configure NGINX Ingress Controller on minikube. For basic ingress support, only the \"ingress\" addon needs to be enabled:
minikube addons enable ingress\n
Optionally, Prometheus and Grafana can be installed to utilize progressive delivery functionality:
# Install Prometheus\nkubectl create ns monitoring\nhelm install prometheus prometheus-community/prometheus -n monitoring -f docs/getting-started/setup/values-prometheus.yaml\n\n# Patch the ingress-nginx-controller pod so that it has the required\n# prometheus annotations. This allows the pod to be scraped by the\n# prometheus server.\nkubectl patch deploy ingress-nginx-controller -n ingress-nginx -p \"$(cat docs/getting-started/setup/ingress-nginx-controller-metrics-scrape.yaml)\"\n
Note
For Minikube version 1.18.1 or earlier, change the -n parameter value (namespace) to kube-system.
# Install grafana along with nginx ingress dashboards\nhelm install grafana grafana/grafana -n monitoring -f docs/getting-started/setup/values-grafana-nginx.yaml\n\n# Grafana UI can be accessed by running:\nminikube service grafana -n monitoring\n
"},{"location":"getting-started/setup/#istio-setup","title":"Istio Setup","text":"The following instructions describe how to configure Istio on minikube.
# Istio on Minikube requires additional memory and CPU\nminikube start --memory=8192mb --cpus=4\n\n# Install istio\nminikube addons enable istio-provisioner\nminikube addons enable istio\n\n# Label the default namespace to enable istio sidecar injection for the namespace\nkubectl label namespace default istio-injection=enabled\n
Istio already comes with a Prometheus database ready to use. To visualize metrics about istio services, Grafana and Istio dashboards can be installed via Helm to leverage progressive delivery functionality:
# Install Grafana and Istio dashboards\nhelm install grafana grafana/grafana -n istio-system -f docs/getting-started/setup/values-grafana-istio.yaml\n\n# Grafana UI can be accessed by running\nminikube service grafana -n istio-system\n
In order for traffic to enter the Istio mesh, the request needs to go through an Istio ingress gateway, which is simply a normal Kubernetes Deployment and Service. One convenient way to reach the gateway using minikube, is using the minikube tunnel command, which assigns Services a LoadBalancer. This command should be run in the background, usually in a separate terminal window:
minikube tunnel\n
While running minikube tunnel, the istio-ingressgateway Service will now have an external IP which can be retrieved via kubectl:
$ kubectl get svc -n istio-system istio-ingressgateway\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nistio-ingressgateway LoadBalancer 10.100.136.45 10.100.136.45 15020:31711/TCP,80:31298/TCP.... 7d22h\n
The LoadBalancer external IP (10.100.136.45 in this example) is now reachable to access services in the Istio mesh. Istio routes requests to the correct pod based on the Host HTTP header. Follow the guide on supplying host headers to learn how to configure your client environment to supply the proper request to reach the pod.
"},{"location":"getting-started/setup/#linkerd-setup","title":"Linkerd Setup","text":"Linkerd can be installed using the linkerd CLI.
brew install linkerd\nlinkerd install | kubectl apply -f -\n
Linkerd does not provide its own ingress controller, choosing instead to work alongside your ingress controller of choice. On minikube, we can use the built-in NGINX ingress addon and reconfigure it to be part of the linkerd mesh.
# Install the NGINX ingress controller addon\nminikube addons enable ingress\n\n# Patch the nginx-ingress-controller deployment to allow injection of the linkerd proxy to the\n# pod, so that it will be part of the mesh.\nkubectl patch deploy ingress-nginx-controller -n kube-system \\\n -p '{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"linkerd.io/inject\":\"enabled\"}}}}}'\n
"},{"location":"getting-started/setup/#supplying-host-headers","title":"Supplying Host Headers","text":"Most ingress controllers and service mesh implementations rely on the Host HTTP request header being supplied in the request in order to determine how to route the request to the correct pod.
"},{"location":"getting-started/setup/#determining-the-hostname-to-ip-mapping","title":"Determining the hostname to IP mapping","text":"For the Host header to be set in the request, the hostname of the service should resolve to the public IP address of the ingress or service mesh. Depending on if you are using an ingress controller or a service mesh, use one of the following techniques to determine the correct hostname to IP mapping:
"},{"location":"getting-started/setup/#ingresses","title":"Ingresses","text":"For traffic which is reaching the cluster network via a normal Kubernetes Ingress, the hostname should map to the IP of the ingress. We can retrieve the external IP of the ingress from the Ingress object itself, using kubectl:
$ kubectl get ing rollouts-demo-stable\nNAME CLASS HOSTS ADDRESS PORTS AGE\nrollouts-demo-stable <none> rollouts-demo.local 192.168.64.2 80 80m\n
In the example above, the hostname rollouts-demo.local should be configured to resolve to the IP 192.168.64.2. The next section describes various ways to configure your local system to resolve the hostname to the desired IP.
"},{"location":"getting-started/setup/#istio","title":"Istio","text":"In the case of Istio, traffic enters the mesh through an Ingress Gateway, which simply is a load balancer sitting at the edge of mesh.
To determine the correct hostname to IP mapping, it largely depends on what was configured in the VirtualService and Gateway. If you are following the Istio getting started guide, the examples use the \"default\" istio ingress gateway, which we can obtain from kubectl:
$ kubectl get svc -n istio-system istio-ingressgateway\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nistio-ingressgateway LoadBalancer 10.100.136.45 10.100.136.45 15020:31711/TCP,80:31298/TCP.... 7d22h\n
In the above example, the hostname rollouts-demo.local should be configured to resolve to the IP 10.100.136.45. The next section describes various ways to configure your local system to resolve the hostname to the desired IP.
"},{"location":"getting-started/setup/#configuring-local-hostname-resolution","title":"Configuring local hostname resolution","text":"Now that you have determined the correct hostname to IP mapping, the next step involves configuring the system so that will resolve properly. There are different techniques to do this:
"},{"location":"getting-started/setup/#dns-entry","title":"DNS Entry","text":"In real, production environments, the Host header is typically achieved by adding a DNS entry for the hostname in the DNS server. However, for local development, this is typically not an easily accessible option.
"},{"location":"getting-started/setup/#etchosts-entry","title":"/etc/hosts Entry","text":"On local workstations, a local entry to /etc/hosts can be added to map the hostname and IP address of the ingress. For example, the following is an example of an /etc/hosts file which maps rollouts-demo.local to IP 10.100.136.45.
##\n# Host Database\n#\n# localhost is used to configure the loopback interface\n# when the system is booting. Do not change this entry.\n##\n127.0.0.1 localhost\n255.255.255.255 broadcasthost\n::1 localhost\n\n10.100.136.45 rollouts-demo.local\n
The advantages of using a host entry, are that it works for all clients (CLIs, browsers). On the other hand, it is harder to maintain if the IP address changes frequently.
"},{"location":"getting-started/setup/#supply-header-in-curl","title":"Supply Header in Curl","text":"Clients such as curl, have the ability to explicitly set a header (the -H flag in curl). For example:
$ curl -I -H 'Host: rollouts-demo.local' http://10.100.136.45/color\nHTTP/1.1 200 OK\ncontent-type: text/plain; charset=utf-8\nx-content-type-options: nosniff\ndate: Wed, 24 Jun 2020 08:44:59 GMT\ncontent-length: 6\nx-envoy-upstream-service-time: 1\nserver: istio-envoy\n
Notice that the same request made without the header, fails with a 404 Not Found error.
$ curl -I http://10.100.136.45/color\nHTTP/1.1 404 Not Found\ndate: Wed, 24 Jun 2020 08:46:07 GMT\nserver: istio-envoy\ntransfer-encoding: chunked\n
"},{"location":"getting-started/setup/#browser-extension","title":"Browser Extension","text":"Similar to curl's ability to explicitly set a header, browsers can also achieve this via browser extensions. One example of a browser extension which can do this, is ModHeader.
"},{"location":"getting-started/smi/","title":"Getting Started - SMI (Service Mesh Interface)","text":"Important
Available since v0.9
This guide covers how Argo Rollouts integrates with the Service Mesh Interface (SMI), using Linkerd and NGINX Ingress Controller for traffic shaping. Since the SMI TrafficSplit resource is supported by multiple service mesh providers, the concepts taught here are applicable to other service mesh providers that support the interface. See the SMI Ecosystem for other projects that support SMI. This guide builds upon the concepts of the basic getting started guide.
"},{"location":"getting-started/smi/#requirements","title":"Requirements","text":" - Kubernetes cluster with Linkerd installed
- Kubernetes cluster with NGINX ingress controller installed and part of the mesh
Tip
See the environment setup guide for linkerd on how to setup a local minikube environment with linkerd and nginx.
"},{"location":"getting-started/smi/#1-deploy-the-rollout-services-and-ingress","title":"1. Deploy the Rollout, Services, and Ingress","text":"When SMI is used as the traffic router, the Rollout canary strategy must define the following mandatory fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n strategy:\n canary:\n # Reference to a Service which the controller will update to point to the canary ReplicaSet\n canaryService: rollouts-demo-canary\n # Reference to a Service which the controller will update to point to the stable ReplicaSet\n stableService: rollouts-demo-stable\n trafficRouting:\n smi: {}\n
Run the following commands to deploy:
- A Rollout with the Linkerd
linkerd.io/inject: enabled annotation - Two Services (stable and canary)
- An Ingress
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/smi/rollout.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/smi/services.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/smi/ingress.yaml\n
After applying the manifests you should see the following rollout, services, and ingress resources in the cluster:
$ kubectl get ro\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE\nrollouts-demo 1 2 1 2\n\n$ kubectl get svc\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nrollouts-demo-canary ClusterIP 10.111.69.188 <none> 80/TCP 23m\nrollouts-demo-stable ClusterIP 10.109.175.248 <none> 80/TCP 23m\n\n$ kubectl get ing\nNAME CLASS HOSTS ADDRESS PORTS AGE\nrollouts-demo-stable <none> rollouts-demo.local 192.168.64.2 80 23m\n
You should also see a TrafficSplit resource which is created automatically and owned by the rollout:
$ kubectl get trafficsplit\nNAME SERVICE\nrollouts-demo rollouts-demo-stable\n
When inspecting the generated TrafficSplit resource, the weights are automatically configured to send 100% traffic to the rollouts-demo-stable service, and 0% traffic to the rollouts-demo-canary. These values will be updated during an update.
apiVersion: split.smi-spec.io/v1alpha1\nkind: TrafficSplit\nmetadata:\n name: rollouts-demo\n namespace: default\nspec:\n backends:\n - service: rollouts-demo-canary\n weight: \"0\"\n - service: rollouts-demo-stable\n weight: \"100\"\n service: rollouts-demo-stable\n
"},{"location":"getting-started/smi/#2-perform-an-update","title":"2. Perform an update","text":"Now perform an update the rollout by changing the image, and wait for it to reached the paused state.
kubectl argo rollouts set image rollouts-demo rollouts-demo=argoproj/rollouts-demo:yellow\nkubectl argo rollouts get rollout rollouts-demo\n
At this point, both the canary and stable version of the Rollout are running, with 5% of the traffic directed to the canary. When inspecting the TrafficSplit generated by the controller, we see that the weight has been updated to reflect the current setWeight: 5 step of the canary deploy.
apiVersion: split.smi-spec.io/v1alpha1\nkind: TrafficSplit\nmetadata:\n name: rollouts-demo\n namespace: default\nspec:\n backends:\n - service: rollouts-demo-canary\n weight: \"5\"\n - service: rollouts-demo-stable\n weight: \"95\"\n service: rollouts-demo-stable\n
As the Rollout progresses through steps, the weights in the TrafficSplit resource will be adjusted to match the current setWeight of the steps.
"},{"location":"security/security/","title":"Security Policy for Argo Rollouts","text":""},{"location":"security/security/#reporting-a-vulnerability","title":"Reporting a Vulnerability","text":"If you find a security related bug in Argo Rollouts, we kindly ask you for responsible disclosure and for giving us appropriate time to react, analyze and develop a fix to mitigate the found security vulnerability.
Please report vulnerabilities by e-mail to the following address:
- cncf-argo-security@lists.cncf.io
All vulnerabilites and associated information will be treated with full confidentiality.
"},{"location":"security/security/#public-disclosure","title":"Public Disclosure","text":"We will publish security advisiories using the GitHub Security Advisories feature to keep our community well informed, and will credit you for your findings (unless you prefer to stay anonymous, of course).
"},{"location":"security/security/#internet-bug-bounty-collaboration","title":"Internet Bug Bounty collaboration","text":"We're happy to announce that the Argo project is collaborating with the great folks over at Hacker One and their Internet Bug Bounty program to reward the awesome people who find security vulnerabilities in the four main Argo projects (CD, Events, Rollouts and Workflows) and then work with us to fix and disclose them in a responsible manner.
If you report a vulnerability to us as outlined in this security policy, we will work together with you to find out whether your finding is eligible for claiming a bounty, and also on how to claim it.
"},{"location":"security/signed-release-assets/","title":"Verification of Argo Rollouts Artifacts","text":""},{"location":"security/signed-release-assets/#prerequisites","title":"Prerequisites","text":" - cosign
v2.0.0 or higher installation instructions - slsa-verifier installation instructions
"},{"location":"security/signed-release-assets/#release-assets","title":"Release Assets","text":"Asset Description argo-rollouts-checksums.txt Checksums of binaries argo-rollouts-cli.intoto.jsonl Attestation of CLI binaries & manifiest dashboard-install.yaml Dashboard install install.yaml Standard installation method kubectl-argo-rollouts-darwin-amd64 CLI Binary kubectl-argo-rollouts-darwin-arm64 CLI Binary kubectl-argo-rollouts-linux-amd64 CLI Binary kubectl-argo-rollouts-linux-arm64 CLI Binary kubectl-argo-rollouts-windows-amd64 CLI Binary namespace-install.yaml Namespace installation notifications-install.yaml Notification installation rollout_cr_schema.json Schema sbom.tar.gz Sbom sbom.tar.gz.pem Certificate used to sign sbom sbom.tar.gz.sig Signature of sbom"},{"location":"security/signed-release-assets/#verification-of-container-images","title":"Verification of container images","text":"Argo Rollouts container images are signed by cosign using identity-based (\"keyless\") signing and transparency. Executing the following command can be used to verify the signature of a container image:
cosign verify \\\n--certificate-identity-regexp https://github.com/argoproj/argo-rollouts/.github/workflows/image-reuse.yaml@refs/tags/v \\\n--certificate-oidc-issuer https://token.actions.githubusercontent.com \\\nquay.io/argoproj/argo-rollouts:v1.5.0 | jq\n
The command should output the following if the container image was correctly verified: The following checks were performed on each of these signatures:\n - The cosign claims were validated\n - Existence of the claims in the transparency log was verified offline\n - The code-signing certificate was verified using trusted certificate authority certificates\n[\n {\n \"critical\": {\n \"identity\": {\n \"docker-reference\": \"quay.io/argoproj/argo-rollouts\"\n },\n \"image\": {\n \"docker-manifest-digest\": \"sha256:519522f8c66c7b4c468f360ebe6c8ba07b8d64f5f948e71ae52c01b9953e1eb9\"\n },\n \"type\": \"cosign container image signature\"\n },\n \"optional\": {\n \"1.3.6.1.4.1.57264.1.1\": \"https://token.actions.githubusercontent.com\",\n \"1.3.6.1.4.1.57264.1.2\": \"push\",\n \"1.3.6.1.4.1.57264.1.3\": \"aa1afcb418fcebcc68b063377c48225f5a9d1511\",\n \"1.3.6.1.4.1.57264.1.4\": \"Release\",\n \"1.3.6.1.4.1.57264.1.5\": \"argoproj/argo-rollouts\",\n \"1.3.6.1.4.1.57264.1.6\": \"refs/tags/v1.5.0\",\n ...\n
"},{"location":"security/signed-release-assets/#verification-of-container-image-attestations","title":"Verification of container image attestations","text":"A SLSA Level 3 provenance is generated using slsa-github-generator.
The following command will verify the signature of an attestation and how it was issued. It will contain the payloadType, payload, and signature.
cosign verify-attestation --type slsaprovenance \\\n--certificate-identity-regexp https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v \\\n--certificate-oidc-issuer https://token.actions.githubusercontent.com \\\nquay.io/argoproj/argo-rollouts:v1.5.0 | jq\n
The payload is a non-falsifiable provenance which is base64 encoded and can be viewed by using the command below: cosign verify-attestation --type slsaprovenance \\\n--certificate-identity-regexp https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v \\\n--certificate-oidc-issuer https://token.actions.githubusercontent.com \\\nquay.io/argoproj/argo-rollouts:v1.5.0 | jq -r .payload | base64 -d | jq\n
Tip
cosign or slsa-verifier can both be used to verify image attestations. Check the documentation of each binary for detailed instructions.
"},{"location":"security/signed-release-assets/#verification-of-cli-artifacts-with-attestations","title":"Verification of CLI artifacts with attestations","text":"A single attestation (argo-rollouts.intoto.jsonl) from each release is provided. This can be used with slsa-verifier to verify that a CLI binary or manifest was generated using Argo Rollouts workflows on GitHub and ensures it was cryptographically signed.
slsa-verifier verify-artifact kubectl-argo-rollouts-linux-amd64 --provenance-path kubectl-argo-rollouts.intoto.jsonl --source-uri github.com/argoproj/argo-rollouts\n
"},{"location":"security/signed-release-assets/#verifying-an-artifact-and-output-the-provenance","title":"Verifying an artifact and output the provenance","text":"slsa-verifier verify-artifact kubectl-argo-rollouts-linux-amd64 --provenance-path kubectl-argo-rollouts.intoto.jsonl --source-uri github.com/argoproj/argo-rollouts --print-provenance | jq\n
"},{"location":"security/signed-release-assets/#verification-of-sbom","title":"Verification of Sbom","text":"cosign verify-blob --signature sbom.tar.gz.sig --certificate sbom.tar.gz.pem \\\n--certificate-identity-regexp ^https://github.com/argoproj/argo-rollouts/.github/workflows/release.yaml@refs/tags/v \\\n--certificate-oidc-issuer https://token.actions.githubusercontent.com \\\n ~/Downloads/sbom.tar.gz | jq\n
"},{"location":"security/signed-release-assets/#verification-on-kubernetes","title":"Verification on Kubernetes","text":""},{"location":"security/signed-release-assets/#policy-controllers","title":"Policy controllers","text":"Note
We encourage all users to verify signatures and provenances with your admission/policy controller of choice. Doing so will verify that an image was built by us before it's deployed on your Kubernetes cluster.
Cosign signatures and SLSA provenances are compatible with several types of admission controllers. Please see the cosign documentation and slsa-github-generator for supported controllers.
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Argo Rollouts - Kubernetes Progressive Delivery Controller","text":""},{"location":"#what-is-argo-rollouts","title":"What is Argo Rollouts?","text":"Argo Rollouts is a Kubernetes controller and set of CRDs which provide advanced deployment capabilities such as blue-green, canary, canary analysis, experimentation, and progressive delivery features to Kubernetes.
Argo Rollouts (optionally) integrates with ingress controllers and service meshes, leveraging their traffic shaping abilities to gradually shift traffic to the new version during an update. Additionally, Rollouts can query and interpret metrics from various providers to verify key KPIs and drive automated promotion or rollback during an update.
Here is a demonstration video (click to watch on Youtube):
"},{"location":"#why-argo-rollouts","title":"Why Argo Rollouts?","text":"The native Kubernetes Deployment Object supports the RollingUpdate strategy which provides a basic set of safety guarantees (readiness probes) during an update. However the rolling update strategy faces many limitations:
- Few controls over the speed of the rollout
- Inability to control traffic flow to the new version
- Readiness probes are unsuitable for deeper, stress, or one-time checks
- No ability to query external metrics to verify an update
- Can halt the progression, but unable to automatically abort and rollback the update
For these reasons, in large scale high-volume production environments, a rolling update is often considered too risky of an update procedure since it provides no control over the blast radius, may rollout too aggressively, and provides no automated rollback upon failures.
"},{"location":"#controller-features","title":"Controller Features","text":" - Blue-Green update strategy
- Canary update strategy
- Fine-grained, weighted traffic shifting
- Automated rollbacks and promotions
- Manual judgement
- Customizable metric queries and analysis of business KPIs
- Ingress controller integration: NGINX, ALB, Apache APISIX
- Service Mesh integration: Istio, Linkerd, SMI
- Simultaneous usage of multiple providers: SMI + NGINX, Istio + ALB, etc.
- Metric provider integration: Prometheus, Wavefront, Kayenta, Web, Kubernetes Jobs, Datadog, New Relic, Graphite, InfluxDB
"},{"location":"#quick-start","title":"Quick Start","text":"kubectl create namespace argo-rollouts\nkubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml\n
Follow the full getting started guide to walk through creating and then updating a rollout object.
"},{"location":"#how-does-it-work","title":"How does it work?","text":"Similar to the deployment object, the Argo Rollouts controller will manage the creation, scaling, and deletion of ReplicaSets. These ReplicaSets are defined by the spec.template field inside the Rollout resource, which uses the same pod template as the deployment object.
When the spec.template is changed, that signals to the Argo Rollouts controller that a new ReplicaSet will be introduced. The controller will use the strategy set within the spec.strategy field in order to determine how the rollout will progress from the old ReplicaSet to the new ReplicaSet. Once that new ReplicaSet is scaled up (and optionally passes an Analysis), the controller will mark it as \"stable\".
If another change occurs in the spec.template during a transition from a stable ReplicaSet to a new ReplicaSet (i.e. you change the application version in the middle of a rollout), then the previously new ReplicaSet will be scaled down, and the controller will try to progress the ReplicasSet that reflects the updated spec.template field. There is more information on the behaviors of each strategy in the spec section.
"},{"location":"#use-cases-of-argo-rollouts","title":"Use cases of Argo Rollouts","text":" -
A user wants to run last-minute functional tests on the new version before it starts to serve production traffic. With the BlueGreen strategy, Argo Rollouts allows users to specify a preview service and an active service. The Rollout will configure the preview service to send traffic to the new version while the active service continues to receive production traffic. Once a user is satisfied, they can promote the preview service to be the new active service. (example)
-
Before a new version starts receiving live traffic, a generic set of steps need to be executed beforehand. With the BlueGreen Strategy, the user can bring up the new version without it receiving traffic from the active service. Once those steps finish executing, the rollout can cut over traffic to the new version.
-
A user wants to give a small percentage of the production traffic to a new version of their application for a couple of hours. Afterward, they want to scale down the new version and look at some metrics to determine if the new version is performant compared to the old version. Then they will decide if they want to roll out the new version for all of the production traffic or stick with the current version. With the canary strategy, the rollout can scale up a ReplicaSet with the new version to receive a specified percentage of traffic, wait for a specified amount of time, set the percentage back to 0, and then wait to rollout out to service all of the traffic once the user is satisfied. (example)
-
A user wants to slowly give the new version more production traffic. They start by giving it a small percentage of the live traffic and wait a while before giving the new version more traffic. Eventually, the new version will receive all the production traffic. With the canary strategy, the user specifies the percentages they want the new version to receive and the amount of time to wait between percentages. (example)
-
A user wants to use the normal Rolling Update strategy from the deployment. If a user uses the canary strategy with no steps, the rollout will use the max surge and max unavailable values to roll to the new version. (example)
"},{"location":"#examples","title":"Examples","text":"You can see more examples of Rollouts at:
- The example directory
- The Argo Rollouts Demo application
"},{"location":"CONTRIBUTING/","title":"Contributing","text":""},{"location":"CONTRIBUTING/#before-you-start","title":"Before You Start","text":"Argo Rollouts is written in Golang. If you do not have a good grounding in Go, try out the tutorial.
"},{"location":"CONTRIBUTING/#pre-requisites","title":"Pre-requisites","text":"Install:
- docker
- golang
- kubectl
- kustomize >= 4.5.5
- k3d recommended
Kustomize is required for unit tests (make test is using it), so you must install it locally if you wish to make code contributions to Argo Rollouts.
Argo Rollout additionally uses the following tools
golangci-lint to lint the project.protoc and swagger-codegen to generate proto related filesyarn to build the UI
Run the following commands to install them:
# macOS\nbrew install golangci-lint\n\n# linux\ngo get -u github.com/golangci/golangci-lint/cmd/golangci-lint\n
Brew users can quickly install the lot:
brew install go kubectl kustomize golangci-lint protobuf swagger-codegen k3d\n
Set up environment variables (e.g. is ~/.bashrc):
export GOPATH=~/go\nexport PATH=$PATH:$GOPATH/bin\n
Checkout the code:
go get -u github.com/argoproj/argo-rollouts\ncd ~/go/src/github.com/argoproj/argo-rollouts\n
"},{"location":"CONTRIBUTING/#building","title":"Building","text":"go.mod is used, so the go build/test commands automatically install the needed dependencies
The make controller command will build the controller.
-
make install-tools-local - Runs scripts to install codegen utility CLIs necessary for codegen.
-
make codegen - Runs the code generator that creates the informers, client, lister, and deepcopies from the types.go and modifies the open-api spec.
"},{"location":"CONTRIBUTING/#running-controller-locally","title":"Running Controller Locally","text":"It is much easier to run and debug if you run Argo Rollout in your local machine than in the Kubernetes cluster.
cd ~/go/src/github.com/argoproj/argo-rollouts\ngo run ./cmd/rollouts-controller/main.go\n
When running locally it will connect to whatever kubernetes cluster you have configured in your kubeconfig. You will need to make sure to install the Argo Rollout CRDs into your local cluster, and have the argo-rollouts namespace.
"},{"location":"CONTRIBUTING/#running-unit-tests","title":"Running Unit Tests","text":"To run unit tests:
make test\n
"},{"location":"CONTRIBUTING/#running-e2e-tests","title":"Running E2E tests","text":"The end-to-end tests need to run against a kubernetes cluster with the Argo Rollouts controller running. The rollout controller can be started with the command:
make start-e2e\n
Start and prepare your cluster for e2e tests:
k3d cluster create\nkubectl create ns argo-rollouts\nkubectl apply -k manifests/crds\nkubectl apply -f test/e2e/crds\n
Then run the e2e tests:
make test-e2e\n
To run a subset of e2e tests, you need to specify the suite with -run, and the specific test regex with -testify.m.
E2E_TEST_OPTIONS=\"-run 'TestCanarySuite' -testify.m 'TestCanaryScaleDownOnAbortNoTrafficRouting'\" make test-e2e\n
"},{"location":"CONTRIBUTING/#running-the-ui","title":"Running the UI","text":"If you'd like to run the UI locally, you first need a running Rollouts controller. This can be a locally running controller with a k3d cluster, as described above, or a controller running in a remote Kubernetes cluster.
In order for the local React app to communicate with the controller and Kubernetes API, run the following to open a port forward to the dashboard:
kubectl argo rollouts dashboard\n
Note that you can also build the API server and run this instead,
make plugin\n./dist/kubectl-argo-rollouts dashboard\n
In another terminal, run the following to start the UI:
cd ui\nyarn install\nyarn start\n
"},{"location":"CONTRIBUTING/#controller-architecture","title":"Controller architecture","text":"Argo Rollouts is actually a collection of individual controllers that handle a specific aspect of Progressive Delivery.
The controllers are:
- Rollout Controller
- Service Controller
- Ingress Controller
- Experiment Controller
- AnalysisRun Controller
"},{"location":"CONTRIBUTING/#tips","title":"Tips","text":" - You can run the tests using a different kubeconfig by setting the
KUBECONFIG environment variable:
KUBECONFIG=~/.kube/minikube make start-e2e\nKUBECONFIG=~/.kube/minikube make test-e2e\n
- To run a specific e2e test, set the
E2E_TEST_OPTIONS environment variable to specify the test (or test regex):
make test-e2e E2E_TEST_OPTIONS=\"-testify.m ^TestRolloutRestart$\"\n
- The e2e tests are designed to run as quickly as possible, eliminating readiness and termination delays. However, it is often desired to artificially slow down the tests for debugging purposes, as well as to understand what the test is doing. To delay startup and termination of pods, set the
E2E_POD_DELAY to an integer value in seconds. This environment variable is often coupled with E2E_TEST_OPTIONS to debug and slow down a specific test.
make test-e2e E2E_POD_DELAY=10\n
- Increasing the timeout. The E2E tests time out waiting on conditions to be met within 60 seconds. If debugging the rollout controller, it may be useful to increase this timeout while say sitting at a debugger breakpoint:
make test-e2e E2E_WAIT_TIMEOUT=999999\n
- The e2e tests leverage a feature of the controller allowing the controller to be sharded with a user-specific \"instance id\" label. This allows the tests to operate only on rollouts with the specified label, and prevents any other controllers (including the system rollout controller), from also operating on the same set of rollouts. This value can be changed (from the default of
argo-rollouts-e2e), using the E2E_INSTANCE_ID environment variable:
make start-e2e E2E_INSTANCE_ID=foo\nmake test-e2e E2E_INSTANCE_ID=foo\n
Alternatively, the e2e tests can be run against the system controller (i.e. without an instance id):
make start-e2e E2E_INSTANCE_ID=''\n
- Working on CRDs? While editing them directly works when you are finding the shape of things you want, the final CRDs are autogenerated. Make sure to regenerate them before submitting PRs. They are controlled by the relevant annotations in the types file:
eg: Analysis Templates are controlled by annotations in pkg/apis/rollouts/v1alpha1/analysis_types.go.
Refer to https://book-v1.book.kubebuilder.io/beyond_basics/generating_crd.html and https://book.kubebuilder.io/reference/markers/crd-validation.html for more info on annotations you can use.
"},{"location":"CONTRIBUTING/#running-local-containers","title":"Running Local Containers","text":"You may need to run containers locally, so here's how:
Create login to Docker Hub, then login.
docker login\n
Add your username as the environment variable, e.g. to your ~/.bash_profile:
export IMAGE_NAMESPACE=argoproj\n
Build the images:
DOCKER_PUSH=true make image\n
Update the manifests:
make manifests\n
Install the manifests:
kubectl -n argo-rollouts apply -f manifests/install.yaml\n
"},{"location":"CONTRIBUTING/#upgrading-kubernetes-libraries","title":"Upgrading Kubernetes Libraries","text":"Argo Rollouts has a dependency on the kubernetes/kubernetes repo for some of the functionality that has not been pushed into the other kubernetes repositories yet. In order to import the kubernetes/kubernetes repo, all of the associated repos have to pinned to the correct version specified by the kubernetes/kubernetes release. The ./hack/update-k8s-dependencies.sh updates all the dependencies to the those correct versions.
"},{"location":"CONTRIBUTING/#upgrading-notifications-engine","title":"Upgrading Notifications Engine","text":"Argo Rollouts has a dependency on the argoproj/notifications-engines repo for the notifications functionality and related documentation.
This is updated by upgrading the Go library in go.mod by running the commands:
go get github.com/argoproj/notifications-engine@LATEST_COMMIT_HASH\ngo mod tidy\n
Next the latest notifications documentation can be imported by running:
make docs\n
"},{"location":"CONTRIBUTING/#documentation-changes","title":"Documentation Changes","text":"Modify contents in docs/ directory.
Preview changes in your browser by visiting http://localhost:8000 after running:
make serve-docs\n
To publish changes, run:
make release-docs\n
"},{"location":"FAQ/","title":"FAQ","text":""},{"location":"FAQ/#general","title":"General","text":""},{"location":"FAQ/#does-argo-rollouts-depend-on-argo-cd-or-any-other-argo-project","title":"Does Argo Rollouts depend on Argo CD or any other Argo project?","text":"Argo Rollouts is a standalone project. Even though it works great with Argo CD and other Argo projects, it can be used on its own for Progressive Delivery scenarios. More specifically, Argo Rollouts does NOT require that you also have installed Argo CD on the same cluster.
"},{"location":"FAQ/#how-does-argo-rollouts-integrate-with-argo-cd","title":"How does Argo Rollouts integrate with Argo CD?","text":"Argo CD understands the health of Argo Rollouts resources via Argo CD\u2019s Lua health check. These Health checks understand when the Argo Rollout objects are Progressing, Suspended, Degraded, or Healthy. Additionally, Argo CD has Lua based Resource Actions that can mutate an Argo Rollouts resource (i.e. unpause a Rollout).
As a result, an operator can build automation to react to the states of the Argo Rollouts resources. For example, if a Rollout created by Argo CD is paused, Argo CD detects that and marks the Application as suspended. Once the new version is verified to be good, the operator can use Argo CD\u2019s resume resource action to unpause the Rollout so it can continue to make progress.
"},{"location":"FAQ/#can-we-run-the-argo-rollouts-kubectl-plugin-commands-via-argo-cd","title":"Can we run the Argo Rollouts kubectl plugin commands via Argo CD?","text":"Argo CD supports running Lua scripts to modify resource kinds (i.e. suspending a CronJob by setting the .spec.suspend to true). These Lua Scripts can be configured in the argocd-cm ConfigMap or upstreamed to the Argo CD's resource_customizations directory. These custom actions have two Lua scripts: one to modify the said resource and another to detect if the action can be executed (i.e. A user should not be able to resuming a unpaused Rollout). Argo CD allows users to execute these actions via the UI or CLI.
In the CLI, a user (or a CI system) can run
argocd app actions run <APP_NAME> <ACTION> \n
This command executes the action listed on the application listed. In the UI, a user can click the hamburger button of a resource and the available actions will appear in a couple of seconds. The user can click and confirm that action to execute it.
Currently, the Rollout action has two available custom actions in Argo CD: resume and restart.
- Resume unpauses a Rollout with a PauseCondition
- Restart: Sets the RestartAt and causes all the pods to be restarted.
"},{"location":"FAQ/#does-argo-rollout-require-a-service-mesh-like-istio","title":"Does Argo Rollout require a Service Mesh like Istio?","text":"Argo Rollouts does not require a service mesh or ingress controller to be used. In the absence of a traffic routing provider, Argo Rollouts manages the replica counts of the canary/stable ReplicaSets to achieve the desired canary weights. Normal Kubernetes Service routing (via kube-proxy) is used to split traffic between the ReplicaSets.
"},{"location":"FAQ/#does-argo-rollout-require-we-follow-gitops-in-my-organization","title":"Does Argo Rollout require we follow GitOps in my organization?","text":"Argo Rollouts is a Kubernetes controller that will react to any manifest change regardless of how the manifest was changed. The manifest can be changed by a Git commit, an API call, another controller or even a manual kubectl command. You can use Argo Rollouts with any traditional CI/CD solution that does not follow the GitOps approach.
"},{"location":"FAQ/#can-we-run-the-argo-rollouts-controller-in-ha-mode","title":"Can we run the Argo Rollouts controller in HA mode?","text":"Yes. A k8s cluster can run multiple replicas of Argo-rollouts controllers to achieve HA. To enable this feature, run the controller with --leader-elect flag and increase the number of replicas in the controller's deployment manifest. The implementation is based on the k8s client-go's leaderelection package. This implementation is tolerant to arbitrary clock skew among replicas. The level of tolerance to skew rate can be configured by setting --leader-election-lease-duration and --leader-election-renew-deadline appropriately. Please refer to the package documentation for details.
"},{"location":"FAQ/#can-we-install-argo-rollouts-centrally-in-a-cluster-and-manage-rollout-resources-in-external-clusters","title":"Can we install Argo Rollouts centrally in a cluster and manage Rollout resources in external clusters?","text":"No you cannot do that (even though Argo CD can work that way). This is by design because the Rollout is a custom resource unknown to vanilla Kubernetes. You need the Rollout CRD as well as the controller in the deployment cluster (every cluster that will use workloads with Rollouts).
"},{"location":"FAQ/#rollouts","title":"Rollouts","text":""},{"location":"FAQ/#which-deployment-strategies-does-argo-rollouts-support","title":"Which deployment strategies does Argo Rollouts support?","text":"Argo Rollouts supports BlueGreen, Canary, and Rolling Update. Additionally, Progressive Delivery features can be enabled on top of the blue-green/canary update, which further provides advanced deployment such as automated analysis and rollback.
"},{"location":"FAQ/#does-the-rollout-object-follow-the-provided-strategy-when-it-is-first-created","title":"Does the Rollout object follow the provided strategy when it is first created?","text":"As with Deployments, Rollouts does not follow the strategy parameters on the initial deploy. The controller tries to get the Rollout into a steady state as fast as possible by creating a fully scaled up ReplicaSet from the provided .spec.template. Once the Rollout has a stable ReplicaSet to transition from, the controller starts using the provided strategy to transition the previous ReplicaSet to the desired ReplicaSet.
"},{"location":"FAQ/#how-does-bluegreen-rollback-work","title":"How does BlueGreen rollback work?","text":"A BlueGreen Rollout keeps the old ReplicaSet up and running for 30 seconds or the value of the scaleDownDelaySeconds. The controller tracks the remaining time before scaling down by adding an annotation called argo-rollouts.argoproj.io/scale-down-deadline to the old ReplicaSet. If the user applies the old Rollout manifest before the old ReplicaSet scales down, the controller does something called a fast rollback. The controller immediately switches the active service\u2019s selector back to the old ReplicaSet\u2019s rollout-pod-template-hash and removes the scaled down annotation from that ReplicaSet. The controller does not do any of the normal operations when trying to introduce a new version since it is trying to revert as fast as possible. A non-fast-track rollback occurs when the scale down annotation has past and the old ReplicaSet has been scaled down. In this case, the Rollout treats the ReplicaSet like any other new ReplicaSet and follows the usual procedure for deploying a new ReplicaSet.
"},{"location":"FAQ/#what-is-the-argo-rolloutsargoprojiomanaged-by-rollouts-annotation","title":"What is the argo-rollouts.argoproj.io/managed-by-rollouts annotation?","text":"Argo Rollouts adds an argo-rollouts.argoproj.io/managed-by-rollouts annotation to Services and Ingresses that the controller modifies. They are used when the Rollout managing these resources is deleted and the controller tries to revert them back into their previous state.
"},{"location":"FAQ/#rollbacks","title":"Rollbacks","text":""},{"location":"FAQ/#does-argo-rollouts-write-back-in-git-when-a-rollback-takes-place","title":"Does Argo Rollouts write back in Git when a rollback takes place?","text":"No. Argo Rollouts doesn't read/write anything to Git. Actually Argo Rollouts knows nothing about Git repositories (only Argo CD has this information if it manages the Rollout). When a rollback takes place, Argo Rollouts marks the application as \"degraded\" and changes the version on the cluster back to the known stable one.
"},{"location":"FAQ/#if-i-use-both-argo-rollouts-and-argo-cd-wouldnt-i-have-an-endless-loop-in-the-case-of-a-rollback","title":"If I use both Argo Rollouts and Argo CD wouldn't I have an endless loop in the case of a Rollback?","text":"No there is no endless loop. As explained already in the previous question, Argo Rollouts doesn't tamper with Git in any way. If you use both Argo projects together, the sequence of events for a rollback is the following:
- Version N runs on the cluster as a Rollout (managed by Argo CD). The Git repository is updated with version N+1 in the Rollout/Deployment manifest
- Argo CD sees the changes in Git and updates the live state in the cluster with the new Rollout object
- Argo Rollouts takes over as it watches for all changes in Rollout Objects. Argo Rollouts is completely oblivious to what is happening in Git. It only cares about what is happening with Rollout objects that are live in the cluster.
- Argo Rollouts tries to apply version N+1 with the selected strategy (e.g. blue/green)
- Version N+1 fails to deploy for some reason
- Argo Rollouts scales back again (or switches traffic back) to version N in the cluster. No change in Git takes place from Argo Rollouts
- Cluster is running version N and is completely healthy
- The Rollout is marked as \"Degraded\" both in ArgoCD and Argo Rollouts.
- Argo CD syncs take no further action as the Rollout object in Git is exactly the same as in the cluster. They both mention version N+1
"},{"location":"FAQ/#so-how-can-i-make-argo-rollouts-write-back-in-git-when-a-rollback-takes-place","title":"So how can I make Argo Rollouts write back in Git when a rollback takes place?","text":"You don't need to do that if you simply want to go back to the previous version using Argo CD. When a deployment fails, Argo Rollouts automatically sets the cluster back to the stable/previous version as explained in the previous question. You don't need to write anything in Git to achieve this. The cluster is still healthy and you have avoided downtime. You are then expected to fix the issue and roll-forward (i.e. deploy the next version) if you want to follow GitOps in a pedantic manner. If you want Argo Rollouts to write back in Git after a failed deployment then you need to orchestrate this with an external system or write custom glue code. But this is normally not needed.
"},{"location":"FAQ/#what-is-the-relationship-between-rollbacks-with-argo-rollouts-and-rollbacks-with-argo-cd","title":"What is the relationship between Rollbacks with Argo Rollouts and Rollbacks with Argo CD?","text":"They are completely unrelated. Argo Rollouts \"rollbacks\" switch the cluster back to the previous version as explained in the previous question. They don't touch or affect Git in any way. Argo CD rollbacks simply point the cluster back a previous Git hash. Normally if you have Argo Rollouts, you don't need to use the Argo CD rollback command.
"},{"location":"FAQ/#how-can-i-deploy-multiple-services-in-a-single-step-and-roll-them-back-according-to-their-dependencies","title":"How can I deploy multiple services in a single step and roll them back according to their dependencies?","text":"The Rollout specification focuses on a single application/deployment. Argo Rollouts knows nothing about application dependencies. If you want to deploy multiple applications together in a smart way (e.g. automatically rollback a frontend if backend deployment fails) you need to write your own solution on top of Argo Rollouts. In most cases, you would need one Rollout resource for each application that you are deploying. Ideally you should also make your services backwards and forwards compatible (i.e. frontend should be able to work with both backend-preview and backend-active).
"},{"location":"FAQ/#how-can-i-run-my-own-custom-tests-eg-smoke-tests-to-decide-if-a-rollback-should-take-place-or-not","title":"How can I run my own custom tests (e.g. smoke tests) to decide if a Rollback should take place or not?","text":"Use a custom Job or Web Analysis. You can pack all your smoke tests in a single container and run them as a Job analysis. Argo Rollouts will use the results of the analysis to automatically rollback if the tests fail.
"},{"location":"FAQ/#experiments","title":"Experiments","text":""},{"location":"FAQ/#why-doesnt-my-experiment-end","title":"Why doesn't my Experiment end?","text":"An Experiment\u2019s duration is controlled by the .spec.duration field and the analyses created for the Experiment. The .spec.duration indicates how long the ReplicaSets created by the Experiment should run. Once the duration passes, the experiment scales down the ReplicaSets it created and marks the AnalysisRuns successful unless the requiredForCompletion field is used in the Experiment. If enabled, the ReplicaSets are still scaled-down, but the Experiment does not finish until the Analysis Run finishes.
Additionally, the .spec.duration is an optional field. If it\u2019s left unset, and the Experiment creates no AnalysisRuns, the ReplicaSets run indefinitely. The Experiment creates AnalysisRuns without the requiredForCompletion field, the Experiment fails only when the AnalysisRun created fails or errors out. If the requiredForCompletion field is set, the Experiment only marks itself as Successful and scales down the created ReplicaSets when the AnalysisRun finishes Successfully.
Additionally, an Experiment ends if the .spec.terminate field is set to true regardless of the state of the Experiment.
"},{"location":"FAQ/#analysis","title":"Analysis","text":""},{"location":"FAQ/#why-doesnt-my-analysisrun-end","title":"Why doesn't my AnalysisRun end?","text":"The AnalysisRun\u2019s duration is controlled by the metrics specified. Each Metric can specify an interval, count, and various limits (ConsecutiveErrorLimit, InconclusiveLimit, FailureLimit). If the interval is omitted, the AnalysisRun takes a single measurement. The count indicates how many measurements should be taken and causes the AnalysisRun to run indefinitely if omitted. The ConsecutiveErrorLimit, InconclusiveLimit, and FailureLimit define the thresholds allowed before putting the rollout into a completed state.
Additionally, an AnalysisRun ends if the .spec.terminate field is set to true regardless of the state of the AnalysisRun.
"},{"location":"FAQ/#what-is-the-difference-between-failures-and-errors","title":"What is the difference between failures and errors?","text":"Failures are when the failure condition evaluates to true or an AnalysisRun without a failure condition evaluates the success condition to false. Errors are when the controller has any kind of issue with taking a measurement (i.e. invalid Prometheus URL).
"},{"location":"architecture/","title":"Architecture","text":"Here is an overview of all the components that take part in a deployment managed by Argo Rollouts.
"},{"location":"architecture/#argo-rollouts-controller","title":"Argo Rollouts controller","text":"This is the main controller that monitors the cluster for events and reacts whenever a resource of type Rollout is changed. The controller will read all the details of the rollout and bring the cluster to the same state as described in the rollout definition.
Note that Argo Rollouts will not tamper with or respond to any changes that happen on normal Deployment Resources. This means that you can install Argo Rollouts in a cluster that is also deploying applications with alternative methods.
To install the controller in your cluster and get started with Progressive Delivery, see the Installation page.
"},{"location":"architecture/#rollout-resource","title":"Rollout resource","text":"The Rollout resource is a custom Kubernetes resource introduced and managed by Argo Rollouts. It is mostly compatible with the native Kubernetes Deployment resource but with extra fields that control the stages, thresholds and methods of advanced deployment methods such as canaries and blue/green deployments.
Note that the Argo Rollouts controller will only respond to those changes that happen in Rollout sources. It will do nothing for normal deployment resources. This means that you need to migrate your Deployments to Rollouts if you want to manage them with Argo Rollouts.
You can see all possible options of a Rollout in the full specification page.
"},{"location":"architecture/#replica-sets-for-old-and-new-version","title":"Replica sets for old and new version","text":"These are instances of the standard Kubernetes ReplicaSet resources. Argo Rollouts puts some extra metadata on them in order to keep track of the different versions that are part of an application.
Note also that the replica sets that take part in a Rollout are fully managed by the controller in an automatic way. You should not tamper with them with external tools.
"},{"location":"architecture/#ingressservice","title":"Ingress/Service","text":"This is the mechanism that traffic from live users enters your cluster and is redirected to the appropriate version. Argo Rollouts use the standard Kubernetes service resource, but with some extra metadata needed for management.
Argo Rollouts is very flexible on networking options. First of all you can have different services during a Rollout, that go only to the new version, only to the old version or both. Specifically for Canary deployments, Argo Rollouts supports several service mesh and ingress solutions for splitting traffic with specific percentages instead of simple balancing based on pod counts and it is possible to use multiple routing providers simultaneously.
"},{"location":"architecture/#analysistemplate-and-analysisrun","title":"AnalysisTemplate and AnalysisRun","text":"Analysis is the capability to connect a Rollout to your metrics provider and define specific thresholds for certain metrics that will decide if an update is successful or not. For each analysis you can define one or more metric queries along with their expected results. A Rollout will progress on its own if metric queries are good, rollback automatically if metrics show failure and pause the rollout if metrics cannot provide a success/failure answer.
For performing an analysis, Argo Rollouts includes two custom Kubernetes resources: AnalysisTemplate and AnalysisRun.
AnalysisTemplate contains instructions on what metrics to query. The actual result that is attached to a Rollout is the AnalysisRun custom resource. You can define an AnalysisTemplate on a specific Rollout or globally on the cluster to be shared by multiple rollouts as a ClusterAnalysisTemplate. The AnalysisRun resource is scoped on a specific rollout.
Note that using an analysis and metrics in a Rollout is completely optional. You can manually pause and promote a rollout or use other external methods (e.g. smoke tests) via the API or the CLI. You don't need a metric solution just to use Argo Rollouts. You can also mix both automated (i.e. analysis based) and manual steps in a Rollout.
Apart from metrics, you can also decide the success of a rollout by running a Kubernetes job or running a webhook.
"},{"location":"architecture/#metric-providers","title":"Metric providers","text":"Argo Rollouts includes native integration for several popular metrics providers that you can use in the Analysis resources to automatically promote or rollback a rollout. See the documentation of each provider for specific setup options.
"},{"location":"architecture/#cli-and-ui-not-shown-in-the-diagram","title":"CLI and UI (Not shown in the diagram)","text":"You can view and manage Rollouts with the Argo Rollouts CLI or the integrated UI. Both are optional.
"},{"location":"best-practices/","title":"Best Practices","text":"This document describes some best practices, tips and tricks when using Argo Rollouts. Be sure to read the FAQ page as well.
"},{"location":"best-practices/#check-application-compatibility","title":"Check application compatibility","text":"Argo Rollouts is a great solution for applications that your team is deploying in a continuous manner (and you have access to the source code). Before using Argo Rollouts you need to contact the developers of the application and verify that you can indeed run multiple versions of the same application at the same time.
Not all applications can work with Argo Rollouts. Applications that use shared resources (e.g. writing to a shared file) will have issues, and \"worker\" type applications (that load data from queues) will rarely work ok without source code modifications.
Note that using Argo Rollouts for \"infrastructure\" applications such as cert-manager, nginx, coredns, sealed-secrets etc is NOT recommended.
"},{"location":"best-practices/#understand-the-scope-of-argo-rollouts","title":"Understand the scope of Argo Rollouts","text":"Currently Argo Rollouts works with a single Kubernetes deployment/application and within a single cluster only. You also need to have the controller deployed on every cluster where a Rollout is running if have more than one clusters using Rollout workloads.
If you want to look at multiple-services on multiple clusters see discussion at issues 2737, 451 and 2088.
Note also that Argo Rollouts is a self-contained solution. It doesn't need Argo CD or any other Argo project to work.
"},{"location":"best-practices/#understand-your-use-case","title":"Understand your use case","text":"Argo Rollouts is perfect for all progressive delivery scenarios as explained in the concepts page.
You should NOT use Argo Rollouts for preview/ephemeral environments. For that use case check the Argo CD Pull Request generator.
The recommended way to use Argo Rollouts is for brief deployments that take 15-20 minutes or maximum 1-2 hours. If you want to run new versions for days or weeks before deciding to promote, then Argo Rollouts is probably not the best solution for you.
Also, if you want to run a wave of multiple versions at the same time (i.e. have 1.1 and 1.2 and 1.3 running at the same time), know that Argo Rollouts was not designed for this scenario.
A version that has just been promoted is assumed to be ready for production and has already passed all your tests (either manual or automated).
"},{"location":"best-practices/#prepare-your-metrics","title":"Prepare your metrics","text":"The end-goal for using Argo Rollouts is to have fully automated deployments that also include rollbacks when needed.
While Argo Rollouts supports manual promotions and other manual pauses, these are best used for experimentation and test reasons.
Ideally you should have proper metrics that tell you in 5-15 minutes if a deployment is successful or not. If you don't have those metrics, then you will miss a lot of value from Argo Rollouts.
Get your metrics in place first and test them with dry-runs before applying them to production deployments.
"},{"location":"best-practices/#there-is-no-argo-rollouts-api","title":"There is no \"Argo Rollouts API\"","text":"A lot of people want to find an official API for managing Rollouts. There isn't any separate Argo Rollouts API. You can always use the Kubernetes API and patching of resources if you want to control a rollout.
But as explained in the previous point the end goal should be fully automated deployments without you having to tell Argo Rollouts to promote or abort.
"},{"location":"best-practices/#integrating-with-other-systems-and-processes","title":"Integrating with other systems and processes","text":"There are two main ways to integrate other systems with Argo Rollouts.
The easiest way is to use Notifications. This means that when a rollout is finished/aborted you send a notification to another system that does other tasks that you want to happen.
Alternatively you can control Rollouts with the CLI or by patching manually the Kubernetes resources.
"},{"location":"best-practices/#use-the-kubernetes-downward-api","title":"Use the Kubernetes Downward API","text":"If you want your applications to know if they are part of a canary or not, you can use Ephemeral labels along with the Kubernetes downward api.
This means that your application will read from files its configuration in a dynamic manner and adapt according to the situation.
"},{"location":"best-practices/#ingress-desiredstable-host-routes","title":"Ingress desired/stable host routes","text":"For various reasons, it is often desired that external services are able to reach the desired pods (aka canary/preview) or stable pods specifically, without the possibility of traffic arbitrarily being split between the two versions. Some use cases include:
- The new version of the service is able to be reach internally/privately (e.g. for manual verification), before exposing it externally.
- An external CI/CD pipeline runs tests against the blue-green preview stack before it is promoted to production.
- Running tests which compare the behavior of old version against the new version.
If you are using an Ingress to route traffic to the service, additional host rules can be added to the ingress rules so that it is possible to specifically reach to the desired (canary/preview) pods or stable pods.
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n name: guestbook\nspec:\n rules:\n # host rule to only reach the desired pods (aka canary/preview)\n - host: guestbook-desired.argoproj.io\n http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n name: guestbook-desired\n port:\n number: 443\n\n # host rule to only reach the stable pods\n - host: guestbook-stable.argoproj.io\n http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n name: guestbook-stable\n port:\n number: 443\n\n # default rule which omits host, and will split traffic between desired vs. stable\n - http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n name: guestbook-root\n port:\n number: 443\n
The above technique has the a benefit in that it would not incur additional cost of allocating additional load balancers.
"},{"location":"best-practices/#reducing-operator-memory-usage","title":"Reducing operator memory usage","text":"On clusters with thousands of rollouts memory usage for the argo-rollouts controller can be reduced significantly by changing the RevisionHistoryLimit property from the default of 10 to a lower number.
One user of Argo Rollouts saw a 27% reduction in memory usage for a cluster with 1290 rollouts by changing RevisionHistoryLimit from 10 to 0.
"},{"location":"best-practices/#rollout-a-configmap-change","title":"Rollout a ConfigMap change","text":"Argo Rollouts is meant to work on a Kubernetes Deployment. When a ConfigMap is mounted inside one the Deployment container and a change occurs inside the ConfigMap, it won't trigger a new Rollout by default.
One technique to trigger the Rollout it to name dynamically the ConfigMap. For example, adding a hash of its content at the end of the name:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: my-config-7270e14e6\n
Each time a change occurs in the ConfigMap, its name will change in the Deployment reference as well, triggering a Rollout.
However, it's not enough to perform correctly progressive rollouts, as the old ConfigMap might get deleted as soon as the new one is created. This would prevent Experiments and rollbacks in case of rollout failure to work correctly.
While no magical solution exist to work aroud that, you can tweak your deployment tool to remove the ConfigMap only when the Rollout is completed successfully.
Example with Argo CD:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: my-config-7270e14e6\n annotations:\n argocd.argoproj.io/sync-options: PruneLast=true\n
"},{"location":"concepts/","title":"Concepts","text":""},{"location":"concepts/#rollout","title":"Rollout","text":"A Rollout is Kubernetes workload resource which is equivalent to a Kubernetes Deployment object. It is intended to replace a Deployment object in scenarios when more advanced deployment or progressive delivery functionality is needed. A Rollout provides the following features which a Kubernetes Deployment cannot:
- blue-green deployments
- canary deployments
- integration with ingress controllers and service meshes for advanced traffic routing
- integration with metric providers for blue-green & canary analysis
- automated promotion or rollback based on successful or failed metrics
"},{"location":"concepts/#progressive-delivery","title":"Progressive Delivery","text":"Progressive delivery is the process of releasing updates of a product in a controlled and gradual manner, thereby reducing the risk of the release, typically coupling automation and metric analysis to drive the automated promotion or rollback of the update.
Progressive delivery is often described as an evolution of continuous delivery, extending the speed benefits made in CI/CD to the deployment process. This is accomplished by limiting the exposure of the new version to a subset of users, observing and analyzing for correct behavior, then progressively increasing the exposure to a broader and wider audience while continuously verifying correctness.
"},{"location":"concepts/#deployment-strategies","title":"Deployment Strategies","text":"While the industry has used a consistent terminology to describe various deployment strategies, the implementations of these strategies tend to differ across tooling. To make it clear how the Argo Rollouts will behave, here are the descriptions of the various deployment strategies implementations offered by the Argo Rollouts.
"},{"location":"concepts/#rolling-update","title":"Rolling Update","text":"A RollingUpdate slowly replaces the old version with the new version. As the new version comes up, the old version is scaled down in order to maintain the overall count of the application. This is the default strategy of the Deployment object.
"},{"location":"concepts/#recreate","title":"Recreate","text":"A Recreate deployment deletes the old version of the application before bring up the new version. As a result, this ensures that two versions of the application never run at the same time, but there is downtime during the deployment.
"},{"location":"concepts/#blue-green","title":"Blue-Green","text":"A Blue-Green deployment (sometimes referred to as a Red-Black) has both the new and old version of the application deployed at the same time. During this time, only the old version of the application will receive production traffic. This allows the developers to run tests against the new version before switching the live traffic to the new version.
"},{"location":"concepts/#canary","title":"Canary","text":"A Canary deployment exposes a subset of users to the new version of the application while serving the rest of the traffic to the old version. Once the new version is verified to be correct, the new version can gradually replace the old version. Ingress controllers and service meshes such as NGINX and Istio, enable more sophisticated traffic shaping patterns for canarying than what is natively available (e.g. achieving very fine-grained traffic splitting, or splitting based on HTTP headers).
The picture above shows a canary with two stages (10% and 33% of traffic goes to new version) but this is just an example. With Argo Rollouts you can define the exact number of stages and percentages of traffic according to your use case.
"},{"location":"concepts/#which-strategy-to-choose","title":"Which strategy to choose","text":"In general Blue/Green is the easier strategy to start with, but also the more limited. We recommend you start with Blue/Green deployments first and as you gain confidence for your metrics and applications switch to Canaries.
You also need to examine if your application can handle canaries or not.
- Blue/Green always works because only one application is active at a time. Not all applications can have different versions running in parallel at the same time (which is what canaries are doing). This can be a showstopper for adopting canary deployments especially for legacy applications.
- Blue/Green is simpler because you can get their full value WITHOUT a traffic manager. While canaries can also work without a traffic manager, most of their advanced features assume a fine-grained way to control traffic. If you don't have a traffic manager, then you can easily get the full value of blue/green deployments but only the basic capabilities of canaries.
- Blue/Green also works with services that use queues and databases (workers that fetch tasks). Argo Rollouts doesn't control traffic flow for connections it doesn't understand (i.e. binary/queue channels).
Here is a summary table for the possible approaches.
Blue/Green Basic Canary Canary with Traffic manager Adoption Complexity Low Medium High Flexibility Low High Maximum Needs traffic provider No No Yes Works with queue workers Yes No No Works with shared/locked resources Yes No No Traffic switch All or nothing Gradual percentage Gradual percentage Traffic control 0% or 100% coarse grained fine grained Traffic depends on deployment state number of canary pods Any split option is possible Advanced routing scenarios No No Yes Failure Blast Radius Massive impact Low impact Low impact Note that that traffic manager can be any compatible Service Mesh or Ingress Controller or Gateway API implementation (via a plugin).
"},{"location":"dashboard/","title":"UI Dashboard","text":"The Argo Rollouts Kubectl plugin can serve a local UI Dashboard to visualize your Rollouts.
To start it, run kubectl argo rollouts dashboard in the namespace that contains your Rollouts. Then visit localhost:3100 to view the user interface.
"},{"location":"dashboard/#list-view","title":"List view","text":""},{"location":"dashboard/#individual-rollout-view","title":"Individual Rollout view","text":""},{"location":"getting-started/","title":"Getting Started","text":"This guide will demonstrate various concepts and features of Argo Rollouts by going through deployment, upgrade, promotion, and abortion of a Rollout.
"},{"location":"getting-started/#requirements","title":"Requirements","text":" - Kubernetes cluster with argo-rollouts controller installed (see install guide)
- kubectl with argo-rollouts plugin installed (see install guide)
"},{"location":"getting-started/#1-deploying-a-rollout","title":"1. Deploying a Rollout","text":"First we deploy a Rollout resource and a Kubernetes Service targeting that Rollout. The example Rollout in this guide utilizes a canary update strategy which sends 20% of traffic to the canary, followed by a manual promotion, and finally gradual automated traffic increases for the remainder of the upgrade. This behavior is described in the following portion of the Rollout spec:
spec:\n replicas: 5\n strategy:\n canary:\n steps:\n - setWeight: 20\n - pause: {}\n - setWeight: 40\n - pause: {duration: 10}\n - setWeight: 60\n - pause: {duration: 10}\n - setWeight: 80\n - pause: {duration: 10}\n
Run the following command to deploy the initial Rollout and Service:
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/basic/rollout.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/basic/service.yaml\n
Initial creations of any Rollout will immediately scale up the replicas to 100% (skipping any canary upgrade steps, analysis, etc...) since there was no upgrade that occurred.
The Argo Rollouts kubectl plugin allows you to visualize the Rollout, its related resources (ReplicaSets, Pods, AnalysisRuns), and presents live state changes as they occur. To watch the rollout as it deploys, run the get rollout --watch command from plugin:
kubectl argo rollouts get rollout rollouts-demo --watch\n
"},{"location":"getting-started/#2-updating-a-rollout","title":"2. Updating a Rollout","text":"Next it is time to perform an update. Just as with Deployments, any change to the Pod template field (spec.template) results in a new version (i.e. ReplicaSet) to be deployed. Updating a Rollout involves modifying the rollout spec, typically changing the container image field with a new version, and then running kubectl apply against the new manifest. As a convenience, the rollouts plugin provides a set image command, which performs these steps against the live rollout object in-place. Run the following command to update the rollouts-demo Rollout with the \"yellow\" version of the container:
kubectl argo rollouts set image rollouts-demo \\\n rollouts-demo=argoproj/rollouts-demo:yellow\n
During a rollout update, the controller will progress through the steps defined in the Rollout's update strategy. The example rollout sets a 20% traffic weight to the canary, and pauses the rollout indefinitely until user action is taken to unpause/promote the rollout. After updating the image, watch the rollout again until it reaches the paused state:
kubectl argo rollouts get rollout rollouts-demo --watch\n
When the demo rollout reaches the second step, we can see from the plugin that the Rollout is in a paused state, and now has 1 of 5 replicas running the new version of the pod template, and 4 of 5 replicas running the old version. This equates to the 20% canary weight as defined by the setWeight: 20 step.
"},{"location":"getting-started/#3-promoting-a-rollout","title":"3. Promoting a Rollout","text":"The rollout is now in a paused state. When a Rollout reaches a pause step with no duration, it will remain in a paused state indefinitely until it is resumed/promoted. To manually promote a rollout to the next step, run the promote command of the plugin:
kubectl argo rollouts promote rollouts-demo\n
After promotion, Rollout will proceed to execute the remaining steps. The remaining rollout steps in our example are fully automated, so the Rollout will eventually complete steps until it has has fully transitioned to the new version. Watch the rollout again until it has completed all steps:
kubectl argo rollouts get rollout rollouts-demo --watch\n
Tip
The promote command also supports the ability to skip all remaining steps and analysis with the --full flag.
Once all steps complete successfully, the new ReplicaSet is marked as the \"stable\" ReplicaSet. Whenever a rollout is aborted during an update, either automatically via a failed canary analysis, or manually by a user, the Rollout will fall back to the \"stable\" version.
"},{"location":"getting-started/#4-aborting-a-rollout","title":"4. Aborting a Rollout","text":"Next we will learn how to manually abort a rollout during an update. First, deploy a new \"red\" version of the container using the set image command, and wait for the rollout to reach the paused step again:
kubectl argo rollouts set image rollouts-demo \\\n rollouts-demo=argoproj/rollouts-demo:red\n
This time, instead of promoting the rollout to the next step, we will abort the update, so that it falls back to the \"stable\" version. The plugin provides an abort command as a way to manually abort a rollout at any time during an update:
kubectl argo rollouts abort rollouts-demo\n
When a rollout is aborted, it will scale up the \"stable\" version of the ReplicaSet (in this case the yellow image), and scale down any other versions. Although the stable version of the ReplicaSet may be running and is healthy, the overall rollout is still considered Degraded, since the desired version (the red image) is not the version which is actually running.
In order to make Rollout considered Healthy again and not Degraded, it is necessary to change the desired state back to the previous, stable version. This typically involves running kubectl apply against the previous Rollout spec. In our case, we can simply re-run the set image command using the previous, \"yellow\" image.
kubectl argo rollouts set image rollouts-demo \\\n rollouts-demo=argoproj/rollouts-demo:yellow\n
After running this command, you should notice that the Rollout immediately becomes Healthy, and there is no activity with regards to new ReplicaSets becoming created.
When a Rollout has not yet reached its desired state (e.g. it was aborted, or in the middle of an update), and the stable manifest were re-applied, the Rollout detects this as a rollback and not a update, and will fast-track the deployment of the stable ReplicaSet by skipping analysis, and the steps.
"},{"location":"getting-started/#summary","title":"Summary","text":"In this guide, we have learned basic capabilities of Argo Rollouts, including:
- Deploying a rollout
- Performing a canary update
- Manual promotion
- Manual abortion
The Rollout in this basic example did not utilize a ingress controller or service mesh provider to route traffic. Instead, it used normal Kubernetes Service networking (i.e. kube-proxy) to achieve an approximate canary weight, based on the closest ratio of new to old replica counts. As a result, this Rollout had a limitation in that it could only achieve a minimum canary weight of 20%, by scaling 1 of 5 pods to run the new version. In order to achieve much finer grained canaries, an ingress controller or service mesh is necessary.
Follow one of the traffic routing guides to see how Argo Rollouts can leverage a networking provider to achieve more advanced traffic shaping.
- ALB Guide
- App Mesh Guide
- Ambassador Guide
- Istio Guide
- Multiple Providers Guide
- NGINX Guide
- SMI Guide
"},{"location":"installation/","title":"Installation","text":""},{"location":"installation/#controller-installation","title":"Controller Installation","text":"Two types of installation:
- install.yaml - Standard installation method.
kubectl create namespace argo-rollouts\nkubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml\n
This will create a new namespace, argo-rollouts, where Argo Rollouts controller will run.
Tip
If you are using another namespace name, please update install.yaml clusterrolebinding's serviceaccount namespace name.
Tip
When installing Argo Rollouts on Kubernetes v1.14 or lower, the CRD manifests must be kubectl applied with the --validate=false option. This is caused by use of new CRD fields introduced in v1.15, which are rejected by default in lower API servers.
Tip
On GKE, you will need grant your account the ability to create new cluster roles:
kubectl create clusterrolebinding YOURNAME-cluster-admin-binding --clusterrole=cluster-admin --user=YOUREMAIL@gmail.com\n
- namespace-install.yaml - Installation of Argo Rollouts which requires only namespace level privileges. An example usage of this installation method would be to run several Argo Rollouts controller instances in different namespaces on the same cluster.
Note: Argo Rollouts CRDs are not included into namespace-install.yaml. and have to be installed separately. The CRD manifests are located in manifests/crds directory. Use the following command to install them:
kubectl apply -k https://github.com/argoproj/argo-rollouts/manifests/crds\\?ref\\=stable\n
You can find released container images of the controller at Quay.io. There are also old releases at Dockerhub, but since the introduction of rate limiting, the Argo project has moved to Quay.
"},{"location":"installation/#kubectl-plugin-installation","title":"Kubectl Plugin Installation","text":"The kubectl plugin is optional, but is convenient for managing and visualizing rollouts from the command line.
"},{"location":"installation/#brew","title":"Brew","text":"brew install argoproj/tap/kubectl-argo-rollouts\n
"},{"location":"installation/#manual","title":"Manual","text":" -
Install Argo Rollouts Kubectl plugin with curl.
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-darwin-amd64\n
For Linux dist, replace darwin with linux
-
Make the kubectl-argo-rollouts binary executable.
chmod +x ./kubectl-argo-rollouts-darwin-amd64\n
-
Move the binary into your PATH.
sudo mv ./kubectl-argo-rollouts-darwin-amd64 /usr/local/bin/kubectl-argo-rollouts\n
Test to ensure the version you installed is up-to-date:
kubectl argo rollouts version\n
"},{"location":"installation/#shell-auto-completion","title":"Shell auto completion","text":"To enable auto completion for the plugin when used with kubectl (version 1.26 or newer), you need to create a shell script on your PATH called kubectl_complete-argo-rollouts which will provide the completions.
cat <<EOF >kubectl_complete-argo-rollouts\n#!/usr/bin/env sh\n\n# Call the __complete command passing it all arguments\nkubectl argo rollouts __complete \"\\$@\"\nEOF\n\nchmod +x kubectl_complete-argo-rollouts\nsudo mv ./kubectl_complete-argo-rollouts /usr/local/bin/\n
To enable auto completion for the CLI run as a standalone binary, the CLI can export shell completion code for several shells.
For bash, ensure you have bash completions installed and enabled. To access completions in your current shell, run $ source <(kubectl-argo-rollouts completion bash). Alternatively, write it to a file and source in .bash_profile.
The completion command supports bash, zsh, fish and powershell.
See the completion command documentation for more details.
"},{"location":"installation/#using-the-cli-with-docker","title":"Using the CLI with Docker","text":"The CLI is also available as a container image at https://quay.io/repository/argoproj/kubectl-argo-rollouts
You can run it like any other Docker image or use it in any CI platform that supports Docker images.
docker run quay.io/argoproj/kubectl-argo-rollouts:master version\n
"},{"location":"installation/#supported-versions","title":"Supported versions","text":"Check e2e testing file to see what the Kubernetes version is being fully tested.
You can switch to different tags to see what relevant Kubernetes versions were being tested for the respective version.
"},{"location":"installation/#upgrading-argo-rollouts","title":"Upgrading Argo Rollouts","text":"Argo Rollouts is a Kubernetes controller that doesn't hold any external state. It is active only when deployments are actually happening.
To upgrade Argo Rollouts:
- Try to find a time period when no deployments are happening
- Delete the previous version of the controller and apply/install the new one
- When a new Rollout takes place the new controller will be activated.
If deployments are happening while you upgrade the controller, then you shouldn't have any downtime. Current Rollouts will be paused and as soon as the new controller becomes active it will resume all in-flight deployments.
"},{"location":"migrating/","title":"Migrating to Rollouts","text":"There are ways to migrate to Rollout:
- Convert an existing Deployment resource to a Rollout resource.
- Reference an existing Deployment from a Rollout using
workloadRef field.
"},{"location":"migrating/#convert-deployment-to-rollout","title":"Convert Deployment to Rollout","text":"When converting a Deployment to a Rollout, it involves changing three fields:
- Replacing the
apiVersion from apps/v1 to argoproj.io/v1alpha1 - Replacing the
kind from Deployment to Rollout - Replacing the deployment strategy with a blue-green or canary strategy
Below is an example of a Rollout resource using the canary strategy.
apiVersion: argoproj.io/v1alpha1 # Changed from apps/v1\nkind: Rollout # Changed from Deployment\nmetadata:\n name: rollouts-demo\nspec:\n selector:\n matchLabels:\n app: rollouts-demo\n template:\n metadata:\n labels:\n app: rollouts-demo\n spec:\n containers:\n - name: rollouts-demo\n image: argoproj/rollouts-demo:blue\n ports:\n - containerPort: 8080\n strategy:\n canary: # Changed from rollingUpdate or recreate\n steps:\n - setWeight: 20\n - pause: {}\n
Warning
When migrating a Deployment which is already serving live production traffic, a Rollout should run next to the Deployment before deleting the Deployment or scaling down the Deployment. Not following this approach might result in downtime. It also allows for the Rollout to be tested before deleting the original Deployment.
"},{"location":"migrating/#reference-deployment-from-rollout","title":"Reference Deployment From Rollout","text":"Instead of removing Deployment you can scale it down to zero and reference it from the Rollout resource:
- Create a Rollout resource.
- Reference an existing Deployment using
workloadRef field. - In the
workloadRef field set the scaleDown attribute, which specifies how the Deployment should be scaled down. There are three options available: never: the Deployment is not scaled downonsuccess: the Deployment is scaled down after the Rollout becomes healthyprogressively: as the Rollout is scaled up the Deployment is scaled down.
Alternatively, manually scale down an existing Deployment by changing replicas field of an existing Deployment to zero. 1. To perform an update, the change should be made to the Pod template field of the Deployment.
Below is an example of a Rollout resource referencing a Deployment.
apiVersion: argoproj.io/v1alpha1 # Create a rollout resource\nkind: Rollout\nmetadata:\n name: rollout-ref-deployment\nspec:\n replicas: 5\n selector:\n matchLabels:\n app: rollout-ref-deployment\n workloadRef: # Reference an existing Deployment using workloadRef field\n apiVersion: apps/v1\n kind: Deployment\n name: rollout-ref-deployment\n scaleDown: onsuccess\n strategy:\n canary:\n steps:\n - setWeight: 20\n - pause: {duration: 10s}\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n labels:\n app.kubernetes.io/instance: rollout-canary\n name: rollout-ref-deployment\nspec:\n replicas: 0 # Scale down existing deployment\n selector:\n matchLabels:\n app: rollout-ref-deployment\n template:\n metadata:\n labels:\n app: rollout-ref-deployment\n spec:\n containers:\n - name: rollouts-demo\n image: argoproj/rollouts-demo:blue\n imagePullPolicy: Always\n ports:\n - containerPort: 8080\n
Consider following if your Deployment runs in production:
"},{"location":"migrating/#running-rollout-and-deployment-side-by-side","title":"Running Rollout and Deployment side-by-side","text":"After creation Rollout will spinup required number of Pods side-by-side with the Deployment Pods. Rollout won't try to manage existing Deployment Pods. That means you can safely update add Rollout to the production environment without any interruption but you are going to run twice more Pods during migration.
Argo-rollouts controller patches the spec of rollout object with an annotation of rollout.argoproj.io/workload-generation, which equals the generation of referenced deployment. Users can detect if the rollout matches desired generation of deployment by checking the workloadObservedGeneration in the rollout status.
"},{"location":"migrating/#traffic-management-during-migration","title":"Traffic Management During Migration","text":"The Rollout offers traffic management functionality that manages routing rules and flows the traffic to different versions of an application. For example Blue-Green deployment strategy manipulates Kubernetes Service selector and direct production traffic to \"green\" instances only.
If you are using this feature then Rollout switches production\u00a0traffic to Pods that it manages. The switch happens only when the required number of Pod is running and healthy so it is safe in production as well. However, if you want to be extra careful then consider creating a temporal Service or Ingress object to validate Rollout behavior. Once testing is done delete temporal Service/Ingress and switch rollout to production one.
"},{"location":"migrating/#migrating-to-deployments","title":"Migrating to Deployments","text":"In case users want to rollback to the deployment kinds from rollouts, there are two scenarios aligned with those in Migrating to Rollouts.
- Convert a Rollout resource to a Deployment resource.
- Reference an existing Deployment from a Rollout using
workloadRef field.
"},{"location":"migrating/#convert-rollout-to-deployment","title":"Convert Rollout to Deployment","text":"When converting a Rollout to a Deployment, it involves changing three fields:
- Changing the apiVersion from argoproj.io/v1alpha1 to apps/v1
- Changing the kind from Rollout to Deployment
- Remove the rollout strategy in
spec.strategy.canary or spec.strategy.blueGreen
Warning
When migrating a Rollout which is already serving live production traffic, a Deployment should run next to the rollout before deleting the rollout or scaling down the rollout. Not following this approach might result in downtime. It also allows for the Deployment to be tested before deleting the original Rollout.
"},{"location":"migrating/#reference-deployment-from-rollout_1","title":"Reference Deployment From Rollout","text":"When a rollout is referencing to a deployment:
- Scale-up an existing Deployment by changing its
replicas field to a desired number of pods. - Wait for the Deployment pods to become Ready.
- Scale-down an existing Rollout by changing its
replicas field to zero.
Please refer to Running Rollout and Deployment side-by-side and Traffic Management During Migration for caveats.
"},{"location":"plugins/","title":"Creating an Argo Rollouts Plugin","text":""},{"location":"plugins/#high-level-overview","title":"High Level Overview","text":"Argo Rollouts plugins depend on hashicorp's go-plugin library. This library provides a way for a plugin to be compiled as a standalone executable and then loaded by the rollouts controller at runtime. This works by having the plugin executable act as a rpc server and the rollouts controller act as a client. The plugin executable is started by the rollouts controller and is a long-lived process and that the rollouts controller connects to over a unix socket.
Here is an overview of how plugins are loaded:
The communication protocol uses golang built in net/rpc library so plugins have to be written in golang.
"},{"location":"plugins/#plugin-repository","title":"Plugin Repository","text":"In order to get plugins listed in the main argo rollouts documentation we ask that the plugin repository be created under the argoproj-labs organization. Please open an issue under argo-rollouts requesting a repo which you would be granted admin access on.
There is also a standard naming convention for plugin names used for configmap registration, as well as what the plugin uses for locating its specific configuration on rollout or analysis resources. The name needs to be in the form of <namespace>/<name> and both and have a regular expression check that matches Github's requirements for username/org and repository name. This requirement is in place to help with allowing multiple creators of the same plugin types to exist such as <org1>/nginx and <org2>/nginx. These names could be based of the repo name such as argoproj-labs/rollouts-plugin-metric-sample-prometheus but it is not a requirement.
There will also be a standard for naming repositories under argoproj-labs in the form of rollouts-plugin-<type>-<tool> where <type> is say metric, or trafficrouter and <tool> is the software the plugin is for say nginx.
"},{"location":"plugins/#plugin-name","title":"Plugin Name","text":"So now that we have an idea on plugin naming and repository standards let's pick a name to use for the rest of this documentation and call our plugin argoproj-labs/nginx.
This name will be used in a few different spots the first is the config map that your plugin users will need to configure. It looks like this below.
kind: ConfigMap\nmetadata:\n name: argo-rollouts-config\ndata:\n metricProviderPlugins: |-\n - name: \"argoproj-labs/metrics\"\n location: \"file:///tmp/argo-rollouts/metric-plugin\"\n args:\n - \"--log-level\"\n - \"debug\"\n trafficRouterPlugins: |-\n - name: \"argoproj-labs/nginx\"\n location: \"file:///tmp/argo-rollouts/traffic-plugin\"\n args:\n - \"--log-level\"\n - \"debug\"\n
As you can see there is a field called name: under both metrics or trafficrouters this is the first place where your end users will need to configure the name of the plugin. The second location is either in the rollout object or the analysis template which you can see the examples below. The third args holds the command line arguments of the plugin.
"},{"location":"plugins/#analysistemplate-example","title":"AnalysisTemplate Example","text":"apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n metrics:\n - name: success-rate\n ...\n provider:\n plugin:\n argoproj-labs/metrics:\n address: http://prometheus.local\n
"},{"location":"plugins/#traffic-router-example","title":"Traffic Router Example","text":"apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: example-plugin-ro\nspec:\n strategy:\n canary:\n canaryService: example-plugin-ro-canary-analysis\n stableService: example-plugin-ro-stable-analysis\n trafficRouting:\n plugins:\n argoproj-labs/nginx:\n stableIngress: canary-demo\n
You can see that we use the plugin name under spec.metrics[].provider.plugin for analysis template and spec.strategy.canary.trafficRouting.plugins for traffic routers. You as a plugin author can then put any configuration you need under argoproj-labs/nginx and you will be able to look up that config in your plugin via the plugin name key. You will also want to document what configuration options your plugin supports.
"},{"location":"plugins/#plugin-interfaces","title":"Plugin Interfaces","text":"Argo Rollouts currently supports two plugin systems as a plugin author your end goal is to implement these interfaces as a hashicorp go-plugin. The two interfaces are MetricsPlugin and TrafficRouterPlugin for each of the respective plugins:
type MetricProviderPlugin interface {\n // InitPlugin initializes the traffic router plugin this gets called once when the plugin is loaded.\n InitPlugin() RpcError\n // Run start a new external system call for a measurement\n // Should be idempotent and do nothing if a call has already been started\n Run(*v1alpha1.AnalysisRun, v1alpha1.Metric) v1alpha1.Measurement\n // Resume Checks if the external system call is finished and returns the current measurement\n Resume(*v1alpha1.AnalysisRun, v1alpha1.Metric, v1alpha1.Measurement) v1alpha1.Measurement\n // Terminate will terminate an in-progress measurement\n Terminate(*v1alpha1.AnalysisRun, v1alpha1.Metric, v1alpha1.Measurement) v1alpha1.Measurement\n // GarbageCollect is used to garbage collect completed measurements to the specified limit\n GarbageCollect(*v1alpha1.AnalysisRun, v1alpha1.Metric, int) RpcError\n // Type gets the provider type\n Type() string\n // GetMetadata returns any additional metadata which providers need to store/display as part\n // of the metric result. For example, Prometheus uses is to store the final resolved queries.\n GetMetadata(metric v1alpha1.Metric) map[string]string\n}\n\ntype TrafficRouterPlugin interface {\n // InitPlugin initializes the traffic router plugin this gets called once when the plugin is loaded.\n InitPlugin() RpcError\n // UpdateHash informs a traffic routing reconciler about new canary, stable, and additionalDestination(s) pod hashes\n UpdateHash(rollout *v1alpha1.Rollout, canaryHash, stableHash string, additionalDestinations []v1alpha1.WeightDestination) RpcError\n // SetWeight sets the canary weight to the desired weight\n SetWeight(rollout *v1alpha1.Rollout, desiredWeight int32, additionalDestinations []v1alpha1.WeightDestination) RpcError\n // SetHeaderRoute sets the header routing step\n SetHeaderRoute(rollout *v1alpha1.Rollout, setHeaderRoute *v1alpha1.SetHeaderRoute) RpcError\n // SetMirrorRoute sets up the traffic router to mirror traffic to a service\n SetMirrorRoute(rollout *v1alpha1.Rollout, setMirrorRoute *v1alpha1.SetMirrorRoute) RpcError\n // VerifyWeight returns true if the canary is at the desired weight and additionalDestinations are at the weights specified\n // Returns nil if weight verification is not supported or not applicable\n VerifyWeight(rollout *v1alpha1.Rollout, desiredWeight int32, additionalDestinations []v1alpha1.WeightDestination) (RpcVerified, RpcError)\n // RemoveManagedRoutes Removes all routes that are managed by rollouts by looking at spec.strategy.canary.trafficRouting.managedRoutes\n RemoveManagedRoutes(ro *v1alpha1.Rollout) RpcError\n // Type returns the type of the traffic routing reconciler\n Type() string\n}\n
"},{"location":"plugins/#plugin-init-function","title":"Plugin Init Function","text":"Each plugin interface has a InitPlugin function, this function is called when the plugin is first started up and is only called once per startup. The InitPlugin function is used as a means to initialize the plugin it gives you the plugin author the ability to either set up a client for a specific metrics provider or in the case of a traffic router construct a client or informer for kubernetes api. The one thing to note about this though is because these calls happen over RPC the plugin author should not depend on state being stored in the plugin struct as it will not be persisted between calls.
"},{"location":"plugins/#kubernetes-rbac","title":"Kubernetes RBAC","text":"The plugin runs as a child process of the rollouts controller and as such it will inherit the same RBAC permissions as the controller. This means that the service account for the rollouts controller will need the correct permissions for the plugin to function. This might mean instructing users to create a role and role binding to the standard rollouts service account for the plugin to use. This will probably affect traffic router plugins more than metrics plugins.
"},{"location":"plugins/#sample-plugins","title":"Sample Plugins","text":"There are two sample plugins within the argo-rollouts repo that you can use as a reference for creating your own plugin.
- Metrics Plugin Sample
- Traffic Router Plugin Sample
"},{"location":"releasing/","title":"Releasing","text":" -
Ensure that the release branch already exist.
-
Checkout the release branch. Example: git fetch upstream && git checkout release-1.5
-
Run the script found at hack/trigger-release.sh as follows:
./hack/trigger-release.sh <version> <remote name>\n
Example:
./hack/trigger-release.sh v1.6.0-rc1 upstream\n
Tip
The tag must be in one of the following formats to trigger the GH workflow: * GA: v<MAJOR>.<MINOR>.<PATCH> * Pre-release: v<MAJOR>.<MINOR>.<PATCH>-rc<RC#>
Once the script is executed successfully, a GitHub workflow will start execution. You can follow its progress under the Actions tab, the name of the action is Release.
- When the action completes, visit the generated draft Github releases and enter the details about the release:
- Getting started (copy from previous release and new version)
- Changelog
"},{"location":"releasing/#update-brew-formula","title":"Update Brew formula","text":" -
Update Brew formula:
-
Fork the repo https://github.com/argoproj/homebrew-tap
- Run the following commands to update the brew formula:
cd homebrew-tap\n./update.sh kubectl-argo-rollouts $VERSION\n
- If there is a new minor version we want to update the versioned formula as well:
- Run the following commands to update the versioned brew formula:
./update.sh kubectl-argo-rollouts $VERSION @<version_without_patch_and_v>\n
- Example: If the new version is
v1.3.2, we want to update the formula for v1.3 as well. ./update.sh kubectl-argo-rollouts v1.3.2 @1.3\n
- Commit and push the changes to your fork
git commit -am \"Update kubectl-argo-rollouts to $VERSION\"\n
- Create a PR with the modified files pointing to upstream/master
- Once the PR is approved by a maintainer, it can be merged.
"},{"location":"releasing/#verify","title":"Verify","text":" -
Install locally using the command below and follow the Getting Started Guide:
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/download/${VERSION}/install.yaml\n
-
Check the Kubectl Argo Rollout plugin:
brew upgrade kubectl-argo-rollouts\nkubectl argo rollouts version\n
"},{"location":"roadmap/","title":"Roadmap","text":"The Argo Rollouts roadmap is maintained in Github Milestones on the Github repository.
"},{"location":"roadmap/#release-cycle","title":"Release Cycle","text":""},{"location":"roadmap/#schedule","title":"Schedule","text":"These are the upcoming releases date estimates:
Release Release Planning Meeting Release Candidate 1 General Availability v1.4 TBD Monday, Dec. 19, 2022 Tuesday, Jan. 9, 2023 v1.5 Monday, Mar. 6, 2023 Monday, Mar. 20, 2023 Monday, Apr. 10, 2023 v1.6 Monday, Jun. 5, 2023 Monday, Jun. 19, 2023 Wednesday, Jul. 12, 2023 v1.7 Monday, Sep. 4, 2023 Monday, Sep. 18, 2023 Monday, Oct. 9, 2023"},{"location":"roadmap/#release-process","title":"Release Process","text":""},{"location":"roadmap/#minor-releases-eg-1x0","title":"Minor Releases (e.g. 1.x.0)","text":"A minor Argo Rollouts release occurs four times a year, once every three months. Each General Availability (GA) release is preceded by several Release Candidates (RCs). The first RC is released three weeks before the scheduled GA date.
These are the approximate release dates:
- The first Monday of January
- The first Monday of April
- The first Monday of July
- The first Monday of October
Dates may be shifted slightly to accommodate holidays. Those shifts should be minimal.
"},{"location":"roadmap/#patch-releases-eg-14x","title":"Patch Releases (e.g. 1.4.x)","text":"Argo Rollouts patch releases occur on an as-needed basis. Only the three most recent minor versions are eligible for patch releases. Versions older than the three most recent minor versions are considered EOL and will not receive bug fixes or security updates.
"},{"location":"roadmap/#feature-acceptance-criteria","title":"Feature Acceptance Criteria","text":"To be eligible for inclusion in a minor release, a new feature must meet the following criteria before the release\u2019s RC date.
If it is a large feature that involves significant design decisions, that feature must be described in a Proposal.
The feature PR must include:
- Tests (passing)
- Documentation
- If necessary, a note in the Upgrading docs for the planned minor release
- The PR must be reviewed, approved, and merged by an Approver.
If these criteria are not met by the RC date, the feature will be ineligible for inclusion in the RC series or GA for that minor release. It will have to wait for the next minor release.
"},{"location":"analysis/cloudwatch/","title":"CloudWatch Metrics","text":"Important
Available since v1.1.0
A CloudWatch using GetMetricData can be used to obtain measurements for analysis.
"},{"location":"analysis/cloudwatch/#setup","title":"Setup","text":"You can use CloudWatch Metrics if you have used to EKS or not. This analysis is required IAM permission for cloudwatch:GetMetricData and you need to define AWS_REGION in Deployment for argo-rollouts.
"},{"location":"analysis/cloudwatch/#eks","title":"EKS","text":"If you create new cluster on EKS, you can attach cluster IAM role or attach IAM roles for service accounts. If you have already cluster on EKS, you can attach IAM roles for service accounts.
"},{"location":"analysis/cloudwatch/#not-eks","title":"not EKS","text":"You need to define access key and secret key.
apiVersion: v1\nkind: Secret\nmetadata:\n name: cloudwatch-secret\ntype: Opaque\nstringData:\n AWS_ACCESS_KEY_ID: <aws-access-key-id>\n AWS_SECRET_ACCESS_KEY: <aws-secret-access-key>\n AWS_REGION: <aws-region>\n
apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: argo-rollouts\nspec:\n template:\n spec:\n containers:\n - name: argo-rollouts\n env:\n - name: AWS_ACCESS_KEY_ID\n valueFrom:\n secretKeyRef:\n name: cloudwatch-secret\n key: AWS_ACCESS_KEY_ID\n - name: AWS_SECRET_ACCESS_KEY\n valueFrom:\n secretKeyRef:\n name: cloudwatch-secret\n key: AWS_SECRET_ACCESS_KEY\n - name: AWS_REGION\n valueFrom:\n secretKeyRef:\n name: cloudwatch-secret\n key: AWS_REGION\n
"},{"location":"analysis/cloudwatch/#configuration","title":"Configuration","text":" metricDataQueries - GetMetricData query: MetricDataQueryinterval - optional interval, e.g. 30m, default: 5m
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n metrics:\n - name: success-rate\n interval: 1m\n successCondition: \"len(result[0].Values) >= 5 and all(result[0].Values, {# <= 0.01})\"\n failureLimit: 3\n provider:\n cloudWatch:\n interval: 30m\n metricDataQueries:\n - {\n \"id\": \"rate\",\n \"expression\": \"errors / requests\"\n }\n - {\n \"id\": \"errors\",\n \"metricStat\": {\n \"metric\": {\n \"namespace\": \"app\",\n \"metricName\": \"errors\"\n },\n \"period\": 300,\n \"stat\": \"Sum\",\n \"unit\": \"Count\"\n },\n \"returnData\": false\n }\n - {\n \"id\": \"requests\",\n \"metricStat\": {\n \"metric\": {\n \"namespace\": \"app\",\n \"metricName\": \"requests\"\n },\n \"period\": 300,\n \"stat\": \"Sum\",\n \"unit\": \"Count\"\n },\n \"returnData\": false\n }\n
"},{"location":"analysis/cloudwatch/#debug","title":"debug","text":"You can confirm the results value in AnalysisRun.
$ kubectl get analysisrun/rollouts-name-xxxxxxxxxx-xx -o yaml\n(snip)\nstatus:\n metricResults:\n - count: 2\n failed: 1\n measurements:\n - finishedAt: \"2021-09-08T17:29:14Z\"\n phase: Failed\n startedAt: \"2021-09-08T17:29:13Z\"\n value: '[[0.0029476787030213707 0.006100422336931018 0.01020408163265306 0.007932573128408527\n 0.00589622641509434 0.006339144215530904]]'\n - finishedAt: \"2021-09-08T17:30:14Z\"\n phase: Successful\n startedAt: \"2021-09-08T17:30:14Z\"\n value: '[[0.004484304932735426 0.0058374494836102376 0.006736068585425597 0.008444444444444444\n 0.006859756097560976 0.0045385779122541605]]'\n name: success-rate\n phase: Running\n successful: 1\n phase: Running\n startedAt: \"2021-09-08T17:29:14Z\"\n
"},{"location":"analysis/datadog/","title":"Datadog Metrics","text":"A Datadog query can be used to obtain measurements for analysis.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: loq-error-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: error-rate\n interval: 5m\n successCondition: result <= 0.01\n failureLimit: 3\n provider:\n datadog:\n apiVersion: v2\n interval: 5m\n query: |\n sum:requests.error.rate{service:{{args.service-name}}}\n
The field apiVersion refers to the API version of Datadog (v1 or v2). Default value is v1 if this is omitted. See \"Working with Datadog API v2\" below for more information.
Datadog api and app tokens can be configured in a kubernetes secret in argo-rollouts namespace.
apiVersion: v1\nkind: Secret\nmetadata:\n name: datadog\ntype: Opaque\nstringData:\n address: https://api.datadoghq.com\n api-key: <datadog-api-key>\n app-key: <datadog-app-key>\n
apiVersion here is different from the apiVersion from the Datadog configuration above.
"},{"location":"analysis/datadog/#working-with-datadog-api-v2","title":"Working with Datadog API v2","text":"Important
While some basic v2 functionality is working in earlier versions, the new properties of formula and queries are only available as of v1.7
"},{"location":"analysis/datadog/#moving-to-v2","title":"Moving to v2","text":"If your old v1 was just a simple metric query - no formula as part of the query - then you can just move to v2 by updating the apiVersion in your existing Analysis Template, and everything should work.
If you have a formula, you will need to update how you configure your metric. Here is a before/after example of what your Analysis Template should look like:
Before:
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: log-error-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: error-rate\n interval: 30s\n successCondition: default(result, 0) < 10\n failureLimit: 3\n provider:\n datadog:\n apiVersion: v1\n interval: 5m\n query: \"moving_rollup(sum:requests.errors{service:{{args.service-name}}}.as_count(), 60, 'sum') / sum:requests{service:{{args.service-name}}}.as_count()\"\n
After:
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: loq-error-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: error-rate\n # Polling rate against the Datadog API\n interval: 30s\n successCondition: default(result, 0) < 10\n failureLimit: 3\n provider:\n datadog:\n apiVersion: v2\n # The window of time we are looking at in DD. Basically we will fetch data from (now-5m) to now.\n interval: 5m\n queries:\n a: sum:requests.errors{service:{{args.service-name}}}.as_count()\n b: sum:requests{service:{{args.service-name}}}.as_count()\n formula: \"moving_rollup(a, 60, 'sum') / b\"\n
"},{"location":"analysis/datadog/#examples","title":"Examples","text":"Simple v2 query with no formula
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: canary-container-restarts\nspec:\n args:\n # This is set in rollout using the valueFrom: podTemplateHashValue functionality\n - name: canary-hash\n - name: service-name\n - name: restarts.initial-delay\n value: \"60s\"\n - name: restarts.max-restarts\n value: \"4\"\n metrics:\n - name: kubernetes.containers.restarts\n initialDelay: \"{{ args.restarts.initial-delay }}\"\n interval: 15s\n failureCondition: default(result, 0) > {{ args.restarts.max-restarts }}\n failureLimit: 0\n provider:\n datadog:\n apiVersion: v2\n interval: 5m\n queries:\n # The key is arbitrary - you will use this key to refer to the query if you use a formula.\n q: \"max:kubernetes.containers.restarts{service-name:{{args.service-name}},rollouts_pod_template_hash:{{args.canary-hash}}}\"\n
"},{"location":"analysis/datadog/#tips","title":"Tips","text":""},{"location":"analysis/datadog/#datadog-results","title":"Datadog Results","text":"Datadog queries can return empty results if the query takes place during a time interval with no metrics. The Datadog provider will return a nil value yielding an error during the evaluation phase like:
invalid operation: < (mismatched types <nil> and float64)\n
However, empty query results yielding a nil value can be handled using the default() function. Here is a succeeding example using the default() function:
successCondition: default(result, 0) < 0.05\n
"},{"location":"analysis/datadog/#metric-aggregation-v2-only","title":"Metric aggregation (v2 only)","text":"By default, Datadog analysis run is configured to use last metric aggregator when querying Datadog v2 API. This value can be overriden by specifying a new aggregator value from a list of supported aggregators (avg,min,max,sum,last,percentile,mean,l2norm,area) for the V2 API (docs).
For example, using count-based distribution metric (count:metric{*}.as_count()) with values 1,9,3,7,5 in a given interval will make last aggregator return 5. To return a sum of all values (25), set aggregator: sum in Datadog provider block and use moving_rollup() function to aggregate values in the specified rollup interval. These functions can be combined in a formula to perform additional calculations:
...<snip>\n metrics:\n - name: error-percentage\n interval: 30s\n successCondition: default(result, 0) < 5\n failureLimit: 3\n provider:\n datadog:\n apiVersion: v2\n interval: 5m\n aggregator: sum # override default aggregator\n queries:\n a: count:requests.errors{service:my-service}.as_count()\n b: count:requests{service:my-service}.as_count()\n formula: \"moving_rollup(a, 300, 'sum') / moving_rollup(b, 300, 'sum') * 100\" # percentage of requests with errors\n
"},{"location":"analysis/datadog/#templates-and-helm","title":"Templates and Helm","text":"Helm and Argo Rollouts both try to parse things between {{ ... }} when rendering templates. If you use Helm to deliver your manifests, you will need to escape {{ args.whatever }}. Using the example above, here it is set up for Helm:
...<snip>\nmetrics:\n - name: kubernetes.containers.restarts\n initialDelay: \"{{ `{{ args.restarts.initial-delay }}` }}\"\n interval: 15s\n failureCondition: default(result, 0) > {{ `{{ args.restarts.max-restarts }}` }}\n failureLimit: 0\n provider:\n datadog:\n apiVersion: v2\n interval: 5m\n queries:\n q: \"{{ `max:kubernetes.containers.restarts{kube_app_name:{{args.kube_app_name}},rollouts_pod_template_hash:{{args.canary-hash}}}` }}\"\n
"},{"location":"analysis/datadog/#rate-limits","title":"Rate Limits","text":"For the v1 API, you ask for an increase on the api/v1/query route.
For the v2 API, the Ratelimit-Name you ask for an increase in is the query_scalar_public.
"},{"location":"analysis/graphite/","title":"Graphite Metrics","text":"A Graphite query can be used to obtain measurements for analysis.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: success-rate\n interval: 5m\n # Note that the Argo Rollouts Graphite metrics provider returns results as an array of float64s with 6 decimal places.\n successCondition: results[0] >= 90.000000\n failureLimit: 3\n provider:\n graphite:\n address: http://graphite.example.com:9090\n query: |\n target=summarize(\n asPercent(\n sumSeries(\n stats.timers.httpServerRequests.app.{{args.service-name}}.exception.*.method.*.outcome.{CLIENT_ERROR,INFORMATIONAL,REDIRECTION,SUCCESS}.status.*.uri.*.count\n ),\n sumSeries(\n stats.timers.httpServerRequests.app.{{args.service-name}}.exception.*.method.*.outcome.*.status.*.uri.*.count\n )\n ),\n '5min',\n 'avg'\n )\n
"},{"location":"analysis/influxdb/","title":"InfluxDB Metrics","text":"An InfluxDB query using Flux can be used to obtain measurements for analysis.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: error-rate\nspec:\n args:\n - name: application-name\n metrics:\n - name: error-rate\n # NOTE: To be consistent with the prometheus metrics provider InfluxDB query results are returned as an array.\n # In the example we're looking at index 0 of the returned array to obtain the value we're using for the success condition\n successCondition: result[0] <= 0.01\n provider:\n influxdb:\n profile: my-influxdb-secret # optional, defaults to 'influxdb'\n query: |\n from(bucket: \"app_istio\")\n |> range(start: -15m)\n |> filter(fn: (r) => r[\"destination_workload\"] == \"{{ args.application-name }}\")\n |> filter(fn: (r) => r[\"_measurement\"] == \"istio:istio_requests_errors_percentage:rate1m:5xx\")\n
An InfluxDB access profile can be configured using a Kubernetes secret in the argo-rollouts namespace. Alternate accounts can be used by creating more secrets of the same format and specifying which secret to use in the metric provider configuration using the profile field.
apiVersion: v1\nkind: Secret\nmetadata:\n name: influxdb\ntype: Opaque\nstringData:\n address: <infuxdb-url>\n authToken: <influxdb-auth-token>\n org: <influxdb-org>\n
"},{"location":"analysis/job/","title":"Job Metrics","text":"A Kubernetes Job can be used to run analysis. When a Job is used, the metric is considered successful if the Job completes and had an exit code of zero, otherwise it is failed.
metrics:\n - name: test\n provider:\n job:\n metadata:\n annotations:\n foo: bar # annotations defined here will be copied to the Job object\n labels:\n foo: bar # labels defined here will be copied to the Job object\n spec:\n backoffLimit: 1\n template:\n spec:\n containers:\n - name: test\n image: my-image:latest\n command: [my-test-script, my-service.default.svc.cluster.local]\n restartPolicy: Never\n
"},{"location":"analysis/kayenta/","title":"Kayenta","text":""},{"location":"analysis/kayenta/#kayenta-eg-mann-whitney-analysis","title":"Kayenta (e.g. Mann-Whitney Analysis)","text":"Analysis can also be done as part of an Experiment.
This example starts both a canary and baseline ReplicaSet. The ReplicaSets run for 1 hour, then scale down to zero. Call out to Kayenta to perform Mann-Whintney analysis against the two pods. Demonstrates ability to start a short-lived experiment and an asynchronous analysis.
This example demonstrates:
- The ability to start an Experiment as part of rollout steps, which launches multiple ReplicaSets (e.g. baseline & canary)
- The ability to reference and supply pod-template-hash to an AnalysisRun
- Kayenta metrics
Rollout apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\n labels:\n app: guestbook\nspec:\n strategy:\n canary:\n steps:\n - experiment:\n duration: 1h\n templates:\n - name: baseline\n specRef: stable\n - name: canary\n specRef: canary\n analyses:\n - templateName: mann-whitney\n args:\n - name: stable-hash\n valueFrom:\n podTemplateHashValue: Stable\n - name: canary-hash\n valueFrom:\n podTemplateHashValue: Latest\n
AnalysisTemplate apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: mann-whitney\nspec:\n args:\n - name: start-time\n - name: end-time\n - name: stable-hash\n - name: canary-hash\n metrics:\n - name: mann-whitney\n provider:\n kayenta:\n address: https://kayenta.example.com\n application: guestbook\n canaryConfigName: my-test\n thresholds:\n pass: 90\n marginal: 75\n scopes:\n - name: default\n controlScope:\n scope: app=guestbook and rollouts-pod-template-hash={{args.stable-hash}}\n step: 60\n start: \"{{args.start-time}}\"\n end: \"{{args.end-time}}\"\n experimentScope:\n scope: app=guestbook and rollouts-pod-template-hash={{args.canary-hash}}\n step: 60\n start: \"{{args.start-time}}\"\n end: \"{{args.end-time}}\"\n
Experiment # This is the resulting experiment that is produced by the step\napiVersion: argoproj.io/v1alpha1\nkind: Experiment\nname:\n name: guestbook-6c54544bf9-0\nspec:\n duration: 1h\n templates:\n - name: baseline\n replicas: 1\n spec:\n containers:\n - name: guestbook\n image: guestbook:v1\n - name: canary\n replicas: 1\n spec:\n containers:\n - name: guestbook\n image: guestbook:v2\n analysis:\n templateName: mann-whitney\n args:\n - name: start-time\n value: \"{{experiment.availableAt}}\"\n - name: end-time\n value: \"{{experiment.finishedAt}}\"\n
In order to perform multiple kayenta runs over some time duration, the interval and count fields can be supplied. When the start and end fields are omitted from the kayenta scopes, the values will be implicitly decided as:
start = if lookback: true start of analysis, otherwise current time - intervalend = current time
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: mann-whitney\nspec:\n args:\n - name: stable-hash\n - name: canary-hash\n metrics:\n - name: mann-whitney\n provider:\n kayenta:\n address: https://kayenta.intuit.com\n application: guestbook\n canaryConfigName: my-test\n interval: 3600\n count: 3\n # loopback will cause start time value to be equal to start of analysis\n # lookback: true\n thresholds:\n pass: 90\n marginal: 75\n scopes:\n - name: default\n controlScope:\n scope: app=guestbook and rollouts-pod-template-hash={{args.stable-hash}}\n step: 60\n experimentScope:\n scope: app=guestbook and rollouts-pod-template-hash={{args.canary-hash}}\n step: 60\n
"},{"location":"analysis/newrelic/","title":"NewRelic Metrics","text":"Important
Available since v0.10.0
A New Relic query using NRQL can be used to obtain measurements for analysis.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: application-name\n metrics:\n - name: success-rate\n successCondition: result.successRate >= 0.95\n provider:\n newRelic:\n profile: my-newrelic-secret # optional, defaults to 'newrelic'\n query: |\n FROM Transaction SELECT percentage(count(*), WHERE httpResponseCode != 500) as successRate where appName = '{{ args.application-name }}'\n
The result evaluated for the condition will always be map or list of maps. The name will follow the pattern of either function or function.field, e.g. SELECT average(duration) from Transaction will yield average.duration. In this case the field result cannot be accessed with dot notation and instead should be accessed like result['average.duration']. Query results can be renamed using the NRQL clause AS as seen above.
A New Relic access profile can be configured using a Kubernetes secret in the argo-rollouts namespace. Alternate accounts can be used by creating more secrets of the same format and specifying which secret to use in the metric provider configuration using the profile field.
apiVersion: v1\nkind: Secret\nmetadata:\n name: newrelic\ntype: Opaque\nstringData:\n personal-api-key: <newrelic-personal-api-key>\n account-id: <newrelic-account-id>\n region: \"us\" # optional, defaults to \"us\" if not set. Only set to \"eu\" if you use EU New Relic\n
To use the New Relic metric provider from behind a proxy, provide a base-url-rest key pointing to the base URL of the New Relic REST API for your proxy, and a base-url-nerdgraph key pointing to the base URL for NerdGraph for your proxy:
apiVersion: v1\nkind: Secret\nmetadata:\n name: newrelic\ntype: Opaque\nstringData:\n personal-api-key: <newrelic-personal-api-key>\n account-id: <newrelic-account-id>\n region: \"us\" # optional, defaults to \"us\" if not set. Only set to \"eu\" if you use EU New Relic\n base-url-rest: <your-base-url>\n base-url-nerdgraph: <your-base-url>\n
"},{"location":"analysis/plugins/","title":"Metric Plugins","text":"Important
Available since v1.5 - Status: Alpha
Argo Rollouts supports getting analysis metrics via 3rd party plugin system. This allows users to extend the capabilities of Rollouts to support metric providers that are not natively supported. Rollout's uses a plugin library called go-plugin to do this. You can find a sample plugin here: rollouts-plugin-metric-sample-prometheus
"},{"location":"analysis/plugins/#using-a-metric-plugin","title":"Using a Metric Plugin","text":"There are two methods of installing and using an argo rollouts plugin. The first method is to mount up the plugin executable into the rollouts controller container. The second method is to use a HTTP(S) server to host the plugin executable.
"},{"location":"analysis/plugins/#mounting-the-plugin-executable-into-the-rollouts-controller-container","title":"Mounting the plugin executable into the rollouts controller container","text":"There are a few ways to mount the plugin executable into the rollouts controller container. Some of these will depend on your particular infrastructure. Here are a few methods:
- Using an init container to download the plugin executable
- Using a Kubernetes volume mount with a shared volume such as NFS, EBS, etc.
- Building the plugin into the rollouts controller container
Then you can use the configmap to point to the plugin executable file location. Example:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-config\ndata:\n metricProviderPlugins: |-\n - name: \"argoproj-labs/sample-prometheus\" # name of the plugin, it must match the name required by the plugin so it can find it's configuration\n location: \"file://./my-custom-plugin\" # supports http(s):// urls and file://\n
"},{"location":"analysis/plugins/#using-a-https-server-to-host-the-plugin-executable","title":"Using a HTTP(S) server to host the plugin executable","text":"Argo Rollouts supports downloading the plugin executable from a HTTP(S) server. To use this method, you will need to configure the controller via the argo-rollouts-config configmap and set pluginLocation to a http(s) url. Example:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-config\ndata:\n metricProviderPlugins: |-\n - name: \"argoproj-labs/sample-prometheus\" # name of the plugin, it must match the name required by the plugin so it can find it's configuration\n location: \"https://github.com/argoproj-labs/rollouts-plugin-metric-sample-prometheus/releases/download/v0.0.4/metric-plugin-linux-amd64\" # supports http(s):// urls and file://\n sha256: \"dac10cbf57633c9832a17f8c27d2ca34aa97dd3d\" #optional sha256 checksum of the plugin executable\n
"},{"location":"analysis/plugins/#some-words-of-caution","title":"Some words of caution","text":"Depending on which method you use to install and the plugin, there are some things to be aware of. The rollouts controller will not start if it can not download or find the plugin executable. This means that if you are using a method of installation that requires a download of the plugin and the server hosting the plugin for some reason is not available and the rollouts controllers pod got deleted while the server was down or is coming up for the first time, it will not be able to start until the server hosting the plugin is available again.
Argo Rollouts will download the plugin at startup only once but if the pod is deleted it will need to download the plugin again on next startup. Running Argo Rollouts in HA mode can help a little with this situation because each pod will download the plugin at startup. So if a single pod gets deleted during a server outage, the other pods will still be able to take over because there will already be a plugin executable available to it. It is the responsibility of the Argo Rollouts administrator to define the plugin installation method considering the risks of each approach.
"},{"location":"analysis/plugins/#list-of-available-plugins-alphabetical-order","title":"List of Available Plugins (alphabetical order)","text":""},{"location":"analysis/plugins/#add-your-plugin-here","title":"Add Your Plugin Here","text":" - If you have created a plugin, please submit a PR to add it to this list.
"},{"location":"analysis/plugins/#rollouts-plugin-metric-sample-prometheus","title":"rollouts-plugin-metric-sample-prometheus","text":" - This is just a sample plugin that can be used as a starting point for creating your own plugin. It is not meant to be used in production. It is based on the built-in prometheus provider.
"},{"location":"analysis/prometheus/","title":"Prometheus Metrics","text":"A Prometheus query can be used to obtain measurements for analysis.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: success-rate\n interval: 5m\n # NOTE: prometheus queries return results in the form of a vector.\n # So it is common to access the index 0 of the returned array to obtain the value\n successCondition: result[0] >= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n # timeout is expressed in seconds\n timeout: 40\n headers:\n - key: X-Scope-OrgID\n value: tenant_a\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
The example shows Istio metrics, but you can use any kind of metric available to your prometheus instance. We suggest you validate your PromQL expression using the Prometheus GUI first.
See the Analysis Overview page for more details on the available options.
"},{"location":"analysis/prometheus/#authorization","title":"Authorization","text":""},{"location":"analysis/prometheus/#utilizing-amazon-managed-prometheus","title":"Utilizing Amazon Managed Prometheus","text":"Amazon Managed Prometheus can be used as the prometheus data source for analysis. In order to do this the namespace where your analysis is running will have to have the appropriate IRSA attached to allow for prometheus queries. Once you ensure the proper permissions are in place to access AMP, you can use an AMP workspace url in your provider block and add a SigV4 config for Sigv4 signing:
provider:\n prometheus:\n address: https://aps-workspaces.$REGION.amazonaws.com/workspaces/$WORKSPACEID\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n authentication:\n sigv4:\n region: $REGION\n profile: $PROFILE\n roleArn: $ROLEARN\n
"},{"location":"analysis/prometheus/#with-oauth2","title":"With OAuth2","text":"You can setup an OAuth2 client credential flow using the following values:
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n # from secret\n - name: oauthSecret # This is the OAuth2 shared secret\n valueFrom:\n secretKeyRef:\n name: oauth-secret\n key: secret\n metrics:\n - name: success-rate\n interval: 5m\n # NOTE: prometheus queries return results in the form of a vector.\n # So it is common to access the index 0 of the returned array to obtain the value\n successCondition: result[0] >= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n # timeout is expressed in seconds\n timeout: 40\n authentication:\n oauth2:\n tokenUrl: https://my-oauth2-provider/token\n clientId: my-cliend-id\n clientSecret: \"{{ args.oauthSecret }}\"\n scopes: [\n \"my-oauth2-scope\"\n ]\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
The AnalysisRun will first get an access token using that information, and provide it as an Authorization: Bearer header for the metric provider call.
"},{"location":"analysis/prometheus/#additional-metadata","title":"Additional Metadata","text":"Any additional metadata from the Prometheus controller, like the resolved queries after substituting the template's arguments, etc. will appear under the Metadata map in the MetricsResult object of AnalysisRun.
"},{"location":"analysis/prometheus/#skip-tls-verification","title":"Skip TLS verification","text":"You can skip the TLS verification of the prometheus host provided by setting the options insecure: true.
provider:\n prometheus:\n address: https://prometheus.example.com\n insecure: true\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
"},{"location":"analysis/skywalking/","title":"Apache SkyWalking Metrics","text":"Important
Available since v1.5.0
A SkyWalking query using GraphQL can be used to obtain measurements for analysis.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: apdex\nspec:\n args:\n - name: service-name\n metrics:\n - name: apdex\n interval: 5m\n successCondition: \"all(result.service_apdex.values.values, {asFloat(.value) >= 9900})\"\n failureLimit: 3\n provider:\n skywalking:\n interval: 3m\n address: http://skywalking-oap.istio-system:12800\n query: |\n query queryData($duration: Duration!) {\n service_apdex: readMetricsValues(\n condition: { name: \"service_apdex\", entity: { scope: Service, serviceName: \"{{ args.service-name }}\", normal: true } },\n duration: $duration) {\n label values { values { value } }\n }\n }\n
The result evaluated for the query depends on the specific GraphQL you give, you can try to run the GraphQL query first and inspect the output format, then compose the condition.
"},{"location":"analysis/wavefront/","title":"Wavefront Metrics","text":"A Wavefront query can be used to obtain measurements for analysis.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: success-rate\n interval: 5m\n successCondition: result >= 0.95\n failureLimit: 3\n provider:\n wavefront:\n address: example.wavefront.com\n query: |\n sum(rate(\n 5m, ts(\"istio.requestcount.count\", response_code!=500 and destination_service=\"{{args.service-name}}\"\n ))) /\n sum(rate(\n 5m, ts(\"istio.requestcount.count\", reporter=client and destination_service=\"{{args.service-name}}\"\n )))\n
Wavefront api tokens can be configured in a kubernetes secret in argo-rollouts namespace.
apiVersion: v1\nkind: Secret\nmetadata:\n name: wavefront-api-tokens\ntype: Opaque\nstringData:\n example1.wavefront.com: <token1>\n example2.wavefront.com: <token2>\n
"},{"location":"analysis/web/","title":"Web Metrics","text":"An HTTP request can be performed against some external service to obtain the measurement. This example makes a HTTP GET request to some URL. The webhook response must return JSON content. The result of the optional jsonPath expression will be assigned to the result variable that can be referenced in the successCondition and failureCondition expressions. If omitted, will use the entire body of the as the result variable.
metrics:\n - name: webmetric\n successCondition: result == true\n provider:\n web:\n url: \"http://my-server.com/api/v1/measurement?service={{ args.service-name }}\"\n timeoutSeconds: 20 # defaults to 10 seconds\n headers:\n - key: Authorization\n value: \"Bearer {{ args.api-token }}\"\n jsonPath: \"{$.data.ok}\"\n
In the following example, given the payload, the measurement will be Successful if the data.ok field was true, and the data.successPercent was greater than 0.90
{\n \"data\": {\n \"ok\": true,\n \"successPercent\": 0.95\n }\n}\n
metrics:\n - name: webmetric\n successCondition: \"result.ok && result.successPercent >= 0.90\"\n provider:\n web:\n url: \"http://my-server.com/api/v1/measurement?service={{ args.service-name }}\"\n headers:\n - key: Authorization\n value: \"Bearer {{ args.api-token }}\"\n jsonPath: \"{$.data}\"\n
NOTE: if the result is a string, two convenience functions asInt and asFloat are provided to convert a result value to a numeric type so that mathematical comparison operators can be used (e.g. >, <, >=, <=).
"},{"location":"analysis/web/#optional-web-methods","title":"Optional web methods","text":"It is possible to use a POST or PUT requests, by specifying the method and either body or jsonBody fields
metrics:\n - name: webmetric\n successCondition: result == true\n provider:\n web:\n method: POST # valid values are GET|POST|PUT, defaults to GET\n url: \"http://my-server.com/api/v1/measurement?service={{ args.service-name }}\"\n timeoutSeconds: 20 # defaults to 10 seconds\n headers:\n - key: Authorization\n value: \"Bearer {{ args.api-token }}\"\n - key: Content-Type # if body is a json, it is recommended to set the Content-Type\n value: \"application/json\"\n body: \"string value\"\n jsonPath: \"{$.data.ok}\"\n
!!! tip In order to send in JSON, you can use jsonBody and Content-Type will be automatically set as json. Setting a body or jsonBody field for a GET request will result in an error. Set either body or jsonBody and setting both will result in an error. metrics:\n - name: webmetric\n successCondition: result == true\n provider:\n web:\n method: POST # valid values are GET|POST|PUT, defaults to GET\n url: \"http://my-server.com/api/v1/measurement?service={{ args.service-name }}\"\n timeoutSeconds: 20 # defaults to 10 seconds\n headers:\n - key: Authorization\n value: \"Bearer {{ args.api-token }}\"\n - key: Content-Type # if body is a json, it is recommended to set the Content-Type\n value: \"application/json\"\n jsonBody: # If using jsonBody Content-Type header will be automatically set to json\n key1: value_1\n key2:\n nestedObj: nested value\n key3: \"{{ args.service-name }}\"\n jsonPath: \"{$.data.ok}\"\n
"},{"location":"analysis/web/#skip-tls-verification","title":"Skip TLS verification","text":"You can skip the TLS verification of the web host provided by setting the options insecure: true.
metrics:\n - name: webmetric\n successCondition: \"result.ok && result.successPercent >= 0.90\"\n provider:\n web:\n url: \"https://my-server.com/api/v1/measurement?service={{ args.service-name }}\"\n insecure: true\n headers:\n - key: Authorization\n value: \"Bearer {{ args.api-token }}\"\n jsonPath: \"{$.data}\"\n
"},{"location":"analysis/web/#authorization","title":"Authorization","text":""},{"location":"analysis/web/#with-oauth2","title":"With OAuth2","text":"You can setup an OAuth2 client credential flow using the following values:
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n # from secret\n - name: oauthSecret # This is the OAuth2 shared secret\n valueFrom:\n secretKeyRef:\n name: oauth-secret\n key: secret\n metrics:\n - name: webmetric\n successCondition: result == true\n provider:\n web:\n url: \"http://my-server.com/api/v1/measurement?service={{ args.service-name }}\"\n timeoutSeconds: 20 # defaults to 10 seconds\n authentication:\n oauth2:\n tokenUrl: https://my-oauth2-provider/token\n clientId: my-cliend-id\n clientSecret: \"{{ args.oauthSecret }}\"\n scopes: [\n \"my-oauth2-scope\"\n ]\n headers:\n - key: Content-Type # if body is a json, it is recommended to set the Content-Type\n value: \"application/json\"\n jsonPath: \"{$.data.ok}\"\n
In that case, no need to provide specifically the Authentication header. The AnalysisRun will first get an access token using that information, and provide it as an Authorization: Bearer header for the metric provider call.
"},{"location":"features/analysis/","title":"Analysis & Progressive Delivery","text":"Argo Rollouts provides several ways to perform analysis to drive progressive delivery. This document describes how to achieve various forms of progressive delivery, varying the point in time analysis is performed, its frequency, and occurrence.
"},{"location":"features/analysis/#custom-resource-definitions","title":"Custom Resource Definitions","text":"CRD Description Rollout A Rollout acts as a drop-in replacement for a Deployment resource. It provides additional blueGreen and canary update strategies. These strategies can create AnalysisRuns and Experiments during the update, which will progress the update, or abort it. AnalysisTemplate An AnalysisTemplate is a template spec which defines how to perform a canary analysis, such as the metrics which it should perform, its frequency, and the values which are considered successful or failed. AnalysisTemplates may be parameterized with inputs values. ClusterAnalysisTemplate A ClusterAnalysisTemplate is like an AnalysisTemplate, but it is not limited to its namespace. It can be used by any Rollout throughout the cluster. AnalysisRun An AnalysisRun is an instantiation of an AnalysisTemplate. AnalysisRuns are like Jobs in that they eventually complete. Completed runs are considered Successful, Failed, or Inconclusive, and the result of the run affect if the Rollout's update will continue, abort, or pause, respectively. Experiment An Experiment is limited run of one or more ReplicaSets for the purposes of analysis. Experiments typically run for a pre-determined duration, but can also run indefinitely until stopped. Experiments may reference an AnalysisTemplate to run during or after the experiment. The canonical use case for an Experiment is to start a baseline and canary deployment in parallel, and compare the metrics produced by the baseline and canary pods for an equal comparison."},{"location":"features/analysis/#background-analysis","title":"Background Analysis","text":"Analysis can be run in the background -- while the canary is progressing through its rollout steps.
The following example gradually increments the canary weight by 20% every 10 minutes until it reaches 100%. In the background, an AnalysisRun is started based on the AnalysisTemplate named success-rate. The success-rate template queries a prometheus server, measuring the HTTP success rates at 5 minute intervals/samples. It has no end time, and continues until stopped or failed. If the metric is measured to be less than 95%, and there are three such measurements, the analysis is considered Failed. The failed analysis causes the Rollout to abort, setting the canary weight back to zero, and the Rollout would be considered in a Degraded. Otherwise, if the rollout completes all of its canary steps, the rollout is considered successful and the analysis run is stopped by the controller.
This example highlights:
- Background analysis style of progressive delivery
- Using a Prometheus query to perform a measurement
- The ability to parameterize the analysis
- Delay starting the analysis run until step 3 (Set Weight 40%)
Rollout apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\nspec:\n...\n strategy:\n canary:\n analysis:\n templates:\n - templateName: success-rate\n startingStep: 2 # delay starting analysis run until setWeight: 40%\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n steps:\n - setWeight: 20\n - pause: {duration: 10m}\n - setWeight: 40\n - pause: {duration: 10m}\n - setWeight: 60\n - pause: {duration: 10m}\n - setWeight: 80\n - pause: {duration: 10m}\n
AnalysisTemplate apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: success-rate\n interval: 5m\n # NOTE: prometheus queries return results in the form of a vector.\n # So it is common to access the index 0 of the returned array to obtain the value\n successCondition: result[0] >= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
"},{"location":"features/analysis/#inline-analysis","title":"Inline Analysis","text":"Analysis can also be performed as a rollout step as an inline \"analysis\" step. When analysis is performed \"inlined,\" an AnalysisRun is started when the step is reached, and blocks the rollout until the run is completed. The success or failure of the analysis run decides if the rollout will proceed to the next step, or abort the rollout completely.
This example sets the canary weight to 20%, pauses for 5 minutes, then runs an analysis. If the analysis was successful, continues with rollout, otherwise aborts.
This example demonstrates:
- The ability to invoke an analysis in-line as part of steps
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\nspec:\n...\n strategy:\n canary:\n steps:\n - setWeight: 20\n - pause: {duration: 5m}\n - analysis:\n templates:\n - templateName: success-rate\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n
In this example, the AnalysisTemplate is identical to the background analysis example, but since no interval is specified, the analysis will perform a single measurement and complete.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n - name: prometheus-port\n value: 9090\n metrics:\n - name: success-rate\n successCondition: result[0] >= 0.95\n provider:\n prometheus:\n address: \"http://prometheus.example.com:{{args.prometheus-port}}\"\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
Multiple measurements can be performed over a longer duration period, by specifying the count and interval fields:
metrics:\n - name: success-rate\n successCondition: result[0] >= 0.95\n interval: 60s\n count: 5\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: ...\n
"},{"location":"features/analysis/#clusteranalysistemplates","title":"ClusterAnalysisTemplates","text":"Important
Available since v0.9.0
A Rollout can reference a Cluster scoped AnalysisTemplate called a ClusterAnalysisTemplate. This can be useful when you want to share an AnalysisTemplate across multiple Rollouts; in different namespaces, and avoid duplicating the same template in every namespace. Use the field clusterScope: true to reference a ClusterAnalysisTemplate instead of an AnalysisTemplate.
Rollout apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\nspec:\n...\n strategy:\n canary:\n steps:\n - setWeight: 20\n - pause: {duration: 5m}\n - analysis:\n templates:\n - templateName: success-rate\n clusterScope: true\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n
ClusterAnalysisTemplate apiVersion: argoproj.io/v1alpha1\nkind: ClusterAnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n - name: prometheus-port\n value: 9090\n metrics:\n - name: success-rate\n successCondition: result[0] >= 0.95\n provider:\n prometheus:\n address: \"http://prometheus.example.com:{{args.prometheus-port}}\"\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
Note
The resulting AnalysisRun will still run in the namespace of the Rollout
"},{"location":"features/analysis/#analysis-with-multiple-templates","title":"Analysis with Multiple Templates","text":"A Rollout can reference multiple AnalysisTemplates when constructing an AnalysisRun. This allows users to compose analysis from multiple AnalysisTemplates. If multiple templates are referenced, then the controller will merge the templates together. The controller combines the metrics and args fields of all the templates.
Rollout apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\nspec:\n...\n strategy:\n canary:\n analysis:\n templates:\n - templateName: success-rate\n - templateName: error-rate\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n
AnalysisTemplate apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: success-rate\n interval: 5m\n successCondition: result[0] >= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n---\napiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: error-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: error-rate\n interval: 5m\n successCondition: result[0] <= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code=~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
AnalysisRun # NOTE: Generated AnalysisRun from the multiple templates\napiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\nmetadata:\n name: guestbook-CurrentPodHash-multiple-templates\nspec:\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n metrics:\n - name: success-rate\n interval: 5m\n successCondition: result[0] >= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n - name: error-rate\n interval: 5m\n successCondition: result[0] <= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code=~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
Note
The controller will error when merging the templates if:
- Multiple metrics in the templates have the same name
- Two arguments with the same name have different default values no matter the argument value in Rollout
"},{"location":"features/analysis/#analysis-template-referencing-other-analysis-templates","title":"Analysis Template referencing other Analysis Templates","text":"AnalysisTemplates and ClusterAnalysisTemplates may reference other templates.
They can be combined with other metrics:
AnalysisTemplate apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: error-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: error-rate\n interval: 5m\n successCondition: result[0] <= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code=~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n---\napiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: rates\nspec:\n args:\n - name: service-name\n metrics:\n - name: success-rate\n interval: 5m\n successCondition: result[0] >= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n templates:\n - templateName: error-rate\n clusterScope: false\n
Or without additional metrics:
AnalysisTemplate apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: success-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: success-rate\n interval: 5m\n successCondition: result[0] >= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n---\napiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: error-rate\nspec:\n args:\n - name: service-name\n metrics:\n - name: error-rate\n interval: 5m\n successCondition: result[0] <= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code=~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n---\napiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: rates\nspec:\n args:\n - name: service-name\n templates:\n - templateName: success-rate\n clusterScope: false\n - templateName: error-rate\n clusterScope: false\n
The result in the AnalysisRun will have the aggregation of metrics of each template:
AnalysisRun # NOTE: Generated AnalysisRun from a single template referencing several templates\napiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\nmetadata:\n name: guestbook-CurrentPodHash-templates-in-template\nspec:\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n metrics:\n - name: success-rate\n interval: 5m\n successCondition: result[0] >= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code!~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n - name: error-rate\n interval: 5m\n successCondition: result[0] <= 0.95\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code=~\"5.*\"}[5m]\n )) /\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\"}[5m]\n ))\n
Note
The same limitations as for the multiple templates feature apply. The controller will error when merging the templates if:
- Multiple metrics in the templates have the same name
- Two arguments with the same name have different default values no matter the argument value in Rollout
However, if the same AnalysisTemplate is referenced several times along the chain of references, the controller will only keep it once and discard the other references.
"},{"location":"features/analysis/#analysis-template-arguments","title":"Analysis Template Arguments","text":"AnalysisTemplates may declare a set of arguments that can be passed by Rollouts. The args can then be used as in metrics configuration and are resolved at the time the AnalysisRun is created. Argument placeholders are defined as {{ args.<name> }}.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nmetadata:\n name: args-example\nspec:\n args:\n # required in Rollout due to no default value\n - name: service-name\n - name: stable-hash\n - name: latest-hash\n # optional in Rollout given the default value\n - name: api-url\n value: http://example/measure\n # from secret\n - name: api-token\n valueFrom:\n secretKeyRef:\n name: token-secret\n key: apiToken\n metrics:\n - name: webmetric\n successCondition: result == 'true'\n provider:\n web:\n # placeholders are resolved when an AnalysisRun is created\n url: \"{{ args.api-url }}?service={{ args.service-name }}\"\n headers:\n - key: Authorization\n value: \"Bearer {{ args.api-token }}\"\n jsonPath: \"{$.results.ok}\"\n
Analysis arguments defined in a Rollout are merged with the args from the AnalysisTemplate when the AnalysisRun is created.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\nspec:\n...\n strategy:\n canary:\n analysis:\n templates:\n - templateName: args-example\n args:\n # required value\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n # override default value\n - name: api-url\n value: http://other-api\n # pod template hash from the stable ReplicaSet\n - name: stable-hash\n valueFrom:\n podTemplateHashValue: Stable\n # pod template hash from the latest ReplicaSet\n - name: latest-hash\n valueFrom:\n podTemplateHashValue: Latest\n
Analysis arguments also support valueFrom for reading metadata fields and passing them as arguments to AnalysisTemplate. An example would be to reference metadata labels like env and region and passing them along to AnalysisTemplate. apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\n labels:\n appType: demo-app\n buildType: nginx-app\n ...\n env: dev\n region: us-west-2\nspec:\n...\n strategy:\n canary:\n analysis:\n templates:\n - templateName: args-example\n args:\n ...\n - name: env\n valueFrom:\n fieldRef:\n fieldPath: metadata.labels['env']\n # region where this app is deployed\n - name: region\n valueFrom:\n fieldRef:\n fieldPath: metadata.labels['region']\n
Important
Available since v1.2
Analysis arguments also support valueFrom for reading any field from Rollout status and passing them as arguments to AnalysisTemplate. Following example references Rollout status field like aws canaryTargetGroup name and passing them along to AnalysisTemplate
from the Rollout status
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\n labels:\n appType: demo-app\n buildType: nginx-app\n ...\n env: dev\n region: us-west-2\nspec:\n...\n strategy:\n canary:\n analysis:\n templates:\n - templateName: args-example\n args:\n ...\n - name: canary-targetgroup-name\n valueFrom:\n fieldRef:\n fieldPath: status.alb.canaryTargetGroup.name\n
"},{"location":"features/analysis/#bluegreen-pre-promotion-analysis","title":"BlueGreen Pre Promotion Analysis","text":"A Rollout using the BlueGreen strategy can launch an AnalysisRun before it switches traffic to the new version using pre-promotion. This can be used to block the Service selector switch until the AnalysisRun finishes successfully. The success or failure of the AnalysisRun decides if the Rollout switches traffic, or abort the Rollout completely.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\nspec:\n...\n strategy:\n blueGreen:\n activeService: active-svc\n previewService: preview-svc\n prePromotionAnalysis:\n templates:\n - templateName: smoke-tests\n args:\n - name: service-name\n value: preview-svc.default.svc.cluster.local\n
In this example, the Rollout creates a pre-promotion AnalysisRun once the new ReplicaSet is fully available. The Rollout will not switch traffic to the new version until the analysis run finishes successfully.
Note: if theautoPromotionSeconds field is specified and the Rollout has waited auto promotion seconds amount of time, the Rollout marks the AnalysisRun successful and switches the traffic to a new version automatically. If the AnalysisRun completes before then, the Rollout will not create another AnalysisRun and wait out the rest of the autoPromotionSeconds.
"},{"location":"features/analysis/#bluegreen-post-promotion-analysis","title":"BlueGreen Post Promotion Analysis","text":"A Rollout using a BlueGreen strategy can launch an analysis run after the traffic switch to the new version using post-promotion analysis. If post-promotion Analysis fails or errors, the Rollout enters an aborted state and switches traffic back to the previous stable Replicaset. When post-analysis is Successful, the Rollout is considered fully promoted and the new ReplicaSet will be marked as stable. The old ReplicaSet will then be scaled down according to scaleDownDelaySeconds (default 30 seconds).
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\nspec:\n...\n strategy:\n blueGreen:\n activeService: active-svc\n previewService: preview-svc\n scaleDownDelaySeconds: 600 # 10 minutes\n postPromotionAnalysis:\n templates:\n - templateName: smoke-tests\n args:\n - name: service-name\n value: preview-svc.default.svc.cluster.local\n
"},{"location":"features/analysis/#failure-conditions-and-failure-limit","title":"Failure Conditions and Failure Limit","text":"failureCondition can be used to cause an analysis run to fail. failureLimit is the maximum number of failed run an analysis is allowed. The following example continually polls the defined Prometheus server to get the total number of errors(i.e., HTTP response code >= 500) every 5 minutes, causing the measurement to fail if ten or more errors are encountered. The entire analysis run is considered as Failed after three failed measurements.
metrics:\n - name: total-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code=~\"5.*\"}[5m]\n ))\n
"},{"location":"features/analysis/#dry-run-mode","title":"Dry-Run Mode","text":"Important
Available since v1.2
dryRun can be used on a metric to control whether or not to evaluate that metric in a dry-run mode. A metric running in the dry-run mode won't impact the final state of the rollout or experiment even if it fails or the evaluation comes out as inconclusive.
The following example queries prometheus every 5 minutes to get the total number of 4XX and 5XX errors, and even if the evaluation of the metric to monitor the 5XX error-rate fail, the analysis run will pass.
dryRun:\n - metricName: total-5xx-errors\n metrics:\n - name: total-5xx-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code~\"5.*\"}[5m]\n ))\n - name: total-4xx-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code~\"4.*\"}[5m]\n ))\n
RegEx matches are also supported. .* can be used to make all the metrics run in the dry-run mode. In the following example, even if one or both metrics fail, the analysis run will pass.
dryRun:\n - metricName: .*\n metrics:\n - name: total-5xx-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code~\"5.*\"}[5m]\n ))\n - name: total-4xx-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code~\"4.*\"}[5m]\n ))\n
"},{"location":"features/analysis/#dry-run-summary","title":"Dry-Run Summary","text":"If one or more metrics are running in the dry-run mode, the summary of the dry-run results gets appended to the analysis run message. Assuming that the total-4xx-errors metric fails in the above example but, the total-5xx-errors succeeds, the final dry-run summary will look like this.
Message: Run Terminated\nRun Summary:\n ...\nDry Run Summary: \n Count: 2\n Successful: 1\n Failed: 1\nMetric Results:\n...\n
"},{"location":"features/analysis/#dry-run-rollouts","title":"Dry-Run Rollouts","text":"If a rollout wants to dry run its analysis, it simply needs to specify the dryRun field to its analysis stanza. In the following example, all the metrics from random-fail and always-pass get merged and executed in the dry-run mode.
kind: Rollout\nspec:\n...\n steps:\n - analysis:\n templates:\n - templateName: random-fail\n - templateName: always-pass\n dryRun:\n - metricName: .*\n
"},{"location":"features/analysis/#dry-run-experiments","title":"Dry-Run Experiments","text":"If an experiment wants to dry run its analysis, it simply needs to specify the dryRun field under its specs. In the following example, all the metrics from analyze-job matching the RegEx rule test.* will be executed in the dry-run mode.
kind: Experiment\nspec:\n templates:\n - name: baseline\n selector:\n matchLabels:\n app: rollouts-demo\n template:\n metadata:\n labels:\n app: rollouts-demo\n spec:\n containers:\n - name: rollouts-demo\n image: argoproj/rollouts-demo:blue\n analyses:\n - name: analyze-job\n templateName: analyze-job\n dryRun:\n - metricName: test.*\n
"},{"location":"features/analysis/#measurements-retention","title":"Measurements Retention","text":"Important
Available since v1.2
measurementRetention can be used to retain other than the latest ten results for the metrics running in any mode (dry/non-dry). Setting this option to 0 would disable it and, the controller will revert to the existing behavior of retaining the latest ten measurements.
The following example queries Prometheus every 5 minutes to get the total number of 4XX and 5XX errors and retains the latest twenty measurements for the 5XX metric run results instead of the default ten.
measurementRetention:\n - metricName: total-5xx-errors\n limit: 20\n metrics:\n - name: total-5xx-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code~\"5.*\"}[5m]\n ))\n - name: total-4xx-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code~\"4.*\"}[5m]\n ))\n
RegEx matches are also supported. .* can be used to apply the same retention rule to all the metrics. In the following example, the controller will retain the latest twenty run results for all the metrics instead of the default ten results.
measurementRetention:\n - metricName: .*\n limit: 20\n metrics:\n - name: total-5xx-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code~\"5.*\"}[5m]\n ))\n - name: total-4xx-errors\n interval: 5m\n failureCondition: result[0] >= 10\n failureLimit: 3\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: |\n sum(irate(\n istio_requests_total{reporter=\"source\",destination_service=~\"{{args.service-name}}\",response_code~\"4.*\"}[5m]\n ))\n
"},{"location":"features/analysis/#measurements-retention-for-rollouts-analysis","title":"Measurements Retention for Rollouts Analysis","text":"If a rollout wants to retain more results of its analysis metrics, it simply needs to specify the measurementRetention field to its analysis stanza. In the following example, all the metrics from random-fail and always-pass get merged, and their latest twenty measurements get retained instead of the default ten.
kind: Rollout\nspec:\n...\n steps:\n - analysis:\n templates:\n - templateName: random-fail\n - templateName: always-pass\n measurementRetention:\n - metricName: .*\n limit: 20\n
"},{"location":"features/analysis/#define-custom-labelsannotations-for-analysisrun","title":"Define custom Labels/Annotations for AnalysisRun","text":"If you would like to annotate/label the AnalysisRun with the custom labels your can do it by specifying analysisRunMetadata field.
kind: Rollout\nspec:\n...\n steps:\n - analysis:\n templates:\n - templateName: my-template\n analysisRunMetadata:\n labels:\n my-custom-label: label-value\n annotations:\n my-custom-annotation: annotation-value\n
"},{"location":"features/analysis/#measurements-retention-for-experiments","title":"Measurements Retention for Experiments","text":"If an experiment wants to retain more results of its analysis metrics, it simply needs to specify the measurementRetention field under its specs. In the following example, all the metrics from analyze-job matching the RegEx rule test.* will have their latest twenty measurements get retained instead of the default ten.
kind: Experiment\nspec:\n templates:\n - name: baseline\n selector:\n matchLabels:\n app: rollouts-demo\n template:\n metadata:\n labels:\n app: rollouts-demo\n spec:\n containers:\n - name: rollouts-demo\n image: argoproj/rollouts-demo:blue\n analyses:\n - name: analyze-job\n templateName: analyze-job\n measurementRetention:\n - metricName: test.*\n limit: 20\n
"},{"location":"features/analysis/#time-to-live-ttl-strategy","title":"Time-to-live (TTL) Strategy","text":"Important
Available since v1.7
ttlStrategy limits the lifetime of an analysis run that has finished execution depending on if it Succeeded or Failed. If this struct is set, once the run finishes, it will be deleted after the time to live expires. If this field is unset, the analysis controller will keep the completed runs, unless they are associated with rollouts using other garbage collection policies (e.g. successfulRunHistoryLimit and unsuccessfulRunHistoryLimit).
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\nspec:\n ...\n ttlStrategy:\n secondsAfterCompletion: 3600\n secondsAfterSuccess: 1800\n secondsAfterFailure: 1800\n
"},{"location":"features/analysis/#inconclusive-runs","title":"Inconclusive Runs","text":"Analysis runs can also be considered Inconclusive, which indicates the run was neither successful, nor failed. Inconclusive runs causes a rollout to become paused at its current step. Manual intervention is then needed to either resume the rollout, or abort. One example of how analysis runs could become Inconclusive, is when a metric defines no success or failure conditions.
metrics:\n - name: my-query\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: ...\n
Inconclusive analysis runs might also happen when both success and failure conditions are specified, but the measurement value did not meet either condition.
metrics:\n - name: success-rate\n successCondition: result[0] >= 0.90\n failureCondition: result[0] < 0.50\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: ...\n
A use case for having Inconclusive analysis runs are to enable Argo Rollouts to automate the execution of analysis runs, and collect the measurement, but still allow human judgement to decide whether or not measurement value is acceptable and decide to proceed or abort.
"},{"location":"features/analysis/#delay-analysis-runs","title":"Delay Analysis Runs","text":"If the analysis run does not need to start immediately (i.e give the metric provider time to collect metrics on the canary version), Analysis Runs can delay the specific metric analysis. Each metric can be configured to have a different delay. In additional to the metric specific delays, the rollouts with background analysis can delay creating an analysis run until a certain step is reached
Delaying a specific analysis metric:
metrics:\n - name: success-rate\n # Do not start this analysis until 5 minutes after the analysis run starts\n initialDelay: 5m\n successCondition: result[0] >= 0.90\n provider:\n prometheus:\n address: http://prometheus.example.com:9090\n query: ...\n
Delaying starting background analysis run until step 3 (Set Weight 40%):
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\nspec:\n strategy:\n canary:\n analysis:\n templates:\n - templateName: success-rate\n startingStep: 2\n steps:\n - setWeight: 20\n - pause: {duration: 10m}\n - setWeight: 40\n - pause: {duration: 10m}\n
"},{"location":"features/analysis/#referencing-secrets","title":"Referencing Secrets","text":"AnalysisTemplates and AnalysisRuns can reference secret objects in .spec.args. This allows users to securely pass authentication information to Metric Providers, like login credentials or API tokens.
An AnalysisRun can only reference secrets from the same namespace as it's running in. This is only relevant for AnalysisRuns, since AnalysisTemplates do not resolve the secret.
In the following example, an AnalysisTemplate references an API token and passes it to a Web metric provider.
This example demonstrates:
- The ability to reference a secret in the AnalysisTemplate
.spec.args - The ability to pass secret arguments to Metric Providers
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisTemplate\nspec:\n args:\n - name: api-token\n valueFrom:\n secretKeyRef:\n name: token-secret\n key: apiToken\n metrics:\n - name: webmetric\n provider:\n web:\n headers:\n - key: Authorization\n value: \"Bearer {{ args.api-token }}\"\n
"},{"location":"features/analysis/#handling-metric-results","title":"Handling Metric Results","text":""},{"location":"features/analysis/#nan-and-infinity","title":"NaN and Infinity","text":"Metric providers can sometimes return values of NaN (not a number) and infinity. Users can edit the successCondition and failureCondition fields to handle these cases accordingly.
Here are three examples where a metric result of NaN is considered successful, inconclusive and failed respectively.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\n ...\n successCondition: isNaN(result) || result >= 0.95\nstatus:\n metricResults:\n - count: 1\n measurements:\n - finishedAt: \"2021-02-10T00:15:26Z\"\n phase: Successful\n startedAt: \"2021-02-10T00:15:26Z\"\n value: NaN\n name: success-rate\n phase: Successful\n successful: 1\n phase: Successful\n startedAt: \"2021-02-10T00:15:26Z\"\n
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\n ...\n successCondition: result >= 0.95\n failureCondition: result < 0.95\nstatus:\n metricResults:\n - count: 1\n measurements:\n - finishedAt: \"2021-02-10T00:15:26Z\"\n phase: Inconclusive\n startedAt: \"2021-02-10T00:15:26Z\"\n value: NaN\n name: success-rate\n phase: Inconclusive\n successful: 1\n phase: Inconclusive\n startedAt: \"2021-02-10T00:15:26Z\"\n
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\n ...\n successCondition: result >= 0.95\nstatus:\n metricResults:\n - count: 1\n measurements:\n - finishedAt: \"2021-02-10T00:15:26Z\"\n phase: Failed\n startedAt: \"2021-02-10T00:15:26Z\"\n value: NaN\n name: success-rate\n phase: Failed\n successful: 1\n phase: Failed\n startedAt: \"2021-02-10T00:15:26Z\"\n
Here are two examples where a metric result of infinity is considered successful and failed respectively.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\n ...\n successCondition: result >= 0.95\nstatus:\n metricResults:\n - count: 1\n measurements:\n - finishedAt: \"2021-02-10T00:15:26Z\"\n phase: Successful\n startedAt: \"2021-02-10T00:15:26Z\"\n value: +Inf\n name: success-rate\n phase: Successful\n successful: 1\n phase: Successful\n startedAt: \"2021-02-10T00:15:26Z\"\n
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\n ...\n failureCondition: isInf(result)\nstatus:\n metricResults:\n - count: 1\n measurements:\n - finishedAt: \"2021-02-10T00:15:26Z\"\n phase: Failed\n startedAt: \"2021-02-10T00:15:26Z\"\n value: +Inf\n name: success-rate\n phase: Failed\n successful: 1\n phase: Failed\n startedAt: \"2021-02-10T00:15:26Z\"\n
"},{"location":"features/analysis/#empty-array","title":"Empty array","text":""},{"location":"features/analysis/#prometheus","title":"Prometheus","text":"Metric providers can sometimes return empty array, e.g., no data returned from prometheus query.
Here are two examples where a metric result of empty array is considered successful and failed respectively.
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\n ...\n successCondition: len(result) == 0 || result[0] >= 0.95\nstatus:\n metricResults:\n - count: 1\n measurements:\n - finishedAt: \"2021-09-08T19:15:49Z\"\n phase: Successful\n startedAt: \"2021-09-08T19:15:49Z\"\n value: '[]'\n name: success-rate\n phase: Successful\n successful: 1\n phase: Successful\n startedAt: \"2021-09-08T19:15:49Z\"\n
apiVersion: argoproj.io/v1alpha1\nkind: AnalysisRun\n ...\n successCondition: len(result) > 0 && result[0] >= 0.95\nstatus:\n metricResults:\n - count: 1\n measurements:\n - finishedAt: \"2021-09-08T19:19:44Z\"\n phase: Failed\n startedAt: \"2021-09-08T19:19:44Z\"\n value: '[]'\n name: success-rate\n phase: Failed\n successful: 1\n phase: Failed\n startedAt: \"2021-09-08T19:19:44Z\"\n
"},{"location":"features/analysis/#datadog","title":"Datadog","text":"Datadog queries can return empty results if the query takes place during a time interval with no metrics. The Datadog provider will return a nil value yielding an error during the evaluation phase like:
invalid operation: < (mismatched types <nil> and float64)\n
However, empty query results yielding a nil value can be handled using the default() function. Here is a succeeding example using the default() function:
successCondition: default(result, 0) < 0.05\n
"},{"location":"features/bluegreen/","title":"BlueGreen Deployment Strategy","text":"A Blue Green Deployment allows users to reduce the amount of time multiple versions running at the same time.
"},{"location":"features/bluegreen/#overview","title":"Overview","text":"In addition to managing ReplicaSets, the rollout controller will modify a Service resource during the BlueGreenUpdate strategy. The Rollout spec has users specify a reference to active service and optionally a preview service in the same namespace. The active Service is used to send regular application traffic to the old version, while the preview Service is used as funnel traffic to the new version. The rollout controller ensures proper traffic routing by injecting a unique hash of the ReplicaSet to these services' selectors. This allows the rollout to define an active and preview stack and a process to migrate replica sets from the preview to the active.
When there is a change to the .spec.template field of a rollout, the controller will create the new ReplicaSet. If the active service is not sending traffic to a ReplicaSet, the controller will immediately start sending traffic to the ReplicaSet. Otherwise, the active service will point at the old ReplicaSet while the ReplicaSet becomes available. Once the new ReplicaSet becomes available, the controller will modify the active service to point at the new ReplicaSet. After waiting some time configured by the .spec.strategy.blueGreen.scaleDownDelaySeconds, the controller will scale down the old ReplicaSet.
Important
When the rollout changes the selector on a service, there is a propagation delay before all the nodes update their IP tables to send traffic to the new pods instead of the old. During this delay, traffic will be directed to the old pods if the nodes have not been updated yet. In order to prevent the packets from being sent to a node that killed the old pod, the rollout uses the scaleDownDelaySeconds field to give nodes enough time to broadcast the IP table changes.
"},{"location":"features/bluegreen/#example","title":"Example","text":"apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollout-bluegreen\nspec:\n replicas: 2\n revisionHistoryLimit: 2\n selector:\n matchLabels:\n app: rollout-bluegreen\n template:\n metadata:\n labels:\n app: rollout-bluegreen\n spec:\n containers:\n - name: rollouts-demo\n image: argoproj/rollouts-demo:blue\n imagePullPolicy: Always\n ports:\n - containerPort: 8080\n strategy:\n blueGreen: \n # activeService specifies the service to update with the new template hash at time of promotion.\n # This field is mandatory for the blueGreen update strategy.\n activeService: rollout-bluegreen-active\n # previewService specifies the service to update with the new template hash before promotion.\n # This allows the preview stack to be reachable without serving production traffic.\n # This field is optional.\n previewService: rollout-bluegreen-preview\n # autoPromotionEnabled disables automated promotion of the new stack by pausing the rollout\n # immediately before the promotion. If omitted, the default behavior is to promote the new\n # stack as soon as the ReplicaSet are completely ready/available.\n # Rollouts can be resumed using: `kubectl argo rollouts promote ROLLOUT`\n autoPromotionEnabled: false\n
"},{"location":"features/bluegreen/#configurable-features","title":"Configurable Features","text":"Here are the optional fields that will change the behavior of BlueGreen deployment:
spec:\n strategy:\n blueGreen:\n autoPromotionEnabled: boolean\n autoPromotionSeconds: *int32\n antiAffinity: object\n previewService: string\n prePromotionAnalysis: object\n postPromotionAnalysis: object\n previewReplicaCount: *int32\n scaleDownDelaySeconds: *int32\n scaleDownDelayRevisionLimit: *int32\n
"},{"location":"features/bluegreen/#sequence-of-events","title":"Sequence of Events","text":"The following describes the sequence of events that happen during a blue-green update.
- Beginning at a fully promoted, steady-state, a revision 1 ReplicaSet is pointed to by both the
activeService and previewService. - A user initiates an update by modifying the pod template (
spec.template.spec). - The revision 2 ReplicaSet is created with size 0.
- The
previewService is modified to point to the revision 2 ReplicaSet. The activeService remains pointing to revision 1. - The revision 2 ReplicaSet is scaled to either
spec.replicas or previewReplicaCount if set. - Once revision 2 ReplicaSet Pods are fully available,
prePromotionAnalysis begins. - Upon success of
prePromotionAnalysis, the blue/green pauses if autoPromotionEnabled is false, or autoPromotionSeconds is non-zero. - The rollout is resumed either manually by a user, or automatically by surpassing
autoPromotionSeconds. - The revision 2 ReplicaSet is scaled to the
spec.replicas, if the previewReplicaCount feature was used. - The rollout \"promotes\" the revision 2 ReplicaSet by updating the
activeService to point to it. At this point, there are no services pointing to revision 1 postPromotionAnalysis analysis begins- Once
postPromotionAnalysis completes successfully, the update is successful and the revision 2 ReplicaSet is marked as stable. The rollout is considered fully-promoted. - After waiting
scaleDownDelaySeconds (default 30 seconds), the revision 1 ReplicaSet is scaled down
"},{"location":"features/bluegreen/#autopromotionenabled","title":"autoPromotionEnabled","text":"The AutoPromotionEnabled will make the rollout automatically promote the new ReplicaSet to the active service once the new ReplicaSet is healthy. This field is defaulted to true if it is not specified.
Defaults to true
"},{"location":"features/bluegreen/#autopromotionseconds","title":"autoPromotionSeconds","text":"The AutoPromotionSeconds will make the rollout automatically promote the new ReplicaSet to active Service after the AutoPromotionSeconds time has passed since the rollout has entered a paused state. If the AutoPromotionEnabled field is set to false, this field will be ignored
Defaults to nil
"},{"location":"features/bluegreen/#antiaffinity","title":"antiAffinity","text":"Check out the Anti Affinity document document for more information.
Defaults to nil
"},{"location":"features/bluegreen/#maxunavailable","title":"maxUnavailable","text":"The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). This can not be 0 if MaxSurge is 0.
Defaults to 0
"},{"location":"features/bluegreen/#prepromotionanalysis","title":"prePromotionAnalysis","text":"Configures the Analysis before it switches traffic to the new version. The AnalysisRun can be used to block the Service selector switch until the AnalysisRun finishes successful. The success or failure of the analysis run decides if the Rollout will switch traffic, or abort the Rollout completely.
Defaults to nil
"},{"location":"features/bluegreen/#postpromotionanalysis","title":"postPromotionAnalysis","text":"Configures the Analysis after the traffic switch to new version. If the analysis run fails or errors out, the Rollout enters an aborted state and switch traffic back to the previous stable Replicaset. If scaleDownDelaySeconds is specified, the controller will cancel any AnalysisRuns at time of scaleDownDelay to scale down the ReplicaSet. If it is omitted, and post analysis is specified, it will scale down the ReplicaSet only after the AnalysisRun completes (with a minimum of 30 seconds).
Defaults to nil
"},{"location":"features/bluegreen/#previewservice","title":"previewService","text":"The PreviewService field references a Service that will be modified to send traffic to the new ReplicaSet before the new one is promoted to receiving traffic from the active service. Once the new ReplicaSet starts receiving traffic from the active service, the preview service will also be modified to send traffic to the new ReplicaSet as well. The Rollout always makes sure that the preview service is sending traffic to the newest ReplicaSet. As a result, if a new version is introduced before the old version is promoted to the active service, the controller will immediately switch over to that brand new version.
This feature is used to provide an endpoint that can be used to test a new version of an application.
Defaults to an empty string
"},{"location":"features/bluegreen/#previewreplicacount","title":"previewReplicaCount","text":"The PreviewReplicaCount field will indicate the number of replicas that the new version of an application should run. Once the application is ready to promote to the active service, the controller will scale the new ReplicaSet to the value of the spec.replicas. The rollout will not switch over the active service to the new ReplicaSet until it matches the spec.replicas count.
This feature is mainly used to save resources during the testing phase. If the application does not need a fully scaled up application for the tests, this feature can help save some resources.
If omitted, the preview ReplicaSet stack will be scaled to 100% of the replicas.
"},{"location":"features/bluegreen/#scaledowndelayseconds","title":"scaleDownDelaySeconds","text":"The ScaleDownDelaySeconds is used to delay scaling down the old ReplicaSet after the active Service is switched to the new ReplicaSet.
Defaults to 30
"},{"location":"features/bluegreen/#scaledowndelayrevisionlimit","title":"scaleDownDelayRevisionLimit","text":"The ScaleDownDelayRevisionLimit limits the number of old active ReplicaSets to keep scaled up while they wait for the scaleDownDelay to pass after being removed from the active service.
If omitted, all ReplicaSets will be retained for the specified scaleDownDelay
"},{"location":"features/canary/","title":"Canary Deployment Strategy","text":"A canary rollout is a deployment strategy where the operator releases a new version of their application to a small percentage of the production traffic.
"},{"location":"features/canary/#overview","title":"Overview","text":"Since there is no agreed upon standard for a canary deployment, the rollouts controller allows users to outline how they want to run their canary deployment. Users can define a list of steps the controller uses to manipulate the ReplicaSets when there is a change to the .spec.template. Each step will be evaluated before the new ReplicaSet is promoted to the stable version, and the old version is completely scaled down.
Each step can have one of two fields. The setWeight field dictates the percentage of traffic that should be sent to the canary, and the pause struct instructs the rollout to pause. When the controller reaches a pause step for a rollout, it will add a PauseCondition struct to the .status.PauseConditions field. If the duration field within the pause struct is set, the rollout will not progress to the next step until it has waited for the value of the duration field. Otherwise, the rollout will wait indefinitely until that Pause condition is removed. By using the setWeight and the pause fields, a user can declaratively describe how they want to progress to the new version. Below is an example of a canary strategy.
Important
If the canary Rollout does not use traffic management, the Rollout makes a best effort attempt to achieve the percentage listed in the last setWeight step between the new and old version. For example, if a Rollout has 10 Replicas and 10% for the first setWeight step, the controller will scale the new desired ReplicaSet to 1 replicas and the old stable ReplicaSet to 9. In the case where the setWeight is 41%, the Rollout attempts to get there by finding the whole number with the smallest delta, rounding up the calculation if the deltas are equals (i.e. the new ReplicaSet has 4 pods since 41% of 10 is closer to 4/10 than 5/10, and the old ReplicaSet has 6 pods). If a user wants to have more fine-grained control of the percentages without a large number of Replicas, that user should use the traffic management functionality.
"},{"location":"features/canary/#example","title":"Example","text":"apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: example-rollout\nspec:\n replicas: 10\n selector:\n matchLabels:\n app: nginx\n template:\n metadata:\n labels:\n app: nginx\n spec:\n containers:\n - name: nginx\n image: nginx:1.15.4\n ports:\n - containerPort: 80\n minReadySeconds: 30\n revisionHistoryLimit: 3\n strategy:\n canary: #Indicates that the rollout should use the Canary strategy\n maxSurge: \"25%\"\n maxUnavailable: 0\n steps:\n - setWeight: 10\n - pause:\n duration: 1h # 1 hour\n - setWeight: 20\n - pause: {} # pause indefinitely\n
"},{"location":"features/canary/#pause-duration","title":"Pause Duration","text":"Pause duration can be specified with an optional time unit suffix. Valid time units are \"s\", \"m\", \"h\". Defaults to \"s\" if not specified.
spec:\n strategy:\n canary:\n steps:\n - pause: { duration: 10 } # 10 seconds\n - pause: { duration: 10s } # 10 seconds\n - pause: { duration: 10m } # 10 minutes\n - pause: { duration: 10h } # 10 hours\n - pause: {} # pause indefinitely\n
If no duration is specified for a pause step, the rollout will be paused indefinitely. To unpause, use the argo kubectl plugin promote command.
# promote to the next step\nkubectl argo rollouts promote <rollout>\n
"},{"location":"features/canary/#dynamic-canary-scale-with-traffic-routing","title":"Dynamic Canary Scale (with Traffic Routing)","text":"By default, the rollout controller will scale the canary to match the current trafficWeight of the current step. For example, if the current weight is 25%, and there are four replicas, then the canary will be scaled to 1, to match the traffic weight.
It is possible to control the canary replica's scale during the steps such that it does not necessary match the traffic weight. Some use cases for this:
- The new version should not yet be exposed to the public (setWeight: 0), but you would like to scale the canary up for testing purposes.
- You wish to scale the canary stack up minimally, and use some header based traffic shaping to the canary, while setWeight is still set to 0.
- You wish to scale the canary up to 100%, in order to facilitate traffic shadowing.
Important
Setting canary scale is only available when using the canary strategy with a traffic router, since the basic canary needs to control canary scale in order to approximate canary weight.
To control canary scales and weights during steps, use the setCanaryScale step and indicate which scale the the canary should use:
- explicit replica count without changing traffic weight (
replicas) - explicit weight percentage of total spec.replicas without changing traffic weight(
weight) - to or not to match current canary's
setWeight step (matchTrafficWeight: true or false)
spec:\n strategy:\n canary:\n steps:\n # explicit count\n - setCanaryScale:\n replicas: 3\n # a percentage of spec.replicas\n - setCanaryScale:\n weight: 25\n # matchTrafficWeight returns to the default behavior of matching the canary traffic weight\n - setCanaryScale:\n matchTrafficWeight: true\n
When using setCanaryScale with explicit values for either replicas or weight, one must be careful if used in conjunction with the setWeight step. If done incorrectly, an imbalanced amount of traffic may be directed to the canary (in proportion to the Rollout's scale). For example, the following set of steps would cause 90% of traffic to only be served by 10% of pods:
spec:\n replicas: 10\n strategy:\n canary:\n steps:\n # 1 canary pod (10% of spec.replicas)\n - setCanaryScale:\n weight: 10\n # 90% of traffic to the 1 canary pod\n - setWeight: 90\n - pause: {}\n
The above situation is caused by the changed behvaior of setWeight after setCanaryScale. To reset, set matchTrafficWeight: true and the setWeight behavior will be restored, i.e., subsequent setWeight will create canary replicas matching the traffic weight.
"},{"location":"features/canary/#dynamic-stable-scale-with-traffic-routing","title":"Dynamic Stable Scale (with Traffic Routing)","text":"Important
Available since v1.1
When using traffic routing, by default the stable ReplicaSet is left scaled to 100% during the update. This has the advantage that if an abort occurs, traffic can be immediately shifted back to the stable ReplicaSet without delay. However, it has the disadvantage that during the update, there will eventually exist double the number of replica pods running (similar to in a blue-green deployment), since the stable ReplicaSet is left scaled up for the full duration of the update.
It is possible to dynamically reduce the scale of the stable ReplicaSet during an update such that it scales down as the traffic weight increases to canary. This would be desirable in scenarios where the Rollout has a high replica count and resource cost is a concern, or in bare-metal situations where it is not possible to create additional node capacity to accommodate double the replicas.
The ability to dynamically scale the stable ReplicaSet can be enabled by setting the canary.dynamicStableScale flag to true:
spec:\n strategy:\n canary:\n dynamicStableScale: true\n
NOTE: that if dynamicStableScale is set, and the rollout is aborted, the canary ReplicaSet will dynamically scale down as traffic shifts back to stable. If you wish to leave the canary ReplicaSet scaled up while aborting, an explicit value for abortScaleDownDelaySeconds can be set:
spec:\n strategy:\n canary:\n dynamicStableScale: true\n abortScaleDownDelaySeconds: 600\n
"},{"location":"features/canary/#mimicking-rolling-update","title":"Mimicking Rolling Update","text":"If the steps field is omitted, the canary strategy will mimic the rolling update behavior. Similar to the deployment, the canary strategy has the maxSurge and maxUnavailable fields to configure how the Rollout should progress to the new version.
"},{"location":"features/canary/#other-configurable-features","title":"Other Configurable Features","text":"Here are the optional fields that will modify the behavior of canary strategy:
spec:\n strategy:\n canary:\n analysis: object\n antiAffinity: object\n canaryService: string\n stableService: string\n maxSurge: stringOrInt\n maxUnavailable: stringOrInt\n trafficRouting: object\n
"},{"location":"features/canary/#analysis","title":"analysis","text":"Configure the background Analysis to execute during the rollout. If the analysis is unsuccessful the rollout will be aborted.
Defaults to nil
"},{"location":"features/canary/#antiaffinity","title":"antiAffinity","text":"Check out the Anti Affinity document document for more information.
Defaults to nil
"},{"location":"features/canary/#canaryservice","title":"canaryService","text":"canaryService references a Service that will be modified to send traffic to only the canary ReplicaSet. This allows users to only hit the canary ReplicaSet.
Defaults to an empty string
"},{"location":"features/canary/#stableservice","title":"stableService","text":"stableService the name of a Service which selects pods with stable version and doesn't select any pods with canary version. This allows users to only hit the stable ReplicaSet.
Defaults to an empty string
"},{"location":"features/canary/#maxsurge","title":"maxSurge","text":"maxSurge defines the maximum number of replicas the rollout can create to move to the correct ratio set by the last setWeight. Max Surge can either be an integer or percentage as a string (i.e. \"20%\")
Defaults to \"25%\".
"},{"location":"features/canary/#maxunavailable","title":"maxUnavailable","text":"The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). This can not be 0 if MaxSurge is 0.
Defaults to 25%
"},{"location":"features/canary/#trafficrouting","title":"trafficRouting","text":"The traffic management rules to apply to control the flow of traffic between the active and canary versions. If not set, the default weighted pod replica based routing will be used.
Defaults to nil
"},{"location":"features/controller-metrics/","title":"Controller Metrics","text":"The Argo Rollouts controller is already instrumented with Prometheus metrics available at /metrics in port 8090. You can use these metrics to look at the health of the controller either via dashboards or via other Prometheus integrations.
"},{"location":"features/controller-metrics/#installing-and-configuring-prometheus","title":"Installing and configuring Prometheus","text":"To take advantage of the metrics you need to have Prometheus installed in your Kubernetes cluster. If you don't have an existing installation of Prometheus you can use any of the common methods to install it in your cluster. Popular options include the Prometheus Helm chart or the Prometheus Operator.
Once Prometheus is running in your cluster you need to make sure that it scrapes the Argo Rollouts endpoint. Prometheus already contains a service discovery mechanism for Kubernetes, but you need to configure it first. Depending on your installation method you might need to take additional actions to scrape the Argo Rollouts endpoint.
For example, if you used the Helm chart of Prometheus you need to annotate your Argo Rollouts Controller with the following:
spec:\n template:\n metadata:\n annotations:\n prometheus.io/scrape: \"true\"\n prometheus.io/path: /metrics\n prometheus.io/port: \"8090\"\n
You can always see if the controller is reached successfully in the Prometheus \"Targets\" screen:
Once the controller metrics are read by your Prometheus instance, you can use them like any other Prometheus data source.
"},{"location":"features/controller-metrics/#creating-grafana-dashboards","title":"Creating Grafana Dashboards","text":"You can easily visualize the metrics from the controller using Grafana dashboards. Install Grafana in your cluster and connect it your Prometheus instance. Then you can create any dashboard by using the available metrics (described in detail in the next sections).
As a starting point you can find an existing dashboard at https://github.com/argoproj/argo-rollouts/blob/master/examples/dashboard.json
You can import this Dashboard in your Grafana installation as a JSON file.
"},{"location":"features/controller-metrics/#available-metrics-for-rollout-objects","title":"Available metrics for Rollout Objects","text":"The Argo Rollouts controller publishes the following prometheus metrics about Argo Rollout objects.
Name Description rollout_info Information about rollout. rollout_info_replicas_available The number of available replicas per rollout. rollout_info_replicas_unavailable The number of unavailable replicas per rollout. rollout_info_replicas_desired The number of desired replicas per rollout. rollout_info_replicas_updated The number of updated replicas per rollout. rollout_phase [DEPRECATED - use rollout_info] Information on the state of the rollout. rollout_reconcile Rollout reconciliation performance. rollout_reconcile_error Error occurring during the rollout. experiment_info Information about Experiment. experiment_phase Information on the state of the experiment. experiment_reconcile Experiments reconciliation performance. experiment_reconcile_error Error occurring during the experiment. analysis_run_info Information about analysis run. analysis_run_metric_phase Information on the duration of a specific metric in the Analysis Run. analysis_run_metric_type Information on the type of a specific metric in the Analysis Runs. analysis_run_phase Information on the state of the Analysis Run. analysis_run_reconcile Analysis Run reconciliation performance. analysis_run_reconcile_error Error occurring during the analysis run."},{"location":"features/controller-metrics/#available-metrics-for-the-controller-itself","title":"Available metrics for the controller itself","text":"The controller also publishes the following Prometheus metrics to describe the controller health.
Name Description controller_clientset_k8s_request_total Number of kubernetes requests executed during application reconciliation. workqueue_adds_total Total number of adds handled by workqueue workqueue_depth Current depth of workqueue workqueue_queue_duration_seconds How long in seconds an item stays in workqueue before being requested. workqueue_work_duration_seconds How long in seconds processing an item from workqueue takes. workqueue_unfinished_work_seconds How many seconds of work has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases. workqueue_longest_running_processor_seconds How many seconds has the longest running processor for workqueue been running workqueue_retries_total Total number of retries handled by workqueue In addition, the Argo-rollouts offers metrics on CPU, memory and file descriptor usage as well as the process start time and memory stats of current Go processes.
"},{"location":"features/ephemeral-metadata/","title":"Ephemeral Metadata","text":"Important
Available for canary rollouts since v0.10.0
Important
Available for blue-green rollouts since v1.0
One use case is for a Rollout to label or annotate the desired/stable pods with user-defined labels/annotations, for only the duration which they are the desired or stable set, and for the labels to be updated/removed as soon as the ReplicaSet switches roles (e.g. from desired to stable). The use case which this enables, is to allow prometheus, wavefront, datadog queries and dashboards to be built, which can rely on a consistent labels, rather than the rollouts-pod-template-hash which is unpredictable and changing from revision to revision.
A Rollout using the canary strategy has the ability to attach ephemeral metadata to the stable or canary Pods using the stableMetadata and canaryMetadata fields respectively.
spec:\n strategy:\n canary:\n stableMetadata:\n labels:\n role: stable\n canaryMetadata:\n labels:\n role: canary\n
A Rollout using the blue-green strategy has the ability to attach ephemeral metadata to the active or preview Pods using the activeMetadata and previewMetadata fields respectively.
spec:\n strategy:\n blueGreen:\n activeMetadata:\n labels:\n role: active\n previewMetadata:\n labels:\n role: preview\n
During an update, the Rollout will create the desired ReplicaSet while also merging the metadata defined in canaryMetadata/previewMetadata to the desired ReplicaSet's spec.template.metadata. This results in all Pods of the ReplicaSet being created with the desired metadata. When the rollout becomes fully promoted, the desired ReplicaSet becomes the stable, and is updated to use the labels and annotations under stableMetadata/activeMetadata. The Pods of the ReplicaSet will then be updated in place to use the stable metadata (without recreating the pods).
Important
In order for tooling to take advantage of this feature, they would need to recognize the change in labels and/or annotations that happen after the Pod has already started. Not all tools may detect this.
"},{"location":"features/experiment/","title":"Experiment CRD","text":""},{"location":"features/experiment/#what-is-the-experiment-crd","title":"What is the Experiment CRD?","text":"The Experiment CRD allows users to have ephemeral runs of one or more ReplicaSets. In addition to running ephemeral ReplicaSets, the Experiment CRD can launch AnalysisRuns alongside the ReplicaSets. Generally, those AnalysisRun is used to confirm that new ReplicaSets are running as expected.
A Service routing traffic to the Experiment ReplicaSet is also generated if a weight (which requires traffic routing) OR the Service attribute for that experiment is set.
"},{"location":"features/experiment/#use-cases-of-experiments","title":"Use cases of Experiments","text":" -
A user wants to run two versions of an application for a specific duration to enable Kayenta-style analysis of their application. The Experiment CRD creates 2 ReplicaSets (a baseline and a canary) based on the spec.templates field of the Experiment and waits until both are healthy. After the duration passes, the Experiment scales down the ReplicaSets, and the user can start the Kayenta analysis run.
-
A user can use experiments to enable A/B/C testing by launching multiple experiments with a different version of their application for a long duration. Each Experiment has one PodSpec template that defines a specific version a user would want to run. The Experiment allows users to launch multiple experiments at once and keep each Experiment self-contained.
-
Launching a new version of an existing application with different labels to avoid receiving traffic from a Kubernetes service. The user can run tests against the new version before continuing the Rollout.
"},{"location":"features/experiment/#experiment-spec","title":"Experiment Spec","text":"Below is an example of an experiment that creates two ReplicaSets with 1 replica each and runs them for 20 minutes once they both become available. Additionally, several AnalysisRuns are run to perform analysis against the pods of the Experiment
apiVersion: argoproj.io/v1alpha1\nkind: Experiment\nmetadata:\n name: example-experiment\nspec:\n # Duration of the experiment, beginning from when all ReplicaSets became healthy (optional)\n # If omitted, will run indefinitely until terminated, or until all analyses which were marked\n # `requiredForCompletion` have completed.\n duration: 20m\n\n # Deadline in seconds in which a ReplicaSet should make progress towards becoming available.\n # If exceeded, the Experiment will fail.\n progressDeadlineSeconds: 30\n\n # List of pod template specs to run in the experiment as ReplicaSets\n templates:\n - name: purple\n # Number of replicas to run (optional). If omitted, will run a single replica\n replicas: 1\n # Flag to create Service for this Experiment (optional)\n # If omitted, a Service won't be created.\n service:\n # Name of the Service (optional). If omitted, service: {} would also be acceptable.\n name: service-name\n selector:\n matchLabels:\n app: canary-demo\n color: purple\n template:\n metadata:\n labels:\n app: canary-demo\n color: purple\n spec:\n containers:\n - name: rollouts-demo\n image: argoproj/rollouts-demo:purple\n imagePullPolicy: Always\n ports:\n - name: http\n containerPort: 8080\n protocol: TCP\n - name: orange\n replicas: 1\n minReadySeconds: 10\n selector:\n matchLabels:\n app: canary-demo\n color: orange\n template:\n metadata:\n labels:\n app: canary-demo\n color: orange\n spec:\n containers:\n - name: rollouts-demo\n image: argoproj/rollouts-demo:orange\n imagePullPolicy: Always\n ports:\n - name: http\n containerPort: 8080\n protocol: TCP\n\n # List of AnalysisTemplate references to perform during the experiment\n analyses:\n - name: purple\n templateName: http-benchmark\n args:\n - name: host\n value: purple\n - name: orange\n templateName: http-benchmark\n args:\n - name: host\n value: orange\n - name: compare-results\n templateName: compare\n # If requiredForCompletion is true for an analysis reference, the Experiment will not complete\n # until this analysis has completed.\n requiredForCompletion: true\n args:\n - name: host\n value: purple\n
"},{"location":"features/experiment/#experiment-lifecycle","title":"Experiment Lifecycle","text":"An Experiment is intended to temporarily run one or more templates. The lifecycle of an Experiment is as follows:
- Create and scale a ReplicaSet for each pod template specified under
spec.templates. If service is specified under a pod template, a Service will also be created for that pod. - Wait for all ReplicaSets reach full availability. If a ReplicaSet does not become available within
spec.progressDeadlineSeconds, the Experiment will fail. Once available, the Experiment will transition from the Pending state to a Running state. - Once an Experiment is considered
Running, it will begin an AnalysisRun for every AnalysisTemplate referenced under spec.analyses. - If a duration is specified under
spec.duration, the Experiment will wait until the duration has elapsed before completing the Experiment. - If an AnalysisRun fails or errors, the Experiment will end prematurely, with a status equal to the unsuccessful AnalysisRun (i.e.
Failed or Error) - If one or more of the referenced AnalysisTemplates is marked with
requiredForCompletion: true, the Experiment will not complete until those AnalysisRuns have completed, even if it exceeds the Experiment duration. - If neither a
spec.duration or requiredForCompletion: true is specified, the Experiment will run indefinitely, until explicitly terminated (by setting spec.terminate: true). - Once an Experiment is complete, the ReplicaSets will be scaled to zero, and any incomplete AnalysisRuns will be terminated.
Note
ReplicaSet names are generated by combining the Experiment name with the template name.
"},{"location":"features/experiment/#integration-with-rollouts","title":"Integration With Rollouts","text":"A rollout using the Canary strategy can create an experiment using an experiment step. The experiment step serves as a blocking step for the Rollout, and a Rollout will not continue until the Experiment succeeds. The Rollout creates an Experiment using the configuration in the experiment step of the Rollout. If the Experiment fails or errors, the Rollout will abort.
Note
Experiment names are generated by combining the Rollout's name, the PodHash of the new ReplicaSet, the current revision of the Rollout, and the current step-index.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\n labels:\n app: guestbook\nspec:\n...\n strategy:\n canary: \n steps:\n - experiment:\n duration: 1h\n templates:\n - name: baseline\n specRef: stable\n - name: canary\n specRef: canary\n analyses:\n - name : mann-whitney\n templateName: mann-whitney\n args:\n - name: baseline-hash\n value: \"{{templates.baseline.podTemplateHash}}\"\n - name: canary-hash\n value: \"{{templates.canary.podTemplateHash}}\"\n
In the example above, during an update of a Rollout, the Rollout will launch an Experiment. The Experiment will create two ReplicaSets: baseline and canary, with one replica each, and will run for one hour. The baseline template uses the PodSpec from the stable ReplicaSet, and the canary template uses the PodSpec from the canary ReplicaSet.
Additionally, the Experiment will perform analysis using the AnalysisTemplate named mann-whitney. The AnalysisRun is supplied with the pod-hash details of the baseline and canary to perform the necessary metrics queries, using the {{templates.baseline.podTemplateHash}} and {{templates.canary.podTemplateHash}} variables respectively.
Note
The pod-hashes of the baseline/canary ReplicaSets created by the Experiment, will have different values than the pod-hashes of the stable/canary ReplicaSets created by the Rollout. This is despite the fact that the PodSpec are the same. This is intentional behavior, in order to allow the metrics of the Experiment's pods to be delineated and queried separately from the metrics of the Rollout pods.
"},{"location":"features/experiment/#weighted-experiment-step-with-traffic-routing","title":"Weighted Experiment Step with Traffic Routing","text":"Important
Available since v1.1
A Rollout using the Canary strategy along with Traffic Routing can split traffic to an experiment stack in a fine-grained manner. When Traffic Routing is enabled, the Rollout Experiment step allows traffic to be shifted to experiment pods.
Note
This feature is currently available only for the SMI, ALB, and Istio Traffic Routers.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\n labels:\n app: guestbook\nspec:\n...\nstrategy:\n canary:\n trafficRouting:\n alb:\n ingress: ingress\n ...\n steps:\n - experiment:\n duration: 1h\n templates:\n - name: experiment-baseline\n specRef: stable\n weight: 5\n - name: experiment-canary\n specRef: canary\n weight: 5\n
In the above example, during an update, the first step would start a baseline vs. canary experiment. When pods are ready (Experiment enters Running phase), the rollout would direct 5% of traffic to experiment-canary and 5% to experiment-baseline, leaving the remaining 90% of traffic to the old stack.
Note
When a weighted experiment step with traffic routing is used, a service is auto-created for each experiment template. The traffic routers use this service to send traffic to the experiment pods.
By default, the generated Service has the name of the ReplicaSet and inherits ports and selector from the specRef definition. It can be accessed in using the {{templates.baseline.replicaset.name}} or {{templates.canary.replicaset.name}} variables respectively.
"},{"location":"features/experiment/#experiment-service-creation-without-weight","title":"Experiment Service Creation without Weight","text":"If you don't want to use traffic routing for your Experiments but still want to create a Service for them, you can set a Service object which takes an optional Name, without having to set a Weight for them.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: guestbook\n labels:\n app: guestbook\nspec:\n...\nstrategy:\n canary:\n steps:\n - experiment:\n duration: 1h\n templates:\n - name: experiment-baseline\n specRef: stable\n service:\n name: test-service\n - name: experiment-canary\n specRef: canary\n
In the above example, during an update, the first step would start a baseline vs. canary experiment. This time, a service would be created for experiment-baseline even without setting a weight for it or traffic routing for the rollout.
"},{"location":"features/helm/","title":"Using Argo Rollouts with Helm","text":"Argo Rollouts will always respond to changes in Rollouts resources regardless of how the change was made. This means that Argo Rollouts is compatible with all templating solutions that you might use to manage your deployments.
Argo Rollouts manifests can be managed with the Helm package manager. If your Helm chart contains Rollout Resources, then as soon as you install/upgrade your chart, Argo Rollouts will take over and initiate the progressive delivery process.
Here is an example Rollout that is managed with Helm:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: {{ template \"helm-guestbook.fullname\" . }}\n labels:\n app: {{ template \"helm-guestbook.name\" . }}\n chart: {{ template \"helm-guestbook.chart\" . }}\n release: {{ .Release.Name }}\n heritage: {{ .Release.Service }}\nspec:\n replicas: {{ .Values.replicaCount }}\n revisionHistoryLimit: 3\n selector:\n matchLabels:\n app: {{ template \"helm-guestbook.name\" . }}\n release: {{ .Release.Name }}\n strategy:\n blueGreen:\n activeService: {{ template \"helm-guestbook.fullname\" . }}\n previewService: {{ template \"helm-guestbook.fullname\" . }}-preview\n template:\n metadata:\n labels:\n app: {{ template \"helm-guestbook.name\" . }}\n release: {{ .Release.Name }}\n spec:\n containers:\n - name: {{ .Chart.Name }}\n image: \"{{ .Values.image.repository }}:{{ .Values.image.tag }}\"\n imagePullPolicy: {{ .Values.image.pullPolicy }}\n ports:\n - name: http\n containerPort: 80\n protocol: TCP\n livenessProbe:\n httpGet:\n path: /\n port: http\n readinessProbe:\n httpGet:\n path: /\n port: http\n resources:\n{{ toYaml .Values.resources | indent 12 }}\n {{- with .Values.nodeSelector }}\n nodeSelector:\n{{ toYaml . | indent 8 }}\n {{- end }}\n {{- with .Values.affinity }}\n affinity:\n{{ toYaml . | indent 8 }}\n {{- end }}\n {{- with .Values.tolerations }}\n tolerations:\n{{ toYaml . | indent 8 }}\n {{- end }}\n
You can find the full example at https://github.com/argoproj/argo-rollouts/tree/master/examples/helm-blue-green.
"},{"location":"features/hpa-support/","title":"Horizontal Pod Autoscaling","text":"Horizontal Pod Autoscaling (HPA) automatically scales the number of pods in owned by a Kubernetes resource based on observed CPU utilization or user-configured metrics. In order to accomplish this behavior, HPA only supports resources with the scale endpoint enabled with a couple of required fields. The scale endpoint allows the HPA to understand the current state of a resource and modify the resource to scale it appropriately. Argo Rollouts added support for the scale endpoint in the 0.3.0 release. After being modified by the HPA, the Argo Rollouts controller is responsible for reconciling that change in replicas. Since the strategies within a Rollout are very different, the Argo Rollouts controller handles the scale endpoint differently for various strategies. Below is the behavior for the different strategies:
"},{"location":"features/hpa-support/#blue-green","title":"Blue Green","text":"The HPA will scale rollouts using the BlueGreen strategy using the metrics from the ReplicaSet receiving traffic from the active service. When the HPA changes the replicas count, the Argo Rollouts controller will first scale up the ReplicaSet receiving traffic from the active service before ReplicaSet receiving traffic from the preview service. The controller will scale up the ReplicaSet receiving traffic from the preview service to prepare it for when the rollout switches the preview to active. If there are no ReplicaSets receiving from the active service, the controller will use all the pods that match the base selector to determine scaling events. In that case, the controller will scale up the latest ReplicaSet to the new count and scale down the older ReplicaSets.
"},{"location":"features/hpa-support/#canary-replicaset-based","title":"Canary (ReplicaSet based)","text":"The HPA will scale rollouts using the Canary Strategy using the metrics of all the ReplicasSets within the rollout. Since the Argo Rollouts controller does not control the service that sends traffic to those ReplicaSets, it assumes that all the ReplicaSets in the rollout are receiving traffic.
"},{"location":"features/hpa-support/#example","title":"Example","text":"Below is an example of a Horizontal Pod Autoscaler that scales a rollout based on CPU metrics:
apiVersion: autoscaling/v1\nkind: HorizontalPodAutoscaler\nmetadata:\n name: hpa-rollout-example\nspec:\n maxReplicas: 6\n minReplicas: 2\n scaleTargetRef:\n apiVersion: argoproj.io/v1alpha1\n kind: Rollout\n name: example-rollout\n targetCPUUtilizationPercentage: 80\n
"},{"location":"features/hpa-support/#requirements","title":"Requirements","text":"In order for the HPA to manipulate the rollout, the Kubernetes cluster hosting the rollout CRD needs the subresources support for CRDs. This feature was introduced as alpha in Kubernetes version 1.10 and transitioned to beta in Kubernetes version 1.11. If a user wants to use HPA on v1.10, the Kubernetes Cluster operator will need to add a custom feature flag to the API server. After 1.10, the flag is turned on by default. Check out the following link for more information on setting the custom feature flag.
"},{"location":"features/kubectl-plugin/","title":"Kubectl Plugin","text":"Kubectl plugins are a way to extend the kubectl command to provide additional behavior. Generally, they are used to add new functionality to kubectl and automate scriptable workflows against a cluster. The official documentation on them is here.
Argo Rollouts offers a Kubectl plugin to enrich the experience with Rollouts, Experiments, and Analysis from the command line. It offers the ability to visualize the Argo Rollouts resources and run routine operations like promote or retry on those resources from the command.
"},{"location":"features/kubectl-plugin/#installation","title":"Installation","text":"See the installation guide for instructions on installing the plugin.
"},{"location":"features/kubectl-plugin/#usage","title":"Usage","text":"The best way to get information on the available Argo Rollouts kubectl plugin commands is by run kubectl argo rollouts. The plugin lists all the available commands that the tool can execute along with a description of each commend. All the plugin's commands interact with the Kubernetes API server and use KubeConfig credentials for authentication. Since the plugin leverages the KubeConfig of the user running the command, the plugin has the permissions of those configs.
Similar to kubectl, the plugin uses many of the same flags as the kubectl. For example, the kubectl argo rollouts get rollout canary-demo -w command starts a watch on the canary-demo rollout object similar to how the kubectl get deployment canary-demo -w command starts a watch on a deployment.
"},{"location":"features/kubectl-plugin/#visualizing-rollouts-and-experiments","title":"Visualizing Rollouts and Experiments","text":"In addition to encapsulating many routine commands, the Argo Rollouts kubectl plugin supports visualizing rollouts and experiments with the get command. The get command provides a clean representation of either the rollouts or the experiments running in a cluster. It returns a bunch of metadata on a resource and a tree view of the child resources created by the parent. As an example, here is a rollout retrieved with a get command:
Here is a table to explain some of the icons on the tree view:
Icon Kind \u27f3 Rollout \u03a3 Experiment \u03b1 AnalysisRun # Revision \u29c9 ReplicaSet \u25a1 Pod \u229e Job If the get command includes the watch flag (-w or --watch), the terminal updates as the rollouts or experiment progress highlighting the progress.
"},{"location":"features/kustomize/","title":"Kustomize Integration","text":"Kustomize can be extended to understand CRD objects through the use of transformer configs. Using transformer configs, kustomize can be \"taught\" about the structure of a Rollout object and leverage kustomize features such as ConfigMap/Secret generators, variable references, and common labels & annotations. To use Rollouts with kustomize:
-
Download rollout-transform.yaml into your kustomize directory.
-
Include rollout-transform.yaml in your kustomize configurations section:
kind: Kustomization\napiVersion: kustomize.config.k8s.io/v1beta1\n\nconfigurations:\n- rollout-transform.yaml\n
An example kustomize app demonstrating the ability to use transformers with Rollouts can be seen here.
- With Kustomize 3.6.1 it is possible to reference the configuration directly from a remote resource:
configurations:\n - https://argoproj.github.io/argo-rollouts/features/kustomize/rollout-transform.yaml\n
- With Kustomize 4.5.5 kustomize can use kubernetes OpenAPI data to get merge key and patch strategy information about resource types. For example, given the following rollout:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollout-canary\nspec:\n strategy:\n canary:\n steps:\n # detail of the canary steps is omitted\n template:\n metadata:\n labels:\n app: rollout-canary\n spec:\n containers:\n - name: rollouts-demo\n image: argoproj/rollouts-demo:blue\n imagePullPolicy: Always\n ports:\n - containerPort: 8080\n
user can update the Rollout via a patch in a kustomization file, to change the image to nginx
apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n- rollout-canary.yaml\n\nopenapi:\n path: https://raw.githubusercontent.com/argoproj/argo-schema-generator/main/schema/argo_all_k8s_kustomize_schema.json\n\npatches:\n- patch: |-\n apiVersion: argoproj.io/v1alpha1\n kind: Rollout\n metadata:\n name: rollout-canary\n spec:\n template:\n spec:\n containers:\n - name: rollouts-demo\n image: nginx\n
The OpenAPI data is auto-generated and defined in this file.
An example kustomize app demonstrating the ability to use OpenAPI data with Rollouts can be seen here.
"},{"location":"features/notifications/","title":"Notifications","text":"Important
Available since v1.1
Argo Rollouts provides notifications powered by the Notifications Engine. Controller administrators can leverage flexible systems of triggers and templates to configure notifications requested by the end users. The end-users can subscribe to the configured triggers by adding an annotation to the Rollout objects.
"},{"location":"features/notifications/#configuration","title":"Configuration","text":"The trigger defines the condition when the notification should be sent as well as the notification content template. Default Argo Rollouts comes with a list of built-in triggers that cover the most important events of Argo Rollout live-cycle. Both triggers and templates are configured in the argo-rollouts-notification-configmap ConfigMap. In order to get started quickly, you can use pre-configured notification templates defined in notifications-install.yaml.
If you are leveraging Kustomize it is recommended to include notifications-install.yaml as a remote resource into your kustomization.yaml file:
apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n- https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml\n- https://github.com/argoproj/argo-rollouts/releases/latest/download/notifications-install.yaml\n
After including the argo-rollouts-notification-configmap ConfigMap the administrator needs to configure integration with the required notifications service such as Slack or MS Teams. An example below demonstrates Slack integration:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n # detail of the templates is omitted\n # detail of the triggers is omitted\n service.slack: |\n token: $slack-token\n---\napiVersion: v1\nkind: Secret\nmetadata:\n name: argo-rollouts-notification-secret\nstringData:\n slack-token: <my-slack-token>\n
Learn more about supported services and configuration settings in services documentation.
"},{"location":"features/notifications/#namespace-based-configuration","title":"Namespace based configuration","text":"Important
Available since v1.6
A common installation method for Argo Rollouts is to install it in a dedicated namespace to manage a whole cluster. In this case, the administrator is the only person who can configure notifications in that namespace generally. However, in some cases, it is required to allow end-users to configure notifications for their Rollout resources. For example, the end-user can configure notifications for their Rollouts in the namespace where they have access to and their rollout is running in.
To use this feature all you need to do is create the same configmap named argo-rollouts-notification-configmap and possibly a secret argo-rollouts-notification-secret in the namespace where the rollout object lives. When it is configured this way the controller will send notifications using both the controller level configuration (the configmap located in the same namespaces as the controller) as well as the configmap located in the same namespaces where the rollout object is at.
To enable you need to add a flag to the controller --self-service-notification-enabled
"},{"location":"features/notifications/#default-trigger-templates","title":"Default Trigger templates","text":"Currently the following triggers have built-in templates.
on-rollout-completed when a rollout is finished and all its steps are completedon-rollout-step-completed when an individual step inside a rollout definition is completedon-rollout-updated when a rollout definition is changedon-scaling-replica-set when the number of replicas in a rollout is changed
"},{"location":"features/notifications/#subscriptions","title":"Subscriptions","text":"The end-users can start leveraging notifications using notifications.argoproj.io/subscribe.<trigger>.<service>: <recipient> annotation. For example, the following annotation subscribes two Slack channels to notifications about canary rollout step completion:
---\napiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollout-canary\n annotations:\n notifications.argoproj.io/subscribe.on-rollout-step-completed.slack: my-channel1;my-channel2\n
Annotation key consists of following parts:
on-rollout-step-completed - trigger nameslack - notification service namemy-channel1;my-channel2 - a semicolon separated list of recipients
"},{"location":"features/notifications/#customization","title":"Customization","text":"The Rollout administrator can customize the notifications by configuring notification templates and custom triggers in argo-rollouts-notification-configmap ConfigMap.
"},{"location":"features/notifications/#templates","title":"Templates","text":"The notification template is a stateless function that generates the notification content. The template is leveraging html/template golang package. It is meant to be reusable and can be referenced by multiple triggers.
An example below demonstrates a sample template:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n template.my-purple-template: |\n message: |\n Rollout {{.rollout.metadata.name}} has purple image\n slack:\n attachments: |\n [{\n \"title\": \"{{ .rollout.metadata.name}}\",\n \"color\": \"#800080\"\n }]\n
Each template has access to the following fields:
rollout holds the rollout object.recipient holds the recipient name.
The message field of the template definition allows creating a basic notification for any notification service. You can leverage notification service-specific fields to create complex notifications. For example using service-specific you can add blocks and attachments for Slack, subject for Email or URL path, and body for Webhook. See corresponding service documentation for more information.
"},{"location":"features/notifications/#custom-triggers","title":"Custom Triggers","text":"In addition to custom notification template administrator and configure custom triggers. Custom trigger defines the condition when the notification should be sent. The definition includes name, condition and notification templates reference. The condition is a predicate expression that returns true if the notification should be sent. The trigger condition evaluation is powered by antonmedv/expr. The condition language syntax is described at Language-Definition.md.
The trigger is configured in argo-rollouts-notification-configmap ConfigMap. For example the following trigger sends a notification when rollout pod spec uses argoproj/rollouts-demo:purple image:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n trigger.on-purple: |\n - send: [my-purple-template]\n when: rollout.spec.template.spec.containers[0].image == 'argoproj/rollouts-demo:purple'\n
Each condition might use several templates. Typically each template is responsible for generating a service-specific notification part.
"},{"location":"features/notifications/#notification-metrics","title":"Notification Metrics","text":"The following prometheus metrics are emitted when notifications are enabled in argo-rollouts. - notification_send_success is a counter that measures how many times the notification is sent successfully. - notification_send_error is a counter that measures how many times the notification failed to send. - notification_send is a histogram that measures performance of sending notification.
"},{"location":"features/restart/","title":"Restarting Rollout Pods","text":"For various reasons, applications often need to be restarted, e.g. for hygiene purposes or to force startup logic to occur again such as reloading of a modified Secret. In these scenarios, it is undesirable to go through an entire blue-green or canary update process. Argo Rollouts supports the ability to restart all of its Pods by performing a rolling recreate of all the Pods in a Rollout while skipping the regular BlueGreen or Canary update strategy.
"},{"location":"features/restart/#how-it-works","title":"How it works","text":"A rollout can be restarted via the kubectl plugin, using the restart command:
kubectl-argo-rollouts restart ROLLOUT\n
Alternatively, if Rollouts is used with Argo CD, the there is a bundled \"restart\" action which can be performed via the Argo CD UI or CLI:
argocd app actions run my-app restart --kind Rollout --resource-name my-rollout\n
Both of these mechanisms updates the Rollout's .spec.restartAt to the current time in the form of a RFC 3339 formatted UTC string (e.g. 2020-03-30T21:19:35Z), which indicates to the Rollout controller that all of a Rollout's Pods should have been created after this timestamp.
During a restart, the controller iterates through each ReplicaSet to see if all the Pods have a creation timestamp which is newer than the restartAt time. For every pod older than the restartAt timestamp, the Pod will be evicted, allowing the ReplicaSet to replace the pod with a recreated one.
To prevent too many Pods from restarting at once, the controller limits itself to deleting up to maxUnavailable Pods at a time. Secondly, since pods are evicted and not deleted, the restart process will honor any PodDisruptionBudgets which are in place. The controller restarts ReplicaSets in the following order: 1. stable ReplicaSet 2. current ReplicaSet 3. all other ReplicaSets beginning with the oldest
If a Rollout's pod template spec (spec.template) is modified in the middle of a restart, the restart is canceled, and the normal blue-green or canary update will occur.
Note: Unlike deployments, where a \"restart\" is nothing but a normal rolling upgrade that happened to be triggered by a timestamp in the pod spec annotation, Argo Rollouts facilitates restarts by terminating pods and allowing the existing ReplicaSet to replace the terminated pods. This design choice was made in order to allow a restart to occur even when a Rollout was in the middle of a long-running blue-green/canary update (e.g. a paused canary). However, some consequences of this are:
- Restarting a Rollout which has a single replica will cause downtime since Argo Rollouts needs to terminate the pod in order to replace it.
- Restarting a rollout will be slower than a deployment's rolling update, since maxSurge is not used to bring up newer pods faster.
- maxUnavailable will be used to restart multiple pods at a time (starting in v0.10). But if maxUnavailable pods is 0, the controller will still restart pods one at a time.
"},{"location":"features/restart/#scheduled-restarts","title":"Scheduled Restarts","text":"Users can schedule a restart on their Rollout by setting the .spec.restartAt field to a time in the future. The controller only starts the restart after the current time is after the restartAt time.
"},{"location":"features/rollback/","title":"Rollback Windows","text":"Important
Available for blue-green and canary rollouts since v1.4
By default, when an older Rollout manifest is re-applied, the controller treats it the same as a spec change, and will execute the full list of steps, and perform analysis too. There are two exceptions to this rule: 1. the controller detects if it is moving back to a blue-green ReplicaSet which exists and is still scaled up (within its scaleDownDelay) 2. the controller detects it is moving back to the canary's \"stable\" ReplicaSet, and the upgrade had not yet completed.
It is often undesirable to re-run analysis and steps for a rollout, when the desired behavior is to rollback as soon as possible. To help with this, a rollback window feature allows users to indicate that the promotion to the ReplicaSet within the window will skip all steps.
Example:
spec:\n rollbackWindow:\n revisions: 3\n\n revisionHistoryLimit: 5\n
Assume a linear revision history: 1, 2, 3, 4, 5 (current). A rollback from revision 5 back to 4 or 3 will fall within the window, so it will be fast tracked.
"},{"location":"features/scaledown-aborted-rs/","title":"Scaledown New Replicaset on Aborted Rollout","text":"Upon an aborted update, we may scale down the new replicaset for all strategies. Users can then choose to leave the new replicaset scaled up indefinitely by setting abortScaleDownDelaySeconds to 0, or adjust the value to something larger (or smaller).
The following table summarizes the behavior under combinations of rollout strategy and abortScaleDownDelaySeconds. Note that abortScaleDownDelaySeconds is not applicable to argo-rollouts v1.0. abortScaleDownDelaySeconds = nil is the default, which means in v1.1 across all rollout strategies, the new replicaset is scaled down in 30 seconds on abort by default.
strategy v1.0 behavior abortScaleDownDelaySeconds v1.1 behavior blue-green does not scale down nil scales down after 30 seconds blue-green does not scale down 0 does not scale down blue-green does not scale down N scales down after N seconds basic canary rolling update back to stable N/A rolling update back to stable canary w/ traffic routing scales down immediately nil scales down after 30 seconds canary w/ traffic routing scales down immediately 0 does not scale down canary w/ traffic routing scales down immediately N scales down after N seconds canary w/ traffic routing + setCanaryScale does not scale down (bug) * should behave like canary w/ traffic routing"},{"location":"features/specification/","title":"Rollout Specification","text":"The following describes all the available fields of a rollout:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: example-rollout-canary\nspec:\n # Number of desired pods.\n # Defaults to 1.\n replicas: 5\n analysis:\n # limits the number of successful analysis runs and experiments to be stored in a history\n # Defaults to 5.\n successfulRunHistoryLimit: 10\n # limits the number of unsuccessful analysis runs and experiments to be stored in a history. \n # Stages for unsuccessful: \"Error\", \"Failed\", \"Inconclusive\"\n # Defaults to 5.\n unsuccessfulRunHistoryLimit: 10\n\n # Label selector for pods. Existing ReplicaSets whose pods are selected by\n # this will be the ones affected by this rollout. It must match the pod\n # template's labels.\n selector:\n matchLabels:\n app: guestbook\n\n # WorkloadRef holds a references to a workload that provides Pod template \n # (e.g. Deployment). If used, then do not use Rollout template property.\n workloadRef: \n apiVersion: apps/v1\n kind: Deployment\n name: rollout-ref-deployment\n # Specifies if the workload (Deployment) is scaled down after migrating to Rollout.\n # The possible options are:\n # \"never\": the Deployment is not scaled down\n # \"onsuccess\": the Deployment is scaled down after the Rollout becomes healthy\n # \"progressively\": as the Rollout is scaled up the Deployment is scaled down\n # If the Rollout fails the Deployment will be scaled back up.\n scaleDown: never|onsuccess|progressively\n\n # Template describes the pods that will be created. Same as deployment.\n # If used, then do not use Rollout workloadRef property. \n template:\n spec:\n containers:\n - name: guestbook\n image: argoproj/rollouts-demo:blue\n\n # Minimum number of seconds for which a newly created pod should be ready\n # without any of its container crashing, for it to be considered available.\n # Defaults to 0 (pod will be considered available as soon as it is ready)\n minReadySeconds: 30\n\n # The number of old ReplicaSets to retain.\n # Defaults to 10\n revisionHistoryLimit: 3\n\n # Pause allows a user to manually pause a rollout at any time. A rollout\n # will not advance through its steps while it is manually paused, but HPA\n # auto-scaling will still occur. Typically not explicitly set the manifest,\n # but controlled via tools (e.g. kubectl argo rollouts pause). If true at\n # initial creation of Rollout, replicas are not scaled up automatically\n # from zero unless manually promoted.\n paused: true\n\n # The maximum time in seconds in which a rollout must make progress during\n # an update, before it is considered to be failed. Argo Rollouts will\n # continue to process failed rollouts and a condition with a\n # ProgressDeadlineExceeded reason will be surfaced in the rollout status.\n # Note that progress will not be estimated during the time a rollout is\n # paused.\n # Defaults to 600s\n progressDeadlineSeconds: 600\n\n # Whether to abort the update when ProgressDeadlineSeconds is exceeded.\n # Optional and default is false.\n progressDeadlineAbort: false\n\n # UTC timestamp in which a Rollout should sequentially restart all of\n # its pods. Used by the `kubectl argo rollouts restart ROLLOUT` command.\n # The controller will ensure all pods have a creationTimestamp greater\n # than or equal to this value.\n restartAt: \"2020-03-30T21:19:35Z\"\n\n # The rollback window provides a way to fast track deployments to\n # previously deployed versions.\n # Optional, and by default is not set.\n rollbackWindow:\n revisions: 3\n\n strategy:\n\n # Blue-green update strategy\n blueGreen:\n\n # Reference to service that the rollout modifies as the active service.\n # Required.\n activeService: active-service\n\n # Pre-promotion analysis run which performs analysis before the service\n # cutover. +optional\n prePromotionAnalysis:\n templates:\n - templateName: success-rate\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n\n # Post-promotion analysis run which performs analysis after the service\n # cutover. +optional\n postPromotionAnalysis:\n templates:\n - templateName: success-rate\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n\n # Name of the service that the rollout modifies as the preview service.\n # +optional\n previewService: preview-service\n\n # The number of replicas to run under the preview service before the\n # switchover. Once the rollout is resumed the new ReplicaSet will be fully\n # scaled up before the switch occurs +optional\n previewReplicaCount: 1\n\n # Indicates if the rollout should automatically promote the new ReplicaSet\n # to the active service or enter a paused state. If not specified, the\n # default value is true. +optional\n autoPromotionEnabled: false\n\n # Automatically promotes the current ReplicaSet to active after the\n # specified pause delay in seconds after the ReplicaSet becomes ready.\n # If omitted, the Rollout enters and remains in a paused state until\n # manually resumed by resetting spec.Paused to false. +optional\n autoPromotionSeconds: 30\n\n # Adds a delay before scaling down the previous ReplicaSet. If omitted,\n # the Rollout waits 30 seconds before scaling down the previous ReplicaSet.\n # A minimum of 30 seconds is recommended to ensure IP table propagation\n # across the nodes in a cluster.\n scaleDownDelaySeconds: 30\n\n # Limits the number of old RS that can run at once before getting scaled\n # down. Defaults to nil\n scaleDownDelayRevisionLimit: 2\n\n # Add a delay in second before scaling down the preview replicaset\n # if update is aborted. 0 means not to scale down. Default is 30 second\n abortScaleDownDelaySeconds: 30\n\n # Anti Affinity configuration between desired and previous ReplicaSet.\n # Only one must be specified\n antiAffinity:\n requiredDuringSchedulingIgnoredDuringExecution: {}\n preferredDuringSchedulingIgnoredDuringExecution:\n weight: 1 # Between 1 - 100\n\n # activeMetadata will be merged and updated in-place into the ReplicaSet's spec.template.metadata\n # of the active pods. +optional\n activeMetadata:\n labels:\n role: active\n\n # Metadata which will be attached to the preview pods only during their preview phase.\n # +optional\n previewMetadata:\n labels:\n role: preview\n\n # Canary update strategy\n canary:\n\n # Reference to a service which the controller will update to select\n # canary pods. Required for traffic routing.\n canaryService: canary-service\n\n # Reference to a service which the controller will update to select\n # stable pods. Required for traffic routing.\n stableService: stable-service\n\n # Metadata which will be attached to the canary pods. This metadata will\n # only exist during an update, since there are no canary pods in a fully\n # promoted rollout.\n canaryMetadata:\n annotations:\n role: canary\n labels:\n role: canary\n\n # metadata which will be attached to the stable pods\n stableMetadata:\n annotations:\n role: stable\n labels:\n role: stable\n\n # The maximum number of pods that can be unavailable during the update.\n # Value can be an absolute number (ex: 5) or a percentage of total pods\n # at the start of update (ex: 10%). Absolute number is calculated from\n # percentage by rounding down. This can not be 0 if MaxSurge is 0. By\n # default, a fixed value of 1 is used. Example: when this is set to 30%,\n # the old RC can be scaled down by 30% immediately when the rolling\n # update starts. Once new pods are ready, old RC can be scaled down\n # further, followed by scaling up the new RC, ensuring that at least 70%\n # of original number of pods are available at all times during the\n # update. +optional\n maxUnavailable: 1\n\n # The maximum number of pods that can be scheduled above the original\n # number of pods. Value can be an absolute number (ex: 5) or a\n # percentage of total pods at the start of the update (ex: 10%). This\n # can not be 0 if MaxUnavailable is 0. Absolute number is calculated\n # from percentage by rounding up. By default, a value of 1 is used.\n # Example: when this is set to 30%, the new RC can be scaled up by 30%\n # immediately when the rolling update starts. Once old pods have been\n # killed, new RC can be scaled up further, ensuring that total number\n # of pods running at any time during the update is at most 130% of\n # original pods. +optional\n maxSurge: \"20%\"\n\n # Adds a delay before scaling down the previous ReplicaSet when the\n # canary strategy is used with traffic routing (default 30 seconds).\n # A delay in scaling down the previous ReplicaSet is needed after\n # switching the stable service selector to point to the new ReplicaSet,\n # in order to give time for traffic providers to re-target the new pods.\n # This value is ignored with basic, replica-weighted canary without\n # traffic routing.\n scaleDownDelaySeconds: 30\n\n # The minimum number of pods that will be requested for each ReplicaSet\n # when using traffic routed canary. This is to ensure high availability\n # of each ReplicaSet. Defaults to 1. +optional\n minPodsPerReplicaSet: 2\n\n # Limits the number of old RS that can run at one time before getting\n # scaled down. Defaults to nil\n scaleDownDelayRevisionLimit: 2\n\n # Background analysis to run during a rollout update. Skipped upon\n # initial deploy of a rollout. +optional\n analysis:\n templates:\n - templateName: success-rate\n args:\n - name: service-name\n value: guestbook-svc.default.svc.cluster.local\n\n # valueFrom.podTemplateHashValue is a convenience to supply the\n # rollouts-pod-template-hash value of either the Stable ReplicaSet\n # or the Latest ReplicaSet\n - name: stable-hash\n valueFrom:\n podTemplateHashValue: Stable\n - name: latest-hash\n valueFrom:\n podTemplateHashValue: Latest\n\n # valueFrom.fieldRef allows metadata about the rollout to be\n # supplied as arguments to analysis.\n - name: region\n valueFrom:\n fieldRef:\n fieldPath: metadata.labels['region']\n\n # Steps define sequence of steps to take during an update of the\n # canary. Skipped upon initial deploy of a rollout. +optional\n steps:\n\n # Sets the ratio of canary ReplicaSet to 20%\n - setWeight: 20\n\n # Pauses the rollout for an hour. Supported units: s, m, h\n - pause:\n duration: 1h\n\n # Pauses indefinitely until manually resumed\n - pause: {}\n\n # set canary scale to a explicit count without changing traffic weight\n # (supported only with trafficRouting)\n - setCanaryScale:\n replicas: 3\n\n # set canary scale to a percentage of spec.replicas without changing traffic weight\n # (supported only with trafficRouting)\n - setCanaryScale:\n weight: 25\n\n # set canary scale to match the canary traffic weight (default behavior)\n - setCanaryScale:\n matchTrafficWeight: true\n\n # Sets header based route with specified header values\n # Setting header based route will send all traffic to the canary for the requests \n # with a specified header, in this case request header \"version\":\"2\"\n # (supported only with trafficRouting, for Istio only at the moment)\n - setHeaderRoute:\n # Name of the route that will be created by argo rollouts this must also be configured\n # in spec.strategy.canary.trafficRouting.managedRoutes\n name: \"header-route-1\"\n # The matching rules for the header route, if this is missing it acts as a removal of the route.\n match:\n # headerName The name of the header to apply the match rules to.\n - headerName: \"version\"\n # headerValue must contain exactly one field of exact, regex, or prefix. Not all traffic routers support \n # all types\n headerValue:\n # Exact will only match if the header value is exactly the same\n exact: \"2\"\n # Will match the rule if the regular expression matches\n regex: \"2.0.(.*)\"\n # prefix will be a prefix match of the header value\n prefix: \"2.0\"\n\n # Sets up a mirror/shadow based route with the specified match rules\n # The traffic will be mirrored at the configured percentage to the canary service\n # during the rollout\n # (supported only with trafficRouting, for Istio only at the moment)\n - setMirrorRoute:\n # Name of the route that will be created by argo rollouts this must also be configured\n # in spec.strategy.canary.trafficRouting.managedRoutes\n name: \"header-route-1\"\n # The percentage of the matched traffic to mirror to the canary\n percentage: 100\n # The matching rules for the header route, if this is missing it acts as a removal of the route.\n # All conditions inside a single match block have AND semantics, while the list of match blocks have OR semantics.\n # Each type within a match (method, path, headers) must have one and only one match type (exact, regex, prefix)\n # Not all match types (exact, regex, prefix) will be supported by all traffic routers.\n match:\n - method: # What HTTP method to match\n exact: \"GET\"\n regex: \"P.*\"\n prefix: \"POST\"\n path: # What HTTP url paths to match.\n exact: \"/test\"\n regex: \"/test/.*\"\n prefix: \"/\"\n headers:\n agent-1b: # What HTTP header name to use in the match.\n exact: \"firefox\"\n regex: \"firefox2(.*)\"\n prefix: \"firefox\"\n\n # an inline analysis step\n - analysis:\n templates:\n - templateName: success-rate\n\n # an inline experiment step\n - experiment:\n duration: 1h\n templates:\n - name: baseline\n specRef: stable\n # optional, creates a service for the experiment if set\n service:\n # optional, service: {} is also acceptable if name is not included\n name: test-service\n - name: canary\n specRef: canary\n # optional, set the weight of traffic routed to this version\n weight: 10\n analyses:\n - name : mann-whitney\n templateName: mann-whitney\n # Metadata which will be attached to the AnalysisRun.\n analysisRunMetadata:\n labels:\n app.service.io/analysisType: smoke-test\n annotations:\n link.argocd.argoproj.io/external-link: http://my-loggin-platform.com/pre-generated-link\n\n # Anti-affinity configuration between desired and previous ReplicaSet.\n # Only one must be specified.\n antiAffinity:\n requiredDuringSchedulingIgnoredDuringExecution: {}\n preferredDuringSchedulingIgnoredDuringExecution:\n weight: 1 # Between 1 - 100\n\n # Traffic routing specifies the ingress controller or service mesh\n # configuration to achieve advanced traffic splitting. If omitted,\n # will achieve traffic split via a weighted replica counts between\n # the canary and stable ReplicaSet.\n trafficRouting:\n # Supports nginx and plugins only: This lets you control the denominator or total weight of traffic.\n # The total weight of traffic. If unspecified, it defaults to 100\n maxTrafficWeight: 1000\n # This is a list of routes that Argo Rollouts has the rights to manage it is currently only required for\n # setMirrorRoute and setHeaderRoute. The order of managedRoutes array also sets the precedence of the route\n # in the traffic router. Argo Rollouts will place these routes in the order specified above any routes already\n # defined in the used traffic router if something exists. The names here must match the names from the \n # setHeaderRoute and setMirrorRoute steps.\n managedRoutes:\n - name: set-header\n - name: mirror-route\n # Istio traffic routing configuration\n istio:\n # Either virtualService or virtualServices can be configured.\n virtualService: \n name: rollout-vsvc # required\n routes:\n - primary # optional if there is a single route in VirtualService, required otherwise\n virtualServices:\n # One or more virtualServices can be configured\n - name: rollouts-vsvc1 # required\n routes:\n - primary # optional if there is a single route in VirtualService, required otherwise\n - name: rollouts-vsvc2 # required\n routes:\n - secondary # optional if there is a single route in VirtualService, required otherwise\n\n # NGINX Ingress Controller routing configuration\n nginx:\n # Either stableIngress or stableIngresses must be configured, but not both.\n stableIngress: primary-ingress\n stableIngresses:\n - primary-ingress\n - secondary-ingress\n - tertiary-ingress\n annotationPrefix: customingress.nginx.ingress.kubernetes.io # optional\n additionalIngressAnnotations: # optional\n canary-by-header: X-Canary\n canary-by-header-value: iwantsit\n\n # ALB Ingress Controller routing configuration\n alb:\n ingress: ingress # required\n servicePort: 443 # required\n annotationPrefix: custom.alb.ingress.kubernetes.io # optional\n\n # Service Mesh Interface routing configuration\n smi:\n rootService: root-svc # optional\n trafficSplitName: rollout-example-traffic-split # optional\n\n # Add a delay in second before scaling down the canary pods when update\n # is aborted for canary strategy with traffic routing (not applicable for basic canary).\n # 0 means canary pods are not scaled down. Default is 30 seconds.\n abortScaleDownDelaySeconds: 30\n\nstatus:\n pauseConditions:\n - reason: StepPause\n startTime: 2019-10-00T1234\n - reason: BlueGreenPause\n startTime: 2019-10-00T1234\n - reason: AnalysisRunInconclusive\n startTime: 2019-10-00T1234 \n
"},{"location":"features/specification/#examples","title":"Examples","text":"You can find examples of Rollouts at:
- The example directory
- The Argo Rollouts Demo application
"},{"location":"features/vpa-support/","title":"Vertical Pod Autoscaling","text":"Vertical Pod Autoscaling (VPA) reduces the maintenance cost and improve utilization of cluster resources by automating configuration of resource requirements.
"},{"location":"features/vpa-support/#vpa-modes","title":"VPA modes","text":"There are four modes in which VPAs operate
-
\"Auto\": VPA assigns resource requests on pod creation as well as updates them on existing pods using the preferred update mechanism. Currently this is equivalent to \"Recreate\" (see below). Once restart free (\"in-place\") update of pod requests is available, it may be used as the preferred update mechanism by the \"Auto\" mode. NOTE: This feature of VPA is experimental and may cause downtime for your applications.
-
\"Recreate\": VPA assigns resource requests on pod creation as well as updates them on existing pods by evicting them when the requested resources differ significantly from the new recommendation (respecting the Pod Disruption Budget, if defined). This mode should be used rarely, only if you need to ensure that the pods are restarted whenever the resource request changes. Otherwise prefer the \"Auto\" mode which may take advantage of restart free updates once they are available. NOTE: This feature of VPA is experimental and may cause downtime for your applications.
-
\"Initial\": VPA only assigns resource requests on pod creation and never changes them later.
-
\"Off\": VPA does not automatically change resource requirements of the pods. The recommendations are calculated and can be inspected in the VPA object.
"},{"location":"features/vpa-support/#example","title":"Example","text":"Below is an example of a Vertical Pod Autoscaler with Argo-Rollouts.
Rollout sample app:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: vpa-demo-rollout\n namespace: test-vpa\nspec:\n replicas: 5\n strategy:\n canary:\n steps:\n - setWeight: 20\n - pause: {duration: 10}\n - setWeight: 40\n - pause: {duration: 10}\n - setWeight: 60\n - pause: {duration: 10}\n - setWeight: 80\n - pause: {duration: 10}\n revisionHistoryLimit: 10\n selector:\n matchLabels:\n app: vpa-demo-rollout\n template:\n metadata:\n labels:\n app: vpa-demo-rollout\n spec:\n containers:\n - name: vpa-demo-rollout\n image: ravihari/nginx:v1\n ports:\n - containerPort: 80\n resources:\n requests:\n cpu: \"5m\" \n memory: \"5Mi\" \n
VPA configuration for Rollout sample app:
apiVersion: \"autoscaling.k8s.io/v1beta2\"\nkind: VerticalPodAutoscaler \nmetadata: \n name: vpa-rollout-example \n namespace: test-vpa \nspec: \n targetRef: \n apiVersion: \"argoproj.io/v1alpha1\" \n kind: Rollout \n name: vpa-demo-rollout \n updatePolicy: \n updateMode: \"Auto\" \n resourcePolicy: \n containerPolicies: \n - containerName: '*' \n minAllowed: \n cpu: 5m \n memory: 5Mi \n maxAllowed: \n cpu: 1 \n memory: 500Mi \n controlledResources: [\"cpu\", \"memory\"] \n
Describe VPA when initially deployed we donot see recommendations as it will take few mins.
Name: kubengix-vpa\nNamespace: test-vpa\nLabels: <none>\nAnnotations: <none>\nAPI Version: autoscaling.k8s.io/v1\nKind: VerticalPodAutoscaler\nMetadata:\n Creation Timestamp: 2022-03-14T12:54:06Z\n Generation: 1\n Managed Fields:\n API Version: autoscaling.k8s.io/v1beta2\n Fields Type: FieldsV1\n fieldsV1:\n f:metadata:\n f:annotations:\n .:\n f:kubectl.kubernetes.io/last-applied-configuration:\n f:spec:\n .:\n f:resourcePolicy:\n .:\n f:containerPolicies:\n f:targetRef:\n .:\n f:apiVersion:\n f:kind:\n f:name:\n f:updatePolicy:\n .:\n f:updateMode:\n Manager: kubectl-client-side-apply\n Operation: Update\n Time: 2022-03-14T12:54:06Z\n Resource Version: 3886\n UID: 4ac64e4c-c84b-478e-92e4-5f072f985971\nSpec:\n Resource Policy:\n Container Policies:\n Container Name: *\n Controlled Resources:\n cpu\n memory\n Max Allowed:\n Cpu: 1\n Memory: 500Mi\n Min Allowed:\n Cpu: 5m\n Memory: 5Mi\n Target Ref:\n API Version: argoproj.io/v1alpha1\n Kind: Rollout\n Name: vpa-demo-rollout\n Update Policy:\n Update Mode: Auto\nEvents: <none>\n
After few minutes when VPA starts to process and provide recommendation:
Name: kubengix-vpa\nNamespace: test-vpa\nLabels: <none>\nAnnotations: <none>\nAPI Version: autoscaling.k8s.io/v1\nKind: VerticalPodAutoscaler\nMetadata:\n Creation Timestamp: 2022-03-14T12:54:06Z\n Generation: 2\n Managed Fields:\n API Version: autoscaling.k8s.io/v1beta2\n Fields Type: FieldsV1\n fieldsV1:\n f:metadata:\n f:annotations:\n .:\n f:kubectl.kubernetes.io/last-applied-configuration:\n f:spec:\n .:\n f:resourcePolicy:\n .:\n f:containerPolicies:\n f:targetRef:\n .:\n f:apiVersion:\n f:kind:\n f:name:\n f:updatePolicy:\n .:\n f:updateMode:\n Manager: kubectl-client-side-apply\n Operation: Update\n Time: 2022-03-14T12:54:06Z\n API Version: autoscaling.k8s.io/v1\n Fields Type: FieldsV1\n fieldsV1:\n f:status:\n .:\n f:conditions:\n f:recommendation:\n .:\n f:containerRecommendations:\n Manager: recommender\n Operation: Update\n Time: 2022-03-14T12:54:52Z\n Resource Version: 3950\n UID: 4ac64e4c-c84b-478e-92e4-5f072f985971\nSpec:\n Resource Policy:\n Container Policies:\n Container Name: *\n Controlled Resources:\n cpu\n memory\n Max Allowed:\n Cpu: 1\n Memory: 500Mi\n Min Allowed:\n Cpu: 5m\n Memory: 5Mi\n Target Ref:\n API Version: argoproj.io/v1alpha1\n Kind: Rollout\n Name: vpa-demo-rollout\n Update Policy:\n Update Mode: Auto\nStatus:\n Conditions:\n Last Transition Time: 2022-03-14T12:54:52Z\n Status: True\n Type: RecommendationProvided\n Recommendation:\n Container Recommendations:\n Container Name: vpa-demo-rollout\n Lower Bound:\n Cpu: 25m\n Memory: 262144k\n Target:\n Cpu: 25m\n Memory: 262144k\n Uncapped Target:\n Cpu: 25m\n Memory: 262144k\n Upper Bound:\n Cpu: 1\n Memory: 500Mi\nEvents: <none>\n
Here we see the recommendation for cpu, memory with lowerbound, upper bound, Target etc., are provided. If we check the status of the pods.. the older pods with initial configuration would get terminated and newer pods get created.
# kubectl get po -n test-vpa -w \nNAME READY STATUS RESTARTS AGE\nvpa-demo-rollout-f5df6d577-65f26 1/1 Running 0 17m\nvpa-demo-rollout-f5df6d577-d55cx 1/1 Running 0 17m\nvpa-demo-rollout-f5df6d577-fdpn2 1/1 Running 0 17m\nvpa-demo-rollout-f5df6d577-jg2pw 1/1 Running 0 17m\nvpa-demo-rollout-f5df6d577-vlx5x 1/1 Running 0 17m\n...\n\nvpa-demo-rollout-f5df6d577-jg2pw 1/1 Terminating 0 17m\nvpa-demo-rollout-f5df6d577-vlx5x 1/1 Terminating 0 17m\nvpa-demo-rollout-f5df6d577-jg2pw 0/1 Terminating 0 18m\nvpa-demo-rollout-f5df6d577-vlx5x 0/1 Terminating 0 18m\nvpa-demo-rollout-f5df6d577-w7tx4 0/1 Pending 0 0s\nvpa-demo-rollout-f5df6d577-w7tx4 0/1 Pending 0 0s\nvpa-demo-rollout-f5df6d577-w7tx4 0/1 ContainerCreating 0 0s\nvpa-demo-rollout-f5df6d577-vdlqq 0/1 Pending 0 0s\nvpa-demo-rollout-f5df6d577-vdlqq 0/1 Pending 0 1s\nvpa-demo-rollout-f5df6d577-jg2pw 0/1 Terminating 0 18m\nvpa-demo-rollout-f5df6d577-jg2pw 0/1 Terminating 0 18m\nvpa-demo-rollout-f5df6d577-vdlqq 0/1 ContainerCreating 0 1s\nvpa-demo-rollout-f5df6d577-w7tx4 1/1 Running 0 6s\nvpa-demo-rollout-f5df6d577-vdlqq 1/1 Running 0 7s\nvpa-demo-rollout-f5df6d577-vlx5x 0/1 Terminating 0 18m\nvpa-demo-rollout-f5df6d577-vlx5x 0/1 Terminating 0 18m\n
If we check the new pod cpu and memory they would be updated as per VPA recommendation:
# kubectl describe po vpa-demo-rollout-f5df6d577-vdlqq -n test-vpa\nName: vpa-demo-rollout-f5df6d577-vdlqq\nNamespace: test-vpa\nPriority: 0\nNode: argo-rollouts-control-plane/172.18.0.2\nStart Time: Mon, 14 Mar 2022 12:55:06 +0000\nLabels: app=vpa-demo-rollout\n rollouts-pod-template-hash=f5df6d577\nAnnotations: vpaObservedContainers: vpa-demo-rollout\n vpaUpdates: Pod resources updated by kubengix-vpa: container 0: cpu request, memory request\nStatus: Running\nIP: 10.244.0.17\nIPs:\n IP: 10.244.0.17\nControlled By: ReplicaSet/vpa-demo-rollout-f5df6d577\nContainers:\n vpa-demo-rollout:\n Container ID: containerd://b79bd88851fe0622d33bc90a1560ca54ef2c27405a3bc9a4fc3a333eef5f9733\n Image: ravihari/nginx:v1\n Image ID: docker.io/ravihari/nginx@sha256:205961b09a80476af4c2379841bf6abec0022101a7e6c5585a88316f7115d17a\n Port: 80/TCP\n Host Port: 0/TCP\n State: Running\n Started: Mon, 14 Mar 2022 12:55:11 +0000\n Ready: True\n Restart Count: 0\n Requests:\n cpu: 25m\n memory: 262144k\n Environment: <none>\n Mounts:\n /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mk4fz (ro)\nConditions:\n Type Status\n Initialized True \n Ready True \n ContainersReady True \n PodScheduled True \nVolumes:\n kube-api-access-mk4fz:\n Type: Projected (a volume that contains injected data from multiple sources)\n TokenExpirationSeconds: 3607\n ConfigMapName: kube-root-ca.crt\n ConfigMapOptional: <nil>\n DownwardAPI: true\nQoS Class: Burstable\nNode-Selectors: <none>\nTolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s\n node.kubernetes.io/unreachable:NoExecute op=Exists for 300s\nEvents:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal Scheduled 38s default-scheduler Successfully assigned test-vpa/vpa-demo-rollout-f5df6d577-vdlqq to argo-rollouts-control-plane\n Normal Pulled 35s kubelet Container image \"ravihari/nginx:v1\" already present on machine\n Normal Created 35s kubelet Created container vpa-demo-rollout\n Normal Started 33s kubelet Started container vpa-demo-rollout\n
"},{"location":"features/vpa-support/#requirements","title":"Requirements","text":"In order for the VPA to manipulate the rollout, the Kubernetes cluster hosting the rollout CRD needs the subresources support for CRDs. This feature was introduced as alpha in Kubernetes version 1.10 and transitioned to beta in Kubernetes version 1.11. If a user wants to use VPA on v1.10, the Kubernetes Cluster operator will need to add a custom feature flag to the API server. After 1.10, the flag is turned on by default. Check out the following link for more information on setting the custom feature flag.
When installing VPA you may need to add the following in RBAC configurations for system:vpa-target-reader cluster role as by default VPA maynot support rollouts in all the versions.
- apiGroups:\n - argoproj.io\n resources:\n - rollouts\n - rollouts/scale\n - rollouts/status\n - replicasets\n verbs:\n - get\n - list\n - watch\n
Makes sure Metrics-Server is installed in the cluster and openssl is upto date for VPA latest version to apply recommendations to the pods properly.
"},{"location":"features/anti-affinity/anti-affinity/","title":"Anti Affinity","text":""},{"location":"features/anti-affinity/anti-affinity/#background","title":"Background","text":"Depending on a cluster's configuration, a Blue Green Rollout (or a Canary rollout that uses traffic management) can cause newly created pods to restart after deploying a new version. This can be problematic, especially for applications that cannot startup quickly or do not gracefully exit.
This behavior occurs because cluster auto-scaler wants to scale down the extra capacity which was created to support a Rollout running in double capacity. When a node is scaled down, the pods it owns are deleted and recreated. This usually happens if a Rollout has its own dedicated instance group since a Rollout has a greater effect on cluster auto-scaling. Therefore, clusters with a large pool of shared nodes experience the behavior less often.
For example, here is a Rollout is running with 8 pods spread across 2 nodes. Each node can hold 6 pods:
When the spec.template of the Rollout changes, the controller creates a new ReplicaSet with the spec update and the total number of pods doubles. In this case, the number of pods increases to 16.
Since each node can only hold 6 pods, the cluster autoscaler must increase the node count to 3 to accommodate all 16 pods. The resulting distribution of pods across nodes is shown here:
Once the Rollout finishes progressing, the old version is scaled down. This leaves the cluster with more nodes than necessary, thus wasting resources (as shown below).
The cluster auto-scaler terminates the extra node and the pods are rescheduled on the remaining 2 nodes.
To reduce the chance of this behavior, a rollout can inject anti-affinity into the ReplicaSet. This prevents new pods from running on nodes which have the previous version's pods.
You can learn more about anti-affinity here.
Repeating the above example with anti-affinity enabled, here is what happens when the .spec.template of the Rollout changes. Due to anti-affinity, the new pods cannot be scheduled on nodes which run the old ReplicaSet's pods. As a result, the cluster auto-scaler must create 2 nodes to host the new ReplicaSet's pods. In this case, pods won't be started since the scaled-down nodes are guaranteed to not have the new pods.
"},{"location":"features/anti-affinity/anti-affinity/#enabling-anti-affinity-in-rollouts","title":"Enabling Anti-Affinity in Rollouts","text":"Anti-affinity is enabled by adding the anti-affinity struct to the Blue-Green or Canary strategy. When the anti-affinity struct is set, controller injects a PodAntiAffinity struct into the ReplicaSet's Affinity. This feature will not modify any of the ReplicaSet's pre-existing affinity rules.
Users have a choice between these scheduling rules: RequiredDuringSchedulingIgnoredDuringExecution and PreferredDuringSchedulingIgnoredDuringExecution.
RequiredDuringSchedulingIgnoredDuringExecution requires a new version's pods to be on a separate node than the previous versions. If this is not possible, the the new version's pods will not be scheduled.
strategy:\n bluegreen:\n antiAffinity:\n requiredDuringSchedulingIgnoredDuringExecution: {}\n
Unlike the Required strategy, PreferredDuringSchedulingIgnoredDuringExecution does not force a new version's pods to be on a separate node than the previous versions. The scheduler attempts to place the new version's pods on separate node(s). If that's not possible, the new version's pods will still be scheduled. The Weight is used to create a priority order for preferred anti-affinity rules.
strategy:\n canary:\n antiAffinity:\n preferredDuringSchedulingIgnoredDuringExecution:\n weight: 1 # Between 1 - 100\n
Important
The main downside to this approach is that deployments can take longer because new nodes are more likely to be created in order to schedule pods with respect to anti-affinity rules. This delay most frequently occurs when a rollout has its own dedicated instance group, since new nodes are more likely to be created to honor anti-affinity rules.
"},{"location":"features/traffic-management/","title":"Traffic management","text":"Traffic management is controlling the data plane to have intelligent routing rules for an application. These routing rules can manipulate the flow of traffic to different versions of an application enabling Progressive Delivery. These controls limit the blast radius of a new release by ensuring a small percentage of users receive a new version while it is verified.
There are various techniques to achieve traffic management:
- Raw percentages (i.e., 5% of traffic should go to the new version while the rest goes to the stable version)
- Header-based routing (i.e., send requests with a specific header to the new version)
- Mirrored traffic where all the traffic is copied and send to the new version in parallel (but the response is ignored)
"},{"location":"features/traffic-management/#traffic-management-tools-in-kubernetes","title":"Traffic Management tools in Kubernetes","text":"The core Kubernetes objects do not have fine-grained tools needed to fulfill all the requirements of traffic management. At most, Kubernetes offers native load balancing capabilities through the Service object by offering an endpoint that routes traffic to a grouping of pods based on that Service's selector. Functionality like traffic mirroring or routing by headers is not possible with the default core Service object, and the only way to control the percentage of traffic to different versions of an application is by manipulating replica counts of those versions.
Service Meshes fill this missing functionality in Kubernetes. They introduce new concepts and functionality to control the data plane through the use of CRDs and other core Kubernetes resources.
"},{"location":"features/traffic-management/#how-does-argo-rollouts-enable-traffic-management","title":"How does Argo Rollouts enable traffic management?","text":"Argo Rollouts enables traffic management by manipulating the Service Mesh resources to match the intent of the Rollout. Argo Rollouts currently supports the following traffic providers:
- AWS ALB Ingress Controller
- Ambassador Edge Stack
- Apache APISIX
- Google Cloud
- Gateway API
- Istio
- Kong Ingress
- Nginx Ingress Controller
- Service Mesh Interface (SMI)
- Traefik Proxy
- Multiple Providers
- File a ticket here if you would like another implementation (or thumbs up it if that issue already exists)
Regardless of the Service Mesh used, the Rollout object has to set a canary Service and a stable Service in its spec. Here is an example with those fields set:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nspec:\n ...\n strategy:\n canary:\n canaryService: canary-service\n stableService: stable-service\n trafficRouting:\n ...\n
The controller modifies these Services to route traffic to the appropriate canary and stable ReplicaSets as the Rollout progresses. These Services are used by the Service Mesh to define what group of pods should receive the canary and stable traffic.
Additionally, the Argo Rollouts controller needs to treat the Rollout object differently when using traffic management. In particular, the Stable ReplicaSet owned by the Rollout remains fully scaled up as the Rollout progresses through the Canary steps.
Since the traffic is controlled independently by the Service Mesh resources, the controller needs to make a best effort to ensure that the Stable and New ReplicaSets are not overwhelmed by the traffic sent to them. By leaving the Stable ReplicaSet scaled up, the controller is ensuring that the Stable ReplicaSet can handle 100% of the traffic at any time1. The New ReplicaSet follows the same behavior as without traffic management. The new ReplicaSet's replica count is equal to the latest SetWeight step percentage multiple by the total replica count of the Rollout. This calculation ensures that the canary version does not receive more traffic than it can handle.
"},{"location":"features/traffic-management/#traffic-routing-with-managed-routes-and-route-precedence","title":"Traffic routing with managed routes and route precedence","text":""},{"location":"features/traffic-management/#traffic-router-support-istio","title":"Traffic router support: (Istio)","text":"When traffic routing is enabled, you have the ability to also let argo rollouts add and manage other routes besides just controlling the traffic weight to the canary. Two such routing rules are header and mirror based routes. When using these routes we also have to set a route precedence with the upstream traffic router. We do this using the spec.strategy.canary.trafficRouting.managedRoutes field which is an array the order of the items in the array determine the precedence. This set of routes will also be placed in the order specified on top of any other routes defined manually.
Warning
All routes listed in managed routes will be removed at the end of a rollout or on an abort. Do not put any manually created routes in the list.
Here is an example:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nspec:\n ...\n strategy:\n canary:\n ...\n trafficRouting:\n managedRoutes:\n - name: priority-route-1\n - name: priority-route-2\n - name: priority-route-3\n
"},{"location":"features/traffic-management/#traffic-routing-based-on-a-header-values-for-canary","title":"Traffic routing based on a header values for Canary","text":""},{"location":"features/traffic-management/#traffic-router-support-istio_1","title":"Traffic router support: (Istio)","text":"Argo Rollouts has ability to send all traffic to the canary-service based on a http request header value. The step for the header based traffic routing is setHeaderRoute and has a list of matchers for the header.
name - name of the header route.
match - header matching rules is an array of headerName, headerValue pairs.
headerName - name of the header to match.
headerValue- contains exactly one of exact - specify the exact header value, regex - value in a regex format, prefix - the prefix of the value could be provided. Not all traffic routers will support all match types.
To disable header based traffic routing just need to specify empty setHeaderRoute with only the name of the route.
Example:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nspec:\n ...\n strategy:\n canary:\n canaryService: canary-service\n stableService: stable-service\n trafficRouting:\n managedRoutes:\n - name: set-header-1\n istio:\n virtualService:\n name: rollouts-demo-vsvc\n steps:\n - setWeight: 20\n - setHeaderRoute: # enable header based traffic routing where\n name: \"set-header-1\"\n match:\n - headerName: Custom-Header1 # Custom-Header1=Mozilla\n headerValue:\n exact: Mozilla\n - headerName: Custom-Header2 # or Custom-Header2 has a prefix Mozilla\n headerValue:\n prefix: Mozilla\n - headerName: Custom-Header3 # or Custom-Header3 value match regex: Mozilla(.*)\n headerValue:\n regex: Mozilla(.*)\n - pause: {}\n - setHeaderRoute:\n name: \"set-header-1\" # disable header based traffic routing\n
"},{"location":"features/traffic-management/#traffic-routing-mirroring-traffic-to-canary","title":"Traffic routing mirroring traffic to canary","text":""},{"location":"features/traffic-management/#traffic-router-support-istio_2","title":"Traffic router support: (Istio)","text":"Argo Rollouts has ability to mirror traffic to the canary-service based on a various matching rules. The step for the mirror based traffic routing is setMirrorRoute and has a list of matchers for the header.
name - name of the mirror route.
percentage - what percentage of the matched traffic to mirror
match - The matching rules for the header route, if this is missing it acts as a removal of the route. All conditions inside a single match block have AND semantics, while the list of match blocks have OR semantics. Each type within a match (method, path, headers) must have one and only one match type (exact, regex, prefix) Not all match types (exact, regex, prefix) will be supported by all traffic routers.
To disable mirror based traffic route you just need to specify a setMirrorRoute with only the name of the route.
This example will mirror 35% of HTTP traffic that matches a GET requests and with the url prefix of /
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nspec:\n ...\n strategy:\n canary:\n canaryService: canary-service\n stableService: stable-service\n trafficRouting:\n managedRoutes:\n - name: mirror-route\n istio:\n virtualService:\n name: rollouts-demo-vsvc\n steps:\n - setCanaryScale:\n weight: 25\n - setMirrorRoute:\n name: mirror-route\n percentage: 35\n match:\n - method:\n exact: GET\n path:\n prefix: /\n - pause:\n duration: 10m\n - setMirrorRoute:\n name: \"mirror-route\" # removes mirror based traffic route\n
-
The Rollout has to assume that the application can handle 100% of traffic if it is fully scaled up. It should outsource to the HPA to detect if the Rollout needs to more replicas if 100% isn't enough.\u00a0\u21a9
"},{"location":"features/traffic-management/alb/","title":"AWS Load Balancer Controller (ALB)","text":""},{"location":"features/traffic-management/alb/#requirements","title":"Requirements","text":" - AWS Load Balancer Controller v1.1.5 or greater
"},{"location":"features/traffic-management/alb/#overview","title":"Overview","text":"AWS Load Balancer Controller (also known as AWS ALB Ingress Controller) enables traffic management through an Ingress object, which configures an AWS Application Load Balancer (ALB) to route traffic to one or more Kubernetes services. ALBs provides advanced traffic splitting capability through the concept of weighted target groups. This feature is supported by the AWS Load Balancer Controller through annotations made to the Ingress object to configure \"actions.\"
"},{"location":"features/traffic-management/alb/#how-it-works","title":"How it works","text":"ALBs are configured via Listeners, and Rules which contain Actions. Listeners define how traffic from a client comes in, and Rules define how to handle those requests with various Actions. One type of Action allows users to forward traffic to multiple TargetGroups (with each being defined as a Kubernetes service). You can read more about ALB concepts here.
An Ingress which is managed by the AWS Load Balancer Controller, controls an ALB's Listener and Rules through the Ingress' annotations and spec. In order to split traffic among multiple target groups (e.g. different Kubernetes services), the AWS Load Balancer controller looks to a specific \"action\" annotation on the Ingress, alb.ingress.kubernetes.io/actions.<service-name>. This annotation is injected and updated automatically by a Rollout during an update according to the desired traffic weights.
"},{"location":"features/traffic-management/alb/#usage","title":"Usage","text":"To configure a Rollout to use the ALB integration and split traffic between the canary and stable services during updates, the Rollout should be configured with the following fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\n...\nspec:\n strategy:\n canary:\n # canaryService and stableService are references to Services which the Rollout will modify\n # to target the canary ReplicaSet and stable ReplicaSet respectively (required).\n canaryService: canary-service\n stableService: stable-service\n trafficRouting:\n alb:\n # The referenced ingress will be injected with a custom action annotation, directing\n # the AWS Load Balancer Controller to split traffic between the canary and stable\n # Service, according to the desired traffic weight (required).\n ingress: ingress\n # If you want to controll multiple ingress resources you can use the ingresses field, if ingresses is specified\n # the ingress field will need to be omitted.\n ingresses:\n - ingress-1\n - ingress-2\n # Reference to a Service that the Ingress must target in one of the rules (optional).\n # If omitted, uses canary.stableService.\n rootService: root-service\n # Service port is the port which the Service listens on (required).\n servicePort: 443\n
The referenced Ingress should be deployed with an ingress rule that matches the Rollout service:
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n name: ingress\n annotations:\n kubernetes.io/ingress.class: alb\nspec:\n rules:\n - http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n # serviceName must match either: canary.trafficRouting.alb.rootService (if specified),\n # or canary.stableService (if rootService is omitted)\n name: root-service\n # servicePort must be the value: use-annotation\n # This instructs AWS Load Balancer Controller to look to annotations on how to direct traffic\n port:\n name: use-annotation\n
During an update, the rollout controller injects the alb.ingress.kubernetes.io/actions.<SERVICE-NAME> annotation, containing a JSON payload understood by the AWS Load Balancer Controller, directing it to split traffic between the canaryService and stableService according to the current canary weight.
The following is an example of our example Ingress after the rollout has injected the custom action annotation that splits traffic between the canary-service and stable-service, with a traffic weight of 10 and 90 respectively:
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n name: ingress\n annotations:\n kubernetes.io/ingress.class: alb\n alb.ingress.kubernetes.io/actions.root-service: |\n {\n \"Type\":\"forward\",\n \"ForwardConfig\":{\n \"TargetGroups\":[\n {\n \"Weight\":10,\n \"ServiceName\":\"canary-service\",\n \"ServicePort\":\"80\"\n },\n {\n \"Weight\":90,\n \"ServiceName\":\"stable-service\",\n \"ServicePort\":\"80\"\n }\n ]\n }\n }\nspec:\n rules:\n - http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n name: root-service\n port:\n name: use-annotation\n
Note
Argo rollouts additionally injects an annotation, rollouts.argoproj.io/managed-alb-actions, to the Ingress for bookkeeping purposes. The annotation indicates which actions are being managed by the Rollout object (since multiple Rollouts can reference one Ingress). Upon a rollout deletion, the rollout controller looks to this annotation to understand that this action is no longer managed, and is reset to point only the stable service with 100 weight.
"},{"location":"features/traffic-management/alb/#rootservice","title":"rootService","text":"By default, a rollout will inject the alb.ingress.kubernetes.io/actions.<SERVICE-NAME> annotation using the service/action name specified under spec.strategy.canary.stableService. However, it may be desirable to specify an explicit service/action name different from the stableService. For example, one pattern is to use a single Ingress containing three different rules to reach the canary, stable, and root service separately (e.g. for testing purposes). In this case, you may want to specify a \"root\" service as the service/action name instead of stable. To do so, reference a service under rootService under the alb specification:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nspec:\n strategy:\n canary:\n canaryService: guestbook-canary\n stableService: guestbook-stable\n trafficRouting:\n alb:\n rootService: guestbook-root\n...\n
"},{"location":"features/traffic-management/alb/#sticky-session","title":"Sticky session","text":"Because at least two target groups (canary and stable) are used, target group stickiness requires additional configuration: Sticky session must be activated on the target group via
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nspec:\n strategy:\n canary:\n...\n trafficRouting:\n alb:\n stickinessConfig:\n enabled: true\n durationSeconds: 3600\n...\n
More information can be found in the AWS ALB API
"},{"location":"features/traffic-management/alb/#zero-downtime-updates-with-aws-targetgroup-verification","title":"Zero-Downtime Updates with AWS TargetGroup Verification","text":"Argo Rollouts contains two features to help ensure zero-downtime updates when used with the AWS LoadBalancer controller: TargetGroup IP verification and TargetGroup weight verification. Both features involve the Rollout controller performing additional safety checks to AWS, to verify the changes made to the Ingress object are reflected in the underlying AWS TargetGroup.
"},{"location":"features/traffic-management/alb/#targetgroup-ip-verification","title":"TargetGroup IP Verification","text":"Note
Target Group IP verification available since Argo Rollouts v1.1
The AWS LoadBalancer controller can run in one of two modes:
- Instance mode
- IP mode
TargetGroup IP Verification is only applicable when the AWS LoadBalancer controller in IP mode. When using the AWS LoadBalancer controller in IP mode (e.g. using the AWS CNI), the ALB LoadBalancer targets individual Pod IPs, as opposed to K8s node instances. Targeting Pod IPs comes with an increased risk of downtime during an update, because the Pod IPs behind the underlying AWS TargetGroup can more easily become outdated from the actual availability and status of pods, causing HTTP 502 errors when the TargetGroup points to pods which have already been scaled down.
To mitigate this risk, AWS recommends the use of pod readiness gate injection when running the AWS LoadBalancer in IP mode. Readiness gates allow for the AWS LoadBalancer controller to verify that TargetGroups are accurate before marking newly created Pods as \"ready\", preventing premature scale down of the older ReplicaSet.
Pod readiness gate injection uses a mutating webhook which decides to inject readiness gates when a pod is created based on the following conditions:
- There exists a service matching the pod labels in the same namespace
- There exists at least one target group binding that refers to the matching service
Another way to describe this is: the AWS LoadBalancer controller injects readiness gates onto Pods only if they are \"reachable\" from an ALB Ingress at the time of pod creation. A pod is considered reachable if an (ALB) Ingress references a Service which matches the pod labels. It ignores all other Pods.
One challenge with this manner of pod readiness gate injection, is that modifications to the Service selector labels (spec.selector) do not allow for the AWS LoadBalancer controller to inject the readiness gates, because by that time the Pod was already created (and readiness gates are immutable). Note that this is an issue when you change Service selectors of any ALB Service, not just ones involved in Argo Rollouts.
Because Argo Rollout's blue-green strategy works by modifying the activeService selector to the new ReplicaSet labels during promotion, it suffers from the issue where readiness gates for the spec.strategy.blueGreen.activeService fail to be injected. This means there is a possibility of downtime in the following problematic scenario during an update from V1 to V2:
- Update is triggered and V2 ReplicaSet stack is scaled up
- V2 ReplicaSet pods become fully available and ready to be promoted
- Rollout promotes V2 by updating the label selectors of the active service to point to the V2 stack (from V1)
- Due to unknown issues (e.g. AWS load balancer controller downtime, AWS rate limiting), registration of the V2 Pod IPs to the TargetGroup does not happen or is delayed.
- V1 ReplicaSet is scaled down to complete the update
After step 5, when the V1 ReplicaSet is scaled down, the outdated TargetGroup would still be pointing to the V1 Pods IPs which no longer exist, causing downtime.
To allow for zero-downtime updates, Argo Rollouts has the ability to perform TargetGroup IP verification as an additional safety measure during an update. When this feature is enabled, whenever a service selector modification is made, the Rollout controller blocks progression of the update until it can verify the TargetGroup is accurately targeting the new Pod IPs of the bluegreen.activeService. Verification is achieved by querying AWS APIs to describe the underlying TargetGroup, iterating its registered IPs, and ensuring all Pod IPs of the activeService's Endpoints list are registered in the TargetGroup. Verification must succeed before running postPromotionAnalysis or scaling down the old ReplicaSet.
Similarly for the canary strategy, after updating the canary.stableService selector labels to point to the new ReplicaSet, the TargetGroup IP verification feature allows the controller to block the scale down of the old ReplicaSet until it verifies the Pods IP behind the stableService TargetGroup are accurate.
"},{"location":"features/traffic-management/alb/#targetgroup-weight-verification","title":"TargetGroup Weight Verification","text":"Note
TargetGroup weight verification available since Argo Rollouts v1.0
TargetGroup weight verification addresses a similar problem to TargetGroup IP verification, but instead of verifying that the Pod IPs of a service are reflected accurately in the TargetGroup, the controller verifies that the traffic weights are accurate from what was set in the ingress annotations. Weight verification is applicable to AWS LoadBalancer controllers which are running either in IP mode or Instance mode.
After Argo Rollouts adjusts a canary weight by updating the Ingress annotation, it moves on to the next step. However, due to external factors (e.g. AWS rate limiting, AWS load balancer controller downtime) it is possible that the weight modifications made to the Ingress, did not take effect in the underlying TargetGroup. This is potentially dangerous as the controller will believe it is safe to scale down the old stable stack when in reality, the outdated TargetGroup may still be pointing to it.
Using the TargetGroup weight verification feature, the rollout controller will additionally verify the canary weight after a setWeight canary step. It accomplishes this by querying AWS LoadBalancer APIs directly, to confirm that the Rules, Actions, and TargetGroups reflect the desire of Ingress annotation.
"},{"location":"features/traffic-management/alb/#usage_1","title":"Usage","text":"To enable AWS target group verification, add --aws-verify-target-group flag to the rollout-controller flags:
apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: argo-rollouts\nspec:\n template:\n spec:\n containers:\n - name: argo-rollouts\n args: [--aws-verify-target-group]\n # NOTE: in v1.0, the --alb-verify-weight flag should be used instead\n
For this feature to work, the argo-rollouts deployment requires the following AWS API permissions under the Elastic Load Balancing API:
{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Action\": [\n \"elasticloadbalancing:DescribeTargetGroups\",\n \"elasticloadbalancing:DescribeLoadBalancers\",\n \"elasticloadbalancing:DescribeListeners\",\n \"elasticloadbalancing:DescribeRules\",\n \"elasticloadbalancing:DescribeTags\",\n \"elasticloadbalancing:DescribeTargetHealth\"\n ],\n \"Resource\": \"*\",\n \"Effect\": \"Allow\"\n }\n ]\n}\n
There are various ways of granting AWS privileges to the argo-rollouts pods, which is highly dependent to your cluster's AWS environment, and out-of-scope of this documentation. Some solutions include:
- AWS access and secret keys
- kiam
- kube2iam
- EKS ServiceAccount IAM Roles
"},{"location":"features/traffic-management/alb/#zero-downtime-updates-with-ping-pong-feature","title":"Zero-Downtime Updates with Ping-Pong feature","text":"Above there was described the recommended way by AWS to solve zero-downtime issue. Is a use a pod readiness gate injection when running the AWS LoadBalancer in IP mode. There is a challenge with that approach, modifications of the Service selector labels (spec.selector) not allowed the AWS LoadBalancer controller to mutate the readiness gates. And Ping-Pong feature helps to deal with that challenge. At some particular moment one of the services (e.g. ping) is \"wearing a hat\" of stable service another one (e.g. pong) is \"wearing a hat\" of canary. At the end of the promotion step all 100% of traffic sending to the \"canary\" (e.g. pong). And then the Rollout swapped the hats of ping and pong services so the pong became a stable one. The Rollout status object holds the value of who is currently the stable ping or pong (status.canary.currentPingPong). And this way allows the rollout to use pod readiness gate injection as the services are not changing their labels at the end of the rollout progress.
Important
Ping-Pong feature available since Argo Rollouts v1.2
"},{"location":"features/traffic-management/alb/#example","title":"Example","text":"apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: example-rollout\nspec:\n selector:\n matchLabels:\n app: nginx\n template:\n metadata:\n labels:\n app: nginx\n spec:\n containers:\n - name: nginx\n image: nginx:1.15.4\n ports:\n - containerPort: 80\n strategy:\n canary:\n pingPong: #Indicates that the ping-pong services enabled\n pingService: ping-service\n pongService: pong-service\n trafficRouting:\n alb:\n ingress: alb-ingress\n servicePort: 80\n steps:\n - setWeight: 20\n - pause: {}\n
"},{"location":"features/traffic-management/alb/#custom-annotations-prefix","title":"Custom annotations-prefix","text":"The AWS Load Balancer Controller allows users to customize the annotation prefix used by the Ingress controller using a flag to the controller, --annotations-prefix (from the default of alb.ingress.kubernetes.io). If your AWS Load Balancer Controller is customized to use a different annotation prefix, annotationPrefix field should be specified such that the Ingress object will be annotated in a manner understood by the cluster's aws load balancer controller.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nspec:\n strategy:\n canary:\n trafficRouting:\n alb:\n annotationPrefix: custom.alb.ingress.kubernetes.io\n
"},{"location":"features/traffic-management/alb/#custom-ingress-class","title":"Custom Ingress Class","text":"By default, Argo Rollout will operate on Ingresses with the annotation:
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n annotations:\n kubernetes.io/ingress.class: alb\n
Or with the ingressClassName:
apiVersion: networking.k8s.io/v1\nkind: Ingress\nspec:\n ingressClassName: alb\n
To configure the controller to operate on Ingresses with a different class name, you can specify a different value through the --alb-ingress-classes flag in the controller command line arguments.
Note that the --alb-ingress-classes flag can be specified multiple times if the Argo Rollouts controller should operate on multiple values. This may be desired when a cluster has multiple Ingress controllers that operate on different kubernetes.io/ingress.class or spec.ingressClassName values.
If the controller needs to operate on any Ingress without the kubernetes.io/ingress.class annotation or spec.ingressClassName, the flag can be specified with an empty string (e.g. --alb-ingress-classes '').
"},{"location":"features/traffic-management/ambassador/","title":"Ambassador Edge Stack","text":"Ambassador Edge Stack provides the functionality you need at the edge your Kubernetes cluster (hence, an \"edge stack\"). This includes an API gateway, ingress controller, load balancer, developer portal, canary traffic routing and more. It provides a group of CRDs that users can configure to enable different functionalities.
Argo-Rollouts provides an integration that leverages Ambassador's canary routing capability. This allows the traffic to your application to be gradually incremented while new versions are being deployed.
"},{"location":"features/traffic-management/ambassador/#how-it-works","title":"How it works","text":"Ambassador Edge Stack provides a resource called Mapping that is used to configure how to route traffic to services. Ambassador canary deployment is achieved by creating 2 mappings with the same URL prefix pointing to different services. Consider the following example:
apiVersion: getambassador.io/v2\nkind: Mapping\nmetadata:\n name: stable-mapping\nspec:\n prefix: /someapp\n rewrite: /\n service: someapp-stable:80\n---\napiVersion: getambassador.io/v2\nkind: Mapping\nmetadata:\n name: canary-mapping\nspec:\n prefix: /someapp\n rewrite: /\n service: someapp-canary:80\n weight: 30\n
In the example above we are configuring Ambassador to route 30% of the traffic coming from <public ingress>/someapp to the service someapp-canary and the rest of the traffic will go to the service someapp-stable. If users want to gradually increase the traffic to the canary service, they have to update the canary-mapping setting the weight to the desired value either manually or automating it somehow.
With Argo-Rollouts there is no need to create the canary-mapping. The process of creating it and gradually updating its weight is fully automated by the Argo-Rollouts controller. The following example shows how to configure the Rollout resource to use Ambassador as a traffic router for canary deployments:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\n...\nspec:\n strategy:\n canary:\n stableService: someapp-stable\n canaryService: someapp-canary\n trafficRouting:\n ambassador:\n mappings:\n - stable-mapping\n steps:\n - setWeight: 30\n - pause: {duration: 60s}\n - setWeight: 60\n - pause: {duration: 60s}\n
Under spec.strategy.canary.trafficRouting.ambassador there are 2 possible attributes:
mappings: Required. At least one Ambassador mapping must be provided for Argo-Rollouts to be able to manage the canary deployment. Multiple mappings are also supported in case there are multiple routes to the service (e.g., your service has multiple ports, or can be accessed via different URLs). If no mapping is provided Argo-Rollouts will send an error event and the rollout will be aborted.
When Ambassador is configured in the trafficRouting attribute of the manifest, the Rollout controller will: 1. Create one canary mapping for each stable mapping provided in the Rollout manifest 1. Proceed with the steps according to the configuration updating the canary mapping weight 1. At the end of the process Argo-Rollout will delete all the canary mappings created
"},{"location":"features/traffic-management/ambassador/#endpoint-resolver","title":"Endpoint Resolver","text":"By default, Ambassador uses kube-proxy to route traffic to Pods. However we should configure it to bypass kube-proxy and route traffic directly to pods. This will provide true L7 load balancing which is desirable in a canary workflow. This approach is called endpoint routing and can be achieve by configuring endpoint resolvers.
To configure Ambassador to use endpoint resolver it is necessary to apply the following resource in the cluster:
apiVersion: getambassador.io/v2\nkind: KubernetesEndpointResolver\nmetadata:\n name: endpoint\n
And then configure the mapping to use it setting the resolver attribute:
apiVersion: getambassador.io/v2\nkind: Mapping\nmetadata:\n name: stable-mapping\nspec:\n resolver: endpoint\n prefix: /someapp\n rewrite: /\n service: someapp-stable:80\n
For more details about the Ambassador and Argo-Rollouts integration, see the Ambassador Argo documentation.
"},{"location":"features/traffic-management/apisix/","title":"Apache APISIX","text":"You can use the Apache APISIX and Apache APISIX Ingress Controller for traffic management with Argo Rollouts.
The ApisixRoute is the object that supports the ability for weighted round robin load balancing when using Apache APISIX Ingress Controller as ingress.
This guide shows you how to integrate ApisixRoute with Argo Rollouts using it as weighted round robin load balancer
"},{"location":"features/traffic-management/apisix/#prerequisites","title":"Prerequisites","text":"Argo Rollouts requires Apache APISIX v2.15 or newer and Apache APISIX Ingress Controller v1.5.0 or newer.
Install Apache APISIX and Apache APISIX Ingress Controller with Helm v3:
helm repo add apisix https://charts.apiseven.com\nkubectl create ns apisix\n\nhelm upgrade -i apisix apisix/apisix --version=0.11.3 \\\n--namespace apisix \\\n--set ingress-controller.enabled=true \\\n--set ingress-controller.config.apisix.serviceNamespace=apisix\n
"},{"location":"features/traffic-management/apisix/#bootstrap","title":"Bootstrap","text":"First, we need to create the ApisixRoute object using its ability for weighted round robin load balancing.
apiVersion: apisix.apache.org/v2\nkind: ApisixRoute\nmetadata:\n name: rollouts-apisix-route\nspec:\n http:\n - name: rollouts-apisix\n match:\n paths:\n - /*\n hosts:\n - rollouts-demo.apisix.local\n backends:\n - serviceName: rollout-apisix-canary-stable\n servicePort: 80\n - serviceName: rollout-apisix-canary-canary\n servicePort: 80\n
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/examples/apisix/route.yaml\n
Notice, we don't specify the weight field. It is necessary to be synced with ArgoCD. If we specify this field and Argo Rollouts controller changes it, then the ArgoCD controller will notice it and will show that this resource is out of sync (if you are using Argo CD to manage your Rollout).
Secondly, we need to create the Argo Rollouts object.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollout-apisix-canary\nspec:\n replicas: 5\n strategy:\n canary:\n canaryService: rollout-apisix-canary-canary\n stableService: rollout-apisix-canary-stable\n trafficRouting:\n managedRoutes:\n - name: set-header\n apisix:\n route:\n name: rollouts-apisix-route\n rules:\n - rollouts-apisix\n steps:\n - setCanaryScale:\n replicas: 1\n setHeaderRoute:\n match:\n - headerName: trace\n headerValue:\n exact: debug\n name: set-header\n - setWeight: 20\n - pause: {}\n - setWeight: 40\n - pause:\n duration: 15\n - setWeight: 60\n - pause:\n duration: 15\n - setWeight: 80\n - pause:\n duration: 15\n revisionHistoryLimit: 2\n selector:\n matchLabels:\n app: rollout-apisix-canary\n template:\n metadata:\n labels:\n app: rollout-apisix-canary\n spec:\n containers:\n - name: rollout-apisix-canary\n image: argoproj/rollouts-demo:blue\n ports:\n - name: http\n containerPort: 8080\n protocol: TCP\n resources:\n requests:\n memory: 32Mi\n cpu: 5m\n
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/examples/apisix/rollout.yaml\n
Finally, we need to create the services for the Argo Rollouts object.
apiVersion: v1\nkind: Service\nmetadata:\n name: rollout-apisix-canary-canary\nspec:\n ports:\n - port: 80\n targetPort: http\n protocol: TCP\n name: http\n selector:\n app: rollout-apisix-canary\n # This selector will be updated with the pod-template-hash of the canary ReplicaSet. e.g.:\n # rollouts-pod-template-hash: 7bf84f9696\n---\napiVersion: v1\nkind: Service\nmetadata:\n name: rollout-apisix-canary-stable\nspec:\n ports:\n - port: 80\n targetPort: http\n protocol: TCP\n name: http\n selector:\n app: rollout-apisix-canary\n # This selector will be updated with the pod-template-hash of the stable ReplicaSet. e.g.:\n # rollouts-pod-template-hash: 789746c88d\n
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/examples/apisix/services.yaml\n
Initial creations of any Rollout will immediately scale up the replicas to 100% (skipping any canary upgrade steps, analysis, etc...) since there was no upgrade that occurred.
The Argo Rollouts kubectl plugin allows you to visualize the Rollout, its related resources (ReplicaSets, Pods, AnalysisRuns), and presents live state changes as they occur. To watch the rollout as it deploys, run the get rollout --watch command from plugin:
kubectl argo rollouts get rollout rollout-apisix-canary --watch\n
"},{"location":"features/traffic-management/apisix/#updating-a-rollout","title":"Updating a Rollout","text":"Next it is time to perform an update. Just as with Deployments, any change to the Pod template field (spec.template) results in a new version (i.e. ReplicaSet) to be deployed. Updating a Rollout involves modifying the rollout spec, typically changing the container image field with a new version, and then running kubectl apply against the new manifest. As a convenience, the rollouts plugin provides a set image command, which performs these steps against the live rollout object in-place. Run the following command to update the rollout-apisix-canary Rollout with the \"yellow\" version of the container:
kubectl argo rollouts set image rollout-apisix-canary \\\n rollouts-demo=argoproj/rollouts-demo:yellow\n
During a rollout update, the controller will progress through the steps defined in the Rollout's update strategy. The example rollout sets a 20% traffic weight to the canary, and pauses the rollout indefinitely until user action is taken to unpause/promote the rollout.
You can check ApisixRoute's backend weights by the following command
kubectl describe apisixroute rollouts-apisix-route\n\n......\nSpec:\n Http:\n Backends:\n Service Name: rollout-apisix-canary-stable\n Service Port: 80\n Weight: 80\n Service Name: rollout-apisix-canary-canary\n Service Port: 80\n Weight: 20\n......\n
The rollout-apisix-canary-canary service gets 20% traffic through the Apache APISIX. You can check SetHeader ApisixRoute's match by the following command
kubectl describe apisixroute set-header\n\n......\nSpec:\n Http:\n Backends:\n Service Name: rollout-apisix-canary-canary\n Service Port: 80\n Weight: 100\n Match:\n Exprs:\n Op: Equal\n Subject:\n Name: trace\n Scope: Header\n Value: debug\n......\n
"},{"location":"features/traffic-management/google-cloud/","title":"Google Cloud","text":"With the introduction of the Kubernetes Gateway API it is now possible to use Argo Rollouts with all compliant implementations that support it. The integration is available with the Argo Rollouts Gateway API plugin currently hosted in Argo Labs.
Useful resources:
- The Gateway API specification
- Support of the Gateway API in Google Cloud
- Argo Rollouts Plugin capabilities
- Plugin for the Gateway API
The process involves the following steps:
- Creating a Kubernetes cluster with support for the Gateway API in Google Cloud
- Creating a Load balancer that is managed by the Gateway API in Google Cloud
- Installing Argo Rollouts + gateway API plugin in the cluster
- Defining a Rollout that takes advantage of the plugin
For a full application that includes all manifests see the plugin example.
"},{"location":"features/traffic-management/istio/","title":"Istio","text":"Istio is a service mesh that offers a rich feature-set to control the flow of traffic to a web service. Istio offers this functionality through a set of CRDs, and Argo Rollouts automates the management of these resources to provide advanced traffic shaping capabilities to the different versions of the Rollout during an update.
"},{"location":"features/traffic-management/istio/#how-it-works","title":"How it works","text":"Traffic splitting is accomplished in Istio by adjusting traffic weights defined in an Istio VirtualService. When using Argo Rollouts with Istio, a user deploys a VirtualService containing at least one HTTP route containing two HTTP route destinations: a route destination targeting the pods of canary ReplicaSet, and a route destination targeting the pods stable ReplicaSet. Istio provides two approaches for weighted traffic splitting, both approaches are available as options in Argo Rollouts:
- Host-level traffic splitting
- Subset-level traffic splitting
"},{"location":"features/traffic-management/istio/#host-level-traffic-splitting","title":"Host-level Traffic Splitting","text":"The first approach to traffic splitting using Argo Rollouts and Istio, is splitting between two hostnames, or Kubernetes Services: a canary Service and a stable Service. This approach is similar to the way all other Argo Rollouts mesh/ingress-controller integrations work (e.g. ALB, SMI, Nginx). Using this approach, the user is required to deploy the following resources:
- Rollout
- Service (canary)
- Service (stable)
- VirtualService
The Rollout should define the following fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollout-example\nspec:\n ...\n strategy:\n canary:\n canaryService: canary-svc # required\n stableService: stable-svc # required\n trafficRouting:\n istio:\n virtualService:\n name: rollout-vsvc # required\n routes:\n - primary # optional if there is a single route in VirtualService, required otherwise\n steps:\n - setWeight: 5\n - pause:\n duration: 10m\n
The VirtualService must contain an HTTP route with a name referenced in the Rollout, containing two route destinations with host values that match the canaryService and stableService referenced in the Rollout. If the VirtualService is defined in a different namespace than the rollout, its name should be rollout-vsvc.<vsvc namespace name>. Note that Istio requires that all weights add to 100, so the initial weights can be 100% to stable, and 0% to canary.
apiVersion: networking.istio.io/v1alpha3\nkind: VirtualService\nmetadata:\n name: rollout-vsvc\nspec:\n gateways:\n - istio-rollout-gateway\n hosts:\n - istio-rollout.dev.argoproj.io\n http:\n - name: primary # referenced in canary.trafficRouting.istio.virtualService.routes\n route:\n - destination:\n host: stable-svc # referenced in canary.stableService\n weight: 100\n - destination:\n host: canary-svc # referenced in canary.canaryService\n weight: 0\n
Finally, a canary and stable Service should be deployed. The selector of these Services will be modified by the Rollout during an update to target the canary and stable ReplicaSet pods. Note that if the VirtualService and destination host resides in different namespaces (e.g., VirtualService and Rollout are not in the same namespace), the namespace should be included in the destination host (e.g. stable-svc.<namespace>).
apiVersion: v1\nkind: Service\nmetadata:\n name: canary-svc\nspec:\n ports:\n - port: 80\n targetPort: http\n protocol: TCP\n name: http\n selector:\n app: rollouts-demo\n # This selector will be updated with the pod-template-hash of the canary ReplicaSet. e.g.:\n # rollouts-pod-template-hash: 7bf84f9696\n---\napiVersion: v1\nkind: Service\nmetadata:\n name: stable-svc\nspec:\n ports:\n - port: 80\n targetPort: http\n protocol: TCP\n name: http\n selector:\n app: rollouts-demo\n # This selector will be updated with the pod-template-hash of the stable ReplicaSet. e.g.:\n # rollouts-pod-template-hash: 123746c88d\n
During the lifecycle of a Rollout update, Argo Rollouts will continuously:
- modify the canary Service
spec.selector to contain the rollouts-pod-template-hash label of the canary ReplicaSet - modify the stable Service
spec.selector to contain the rollouts-pod-template-hash label of the stable ReplicaSet - modify the VirtualService
spec.http[].route[].weight to match the current desired canary weight
Note
Rollout does not make any other assumptions about the fields within the VirtualService or the Istio mesh. The user could specify additional configurations for the VirtualService like URI rewrite rules on the primary route or any other route if desired. The user can also create specific DestinationRules for each of the services.
"},{"location":"features/traffic-management/istio/#subset-level-traffic-splitting","title":"Subset-level Traffic Splitting","text":"Important
Available since v1.0
The second approach to traffic splitting using Argo Rollouts and Istio, is splitting between two Istio DestinationRule Subsets: a canary subset and a stable subset. When splitting by DestinationRule subsets, the user is required to deploy the following resources:
- Rollout
- Service
- VirtualService
- DestinationRule
The Rollout should define the following fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollout-example\nspec:\n ...\n strategy:\n canary:\n trafficRouting:\n istio:\n virtualService:\n name: rollout-vsvc # required\n routes:\n - primary # optional if there is a single route in VirtualService, required otherwise\n destinationRule:\n name: rollout-destrule # required\n canarySubsetName: canary # required\n stableSubsetName: stable # required\n steps:\n - setWeight: 5\n - pause:\n duration: 10m\n
A single service should be defined, which targets the Rollout pods. Note that unlike the first approach, where traffic splitting is against multiple Services which are modified to contain the rollout-pod-template-hash of the canary/stable ReplicaSets, this Service is not modified by the rollout controller.
apiVersion: v1\nkind: Service\nmetadata:\n name: rollout-example\nspec:\n ports:\n - port: 80\n targetPort: http\n protocol: TCP\n name: http\n selector:\n app: rollout-example\n
The VirtualService must contain an HTTP route with a name referenced in the Rollout, containing two route destinations with subset values that match the canarySubsetName and stableSubsetName referenced in the Rollout. Note that Istio requires that all weights add to 100, so the initial weights can be 100% to stable, and 0% to canary.
apiVersion: networking.istio.io/v1alpha3\nkind: VirtualService\nmetadata:\n name: rollout-vsvc\nspec:\n gateways:\n - istio-rollout-gateway\n hosts:\n - istio-rollout.dev.argoproj.io\n http:\n - name: primary # referenced in canary.trafficRouting.istio.virtualService.routes\n route:\n - destination:\n host: rollout-example\n subset: stable # referenced in canary.trafficRouting.istio.destinationRule.stableSubsetName\n weight: 100\n - destination:\n host: rollout-example\n subset: canary # referenced in canary.trafficRouting.istio.destinationRule.canarySubsetName\n weight: 0\n
Finally, the DestinationRule containing the canary and stable subsets referenced in the Rollout.
apiVersion: networking.istio.io/v1alpha3\nkind: DestinationRule\nmetadata:\n name: rollout-destrule\nspec:\n host: rollout-example\n subsets:\n - name: canary # referenced in canary.trafficRouting.istio.destinationRule.canarySubsetName\n labels: # labels will be injected with canary rollouts-pod-template-hash value\n app: rollout-example\n - name: stable # referenced in canary.trafficRouting.istio.destinationRule.stableSubsetName\n labels: # labels will be injected with stable rollouts-pod-template-hash value\n app: rollout-example\n
During the lifecycle of a Rollout using Istio DestinationRule, Argo Rollouts will continuously:
- modify the VirtualService
spec.http[].route[].weight to match the current desired canary weight - modify the DestinationRule
spec.subsets[].labels to contain the rollouts-pod-template-hash label of the canary and stable ReplicaSets
"},{"location":"features/traffic-management/istio/#tcp-traffic-splitting","title":"TCP Traffic Splitting","text":"Important
Available since v1.2.2
Support for splitting TCP traffic was introduced and requires the Rollout to define the following fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollout-example\nspec:\n ...\n strategy:\n canary:\n canaryService: canary-svc # required\n stableService: stable-svc # required\n trafficRouting:\n istio:\n virtualService:\n name: rollout-vsvc # required\n tcpRoutes:\n # Below fields are optional but if defined, they should match exactly with at least one of the TCP route match rules in your VirtualService\n - port: 3000 # Only required if you want to match any rule in your VirtualService which contains this port\n steps:\n - setWeight: 5\n - pause:\n duration: 10m\n
The VirtualService must contain a TCP route with a matching port referenced in the Rollout
apiVersion: networking.istio.io/v1alpha3\nkind: VirtualService\nmetadata:\n name: rollout-vsvc\nspec:\n gateways:\n - istio-rollout-gateway\n hosts:\n - istio-rollout.dev.argoproj.io\n tcp:\n - match:\n - port: 3000\n route:\n - destination:\n host: stable-svc # referenced in canary.stableService\n weight: 100\n - destination:\n host: canary-svc # referenced in canary.canaryService\n weight: 0\n
"},{"location":"features/traffic-management/istio/#multicluster-setup","title":"Multicluster Setup","text":"If you have Istio multicluster setup where the primary Istio cluster is different than the cluster where the Argo Rollout controller is running, then you need to do the following setup:
- Create a
ServiceAccount in the Istio primary cluster. apiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: argo-rollouts-istio-primary\n namespace: <any-namespace-preferrably-config-namespace>\n
- Create a
ClusterRole that provides access to Rollout controller in the Istio primary cluster. apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRole\nmetadata:\n name: argo-rollouts-istio-primary\nrules:\n- apiGroups:\n - networking.istio.io\n resources:\n - virtualservices\n - destinationrules\n verbs:\n - get\n - list\n - watch\n - update\n - patch\n
Note: If Argo Rollout controller is also installed in the Istio primary cluster, then you can reuse the argo-rollouts-clusterrole ClusterRole instead of creating a new one. - Link the
ClusterRole with the ServiceAccount in the Istio primary cluster. apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n name: argo-rollouts-istio-primary\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: argo-rollouts-istio-primary\nsubjects:\n- kind: ServiceAccount\n name: argo-rollouts-istio-primary\n namespace: <namespace-of-the-service-account>\n
- Now, use the following command to generate a secret for Rollout controller to access the Istio primary cluster. This secret will be applied to the cluster where Argo Rollout is running (i.e, Istio remote cluster), but will be generated from the Istio primary cluster. This secret can be generated right after Step 1, it only requires
ServiceAccount to exist. Reference to the command. istioctl x create-remote-secret --type remote --name <cluster-name> \\\n --namespace <namespace-of-the-service-account> \\\n --service-account <service-account-created-in-step1> \\\n --context=\"<ISTIO_PRIMARY_CLUSTER>\" | \\\n kubectl apply -f - --context=\"<ARGO_ROLLOUT_CLUSTER/ISTIO_REMOTE_CLUSTER>\"\n
- Label the secret.
kubectl label secret <istio-remote-secret> istio.argoproj.io/primary-cluster=\"true\" -n <namespace-of-the-secret>\n
"},{"location":"features/traffic-management/istio/#comparison-between-approaches","title":"Comparison Between Approaches","text":"There are some advantages and disadvantages of host-level traffic splitting vs. subset-level traffic splitting.
"},{"location":"features/traffic-management/istio/#dns-requirements","title":"DNS requirements","text":"With host-level splitting, the VirtualService requires different host values to split among the two destinations. However, using two host values implies the use of different DNS names (one for the canary, the other for the stable). For north-south traffic, which reaches the Service through the Istio Gateway, having multiple DNS names to reach the canary vs. stable pods may not matter. However, for east-west or intra-cluster traffic, it forces microservice-to-microservice communication to choose whether to hit the stable or the canary DNS name, go through the gateway, or add DNS entries for the VirtualServices. In this situation, the DestinationRule subset traffic splitting would be a better option for intra-cluster canarying.
"},{"location":"features/traffic-management/istio/#metrics","title":"Metrics","text":"Depending on the choice of host-level splitting vs. subset-level splitting, there will be different styles of prometheus metrics available. For example, if using host-level splitting, the metrics of the canary vs. stable would appear in the Istio Service metrics dashboard:
On the other hand, when splitting via subsets, it would be necessary to query prometheus using different parameters, such as the workload name:
"},{"location":"features/traffic-management/istio/#integrating-with-gitops","title":"Integrating with GitOps","text":"Earlier it was explained that VirtualServices should be deployed with an initial canary and stable weight of 0 and 100, respectively, such as in the following example:
http:\n - name: primary\n route:\n - destination:\n host: stable-svc\n weight: 100\n - destination:\n host: canary-svc\n weight: 0\n
This introduces a problem for users practicing GitOps. Since a Rollout will modify these VirtualService weights as the Rollout progresses through its steps, it unfortunately causes the VirtualService to become OutOfSync with the version in git. Additionally, if the VirtualService in git were to be applied while the Rollout is in this state (splitting traffic between the services), the apply would revert the weights back to the values in git (i.e. 100 to stable, 0 to canary).
One protection which is implemented in Argo Rollouts, is that it continually watches for changes to managed VirtualServices. In the event that a kubectl apply were to happen using the VirtualService in git, the change would be detected immediately by the rollout controller, and the controller will instantly set the VirtualService weights back to the canary weight appropriate for the given step of the Rollout. But since there is momentary flapping of weights, this behavior should be understood.
Some best practices to follow when using Argo CD with Argo Rollouts to prevent this behavior, is to leverage the following Argo CD features:
-
Configure the application to ignore differences in the VirtualService. e.g.:
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n name: guestbook\nspec:\n ignoreDifferences:\n - group: networking.istio.io\n kind: VirtualService\n jsonPointers:\n - /spec/http/0\n
Ignoring the differences in the VirtualServices HTTP route, prevents gitops differences in the VirtualService HTTP routes to contribute to the overall sync status of the Argo CD application. This adds the additional benefit of prevent auto-sync operations from being triggered.
-
Configure the Application to only apply OutOfSync resources:
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n name: guestbook\nspec:\n syncPolicy:\n syncOptions:\n - ApplyOutOfSyncOnly=true\n
By default, when Argo CD syncs an application, it runs kubectl apply against all resources in git which are part of the application. The ApplyOutOfSyncOnly=true sync option indicates to Argo CD to skip applying resources which it already considers Synced, and only apply the ones which are OutOfSync. This option, when used in conjunction with the ignoreDifferences feature, provides a way to manage the conflict in the desired state of a VirtualService between Argo CD and Argo Rollouts.
Argo CD also has an open issue here which would help address this problem. The proposed solution is to introduce an annotation to resources, which indicates to Argo CD to respect and preserve the differences at a specified path, in order to allow other controllers (e.g. Argo Rollouts) controller manage them instead.
"},{"location":"features/traffic-management/istio/#ping-pong","title":"Ping Pong","text":"Important
Available since v1.7
Argo Rollouts also supports ping pong when using Istio this was added to support configuring both ALB and Istio traffic routers at the same time. When using an ALB, ping-pong is generally a best practice especially with ALB readiness gates enabled. However, when we change the service selectors when a rollout is aborted back to stable pod hash it causes a blip of traffic outage because the ALB controller will set the pod readiness gates to false for a short while due to the label changes. If we configure both ALB and Istio with ping-pong this selector change does not happen and hence we do not see any outages.
"},{"location":"features/traffic-management/istio/#alternatives-considered","title":"Alternatives Considered","text":""},{"location":"features/traffic-management/istio/#rollout-ownership-over-the-virtual-service","title":"Rollout ownership over the Virtual Service","text":"An early design alternative was that instead of the controller modifying a referenced VirtualService, the Rollout controller would create, manage, and own a Virtual Service. While this approach is GitOps friendly, it introduces other issues:
- To provide the same flexibility as referencing VirtualService within a Rollout, the Rollout needs to inline a large portion of the Istio spec. However, networking is outside the responsibility of the Rollout and makes the Rollout spec unnecessarily complicated.
- If Istio introduces a feature, that feature will not be available in Argo Rollouts until implemented within Argo Rollouts.
Both of these issues adds more complexity to the users and Argo Rollouts developers compared to referencing a Virtual Service.
"},{"location":"features/traffic-management/istio/#istio-support-through-the-smi-adapter-for-istio","title":"Istio support through the SMI Adapter for Istio","text":"SMI is the Service Mesh Interface, which serves as a standard interface for all common features of a service mesh. This feature is GitOps friendly, but native Istio has extra functionality that SMI does not currently provide.
"},{"location":"features/traffic-management/kong/","title":"Kong Ingress","text":"With the introduction of the Kubernetes Gateway API it is now possible to use Argo Rollouts with all compliant implementations that support it. The integration is available with the Argo Rollouts Gateway API plugin currently hosted in Argo Labs.
Useful resources:
- The Gateway API specification
- Support of the Gateway API in Kong
- Argo Rollouts Plugin capabilities
- Plugin for the Gateway API
The process involves the following steps:
- Installing the Gateway API CRDs in your cluster
- Installing Kong and enabling the Gateway API support feature
- Creating a GatewayClass and Gateway resources
- Installing Argo Rollouts + gateway API plugin in the cluster
- Defining a Rollout that takes advantage of the plugin
For a full application that includes all manifests see the plugin example.
"},{"location":"features/traffic-management/mixed/","title":"Multiple Providers","text":"Note
Multiple trafficRouting is available since Argo Rollouts v1.2
The usage of multiple providers tries to cover scenarios where, for some reason, we have to use different providers on North-South and West-East traffic routing or any other hybrid architecture that requires the use of multiple providers.
"},{"location":"features/traffic-management/mixed/#examples-of-when-you-can-use-multiple-providers","title":"Examples of when you can use multiple providers","text":""},{"location":"features/traffic-management/mixed/#avoid-injecting-sidecars-on-your-ingress-controller","title":"Avoid injecting sidecars on your Ingress controller","text":"This is a common requirement of the service mesh and with multiple trafficRoutings you can leverage North-South traffic shifting to NGiNX and West-East traffic shifting to SMI, avoiding the need of adding the Ingress controller inside the mesh.
"},{"location":"features/traffic-management/mixed/#avoid-manipulation-of-the-host-header-at-the-ingress","title":"Avoid manipulation of the host header at the Ingress","text":"Another common side effect of adding some of the Ingress controllers into the mesh, and is caused by the usage of those mesh host headers to be pointing into a mesh hostname in order to be routed.
"},{"location":"features/traffic-management/mixed/#avoid-big-bang","title":"Avoid Big-Bang","text":"This takes place on existing fleets where downtime is very reduced or nearly impossible. To avoid big-bang-adoption the use of multiple providers can ease how teams can implement gradually new technologies. An example, where an existing fleet that is using a provider such as Ambassador and is already performing canary in a North-South fashion as part of their rollouts can gradually implement more providers such as Istio, SMI, etc.
"},{"location":"features/traffic-management/mixed/#hybrid-scenarios","title":"Hybrid Scenarios","text":"In this case, its very similar to avoiding the Big-Bang, either if it is part of the platform roadmap or a new redesign of the architecture, there are multiple scenarios where having the capacity of using multiple trafficRoutings is very much in need: gradual implementation, eased rollback of architecture or even for a fallback.
"},{"location":"features/traffic-management/mixed/#requirements","title":"Requirements","text":"The use of multiple providers requires that both providers comply with its minimum requirements independently. By example, if you want to use NGiNX and SMI you would need to have both SMI and NGiNX in place and produce the rollout configuration for both.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n strategy:\n canary:\n # Reference to a Service which the controller will update to point to the canary ReplicaSet\n canaryService: rollouts-demo-canary\n # Reference to a Service which the controller will update to point to the stable ReplicaSet\n stableService: rollouts-demo-stable\n trafficRouting:\n nginx:\n # Reference to an Ingress which has a rule pointing to the stable service (e.g. rollouts-demo-stable)\n # This ingress will be cloned with a new name, in order to achieve NGINX traffic splitting.\n stableIngress: rollouts-demo-stable\n smi: {}\n
"},{"location":"features/traffic-management/nginx/","title":"Nginx","text":"The Nginx Ingress Controller enables traffic management through one or more Ingress objects to configure an Nginx deployment that routes traffic directly to pods. Each Nginx Ingress contains multiple annotations that modify the behavior of the Nginx Deployment. For traffic management between different versions of an application, the Nginx Ingress controller provides the capability to split traffic by introducing a second Ingress object (referred to as the canary Ingress) with some special annotations. You can read more about these canary annotations on the official canary annotations documentation page. The canary Ingress ignores any other non-canary nginx annotations. Instead, it leverages the annotation settings from the primary Ingress.
The Rollout controller will always set the following two annotations on the canary Ingress (using your configured or the default nginx.ingress.kubernetes.io prefix):
canary: true to indicate that this is the canary Ingresscanary-weight: <num> to indicate what percentage of traffic to send to the canary. If all traffic is routed to the stable Service, this is set to 0
You can provide additional annotations to add to the canary Ingress via the additionalIngressAnnotations field to enable features like routing by header or cookie.
"},{"location":"features/traffic-management/nginx/#integration-with-argo-rollouts","title":"Integration with Argo Rollouts","text":"There are a couple of required fields in a Rollout to send split traffic between versions using Nginx. Below is an example of a Rollout with those fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nspec:\n ...\n strategy:\n canary:\n canaryService: canary-service # required\n stableService: stable-service # required\n trafficRouting:\n nginx:\n # Either stableIngress or stableIngresses must be configured, but not both.\n stableIngress: primary-ingress\n stableIngresses:\n - primary-ingress\n - secondary-ingress\n - tertiary-ingress\n annotationPrefix: customingress.nginx.ingress.kubernetes.io # optional\n additionalIngressAnnotations: # optional\n canary-by-header: X-Canary\n canary-by-header-value: iwantsit\n
The stable Ingress field is a reference to an Ingress in the same namespace of the Rollout. The Rollout requires the primary Ingress routes traffic to the stable Service. The Rollout checks that condition by confirming the Ingress has a backend that matches the Rollout's stableService.
The controller routes traffic to the canary Service by creating a second Ingress with the canary annotations. As the Rollout progresses through the Canary steps, the controller updates the canary Ingress's canary annotations to reflect the desired state of the Rollout enabling traffic splitting between two different versions.
Since the Nginx Ingress controller allows users to configure the annotation prefix used by the Ingress controller, Rollouts can specify the optional annotationPrefix field. The canary Ingress uses that prefix instead of the default nginx.ingress.kubernetes.io if the field set.
"},{"location":"features/traffic-management/nginx/#using-argo-rollouts-with-multiple-nginx-ingress-controllers-per-service","title":"Using Argo Rollouts with multiple NGINX ingress controllers per service","text":"Starting with v1.5, argo rollouts supports multiple Nginx ingress controllers pointing at one service with canary deployments. If only one ingress controller is needed, utilize the existing key stableIngress. If multiple ingress controllers are needed (e.g., separating internal vs external traffic), use the key stableIngresses instead. It takes an array of string values that are the names of the ingress controllers. Canary steps are applied identically across all ingress controllers.
"},{"location":"features/traffic-management/nginx/#using-argo-rollouts-with-custom-nginx-ingress-controller-names","title":"Using Argo Rollouts with custom NGINX ingress controller names","text":"As a default, the Argo Rollouts controller only operates on ingresses with the kubernetes.io/ingress.class annotation or spec.ingressClassName set to nginx. A user can configure the controller to operate on Ingresses with different class name by specifying the --nginx-ingress-classes flag. A user can list the --nginx-ingress-classes flag multiple times if the Argo Rollouts controller should operate on multiple values. This solves the case where a cluster has multiple Ingress controllers operating on different class values.
If the user would like the controller to operate on any Ingress without the kubernetes.io/ingress.class annotation or spec.ingressClassName, a user should add the following --nginx-ingress-classes ''.
"},{"location":"features/traffic-management/plugins/","title":"Traffic Router Plugins","text":"Important
Available since v1.5 - Status: Alpha
Argo Rollouts supports getting analysis metrics via 3rd party plugin system. This allows users to extend the capabilities of Rollouts to support metric providers that are not natively supported. Rollout's uses a plugin library called go-plugin to do this. You can find a sample plugin here: rollouts-plugin-trafficrouter-sample-nginx
"},{"location":"features/traffic-management/plugins/#using-a-traffic-router-plugin","title":"Using a Traffic Router Plugin","text":"There are two methods of installing and using an argo rollouts plugin. The first method is to mount up the plugin executable into the rollouts controller container. The second method is to use a HTTP(S) server to host the plugin executable.
"},{"location":"features/traffic-management/plugins/#mounting-the-plugin-executable-into-the-rollouts-controller-container","title":"Mounting the plugin executable into the rollouts controller container","text":"There are a few ways to mount the plugin executable into the rollouts controller container. Some of these will depend on your particular infrastructure. Here are a few methods:
- Using an init container to download the plugin executable
- Using a Kubernetes volume mount with a shared volume such as NFS, EBS, etc.
- Building the plugin into the rollouts controller container
Then you can use the configmap to point to the plugin executable file location. Example:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-config\ndata:\n trafficRouterPlugins: |-\n - name: \"argoproj-labs/sample-nginx\" # name of the plugin, it must match the name required by the plugin so it can find it's configuration\n location: \"file://./my-custom-plugin\" # supports http(s):// urls and file://\n
"},{"location":"features/traffic-management/plugins/#using-a-https-server-to-host-the-plugin-executable","title":"Using a HTTP(S) server to host the plugin executable","text":"Argo Rollouts supports downloading the plugin executable from a HTTP(S) server. To use this method, you will need to configure the controller via the argo-rollouts-config configmap and set pluginLocation to a http(s) url. Example:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-config\ndata:\n trafficRouterPlugins: |-\n - name: \"argoproj-labs/sample-nginx\" # name of the plugin, it must match the name required by the plugin so it can find it's configuration\n location: \"https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-sample-nginx/releases/download/v0.0.1/metric-plugin-linux-amd64\" # supports http(s):// urls and file://\n sha256: \"08f588b1c799a37bbe8d0fc74cc1b1492dd70b2c\" #optional sha256 checksum of the plugin executable\n
"},{"location":"features/traffic-management/plugins/#some-words-of-caution","title":"Some words of caution","text":"Depending on which method you use to install and the plugin, there are some things to be aware of. The rollouts controller will not start if it can not download or find the plugin executable. This means that if you are using a method of installation that requires a download of the plugin and the server hosting the plugin for some reason is not available and the rollouts controllers pod got deleted while the server was down or is coming up for the first time, it will not be able to start until the server hosting the plugin is available again.
Argo Rollouts will download the plugin at startup only once but if the pod is deleted it will need to download the plugin again on next startup. Running Argo Rollouts in HA mode can help a little with this situation because each pod will download the plugin at startup. So if a single pod gets deleted during a server outage, the other pods will still be able to take over because there will already be a plugin executable available to it. It is the responsibility of the Argo Rollouts administrator to define the plugin installation method considering the risks of each approach.
"},{"location":"features/traffic-management/plugins/#list-of-available-plugins-alphabetical-order","title":"List of Available Plugins (alphabetical order)","text":""},{"location":"features/traffic-management/plugins/#add-your-plugin-here","title":"Add Your Plugin Here","text":" - If you have created a plugin, please submit a PR to add it to this list.
"},{"location":"features/traffic-management/plugins/#rollouts-plugin-trafficrouter-sample-nginx","title":"rollouts-plugin-trafficrouter-sample-nginx","text":" - This is just a sample plugin that can be used as a starting point for creating your own plugin. It is not meant to be used in production. It is based on the built-in prometheus provider.
"},{"location":"features/traffic-management/plugins/#consul","title":"Consul","text":" - This is a plugin that allows argo-rollouts to work with Consul's service mesh for traffic shaping patterns.
"},{"location":"features/traffic-management/plugins/#contour","title":"Contour","text":" - This is a plugin that allows argo-rollouts to work with contour's resource: HTTPProxy. It enables traffic shaping patterns such as canary releases and more.
"},{"location":"features/traffic-management/plugins/#gateway-api","title":"Gateway API","text":" - Provide support for Gateway API, which includes Kuma, Traefix, cilium, Contour, GloodMesh, HAProxy, and many others.
"},{"location":"features/traffic-management/smi/","title":"Service Mesh Interface (SMI)","text":"Important
Available since v0.9.0
Warning
The Cloud Native Computing Foundation has archived the SMI Spec. The recommended way forward is to look at the Gateway API, Project Gamma and the Argo Rollouts Gateway API Plugin.
Service Mesh Interface (SMI) is a standard interface for service meshes on Kubernetes leveraged by many Service Mesh implementations (like Linkerd). SMI offers this functionality through a set of CRDs, and the Argo Rollouts controller creates these resources to manipulate the traffic routing into the desired state.
The Argo Rollout controller achieves traffic shaping by creating and manipulating the TrafficSplit CR. A TrafficSplit describes the desired traffic routing for an application and relies on the underlying Service Meshes implement that desired state. Instead of worrying about the details of a specific service mesh, a user needs to specify a root Service that clients use to communicate and a list of backends consisting of a Service and weight. The Service Mesh implementing SMI uses this spec to route traffic to the backends Services based on the weights of the backends. For Rollout users, the Argo Rollout controller creates and manipulates the TrafficSplit using the following information:
- Canary Service: Name of the service that sends traffic only to the canary pods
- Stable Service: Name of the service that sends traffic only to the stable pods
- Root Service: Name of the service that clients use to communicate. If a request comes to this root service not through a proxy, the standard Kubernetes service routing will be used.
Below is an example of a Rollout with all the required fields configured:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollout-example\nspec:\n ...\n strategy:\n canary:\n steps:\n - setWeight: 5\n - pause:\n duration: 600\n canaryService: canary-svc # required\n stableService: stable-svc # required\n trafficRouting:\n smi:\n rootService: root-svc # optional\n trafficSplitName: rollout-example-traffic-split # optional\n
With the above configuration, the controller can automate dynamic traffic splitting. First, the controller manipulates the canary and stable Service listed in the Rollout to make them only receive traffic from the respective canary and stable ReplicaSets. The controller achieves this by adding the ReplicaSet's unique pod template hash to that Service's selector. With the stable and canary Services configured, the controller creates a TrafficSplit using these Services in the backend, and the weights of the backend are dynamically configured from the current desired weight of the Rollout's canary steps. The controller sets the TrafficSplit's root service to the stableService unless the Rollout has the rootService field specified. This configured TrafficSplit along with the Service and Rollout resources enable fine-grained percentages of traffic between two versions of an application. Optionally, the user can specify a name for the traffic split. If there is no name listed in the Rollout, the controller uses the Rollout's name for the TrafficSplit. If a TrafficSplit with that name already exists and isn't owned by that Rollout, the controller marks the Rollout as an error state.
Here is the TrafficSplit created from the above Rollout:
apiVersion: split.smi-spec.io/v1alpha1\nkind: TrafficSplit\nmetadata:\n name: rollout-example-traffic-split\nspec:\n service: root-svc # controller uses the stableService if Rollout does not specify the rootService field\n backends:\n - service: stable-svc\n weight: 95\n - service: canary-svc\n weight: 5\n
As a Rollout progresses through all its steps, the controller updates the TrafficSplit's backend weights to reflect the current weight of the Rollout. When the Rollout has successfully finished executing all the steps, the controller modifies the stable Service's selector to point at the desired ReplicaSet and TrafficSplit's weight to send 100% of traffic to the stable Service.
Note
The controller defaults to using the v1alpha1 version of the TrafficSplit. The Argo Rollouts operator can change the api version used by specifying a --traffic-split-api-version flag in the controller args.
"},{"location":"features/traffic-management/traefik/","title":"Traefik","text":"You can use the Traefik Proxy for traffic management with Argo Rollouts.
The TraefikService is the object that supports the ability for weighted round robin load balancing and traffic mirroring when using Traefik as ingress.
Note
Traefik is also supported via the Argo Rollouts Gateway API plugin.
"},{"location":"features/traffic-management/traefik/#how-to-integrate-traefikservice-with-argo-rollouts-using-it-as-weighted-round-robin-load-balancer","title":"How to integrate TraefikService with Argo Rollouts using it as weighted round robin load balancer","text":"First, we need to create the TraefikService object using its ability for weighted round robin load balancing.
apiVersion: traefik.containo.us/v1alpha1\nkind: TraefikService\nmetadata:\n name: traefik-service\nspec:\n weighted:\n services:\n - name: stable-rollout # k8s service name that you need to create for stable application version\n port: 80\n - name: canary-rollout # k8s service name that you need to create for new application version\n port: 80\n
Notice, we don't specify the weight field. It is necessary to be synced with ArgoCD. If we specify this field and Argo Rollouts controller changes it, then the ArgoCD controller will notice it and will show that this resource is out of sync (if you are using Argo CD to manage your Rollout).
Secondly, we need to create the Argo Rollouts object.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n replicas: 5\n strategy:\n canary:\n canaryService: canary-rollout\n stableService: stable-rollout\n trafficRouting:\n traefik:\n weightedTraefikServiceName: traefik-service # specify traefikService resource name that we have created before\n steps:\n - setWeight: 30\n - pause: {}\n - setWeight: 40\n - pause: {duration: 10}\n - setWeight: 60\n - pause: {duration: 10}\n - setWeight: 80\n - pause: {duration: 10}\n ...\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts/","title":"Rollouts","text":"Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts/#synopsis","title":"Synopsis","text":"This command consists of multiple subcommands which can be used to manage Argo Rollouts.
kubectl argo rollouts COMMAND [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts/#examples","title":"Examples","text":"# Get guestbook rollout and watch progress\nkubectl argo rollouts get rollout guestbook -w\n\n# Pause the guestbook rollout\nkubectl argo rollouts pause guestbook\n\n# Promote the guestbook rollout\nkubectl argo rollouts promote guestbook\n\n# Abort the guestbook rollout\nkubectl argo rollouts abort guestbook\n\n# Retry the guestbook rollout\nkubectl argo rollouts retry guestbook\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts/#options","title":"Options","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n -h, --help help for kubectl-argo-rollouts\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts/#available-commands","title":"Available Commands","text":" - rollouts abort - Abort a rollout
- rollouts completion - Generate completion script
- rollouts create - Create a Rollout, Experiment, AnalysisTemplate, ClusterAnalysisTemplate, or AnalysisRun resource
- rollouts dashboard - Start UI dashboard
- rollouts get - Get details about rollouts and experiments
- rollouts lint - Lint and validate a Rollout
- rollouts list - List rollouts or experiments
- rollouts notifications - Set of CLI commands that helps manage notifications settings
- rollouts pause - Pause a rollout
- rollouts promote - Promote a rollout
- rollouts restart - Restart the pods of a rollout
- rollouts retry - Retry a rollout or experiment
- rollouts set - Update various values on resources
- rollouts status - Show the status of a rollout
- rollouts terminate - Terminate an AnalysisRun or Experiment
- rollouts undo - Undo a rollout
- rollouts version - Print version
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_abort/","title":"Rollouts Abort","text":"Abort a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_abort/#synopsis","title":"Synopsis","text":"This command stops progressing the current rollout and reverts all steps. The previous ReplicaSet will be active.
Note the 'spec.template' still represents the new rollout version. If the Rollout leaves the aborted state, it will try to go to the new version. Updating the 'spec.template' back to the previous version will fully revert the rollout.
kubectl argo rollouts abort ROLLOUT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_abort/#examples","title":"Examples","text":"# Abort a rollout\nkubectl argo rollouts abort guestbook\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_abort/#options","title":"Options","text":" -h, --help help for abort\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_abort/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_abort/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_completion/","title":"Rollouts Completion","text":"Generate completion script
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_completion/#synopsis","title":"Synopsis","text":"To load completions:
Bash:\n\n $ source <(yourprogram completion bash)\n\n # To load completions for each session, execute once:\n # Linux:\n $ yourprogram completion bash > /etc/bash_completion.d/yourprogram\n # macOS:\n $ yourprogram completion bash > /usr/local/etc/bash_completion.d/yourprogram\n\nZsh:\n\n # If shell completion is not already enabled in your environment,\n # you will need to enable it. You can execute the following once:\n\n $ echo \"autoload -U compinit; compinit\" >> ~/.zshrc\n\n # To load completions for each session, execute once:\n $ yourprogram completion zsh > \"${fpath[1]}/_yourprogram\"\n\n # You will need to start a new shell for this setup to take effect.\n\nfish:\n\n $ yourprogram completion fish | source\n\n # To load completions for each session, execute once:\n $ yourprogram completion fish > ~/.config/fish/completions/yourprogram.fish\n\nPowerShell:\n\n PS> yourprogram completion powershell | Out-String | Invoke-Expression\n\n # To load completions for every new session, run:\n PS> yourprogram completion powershell > yourprogram.ps1\n # and source this file from your PowerShell profile.\n
kubectl argo rollouts completion [bash|zsh|fish|powershell]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_completion/#options","title":"Options","text":" -h, --help help for completion\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_completion/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_completion/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create/","title":"Rollouts Create","text":"Create a Rollout, Experiment, AnalysisTemplate, ClusterAnalysisTemplate, or AnalysisRun resource
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create/#synopsis","title":"Synopsis","text":"This command creates a new Rollout, Experiment, AnalysisTemplate, ClusterAnalysisTemplate, or AnalysisRun resource from a file.
kubectl argo rollouts create [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create/#examples","title":"Examples","text":"# Create an experiment and watch it\nkubectl argo rollouts create -f my-experiment.yaml -w\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create/#options","title":"Options","text":" -f, --filename stringArray Files to use to create the resource\n -h, --help help for create\n --no-color Do not colorize output\n -w, --watch Watch live updates to the resource after creating\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create/#available-commands","title":"Available Commands","text":" - rollouts create analysisrun - Create an AnalysisRun from an AnalysisTemplate or a ClusterAnalysisTemplate
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create_analysisrun/","title":"Rollouts Create Analysisrun","text":"Create an AnalysisRun from an AnalysisTemplate or a ClusterAnalysisTemplate
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create_analysisrun/#synopsis","title":"Synopsis","text":"This command creates a new AnalysisRun from an existing AnalysisTemplate resources or from an AnalysisTemplate file.
kubectl argo rollouts create analysisrun [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create_analysisrun/#examples","title":"Examples","text":"# Create an AnalysisRun from a local AnalysisTemplate file\nkubectl argo rollouts create analysisrun --from-file my-analysis-template.yaml\n\n# Create an AnalysisRun from a AnalysisTemplate in the cluster\nkubectl argo rollouts create analysisrun --from my-analysis-template\n\n# Create an AnalysisRun from a local ClusterAnalysisTemplate file\nkubectl argo rollouts create analysisrun --global --from my-analysis-cluster-template.yaml\n\n# Create an AnalysisRun from a ClusterAnalysisTemplate in the cluster\nkubectl argo rollouts create analysisrun --global --from my-analysis-cluster-template\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create_analysisrun/#options","title":"Options","text":" -a, --argument stringArray Arguments to the parameter template\n --from string Create an AnalysisRun from an AnalysisTemplate or ClusterAnalysisTemplate in the cluster\n --from-file string Create an AnalysisRun from an AnalysisTemplate or ClusterAnalysisTemplate in a local file\n --generate-name string Use the specified generateName for the run\n --global Use a ClusterAnalysisTemplate instead of a AnalysisTemplate\n -h, --help help for analysisrun\n --instance-id string Instance-ID for the AnalysisRun\n --name string Use the specified name for the run\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create_analysisrun/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_create_analysisrun/#see-also","title":"See Also","text":" - rollouts create - Create a Rollout, Experiment, AnalysisTemplate, ClusterAnalysisTemplate, or AnalysisRun resource
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_dashboard/","title":"Rollouts Dashboard","text":"Start UI dashboard
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_dashboard/#synopsis","title":"Synopsis","text":"Start UI dashboard
kubectl argo rollouts dashboard [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_dashboard/#examples","title":"Examples","text":"# Start UI dashboard\nkubectl argo rollouts dashboard\n\n# Start UI dashboard on a specific port\nkubectl argo rollouts dashboard --port 8080\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_dashboard/#options","title":"Options","text":" -h, --help help for dashboard\n -p, --port int port to listen on (default 3100)\n --root-path string changes the root path of the dashboard (default \"rollouts\")\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_dashboard/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_dashboard/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get/","title":"Rollouts Get","text":"Get details about rollouts and experiments
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get/#synopsis","title":"Synopsis","text":"This command consists of multiple subcommands which can be used to get extended information about a rollout or experiment.
kubectl argo rollouts get <rollout|experiment> RESOURCE_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get/#examples","title":"Examples","text":"# Get a rollout\nkubectl argo rollouts get rollout guestbook\n\n# Watch a rollouts progress\nkubectl argo rollouts get rollout guestbook -w\n\n# Get an experiment\nkubectl argo rollouts get experiment my-experiment\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get/#options","title":"Options","text":" -h, --help help for get\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get/#available-commands","title":"Available Commands","text":" - rollouts get experiment - Get details about an Experiment
- rollouts get rollout - Get details about a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_experiment/","title":"Rollouts Get Experiment","text":"Get details about an Experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_experiment/#synopsis","title":"Synopsis","text":"Get details about and visual representation of a experiment. It returns a bunch of metadata on a resource and a tree view of the child resources created by the parent.
Tree view icons
Icon Kind \u27f3 Rollout \u03a3 Experiment \u03b1 AnalysisRun # Revision \u29c9 ReplicaSet \u25a1 Pod \u229e Job kubectl argo rollouts get experiment EXPERIMENT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_experiment/#examples","title":"Examples","text":"# Get an experiment\nkubectl argo rollouts get experiment my-experiment\n\n# Watch experiment progress\nkubectl argo rollouts get experiment my-experiment -w\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_experiment/#options","title":"Options","text":" -h, --help help for experiment\n --no-color Do not colorize output\n -w, --watch Watch live updates to the rollout\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_experiment/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_experiment/#see-also","title":"See Also","text":" - rollouts get - Get details about rollouts and experiments
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_rollout/","title":"Rollouts Get Rollout","text":"Get details about a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_rollout/#synopsis","title":"Synopsis","text":"Get details about and visual representation of a rollout. It returns a bunch of metadata on a resource and a tree view of the child resources created by the parent.
Tree view icons
Icon Kind \u27f3 Rollout \u03a3 Experiment \u03b1 AnalysisRun # Revision \u29c9 ReplicaSet \u25a1 Pod \u229e Job kubectl argo rollouts get rollout ROLLOUT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_rollout/#examples","title":"Examples","text":"# Get a rollout\nkubectl argo rollouts get rollout guestbook\n\n# Watch progress of a rollout\nkubectl argo rollouts get rollout guestbook -w\n\n# Watch the rollout, fail if it takes more than 60 seconds\nkubectl argo rollouts get rollout guestbook -w --timeout-seconds 60\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_rollout/#options","title":"Options","text":" -h, --help help for rollout\n --no-color Do not colorize output\n --timeout-seconds int Timeout after specified seconds\n -w, --watch Watch live updates to the rollout\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_rollout/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_get_rollout/#see-also","title":"See Also","text":" - rollouts get - Get details about rollouts and experiments
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_lint/","title":"Rollouts Lint","text":"Lint and validate a Rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_lint/#synopsis","title":"Synopsis","text":"This command lints and validates a new Rollout resource from a file.
kubectl argo rollouts lint [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_lint/#examples","title":"Examples","text":"# Lint a rollout\nkubectl argo rollouts lint -f my-rollout.yaml\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_lint/#options","title":"Options","text":" -f, --filename string File to lint\n -h, --help help for lint\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_lint/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_lint/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list/","title":"Rollouts List","text":"List rollouts or experiments
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list/#synopsis","title":"Synopsis","text":"This command consists of multiple subcommands which can be used to lists all of the rollouts or experiments for a specified namespace (uses current namespace context if namespace not specified).
kubectl argo rollouts list <rollout|experiment> [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list/#examples","title":"Examples","text":"# List rollouts\nkubectl argo rollouts list rollouts\n\n# List experiments\nkubectl argo rollouts list experiments\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list/#options","title":"Options","text":" -h, --help help for list\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list/#available-commands","title":"Available Commands","text":" - rollouts list experiments - List experiments
- rollouts list rollouts - List rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_experiments/","title":"Rollouts List Experiments","text":"List experiments
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_experiments/#synopsis","title":"Synopsis","text":"This command lists all of the experiments for a specified namespace (uses current namespace context if namespace not specified).
kubectl argo rollouts list experiments [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_experiments/#examples","title":"Examples","text":"# List rollouts\nkubectl argo rollouts list experiments\n\n# List rollouts from all namespaces\nkubectl argo rollouts list experiments --all-namespaces\n\n# List rollouts and watch for changes\nkubectl argo rollouts list experiments --watch\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_experiments/#options","title":"Options","text":" -A, --all-namespaces Include all namespaces\n -h, --help help for experiments\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_experiments/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_experiments/#see-also","title":"See Also","text":" - rollouts list - List rollouts or experiments
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_rollouts/","title":"Rollouts List Rollouts","text":"List rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_rollouts/#synopsis","title":"Synopsis","text":"This command lists all of the rollouts for a specified namespace (uses current namespace context if namespace not specified).
kubectl argo rollouts list rollouts [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_rollouts/#examples","title":"Examples","text":"# List rollouts\nkubectl argo rollouts list rollouts\n\n# List rollouts with a specific name\nkubectl argo rollouts list rollouts --name my-rollout\n\n# List rollouts from all namespaces\nkubectl argo rollouts list rollouts --all-namespaces\n\n# List rollouts and watch for changes\nkubectl argo rollouts list rollouts --watch\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_rollouts/#options","title":"Options","text":" -A, --all-namespaces Include all namespaces\n -h, --help help for rollouts\n --name string Only show rollout with specified name\n --timestamps Print timestamps on updates\n -w, --watch Watch for changes\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_rollouts/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_list_rollouts/#see-also","title":"See Also","text":" - rollouts list - List rollouts or experiments
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications/","title":"Rollouts Notifications","text":"Set of CLI commands that helps manage notifications settings
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications/#synopsis","title":"Synopsis","text":"Set of CLI commands that helps manage notifications settings
kubectl argo rollouts notifications [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications/#options","title":"Options","text":" --as string Username to impersonate for the operation\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --config-map string argo-rollouts-notification-configmap.yaml file path\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n -h, --help help for notifications\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n --kubeconfig string Path to a kube config. Only required if out-of-cluster\n -n, --namespace string If present, the namespace scope for this CLI request\n --password string Password for basic authentication to the API server\n --proxy-url string If provided, this URL will be used to connect via proxy\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --secret string argo-rollouts-notification-secret.yaml file path. Use empty secret if provided value is ':empty'\n --server string The address and port of the Kubernetes API server\n --tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n --username string Username for basic authentication to the API server\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n -v, --kloglevel int Log level for kubernetes client library\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications/#available-commands","title":"Available Commands","text":" - rollouts notifications template - Notification templates related commands
- rollouts notifications trigger - Notification triggers related commands
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template/","title":"Rollouts Notifications Template","text":"Notification templates related commands
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template/#synopsis","title":"Synopsis","text":"Notification templates related commands
kubectl argo rollouts notifications template [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template/#options","title":"Options","text":" -h, --help help for template\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --config-map string argo-rollouts-notification-configmap.yaml file path\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to a kube config. Only required if out-of-cluster\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --password string Password for basic authentication to the API server\n --proxy-url string If provided, this URL will be used to connect via proxy\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --secret string argo-rollouts-notification-secret.yaml file path. Use empty secret if provided value is ':empty'\n --server string The address and port of the Kubernetes API server\n --tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n --username string Username for basic authentication to the API server\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template/#available-commands","title":"Available Commands","text":" - rollouts notifications template get - Prints information about configured templates
- rollouts notifications template notify - Generates notification using the specified template and send it to specified recipients
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template/#see-also","title":"See Also","text":" - rollouts notifications - Set of CLI commands that helps manage notifications settings
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_get/","title":"Rollouts Notifications Template Get","text":"Prints information about configured templates
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_get/#synopsis","title":"Synopsis","text":"Prints information about configured templates
kubectl argo rollouts notifications template get [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_get/#examples","title":"Examples","text":"# prints all templates\nkubectl argo rollouts notifications template get\n# print YAML formatted app-sync-succeeded template definition\nkubectl argo rollouts notifications template get app-sync-succeeded -o=yaml\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_get/#options","title":"Options","text":" -h, --help help for get\n -o, --output string Output format. One of:json|yaml|wide|name (default \"wide\")\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_get/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --config-map string argo-rollouts-notification-configmap.yaml file path\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to a kube config. Only required if out-of-cluster\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --password string Password for basic authentication to the API server\n --proxy-url string If provided, this URL will be used to connect via proxy\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --secret string argo-rollouts-notification-secret.yaml file path. Use empty secret if provided value is ':empty'\n --server string The address and port of the Kubernetes API server\n --tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n --username string Username for basic authentication to the API server\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_get/#see-also","title":"See Also","text":" - rollouts notifications template - Notification templates related commands
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_notify/","title":"Rollouts Notifications Template Notify","text":"Generates notification using the specified template and send it to specified recipients
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_notify/#synopsis","title":"Synopsis","text":"Generates notification using the specified template and send it to specified recipients
kubectl argo rollouts notifications template notify NAME RESOURCE_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_notify/#examples","title":"Examples","text":"# Trigger notification using in-cluster config map and secret\nkubectl argo rollouts notifications template notify app-sync-succeeded guestbook --recipient slack:my-slack-channel\n\n# Render notification render generated notification in console\nkubectl argo rollouts notifications template notify app-sync-succeeded guestbook\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_notify/#options","title":"Options","text":" -h, --help help for notify\n --recipient stringArray List of recipients (default [console:stdout])\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_notify/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --config-map string argo-rollouts-notification-configmap.yaml file path\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to a kube config. Only required if out-of-cluster\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --password string Password for basic authentication to the API server\n --proxy-url string If provided, this URL will be used to connect via proxy\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --secret string argo-rollouts-notification-secret.yaml file path. Use empty secret if provided value is ':empty'\n --server string The address and port of the Kubernetes API server\n --tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n --username string Username for basic authentication to the API server\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_template_notify/#see-also","title":"See Also","text":" - rollouts notifications template - Notification templates related commands
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger/","title":"Rollouts Notifications Trigger","text":"Notification triggers related commands
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger/#synopsis","title":"Synopsis","text":"Notification triggers related commands
kubectl argo rollouts notifications trigger [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger/#options","title":"Options","text":" -h, --help help for trigger\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --config-map string argo-rollouts-notification-configmap.yaml file path\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to a kube config. Only required if out-of-cluster\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --password string Password for basic authentication to the API server\n --proxy-url string If provided, this URL will be used to connect via proxy\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --secret string argo-rollouts-notification-secret.yaml file path. Use empty secret if provided value is ':empty'\n --server string The address and port of the Kubernetes API server\n --tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n --username string Username for basic authentication to the API server\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger/#available-commands","title":"Available Commands","text":" - rollouts notifications trigger get - Prints information about configured triggers
- rollouts notifications trigger run - Evaluates specified trigger condition and prints the result
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger/#see-also","title":"See Also","text":" - rollouts notifications - Set of CLI commands that helps manage notifications settings
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_get/","title":"Rollouts Notifications Trigger Get","text":"Prints information about configured triggers
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_get/#synopsis","title":"Synopsis","text":"Prints information about configured triggers
kubectl argo rollouts notifications trigger get [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_get/#examples","title":"Examples","text":"# prints all triggers\nkubectl argo rollouts notifications trigger get\n# print YAML formatted on-sync-failed trigger definition\nkubectl argo rollouts notifications trigger get on-sync-failed -o=yaml\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_get/#options","title":"Options","text":" -h, --help help for get\n -o, --output string Output format. One of:json|yaml|wide|name (default \"wide\")\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_get/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --config-map string argo-rollouts-notification-configmap.yaml file path\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to a kube config. Only required if out-of-cluster\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --password string Password for basic authentication to the API server\n --proxy-url string If provided, this URL will be used to connect via proxy\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --secret string argo-rollouts-notification-secret.yaml file path. Use empty secret if provided value is ':empty'\n --server string The address and port of the Kubernetes API server\n --tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n --username string Username for basic authentication to the API server\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_get/#see-also","title":"See Also","text":" - rollouts notifications trigger - Notification triggers related commands
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_run/","title":"Rollouts Notifications Trigger Run","text":"Evaluates specified trigger condition and prints the result
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_run/#synopsis","title":"Synopsis","text":"Evaluates specified trigger condition and prints the result
kubectl argo rollouts notifications trigger run NAME RESOURCE_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_run/#examples","title":"Examples","text":"# Execute trigger configured in 'argocd-notification-cm' ConfigMap\nkubectl argo rollouts notifications trigger run on-sync-status-unknown ./sample-app.yaml\n\n# Execute trigger using my-config-map.yaml instead of 'argo-rollouts-notification-configmap' ConfigMap\nkubectl argo rollouts notifications trigger run on-sync-status-unknown ./sample-app.yaml \\\n--config-map ./my-config-map.yaml\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_run/#options","title":"Options","text":" -h, --help help for run\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_run/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --config-map string argo-rollouts-notification-configmap.yaml file path\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to a kube config. Only required if out-of-cluster\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --password string Password for basic authentication to the API server\n --proxy-url string If provided, this URL will be used to connect via proxy\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --secret string argo-rollouts-notification-secret.yaml file path. Use empty secret if provided value is ':empty'\n --server string The address and port of the Kubernetes API server\n --tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n --username string Username for basic authentication to the API server\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_notifications_trigger_run/#see-also","title":"See Also","text":" - rollouts notifications trigger - Notification triggers related commands
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_pause/","title":"Rollouts Pause","text":"Pause a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_pause/#synopsis","title":"Synopsis","text":"Set the rollout paused state to 'true'
kubectl argo rollouts pause ROLLOUT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_pause/#examples","title":"Examples","text":"# Pause a rollout\nkubectl argo rollouts pause guestbook\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_pause/#options","title":"Options","text":" -h, --help help for pause\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_pause/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_pause/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_promote/","title":"Rollouts Promote","text":"Promote a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_promote/#synopsis","title":"Synopsis","text":"Promote a rollout
Promotes a rollout paused at a canary step, or a paused blue-green pre-promotion. To skip analysis, pauses and steps entirely, use '--full' to fully promote the rollout
kubectl argo rollouts promote ROLLOUT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_promote/#examples","title":"Examples","text":"# Promote a paused rollout\nkubectl argo rollouts promote guestbook\n\n# Fully promote a rollout to desired version, skipping analysis, pauses, and steps\nkubectl argo rollouts promote guestbook --full\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_promote/#options","title":"Options","text":" --full Perform a full promotion, skipping analysis, pauses, and steps\n -h, --help help for promote\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_promote/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_promote/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_restart/","title":"Rollouts Restart","text":"Restart the pods of a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_restart/#synopsis","title":"Synopsis","text":"Restart the pods of a rollout
kubectl argo rollouts restart ROLLOUT [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_restart/#examples","title":"Examples","text":"# Restart the pods of a rollout in now\nkubectl argo rollouts restart ROLLOUT_NAME\n\n# Restart the pods of a rollout in ten seconds\nkubectl argo rollouts restart ROLLOUT_NAME --in 10s\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_restart/#options","title":"Options","text":" -h, --help help for restart\n -i, --in string Amount of time before a restart. (e.g. 30s, 5m, 1h)\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_restart/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_restart/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry/","title":"Rollouts Retry","text":"Retry a rollout or experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry/#synopsis","title":"Synopsis","text":"This command consists of multiple subcommands which can be used to restart an aborted rollout or a failed experiment.
kubectl argo rollouts retry <rollout|experiment> RESOURCE_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry/#examples","title":"Examples","text":"# Retry an aborted rollout\nkubectl argo rollouts retry rollout guestbook\n\n# Retry a failed experiment\nkubectl argo rollouts retry experiment my-experiment\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry/#options","title":"Options","text":" -h, --help help for retry\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry/#available-commands","title":"Available Commands","text":" - rollouts retry experiment - Retry an experiment
- rollouts retry rollout - Retry an aborted rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_experiment/","title":"Rollouts Retry Experiment","text":"Retry an experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_experiment/#synopsis","title":"Synopsis","text":"Retry a failed experiment.
kubectl argo rollouts retry experiment EXPERIMENT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_experiment/#examples","title":"Examples","text":"# Retry an experiment\nkubectl argo rollouts retry experiment my-experiment\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_experiment/#options","title":"Options","text":" -h, --help help for experiment\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_experiment/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_experiment/#see-also","title":"See Also","text":" - rollouts retry - Retry a rollout or experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_rollout/","title":"Rollouts Retry Rollout","text":"Retry an aborted rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_rollout/#synopsis","title":"Synopsis","text":"Retry an aborted rollout
kubectl argo rollouts retry rollout ROLLOUT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_rollout/#examples","title":"Examples","text":"# Retry an aborted rollout\nkubectl argo rollouts retry rollout guestbook\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_rollout/#options","title":"Options","text":" -h, --help help for rollout\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_rollout/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_retry_rollout/#see-also","title":"See Also","text":" - rollouts retry - Retry a rollout or experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set/","title":"Rollouts Set","text":"Update various values on resources
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set/#synopsis","title":"Synopsis","text":"This command consists of multiple subcommands which can be used to update rollout resources.
kubectl argo rollouts set COMMAND [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set/#examples","title":"Examples","text":"# Set rollout image\nkubectl argo rollouts set image my-rollout demo=argoproj/rollouts-demo:yellow\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set/#options","title":"Options","text":" -h, --help help for set\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set/#available-commands","title":"Available Commands","text":" - rollouts set image - Update the image of a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set_image/","title":"Rollouts Set Image","text":"Update the image of a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set_image/#synopsis","title":"Synopsis","text":"Update the image of a rollout
kubectl argo rollouts set image ROLLOUT_NAME CONTAINER=IMAGE [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set_image/#examples","title":"Examples","text":"# Set rollout image (containers contains 'initContainer', 'container', 'ephemeralContainer')\nkubectl argo rollouts set image my-rollout containerName=imageName\n\n# Set rollout image for all containers\nkubectl argo rollouts set image my-rollout *=imageName\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set_image/#options","title":"Options","text":" -h, --help help for image\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set_image/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_set_image/#see-also","title":"See Also","text":" - rollouts set - Update various values on resources
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_status/","title":"Rollouts Status","text":"Show the status of a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_status/#synopsis","title":"Synopsis","text":"Watch rollout until it finishes or the timeout is exceeded. Returns success if the rollout is healthy upon completion and an error otherwise.
kubectl argo rollouts status ROLLOUT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_status/#examples","title":"Examples","text":"# Watch the rollout until it succeeds\nkubectl argo rollouts status guestbook\n\n# Show the rollout status\nkubectl argo rollouts status guestbook --watch false\n\n# Watch the rollout until it succeeds, fail if it takes more than 60 seconds\nkubectl argo rollouts status --timeout 60s guestbook\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_status/#options","title":"Options","text":" -h, --help help for status\n -t, --timeout duration The length of time to watch before giving up. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). Zero means wait forever\n -w, --watch Watch the status of the rollout until it's done (default true)\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_status/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_status/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate/","title":"Rollouts Terminate","text":"Terminate an AnalysisRun or Experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate/#synopsis","title":"Synopsis","text":"This command consists of multiple subcommands which can be used to terminate an AnalysisRun or Experiment that is in progress.
kubectl argo rollouts terminate <analysisrun|experiment> RESOURCE_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate/#examples","title":"Examples","text":"# Terminate an AnalysisRun\nkubectl argo rollouts terminate analysisrun guestbook-877894d5b-4-success-rate.1\n\n# Terminate a failed experiment\nkubectl argo rollouts terminate experiment my-experiment\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate/#options","title":"Options","text":" -h, --help help for terminate\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate/#available-commands","title":"Available Commands","text":" - rollouts terminate analysisrun - Terminate an AnalysisRun
- rollouts terminate experiment - Terminate an experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_analysisrun/","title":"Rollouts Terminate Analysisrun","text":"Terminate an AnalysisRun
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_analysisrun/#synopsis","title":"Synopsis","text":"This command terminates an AnalysisRun.
kubectl argo rollouts terminate analysisrun ANALYSISRUN_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_analysisrun/#examples","title":"Examples","text":"# Terminate an AnalysisRun\nkubectl argo rollouts terminate analysisrun guestbook-877894d5b-4-success-rate.1\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_analysisrun/#options","title":"Options","text":" -h, --help help for analysisrun\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_analysisrun/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_analysisrun/#see-also","title":"See Also","text":" - rollouts terminate - Terminate an AnalysisRun or Experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_experiment/","title":"Rollouts Terminate Experiment","text":"Terminate an experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_experiment/#synopsis","title":"Synopsis","text":"This command terminates an Experiment.
kubectl argo rollouts terminate experiment EXPERIMENT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_experiment/#examples","title":"Examples","text":"# Terminate an experiment\nkubectl argo rollouts terminate experiment my-experiment\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_experiment/#options","title":"Options","text":" -h, --help help for experiment\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_experiment/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_terminate_experiment/#see-also","title":"See Also","text":" - rollouts terminate - Terminate an AnalysisRun or Experiment
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_undo/","title":"Rollouts Undo","text":"Undo a rollout
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_undo/#synopsis","title":"Synopsis","text":"Rollback to the previous rollout.
kubectl argo rollouts undo ROLLOUT_NAME [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_undo/#examples","title":"Examples","text":"# Undo a rollout\nkubectl argo rollouts undo guestbook\n\n# Undo a rollout to revision 3\nkubectl argo rollouts undo guestbook --to-revision=3\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_undo/#options","title":"Options","text":" -h, --help help for undo\n --to-revision int The revision to rollback to. Default to 0 (last revision).\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_undo/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_undo/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_version/","title":"Rollouts Version","text":"Print version
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_version/#synopsis","title":"Synopsis","text":"Show the version and build information of the Argo Rollouts plugin.
kubectl argo rollouts version [flags]\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_version/#examples","title":"Examples","text":"# Get full version info\nkubectl argo rollouts version\n\n# Get just plugin version number\nkubectl argo rollouts version --short\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_version/#options","title":"Options","text":" -h, --help help for version\n --short print just the version number\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_version/#options-inherited-from-parent-commands","title":"Options inherited from parent commands","text":" --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace.\n --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --as-uid string UID to impersonate for the operation.\n --cache-dir string Default cache directory (default \"$HOME/.kube/cache\")\n --certificate-authority string Path to a cert file for the certificate authority\n --client-certificate string Path to a client certificate file for TLS\n --client-key string Path to a client key file for TLS\n --cluster string The name of the kubeconfig cluster to use\n --context string The name of the kubeconfig context to use\n --disable-compression If true, opt-out of response compression for all requests to the server\n --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -v, --kloglevel int Log level for kubernetes client library\n --kubeconfig string Path to the kubeconfig file to use for CLI requests.\n --loglevel string Log level for kubectl argo rollouts (default \"info\")\n -n, --namespace string If present, the namespace scope for this CLI request\n --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n -s, --server string The address and port of the Kubernetes API server\n --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used\n --token string Bearer token for authentication to the API server\n --user string The name of the kubeconfig user to use\n
"},{"location":"generated/kubectl-argo-rollouts/kubectl-argo-rollouts_version/#see-also","title":"See Also","text":" - rollouts - Manage argo rollouts
"},{"location":"generated/notification-services/alertmanager/","title":"Alertmanager","text":""},{"location":"generated/notification-services/alertmanager/#parameters","title":"Parameters","text":"The notification service is used to push events to Alertmanager, and the following settings need to be specified:
targets - the alertmanager service address, array typescheme - optional, default is \"http\", e.g. http or httpsapiPath - optional, default is \"/api/v2/alerts\"insecureSkipVerify - optional, default is \"false\", when scheme is https whether to skip the verification of cabasicAuth - optional, server authbearerToken - optional, server authtimeout - optional, the timeout in seconds used when sending alerts, default is \"3 seconds\"
basicAuth or bearerToken is used for authentication, you can choose one. If the two are set at the same time, basicAuth takes precedence over bearerToken.
"},{"location":"generated/notification-services/alertmanager/#example","title":"Example","text":""},{"location":"generated/notification-services/alertmanager/#prometheus-alertmanager-config","title":"Prometheus Alertmanager config","text":"global:\n resolve_timeout: 5m\n\nroute:\n group_by: ['alertname']\n group_wait: 10s\n group_interval: 10s\n repeat_interval: 1h\n receiver: 'default'\nreceivers:\n- name: 'default'\n webhook_configs:\n - send_resolved: false\n url: 'http://10.5.39.39:10080/api/alerts/webhook'\n
You should turn off \"send_resolved\" or you will receive unnecessary recovery notifications after \"resolve_timeout\".
"},{"location":"generated/notification-services/alertmanager/#send-one-alertmanager-without-auth","title":"Send one alertmanager without auth","text":"apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.alertmanager: |\n targets:\n - 10.5.39.39:9093\n
"},{"location":"generated/notification-services/alertmanager/#send-alertmanager-cluster-with-custom-api-path","title":"Send alertmanager cluster with custom api path","text":"If your alertmanager has changed the default api, you can customize \"apiPath\".
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.alertmanager: |\n targets:\n - 10.5.39.39:443\n scheme: https\n apiPath: /api/events\n insecureSkipVerify: true\n
"},{"location":"generated/notification-services/alertmanager/#send-high-availability-alertmanager-with-auth","title":"Send high availability alertmanager with auth","text":"Store auth token in argo-rollouts-notification-secret Secret and use configure in argo-rollouts-notification-configmap ConfigMap.
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n alertmanager-username: <username>\n alertmanager-password: <password>\n alertmanager-bearer-token: <token>\n
- with basicAuth
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.alertmanager: |\n targets:\n - 10.5.39.39:19093\n - 10.5.39.39:29093\n - 10.5.39.39:39093\n scheme: https\n apiPath: /api/v2/alerts\n insecureSkipVerify: true\n basicAuth:\n username: $alertmanager-username\n password: $alertmanager-password \n
- with bearerToken
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.alertmanager: |\n targets:\n - 10.5.39.39:19093\n - 10.5.39.39:29093\n - 10.5.39.39:39093\n scheme: https\n apiPath: /api/v2/alerts\n insecureSkipVerify: true\n bearerToken: $alertmanager-bearer-token\n
"},{"location":"generated/notification-services/alertmanager/#templates","title":"Templates","text":" labels - at least one label pair required, implement different notification strategies according to alertmanager routingannotations - optional, specifies a set of information labels, which can be used to store longer additional information, but only for displaygeneratorURL - optional, default is '{{.app.spec.source.repoURL}}', backlink used to identify the entity that caused this alert in the client
the label or annotations or generatorURL values can be templated.
context: |\n argocdUrl: https://example.com/argocd\n\ntemplate.app-deployed: |\n message: Application {{.app.metadata.name}} has been healthy.\n alertmanager:\n labels:\n fault_priority: \"P5\"\n event_bucket: \"deploy\"\n event_status: \"succeed\"\n recipient: \"{{.recipient}}\"\n annotations:\n application: '<a href=\"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}\">{{.app.metadata.name}}</a>'\n author: \"{{(call .repo.GetCommitMetadata .app.status.sync.revision).Author}}\"\n message: \"{{(call .repo.GetCommitMetadata .app.status.sync.revision).Message}}\"\n
You can do targeted push on Alertmanager according to labels.
template.app-deployed: |\n message: Application {{.app.metadata.name}} has been healthy.\n alertmanager:\n labels:\n alertname: app-deployed\n fault_priority: \"P5\"\n event_bucket: \"deploy\"\n
There is a special label alertname. If you don\u2019t set its value, it will be equal to the template name by default.
"},{"location":"generated/notification-services/awssqs/","title":"AWS SQS","text":""},{"location":"generated/notification-services/awssqs/#parameters","title":"Parameters","text":"This notification service is capable of sending simple messages to AWS SQS queue.
queue - name of the queue you are intending to send messages to. Can be overridden with target destination annotation.region - region of the sqs queue can be provided via env variable AWS_DEFAULT_REGIONkey - optional, aws access key must be either referenced from a secret via variable or via env variable AWS_ACCESS_KEY_IDsecret - optional, aws access secret must be either referenced from a secret via variable or via env variable AWS_SECRET_ACCESS_KEYaccount optional, external accountId of the queueendpointUrl optional, useful for development with localstack
"},{"location":"generated/notification-services/awssqs/#example","title":"Example","text":""},{"location":"generated/notification-services/awssqs/#using-secret-for-credential-retrieval","title":"Using Secret for credential retrieval:","text":"Resource Annotation:
apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: nginx-deployment\n annotations:\n notifications.argoproj.io/subscribe.on-deployment-ready.awssqs: \"overwrite-myqueue\"\n
- ConfigMap
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.awssqs: |\n region: \"us-east-2\"\n queue: \"myqueue\"\n account: \"1234567\"\n key: \"$awsaccess_key\"\n secret: \"$awsaccess_secret\"\n\n template.deployment-ready: |\n message: |\n Deployment {{.obj.metadata.name}} is ready!\n\n trigger.on-deployment-ready: |\n - when: any(obj.status.conditions, {.type == 'Available' && .status == 'True'})\n send: [deployment-ready]\n - oncePer: obj.metadata.annotations[\"generation\"]\n
Secret apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n awsaccess_key: test\n awsaccess_secret: test\n
"},{"location":"generated/notification-services/awssqs/#minimal-configuration-using-aws-env-variables","title":"Minimal configuration using AWS Env variables","text":"Ensure the following list of environment variables are injected via OIDC, or another method. And assuming SQS is local to the account. You may skip usage of secret for sensitive data and omit other parameters. (Setting parameters via ConfigMap takes precedent.)
Variables:
export AWS_ACCESS_KEY_ID=\"test\"\nexport AWS_SECRET_ACCESS_KEY=\"test\"\nexport AWS_DEFAULT_REGION=\"us-east-1\"\n
Resource Annotation:
apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: nginx-deployment\n annotations:\n notifications.argoproj.io/subscribe.on-deployment-ready.awssqs: \"\"\n
- ConfigMap
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.awssqs: |\n queue: \"myqueue\"\n\n template.deployment-ready: |\n message: |\n Deployment {{.obj.metadata.name}} is ready!\n\n trigger.on-deployment-ready: |\n - when: any(obj.status.conditions, {.type == 'Available' && .status == 'True'})\n send: [deployment-ready]\n - oncePer: obj.metadata.annotations[\"generation\"]\n
"},{"location":"generated/notification-services/awssqs/#fifo-sqs-queues","title":"FIFO SQS Queues","text":"FIFO queues require a MessageGroupId to be sent along with every message, every message with a matching MessageGroupId will be processed one by one in order.
To send to a FIFO SQS Queue you must include a messageGroupId in the template such as in the example below:
template.deployment-ready: |\n message: |\n Deployment {{.obj.metadata.name}} is ready!\n messageGroupId: {{.obj.metadata.name}}-deployment\n
"},{"location":"generated/notification-services/email/","title":"Email","text":""},{"location":"generated/notification-services/email/#parameters","title":"Parameters","text":"The Email notification service sends email notifications using SMTP protocol and requires specifying the following settings:
host - the SMTP server host nameport - the SMTP server portusername - usernamepassword - passwordfrom - from email addresshtml - optional bool, true or falseinsecure_skip_verify - optional bool, true or false
"},{"location":"generated/notification-services/email/#example","title":"Example","text":"The following snippet contains sample Gmail service configuration:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.email.gmail: |\n username: $email-username\n password: $email-password\n host: smtp.gmail.com\n port: 465\n from: $email-username\n
Without authentication:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.email.example: |\n host: smtp.example.com\n port: 587\n from: $email-username\n
"},{"location":"generated/notification-services/email/#template","title":"Template","text":"Notification templates support specifying subject for email notifications:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n template.app-sync-succeeded: |\n email:\n subject: Application {{.app.metadata.name}} has been successfully synced.\n message: |\n {{if eq .serviceType \"slack\"}}:white_check_mark:{{end}} Application {{.app.metadata.name}} has been successfully synced at {{.app.status.operationState.finishedAt}}.\n Sync operation details are available at: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true .\n
"},{"location":"generated/notification-services/github/","title":"GitHub","text":""},{"location":"generated/notification-services/github/#parameters","title":"Parameters","text":"The GitHub notification service changes commit status using GitHub Apps and requires specifying the following settings:
appID - the app idinstallationID - the app installation idprivateKey - the app private keyenterpriseBaseURL - optional URL, e.g. https://git.example.com/
"},{"location":"generated/notification-services/github/#configuration","title":"Configuration","text":" - Create a GitHub Apps using https://github.com/settings/apps/new
- Change repository permissions to enable write commit statuses and/or deployments and/or pull requests comments
- Generate a private key, and download it automatically
- Install app to account
- Store privateKey in
argo-rollouts-notification-secret Secret and configure GitHub integration in argo-rollouts-notification-configmap ConfigMap
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.github: |\n appID: <app-id>\n installationID: <installation-id>\n privateKey: $github-privateKey\n
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n github-privateKey: |\n -----BEGIN RSA PRIVATE KEY-----\n (snip)\n -----END RSA PRIVATE KEY-----\n
- Create subscription for your GitHub integration
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.<trigger-name>.github: \"\"\n
"},{"location":"generated/notification-services/github/#templates","title":"Templates","text":"template.app-deployed: |\n message: |\n Application {{.app.metadata.name}} is now running new version of deployments manifests.\n github:\n repoURLPath: \"{{.app.spec.source.repoURL}}\"\n revisionPath: \"{{.app.status.operationState.syncResult.revision}}\"\n status:\n state: success\n label: \"continuous-delivery/{{.app.metadata.name}}\"\n targetURL: \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true\"\n deployment:\n state: success\n environment: production\n environmentURL: \"https://{{.app.metadata.name}}.example.com\"\n logURL: \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true\"\n requiredContexts: []\n autoMerge: true\n transientEnvironment: false\n pullRequestComment:\n content: |\n Application {{.app.metadata.name}} is now running new version of deployments manifests.\n See more here: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true\n
Notes: - If the message is set to 140 characters or more, it will be truncated. - If github.repoURLPath and github.revisionPath are same as above, they can be omitted. - Automerge is optional and true by default for github deployments to ensure the requested ref is up to date with the default branch. Setting this option to false is required if you would like to deploy older refs in your default branch. For more information see the GitHub Deployment API Docs. - If github.pullRequestComment.content is set to 65536 characters or more, it will be truncated.
"},{"location":"generated/notification-services/googlechat/","title":"Google Chat","text":""},{"location":"generated/notification-services/googlechat/#parameters","title":"Parameters","text":"The Google Chat notification service send message notifications to a google chat webhook. This service uses the following settings:
webhooks - a map of the form webhookName: webhookUrl
"},{"location":"generated/notification-services/googlechat/#configuration","title":"Configuration","text":" - Open
Google chat and go to the space to which you want to send messages - From the menu at the top of the page, select Configure Webhooks
- Under Incoming Webhooks, click Add Webhook
- Give a name to the webhook, optionally add an image and click Save
- Copy the URL next to your webhook
- Store the URL in
argocd-notification-secret and declare it in argo-rollouts-notification-configmap
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.googlechat: |\n webhooks:\n spaceName: $space-webhook-url\n
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n space-webhook-url: https://chat.googleapis.com/v1/spaces/<space_id>/messages?key=<key>&token=<token> \n
- Create a subscription for your space
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.googlechat: spaceName\n
"},{"location":"generated/notification-services/googlechat/#templates","title":"Templates","text":"You can send simple text or card messages to a Google Chat space. A simple text message template can be defined as follows:
template.app-sync-succeeded: |\n message: The app {{ .app.metadata.name }} has successfully synced!\n
A card message can be defined as follows:
template.app-sync-succeeded: |\n googlechat:\n cardsV2: |\n - header:\n title: ArgoCD Bot Notification\n sections:\n - widgets:\n - decoratedText:\n text: The app {{ .app.metadata.name }} has successfully synced!\n - widgets:\n - decoratedText:\n topLabel: Repository\n text: {{ call .repo.RepoURLToHTTPS .app.spec.source.repoURL }}\n - decoratedText:\n topLabel: Revision\n text: {{ .app.spec.source.targetRevision }}\n - decoratedText:\n topLabel: Author\n text: {{ (call .repo.GetCommitMetadata .app.status.sync.revision).Author }}\n
All Card fields are supported and can be used in notifications. It is also possible to use the previous (now deprecated) cards key to use the legacy card fields, but this is not recommended as Google has deprecated this field and recommends using the newer cardsV2. The card message can be written in JSON too.
"},{"location":"generated/notification-services/googlechat/#chat-threads","title":"Chat Threads","text":"It is possible send both simple text and card messages in a chat thread by specifying a unique key for the thread. The thread key can be defined as follows:
template.app-sync-succeeded: |\n message: The app {{ .app.metadata.name }} has successfully synced!\n googlechat:\n threadKey: {{ .app.metadata.name }}\n
"},{"location":"generated/notification-services/grafana/","title":"Grafana","text":"To be able to create Grafana annotation with argocd-notifications you have to create an API Key inside your Grafana.
Available parameters :
apiURL - the server url, e.g. https://grafana.example.comapiKey - the API key for the serviceaccount-
insecureSkipVerify - optional bool, true or false
-
Login to your Grafana instance as admin
- On the left menu, go to Configuration / API Keys
- Click \"Add API Key\"
- Fill the Key with name
ArgoCD Notification, role Editor and Time to Live 10y (for example) - Click on Add button
- Store apiKey in
argo-rollouts-notification-secret Secret and Copy your API Key and define it in argo-rollouts-notification-configmap ConfigMap
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.grafana: |\n apiUrl: https://grafana.example.com/api\n apiKey: $grafana-api-key\n
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n grafana-api-key: api-key\n
- Create subscription for your Grafana integration
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.<trigger-name>.grafana: tag1|tag2 # list of tags separated with |\n
- Change the annotations settings
"},{"location":"generated/notification-services/mattermost/","title":"Mattermost","text":""},{"location":"generated/notification-services/mattermost/#parameters","title":"Parameters","text":" apiURL - the server url, e.g. https://mattermost.example.comtoken - the bot tokeninsecureSkipVerify - optional bool, true or false
"},{"location":"generated/notification-services/mattermost/#configuration","title":"Configuration","text":" - Create a bot account and copy token after creating it
- Invite team
- Store token in
argo-rollouts-notification-secret Secret and configure Mattermost integration in argo-rollouts-notification-configmap ConfigMap
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.mattermost: |\n apiURL: <api-url>\n token: $mattermost-token\n
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n mattermost-token: token\n
-
Copy channel id
-
Create subscription for your Mattermost integration
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.<trigger-name>.mattermost: <channel-id>\n
"},{"location":"generated/notification-services/mattermost/#templates","title":"Templates","text":"You can reuse the template of slack. Mattermost is compatible with attachments of Slack. See Mattermost Integration Guide.
template.app-deployed: |\n message: |\n Application {{.app.metadata.name}} is now running new version of deployments manifests.\n mattermost:\n attachments: |\n [{\n \"title\": \"{{.app.metadata.name}}\",\n \"title_link\": \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}\",\n \"color\": \"#18be52\",\n \"fields\": [{\n \"title\": \"Sync Status\",\n \"value\": \"{{.app.status.sync.status}}\",\n \"short\": true\n }, {\n \"title\": \"Repository\",\n \"value\": \"{{.app.spec.source.repoURL}}\",\n \"short\": true\n }]\n }]\n
"},{"location":"generated/notification-services/newrelic/","title":"NewRelic","text":""},{"location":"generated/notification-services/newrelic/#parameters","title":"Parameters","text":" apiURL - the api server url, e.g. https://api.newrelic.comapiKey - a NewRelic ApiKey
"},{"location":"generated/notification-services/newrelic/#configuration","title":"Configuration","text":" - Create a NewRelic Api Key
- Store apiKey in
argo-rollouts-notification-secret Secret and configure NewRelic integration in argo-rollouts-notification-configmap ConfigMap
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.newrelic: |\n apiURL: <api-url>\n apiKey: $newrelic-apiKey\n
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n newrelic-apiKey: apiKey\n
- Copy Application ID
- Create subscription for your NewRelic integration
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.<trigger-name>.newrelic: <app-id>\n
"},{"location":"generated/notification-services/newrelic/#templates","title":"Templates","text":" description - optional, high-level description of this deployment, visible in the Summary page and on the Deployments page when you select an individual deployment.- Defaults to
message changelog - optional, A summary of what changed in this deployment, visible in the Deployments page when you select (selected deployment) > Change log.- Defaults to
{{(call .repo.GetCommitMetadata .app.status.sync.revision).Message}} user - optional, A username to associate with the deployment, visible in the Summary and on the Deployments.- Defaults to
{{(call .repo.GetCommitMetadata .app.status.sync.revision).Author}}
context: |\n argocdUrl: https://example.com/argocd\n\ntemplate.app-deployed: |\n message: Application {{.app.metadata.name}} has successfully deployed.\n newrelic:\n description: Application {{.app.metadata.name}} has successfully deployed\n
"},{"location":"generated/notification-services/opsgenie/","title":"Opsgenie","text":"To be able to send notifications with argocd-notifications you have to create an API Integration inside your Opsgenie Team.
- Login to Opsgenie at https://app.opsgenie.com or https://app.eu.opsgenie.com (if you have an account in the european union)
- Make sure you already have a team, if not follow this guide https://docs.opsgenie.com/docs/teams
- Click \"Teams\" in the Menu on the left
- Select the team that you want to notify
- In the teams configuration menu select \"Integrations\"
- Click \"Add Integration\" in the top right corner
- Select \"API\" integration
- Give your integration a name, copy the \"API key\" and safe it somewhere for later
- Click \"Edit\" in the integration settings
- Make sure the checkbox for \"Create and Update Access\" is selected, disable the other checkboxes to remove unnecessary permissions
- Click \"Save\" at the bottom
- Click \"Turn on integration\" in the top right corner
- Check your browser for the correct server apiURL. If it is \"app.opsgenie.com\" then use the US/international api url
api.opsgenie.com in the next step, otherwise use api.eu.opsgenie.com (European API). - You are finished with configuring Opsgenie. Now you need to configure argocd-notifications. Use the apiUrl, the team name and the apiKey to configure the Opsgenie integration in the
argo-rollouts-notification-secret secret. - You can find the example
argo-rollouts-notification-configmap configuration at the below.
Option Required Type Description Example description True string Description field of the alert that is generally used to provide a detailed information about the alert. Hello from Argo CD! priority False string Priority level of the alert. Possible values are P1, P2, P3, P4 and P5. Default value is P3. P1 alias False string Client-defined identifier of the alert, that is also the key element of Alert De-Duplication. Life is too short for no alias note False string Additional note that will be added while creating the alert. Error from Argo CD! apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.opsgenie: |\n apiUrl: <api-url>\n apiKeys:\n <your-team>: <integration-api-key>\n template.opsgenie: |\n message: |\n [Argo CD] Application {{.app.metadata.name}} has a problem.\n opsgenie:\n description: |\n Application: {{.app.metadata.name}}\n Health Status: {{.app.status.health.status}}\n Operation State Phase: {{.app.status.operationState.phase}}\n Sync Status: {{.app.status.sync.status}}\n priority: P1\n alias: {{.app.metadata.name}}\n note: Error from Argo CD!\n trigger.on-a-problem: |\n - description: Application has a problem.\n send:\n - opsgenie\n when: app.status.health.status == 'Degraded' or app.status.operationState.phase in ['Error', 'Failed'] or app.status.sync.status == 'Unknown'\n
- Add annotation in application yaml file to enable notifications for specific Argo CD app.
apiVersion: argoproj.io/v1alpha1\n kind: Application\n metadata:\n annotations:\n notifications.argoproj.io/subscribe.on-a-problem.opsgenie: <your-team>\n
"},{"location":"generated/notification-services/overview/","title":"Overview","text":"The notification services represent integration with services such as slack, email or custom webhook. Services are configured in argo-rollouts-notification-configmap ConfigMap using service.<type>.(<custom-name>) keys and might reference sensitive data from argo-rollouts-notification-secret Secret. Following example demonstrates slack service configuration:
service.slack: |\n token: $slack-token\n
The slack indicates that service sends slack notification; name is missing and defaults to slack.
"},{"location":"generated/notification-services/overview/#sensitive-data","title":"Sensitive Data","text":"Sensitive data like authentication tokens should be stored in <secret-name> Secret and can be referenced in service configuration using $<secret-key> format. For example $slack-token referencing value of key slack-token in <secret-name> Secret.
"},{"location":"generated/notification-services/overview/#custom-names","title":"Custom Names","text":"Service custom names allow configuring two instances of the same service type.
service.slack.workspace1: |\n token: $slack-token-workspace1\n service.slack.workspace2: |\n token: $slack-token-workspace2\n
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.workspace1: my-channel\n notifications.argoproj.io/subscribe.on-sync-succeeded.workspace2: my-channel\n
"},{"location":"generated/notification-services/overview/#service-types","title":"Service Types","text":" - AwsSqs
- Email
- GitHub
- Slack
- Mattermost
- Opsgenie
- Grafana
- Webhook
- Telegram
- Teams
- Google Chat
- Rocket.Chat
- Pushover
- Alertmanager
"},{"location":"generated/notification-services/pagerduty/","title":"PagerDuty","text":""},{"location":"generated/notification-services/pagerduty/#parameters","title":"Parameters","text":"The PagerDuty notification service is used to create PagerDuty incidents and requires specifying the following settings:
pagerdutyToken - the PagerDuty auth tokenfrom - email address of a valid user associated with the account making the request.serviceID - The ID of the resource.
"},{"location":"generated/notification-services/pagerduty/#example","title":"Example","text":"The following snippet contains sample PagerDuty service configuration:
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n pagerdutyToken: <pd-api-token>\n
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.pagerduty: |\n token: $pagerdutyToken\n from: <emailid>\n
"},{"location":"generated/notification-services/pagerduty/#template","title":"Template","text":"Notification templates support specifying subject for PagerDuty notifications:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n template.rollout-aborted: |\n message: Rollout {{.rollout.metadata.name}} is aborted.\n pagerduty:\n title: \"Rollout {{.rollout.metadata.name}}\"\n urgency: \"high\"\n body: \"Rollout {{.rollout.metadata.name}} aborted \"\n priorityID: \"<priorityID of incident>\"\n
NOTE: A Priority is a label representing the importance and impact of an incident. This is only available on Standard and Enterprise plans of pagerduty.
"},{"location":"generated/notification-services/pagerduty/#annotation","title":"Annotation","text":"Annotation sample for pagerduty notifications:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-rollout-aborted.pagerduty: \"<serviceID for PagerDuty>\"\n
"},{"location":"generated/notification-services/pagerduty_v2/","title":"PagerDuty V2","text":""},{"location":"generated/notification-services/pagerduty_v2/#parameters","title":"Parameters","text":"The PagerDuty notification service is used to trigger PagerDuty events and requires specifying the following settings:
serviceKeys - a dictionary with the following structure:service-name: $pagerduty-key-service-name where service-name is the name you want to use for the service to make events for, and $pagerduty-key-service-name is a reference to the secret that contains the actual PagerDuty integration key (Events API v2 integration)
If you want multiple Argo apps to trigger events to their respective PagerDuty services, create an integration key in each service you want to setup alerts for.
To create a PagerDuty integration key, follow these instructions to add an Events API v2 integration to the service of your choice.
"},{"location":"generated/notification-services/pagerduty_v2/#configuration","title":"Configuration","text":"The following snippet contains sample PagerDuty service configuration. It assumes the service you want to alert on is called my-service.
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n pagerduty-key-my-service: <pd-integration-key>\n
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.pagerdutyv2: |\n serviceKeys:\n my-service: $pagerduty-key-my-service\n
"},{"location":"generated/notification-services/pagerduty_v2/#template","title":"Template","text":"Notification templates support specifying subject for PagerDuty notifications:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n template.rollout-aborted: |\n message: Rollout {{.rollout.metadata.name}} is aborted.\n pagerdutyv2:\n summary: \"Rollout {{.rollout.metadata.name}} is aborted.\"\n severity: \"critical\"\n source: \"{{.rollout.metadata.name}}\"\n
The parameters for the PagerDuty configuration in the template generally match with the payload for the Events API v2 endpoint. All parameters are strings.
summary - (required) A brief text summary of the event, used to generate the summaries/titles of any associated alerts.severity - (required) The perceived severity of the status the event is describing with respect to the affected system. Allowed values: critical, warning, error, infosource - (required) The unique location of the affected system, preferably a hostname or FQDN.component - Component of the source machine that is responsible for the event.group - Logical grouping of components of a service.class - The class/type of the event.url - The URL that should be used for the link \"View in ArgoCD\" in PagerDuty.
The timestamp and custom_details parameters are not currently supported.
"},{"location":"generated/notification-services/pagerduty_v2/#annotation","title":"Annotation","text":"Annotation sample for PagerDuty notifications:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-rollout-aborted.pagerdutyv2: \"<serviceID for PagerDuty>\"\n
"},{"location":"generated/notification-services/pushover/","title":"Pushover","text":" - Create an app at pushover.net.
- Store the API key in
<secret-name> Secret and define the secret name in argo-rollouts-notification-configmap ConfigMap:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.pushover: |\n token: $pushover-token\n
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n pushover-token: avtc41pn13asmra6zaiyf7dh6cgx97\n
- Add your user key to your Application resource:
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.pushover: uumy8u4owy7bgkapp6mc5mvhfsvpcd\n
"},{"location":"generated/notification-services/rocketchat/","title":"Rocket.Chat","text":""},{"location":"generated/notification-services/rocketchat/#parameters","title":"Parameters","text":"The Rocket.Chat notification service configuration includes following settings:
email - the Rocker.Chat user's emailpassword - the Rocker.Chat user's passwordalias - optional alias that should be used to post messageicon - optional message iconavatar - optional message avatarserverUrl - optional Rocket.Chat server url
"},{"location":"generated/notification-services/rocketchat/#configuration","title":"Configuration","text":" - Login to your RocketChat instance
- Go to user management
- Add new user with
bot role. Also note that Require password change checkbox mus be not checked
- Copy username and password that you was created for bot user
- Create a public or private channel, or a team, for this example
my_channel - Add your bot to this channel otherwise it won't work
- Store email and password in argocd_notifications-secret Secret
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n rocketchat-email: <email>\n rocketchat-password: <password>\n
- Finally, use these credentials to configure the RocketChat integration in the
argocd-configmap config map:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.rocketchat: |\n email: $rocketchat-email\n password: $rocketchat-password\n
- Create a subscription for your Rocket.Chat integration:
Note: channel, team or user must be prefixed with # or @ elsewhere we will be interpretative destination as a room ID
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.rocketchat: #my_channel\n
"},{"location":"generated/notification-services/rocketchat/#templates","title":"Templates","text":"Notification templates can be customized with RocketChat attachments.
Note: Attachments structure in Rocketchat is same with Slack attachments feature.
The message attachments can be specified in attachments string fields under rocketchat field:
template.app-sync-status: |\n message: |\n Application {{.app.metadata.name}} sync is {{.app.status.sync.status}}.\n Application details: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}.\n rocketchat:\n attachments: |\n [{\n \"title\": \"{{.app.metadata.name}}\",\n \"title_link\": \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}\",\n \"color\": \"#18be52\",\n \"fields\": [{\n \"title\": \"Sync Status\",\n \"value\": \"{{.app.status.sync.status}}\",\n \"short\": true\n }, {\n \"title\": \"Repository\",\n \"value\": \"{{.app.spec.source.repoURL}}\",\n \"short\": true\n }]\n }]\n
"},{"location":"generated/notification-services/slack/","title":"Slack","text":"If you want to send message using incoming webhook, you can use webhook.
"},{"location":"generated/notification-services/slack/#parameters","title":"Parameters","text":"The Slack notification service configuration includes following settings:
Option Required Type Description Example apiURL False string The server URL. https://example.com/api channels False list[string] [\"my-channel-1\", \"my-channel-2\"] icon False string The app icon. :robot_face: or https://example.com/image.png insecureSkipVerify False bool true signingSecret False string 8f742231b10e8888abcd99yyyzzz85a5 token True string The app's OAuth access token. xoxb-1234567890-1234567890123-5n38u5ed63fgzqlvuyxvxcx6 username False string The app username. argocd disableUnfurl False bool Disable slack unfurling links in messages true"},{"location":"generated/notification-services/slack/#configuration","title":"Configuration","text":" - Create Slack Application using https://api.slack.com/apps?new_app=1
- Once application is created navigate to
Enter OAuth & Permissions - Click
Permissions under Add features and functionality section and add chat:write scope. To use the optional username and icon overrides in the Slack notification service also add the chat:write.customize scope. - Scroll back to the top, click 'Install App to Workspace' button and confirm the installation.
-
Once installation is completed copy the OAuth token.
-
Create a public or private channel, for this example my_channel
- Invite your slack bot to this channel otherwise slack bot won't be able to deliver notifications to this channel
-
Store Oauth access token in argo-rollouts-notification-secret secret
apiVersion: v1\n kind: Secret\n metadata:\n name: <secret-name>\n stringData:\n slack-token: <Oauth-access-token>\n
-
Define service type slack in data section of argo-rollouts-notification-configmap configmap:
apiVersion: v1\n kind: ConfigMap\n metadata:\n name: argo-rollouts-notification-configmap\n data:\n service.slack: |\n token: $slack-token\n
-
Add annotation in application yaml file to enable notifications for specific argocd app. The following example uses the on-sync-succeeded trigger:
apiVersion: argoproj.io/v1alpha1\n kind: Application\n metadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.slack: my_channel\n
-
Annotation with more than one trigger, with multiple destinations and recipients
apiVersion: argoproj.io/v1alpha1\n kind: Application\n metadata:\n annotations:\n notifications.argoproj.io/subscriptions: |\n - trigger: [on-scaling-replica-set, on-rollout-updated, on-rollout-step-completed]\n destinations:\n - service: slack\n recipients: [my-channel-1, my-channel-2]\n - service: email\n recipients: [recipient-1, recipient-2, recipient-3 ]\n - trigger: [on-rollout-aborted, on-analysis-run-failed, on-analysis-run-error]\n destinations:\n - service: slack\n recipients: [my-channel-21, my-channel-22]\n
"},{"location":"generated/notification-services/slack/#templates","title":"Templates","text":"Notification templates can be customized to leverage slack message blocks and attachments feature.
The message blocks and attachments can be specified in blocks and attachments string fields under slack field:
template.app-sync-status: |\n message: |\n Application {{.app.metadata.name}} sync is {{.app.status.sync.status}}.\n Application details: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}.\n slack:\n attachments: |\n [{\n \"title\": \"{{.app.metadata.name}}\",\n \"title_link\": \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}\",\n \"color\": \"#18be52\",\n \"fields\": [{\n \"title\": \"Sync Status\",\n \"value\": \"{{.app.status.sync.status}}\",\n \"short\": true\n }, {\n \"title\": \"Repository\",\n \"value\": \"{{.app.spec.source.repoURL}}\",\n \"short\": true\n }]\n }]\n
The messages can be aggregated to the slack threads by grouping key which can be specified in a groupingKey string field under slack field. groupingKey is used across each template and works independently on each slack channel. When multiple applications will be updated at the same time or frequently, the messages in slack channel can be easily read by aggregating with git commit hash, application name, etc. Furthermore, the messages can be broadcast to the channel at the specific template by notifyBroadcast field.
template.app-sync-status: |\n message: |\n Application {{.app.metadata.name}} sync is {{.app.status.sync.status}}.\n Application details: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}.\n slack:\n attachments: |\n [{\n \"title\": \"{{.app.metadata.name}}\",\n \"title_link\": \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}\",\n \"color\": \"#18be52\",\n \"fields\": [{\n \"title\": \"Sync Status\",\n \"value\": \"{{.app.status.sync.status}}\",\n \"short\": true\n }, {\n \"title\": \"Repository\",\n \"value\": \"{{.app.spec.source.repoURL}}\",\n \"short\": true\n }]\n }]\n # Aggregate the messages to the thread by git commit hash\n groupingKey: \"{{.app.status.sync.revision}}\"\n notifyBroadcast: false\ntemplate.app-sync-failed: |\n message: |\n Application {{.app.metadata.name}} sync is {{.app.status.sync.status}}.\n Application details: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}.\n slack:\n attachments: |\n [{\n \"title\": \"{{.app.metadata.name}}\",\n \"title_link\": \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}\",\n \"color\": \"#ff0000\",\n \"fields\": [{\n \"title\": \"Sync Status\",\n \"value\": \"{{.app.status.sync.status}}\",\n \"short\": true\n }, {\n \"title\": \"Repository\",\n \"value\": \"{{.app.spec.source.repoURL}}\",\n \"short\": true\n }]\n }]\n # Aggregate the messages to the thread by git commit hash\n groupingKey: \"{{.app.status.sync.revision}}\"\n notifyBroadcast: true\n
The message is sent according to the deliveryPolicy string field under the slack field. The available modes are Post (default), PostAndUpdate, and Update. The PostAndUpdate and Update settings require groupingKey to be set.
"},{"location":"generated/notification-services/teams/","title":"Teams","text":""},{"location":"generated/notification-services/teams/#parameters","title":"Parameters","text":"The Teams notification service send message notifications using Teams bot and requires specifying the following settings:
recipientUrls - the webhook url map, e.g. channelName: https://example.com
"},{"location":"generated/notification-services/teams/#configuration","title":"Configuration","text":" - Open
Teams and goto Apps - Find
Incoming Webhook microsoft app and click on it - Press
Add to a team -> select team and channel -> press Set up a connector - Enter webhook name and upload image (optional)
- Press
Create then copy webhook url and store it in argo-rollouts-notification-secret and define it in argo-rollouts-notification-configmap
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.teams: |\n recipientUrls:\n channelName: $channel-teams-url\n
apiVersion: v1\nkind: Secret\nmetadata:\n name: <secret-name>\nstringData:\n channel-teams-url: https://example.com\n
- Create subscription for your Teams integration:
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.teams: channelName\n
"},{"location":"generated/notification-services/teams/#templates","title":"Templates","text":"Notification templates can be customized to leverage teams message sections, facts, themeColor, summary and potentialAction feature.
template.app-sync-succeeded: |\n teams:\n themeColor: \"#000080\"\n sections: |\n [{\n \"facts\": [\n {\n \"name\": \"Sync Status\",\n \"value\": \"{{.app.status.sync.status}}\"\n },\n {\n \"name\": \"Repository\",\n \"value\": \"{{.app.spec.source.repoURL}}\"\n }\n ]\n }]\n potentialAction: |-\n [{\n \"@type\":\"OpenUri\",\n \"name\":\"Operation Details\",\n \"targets\":[{\n \"os\":\"default\",\n \"uri\":\"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true\"\n }]\n }]\n title: Application {{.app.metadata.name}} has been successfully synced\n text: Application {{.app.metadata.name}} has been successfully synced at {{.app.status.operationState.finishedAt}}.\n summary: \"{{.app.metadata.name}} sync succeeded\"\n
"},{"location":"generated/notification-services/teams/#facts-field","title":"facts field","text":"You can use facts field instead of sections field.
template.app-sync-succeeded: |\n teams:\n facts: |\n [{\n \"name\": \"Sync Status\",\n \"value\": \"{{.app.status.sync.status}}\"\n },\n {\n \"name\": \"Repository\",\n \"value\": \"{{.app.spec.source.repoURL}}\"\n }]\n
"},{"location":"generated/notification-services/teams/#theme-color-field","title":"theme color field","text":"You can set theme color as hex string for the message.
template.app-sync-succeeded: |\n teams:\n themeColor: \"#000080\"\n
"},{"location":"generated/notification-services/teams/#summary-field","title":"summary field","text":"You can set a summary of the message that will be shown on Notification & Activity Feed
template.app-sync-succeeded: |\n teams:\n summary: \"Sync Succeeded\"\n
"},{"location":"generated/notification-services/telegram/","title":"Telegram","text":" - Get an API token using @Botfather.
- Store token in
<secret-name> Secret and configure telegram integration in argo-rollouts-notification-configmap ConfigMap:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.telegram: |\n token: $telegram-token\n
- Create new Telegram channel.
- Add your bot as an administrator.
- Use this channel
username (public channel) or chatID (private channel) in the subscription for your Telegram integration:
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.telegram: username\n
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.telegram: -1000000000000\n
If your private chat contains threads, you can optionally specify a thread id by seperating it with a |:
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.telegram: -1000000000000|2\n
"},{"location":"generated/notification-services/webex/","title":"Webex Teams","text":""},{"location":"generated/notification-services/webex/#parameters","title":"Parameters","text":"The Webex Teams notification service configuration includes following settings:
token - the app token
"},{"location":"generated/notification-services/webex/#configuration","title":"Configuration","text":" - Create a Webex Bot
-
Copy the bot access token and store it in the argo-rollouts-notification-secret Secret and configure Webex Teams integration in argo-rollouts-notification-configmap ConfigMap
apiVersion: v1\nkind: Secret\nmetadata:\nname: <secret-name>\nstringData:\nwebex-token: <bot access token>\n
apiVersion: v1\nkind: ConfigMap\nmetadata:\nname: argo-rollouts-notification-configmap\ndata:\nservice.webex: |\n token: $webex-token\n
-
Create subscription for your Webex Teams integration
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\nannotations:\n notifications.argoproj.io/subscribe.<trigger-name>.webex: <personal email or room id>\n
"},{"location":"generated/notification-services/webhook/","title":"Webhook","text":"The webhook notification service allows sending a generic HTTP request using the templatized request body and URL. Using Webhook you might trigger a Jenkins job, update GitHub commit status.
"},{"location":"generated/notification-services/webhook/#parameters","title":"Parameters","text":"The Webhook notification service configuration includes following settings:
url - the url to send the webhook toheaders - optional, the headers to pass along with the webhookbasicAuth - optional, the basic authentication to pass along with the webhookinsecureSkipVerify - optional bool, true or falseretryWaitMin - Optional, the minimum wait time between retries. Default value: 1s.retryWaitMax - Optional, the maximum wait time between retries. Default value: 5s.retryMax - Optional, the maximum number of retries. Default value: 3.
"},{"location":"generated/notification-services/webhook/#retry-behavior","title":"Retry Behavior","text":"The webhook service will automatically retry the request if it fails due to network errors or if the server returns a 5xx status code. The number of retries and the wait time between retries can be configured using the retryMax, retryWaitMin, and retryWaitMax parameters.
The wait time between retries is between retryWaitMin and retryWaitMax. If all retries fail, the Send method will return an error.
"},{"location":"generated/notification-services/webhook/#configuration","title":"Configuration","text":"Use the following steps to configure webhook:
1 Register webhook in argo-rollouts-notification-configmap ConfigMap:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.webhook.<webhook-name>: |\n url: https://<hostname>/<optional-path>\n headers: #optional headers\n - name: <header-name>\n value: <header-value>\n basicAuth: #optional username password\n username: <username>\n password: <api-key>\n insecureSkipVerify: true #optional bool\n
2 Define template that customizes webhook request method, path and body:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n template.github-commit-status: |\n webhook:\n <webhook-name>:\n method: POST # one of: GET, POST, PUT, PATCH. Default value: GET \n path: <optional-path-template>\n body: |\n <optional-body-template>\n trigger.<trigger-name>: |\n - when: app.status.operationState.phase in ['Succeeded']\n send: [github-commit-status]\n
3 Create subscription for webhook integration:
apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n annotations:\n notifications.argoproj.io/subscribe.<trigger-name>.<webhook-name>: \"\"\n
"},{"location":"generated/notification-services/webhook/#examples","title":"Examples","text":""},{"location":"generated/notification-services/webhook/#set-github-commit-status","title":"Set GitHub commit status","text":"apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.webhook.github: |\n url: https://api.github.com\n headers: #optional headers\n - name: Authorization\n value: token $github-token\n
2 Define template that customizes webhook request method, path and body:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.webhook.github: |\n url: https://api.github.com\n headers: #optional headers\n - name: Authorization\n value: token $github-token\n\n template.github-commit-status: |\n webhook:\n github:\n method: POST\n path: /repos/{{call .repo.FullNameByRepoURL .app.spec.source.repoURL}}/statuses/{{.app.status.operationState.operation.sync.revision}}\n body: |\n {\n {{if eq .app.status.operationState.phase \"Running\"}} \"state\": \"pending\"{{end}}\n {{if eq .app.status.operationState.phase \"Succeeded\"}} \"state\": \"success\"{{end}}\n {{if eq .app.status.operationState.phase \"Error\"}} \"state\": \"error\"{{end}}\n {{if eq .app.status.operationState.phase \"Failed\"}} \"state\": \"error\"{{end}},\n \"description\": \"ArgoCD\",\n \"target_url\": \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}\",\n \"context\": \"continuous-delivery/{{.app.metadata.name}}\"\n }\n
"},{"location":"generated/notification-services/webhook/#start-jenkins-job","title":"Start Jenkins Job","text":"apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.webhook.jenkins: |\n url: http://<jenkins-host>/job/<job-name>/build?token=<job-secret>\n basicAuth:\n username: <username>\n password: <api-key>\n\ntype: Opaque\n
"},{"location":"generated/notification-services/webhook/#send-form-data","title":"Send form-data","text":"apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.webhook.form: |\n url: https://form.example.com\n headers:\n - name: Content-Type\n value: application/x-www-form-urlencoded\n\n template.form-data: |\n webhook:\n form:\n method: POST\n body: key1=value1&key2=value2\n
"},{"location":"generated/notification-services/webhook/#send-slack","title":"Send Slack","text":"apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: argo-rollouts-notification-configmap\ndata:\n service.webhook.slack_webhook: |\n url: https://hooks.slack.com/services/xxxxx\n headers:\n - name: Content-Type\n value: application/json\n\n template.send-slack: |\n webhook:\n slack_webhook:\n method: POST\n body: |\n {\n \"attachments\": [{\n \"title\": \"{{.app.metadata.name}}\",\n \"title_link\": \"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}\",\n \"color\": \"#18be52\",\n \"fields\": [{\n \"title\": \"Sync Status\",\n \"value\": \"{{.app.status.sync.status}}\",\n \"short\": true\n }, {\n \"title\": \"Repository\",\n \"value\": \"{{.app.spec.source.repoURL}}\",\n \"short\": true\n }]\n }]\n }\n
"},{"location":"getting-started/alb/","title":"Getting Started - AWS Load Balancer Controller","text":"This guide covers how Argo Rollouts integrates with the AWS Load Balancer Controller for traffic shaping. This guide builds upon the concepts of the basic getting started guide.
"},{"location":"getting-started/alb/#requirements","title":"Requirements","text":" - Kubernetes cluster with AWS ALB Ingress Controller installed
Tip
See the Load Balancer Controller Installation instructions on how to install the AWS Load Balancer Controller
"},{"location":"getting-started/alb/#1-deploy-the-rollout-services-and-ingress","title":"1. Deploy the Rollout, Services, and Ingress","text":"When an AWS ALB Ingress is used as the traffic router, the Rollout canary strategy must define the following fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n strategy:\n canary:\n # canaryService and stableService are references to Services which the Rollout will modify\n # to target the canary ReplicaSet and stable ReplicaSet respectively (required).\n canaryService: rollouts-demo-canary\n stableService: rollouts-demo-stable\n trafficRouting:\n alb:\n # The referenced ingress will be injected with a custom action annotation, directing\n # the AWS Load Balancer Controller to split traffic between the canary and stable\n # Service, according to the desired traffic weight (required).\n ingress: rollouts-demo-ingress\n # Reference to a Service that the Ingress must target in one of the rules (optional).\n # If omitted, uses canary.stableService.\n rootService: rollouts-demo-root\n # Service port is the port which the Service listens on (required).\n servicePort: 443\n...\n
The Ingress referenced by the Rollout must have a rule which matches one of Rollout services. This should be canary.trafficRouting.alb.rootService (if specified), otherwise the rollout will use canary.stableService.
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n name: rollouts-demo-ingress\n annotations:\n kubernetes.io/ingress.class: alb\nspec:\n rules:\n - http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n # serviceName must match either: canary.trafficRouting.alb.rootService (if specified),\n # or canary.stableService (if rootService is omitted)\n name: rollouts-demo-root\n # servicePort must be the value: use-annotation\n # This instructs AWS Load Balancer Controller to look to annotations on how to direct traffic\n port:\n name: use-annotation\n
During an update, the Ingress will be injected with a custom action annotation, which directs the ALB to splits traffic between the stable and canary Services referenced by the Rollout. In this example, those Services are named: rollouts-demo-stable and rollouts-demo-canary respectively.
Run the following commands to deploy:
- A Rollout
- Three Services (root, stable, canary)
- An Ingress
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/alb/rollout.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/alb/services.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/alb/ingress.yaml\n
After applying the manifests you should see the following rollout, services, and ingress resources in the cluster:
$ kubectl get ro\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE\nrollouts-demo 1 1 1 1\n\n$ kubectl get svc\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nrollouts-demo-root NodePort 10.100.16.123 <none> 80:30225/TCP 2m43s\nrollouts-demo-canary NodePort 10.100.16.64 <none> 80:30224/TCP 2m43s\nrollouts-demo-stable NodePort 10.100.146.232 <none> 80:31135/TCP 2m43s\n\n$ kubectl get ingress\nNAME HOSTS ADDRESS PORTS AGE\nrollouts-demo-ingress * b0548428-default-rolloutsd-6951-1972570952.ap-northeast-1.elb.amazonaws.com 80 6m36s\n
kubectl argo rollouts get rollout rollouts-demo\n
"},{"location":"getting-started/alb/#2-perform-an-update","title":"2. Perform an update","text":"Update the rollout by changing the image, and wait for it to reach the paused state.
kubectl argo rollouts set image rollouts-demo rollouts-demo=argoproj/rollouts-demo:yellow\nkubectl argo rollouts get rollout rollouts-demo\n
At this point, both the canary and stable version of the Rollout are running, with 5% of the traffic directed to the canary. To understand how this works, inspect the listener rules for the ALB. When looking at the listener rules, we see that the forward action weights have been modified by the controller to reflect the current weight of the canary.
The controller has added rollouts-pod-template-hash selector to the Services and attached the same label to the Pods. Therefore, you can split the traffic by simply forwarding the requests to the Services according to the weights.
As the Rollout progresses through steps, the forward action weights will be adjusted to match the current setWeight of the steps.
"},{"location":"getting-started/ambassador/","title":"Argo Rollouts and Ambassador Quick Start","text":"This tutorial will walk you through the process of configuring Argo Rollouts to work with Ambassador to facilitate canary releases. All files used in this guide are available in the examples directory of this repository.
"},{"location":"getting-started/ambassador/#requirements","title":"Requirements","text":" - Kubernetes cluster
- Argo-Rollouts installed in the cluster
Note If using Ambassador Edge Stack or Emissary-ingress 2.0+, you will need to install Argo-Rollouts version v1.1+, and you will need to supply --ambassador-api-version getambassador.io/v3alpha1 to your argo-rollouts deployment.
"},{"location":"getting-started/ambassador/#1-install-and-configure-ambassador-edge-stack","title":"1. Install and configure Ambassador Edge Stack","text":"If you don't have Ambassador in your cluster you can install it following the Edge Stack documentation.
By default, Edge Stack routes via Kubernetes services. For best performance with canaries, we recommend you use endpoint routing. Enable endpoint routing on your cluster by saving the following configuration in a file called resolver.yaml:
apiVersion: getambassador.io/v2\nkind: KubernetesEndpointResolver\nmetadata:\n name: endpoint\n
Apply this configuration to your cluster: kubectl apply -f resolver.yaml.
"},{"location":"getting-started/ambassador/#2-create-the-kubernetes-services","title":"2. Create the Kubernetes Services","text":"We'll create two Kubernetes services, named echo-stable and echo-canary. Save this configuration to the file echo-service.yaml.
apiVersion: v1\nkind: Service\nmetadata:\n labels:\n app: echo\n name: echo-stable\nspec:\n type: ClusterIP\n ports:\n - name: http\n port: 80\n protocol: TCP\n targetPort: 8080\n selector:\n app: echo\n---\napiVersion: v1\nkind: Service\nmetadata:\n labels:\n app: echo\n name: echo-canary\nspec:\n type: ClusterIP\n ports:\n - name: http\n port: 80\n protocol: TCP\n targetPort: 8080\n selector:\n app: echo \n
We'll also create an Edge Stack route to the services. Save the following configuration to a file called echo-mapping.yaml.
apiVersion: getambassador.io/v2\nkind: Mapping\nmetadata:\n name: echo\nspec:\n prefix: /echo\n rewrite: /echo\n service: echo-stable:80\n resolver: endpoint\n
Apply both of these configurations to the Kubernetes cluster:
kubectl apply -f echo-service.yaml\nkubectl apply -f echo-mapping.yaml\n
"},{"location":"getting-started/ambassador/#3-deploy-the-echo-service","title":"3. Deploy the Echo Service","text":"Create a Rollout resource and save it to a file called rollout.yaml. Note the trafficRouting attribute, which tells Argo to use Ambassador Edge Stack for routing.
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: echo-rollout\nspec:\n selector:\n matchLabels:\n app: echo\n template:\n metadata:\n labels:\n app: echo\n spec:\n containers:\n - image: hashicorp/http-echo\n args:\n - \"-text=VERSION 1\"\n - -listen=:8080\n imagePullPolicy: Always\n name: echo-v1\n ports:\n - containerPort: 8080\n strategy:\n canary:\n stableService: echo-stable\n canaryService: echo-canary\n trafficRouting:\n ambassador:\n mappings:\n - echo\n steps:\n - setWeight: 30\n - pause: {duration: 30s}\n - setWeight: 60\n - pause: {duration: 30s}\n - setWeight: 100\n - pause: {duration: 10}\n
Apply the rollout to your cluster kubectl apply -f rollout.yaml. Note that no canary rollout will occur, as this is the first version of the service being deployed.
"},{"location":"getting-started/ambassador/#4-test-the-service","title":"4. Test the service","text":"We'll now test that this rollout works as expected. Open a new terminal window. We'll use it to send requests to the cluster. Get the external IP address for Edge Stack:
export AMBASSADOR_LB_ENDPOINT=$(kubectl -n ambassador get svc ambassador -o \"go-template={{range .status.loadBalancer.ingress}}{{or .ip .hostname}}{{end}}\")\n
Send a request to the echo service:
curl -Lk \"https://$AMBASSADOR_LB_ENDPOINT/echo/\"\n
You should get a response of \"VERSION 1\".
"},{"location":"getting-started/ambassador/#5-rollout-a-new-version","title":"5. Rollout a new version","text":"It's time to rollout a new version of the service. Update the echo container in the rollout.yaml to display \"VERSION 2\":
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: echo-rollout\nspec:\n selector:\n matchLabels:\n app: echo\n template:\n metadata:\n labels:\n app: echo\n spec:\n containers:\n - image: hashicorp/http-echo\n args:\n - \"-text=VERSION 2\"\n - -listen=:8080\n imagePullPolicy: Always\n name: echo-v1\n ports:\n - containerPort: 8080\n strategy:\n canary:\n stableService: echo-stable\n canaryService: echo-canary\n trafficRouting:\n ambassador:\n mappings:\n - echo\n steps:\n - setWeight: 30\n - pause: {duration: 30s}\n - setWeight: 60\n - pause: {duration: 30s}\n - setWeight: 100\n - pause: {duration: 10}\n
Apply the rollout to the cluster by typing kubectl apply -f rollout.yaml. This will rollout a version 2 of the service by routing 30% of traffic to the service for 30 seconds, followed by 60% of traffic for another 30 seconds.
You can monitor the status of your rollout at the command line:
kubectl argo rollouts get rollout echo-rollout --watch\n
Will display an output similar to the following:
Name: echo-rollout\nNamespace: default\nStatus: \u0965 Paused\nMessage: CanaryPauseStep\nStrategy: Canary\n Step: 1/6\n SetWeight: 30\n ActualWeight: 30\nImages: hashicorp/http-echo (canary, stable)\nReplicas:\n Desired: 1\n Current: 2\n Updated: 1\n Ready: 2\n Available: 2\n\nNAME KIND STATUS AGE INFO\n\u27f3 echo-rollout Rollout \u0965 Paused 2d21h\n\u251c\u2500\u2500# revision:3\n\u2502 \u2514\u2500\u2500\u29c9 echo-rollout-64fb847897 ReplicaSet \u2714 Healthy 2s canary\n\u2502 \u2514\u2500\u2500\u25a1 echo-rollout-64fb847897-49sg6 Pod \u2714 Running 2s ready:1/1\n\u251c\u2500\u2500# revision:2\n\u2502 \u2514\u2500\u2500\u29c9 echo-rollout-578bfdb4b8 ReplicaSet \u2714 Healthy 3h5m stable\n\u2502 \u2514\u2500\u2500\u25a1 echo-rollout-578bfdb4b8-86z6n Pod \u2714 Running 3h5m ready:1/1\n\u2514\u2500\u2500# revision:1\n \u2514\u2500\u2500\u29c9 echo-rollout-948d9c9f9 ReplicaSet \u2022 ScaledDown 2d21h\n
In your other terminal window, you can verify that the canary is progressing appropriately by sending requests in a loop:
while true; do curl -k https://$AMBASSADOR_LB_ENDPOINT/echo/; sleep 0.2; done\n
This will display a running list of responses from the service that will gradually transition from VERSION 1 strings to VERSION 2 strings.
For more details about the Ambassador and Argo-Rollouts integration, see the Ambassador Argo documentation.
"},{"location":"getting-started/appmesh/","title":"Getting Started - App Mesh","text":"This guide covers how Argo Rollouts integrates with service-meshes managed by AWS App Mesh. This guide builds upon the concepts of the basic getting started guide.
"},{"location":"getting-started/appmesh/#requirements","title":"Requirements","text":" - Kubernetes cluster with AWS App Mesh Controller for K8s installed
Tip
See the App Mesh Controler Installation instructions on how to get started using App Mesh with Kubernetes.
"},{"location":"getting-started/appmesh/#1-deploy-the-rollout-services-app-mesh-crd","title":"1. Deploy the Rollout, Services, App Mesh CRD","text":"When App Mesh is used as the traffic router, the Rollout canary strategy must define the following mandatory fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: my-rollout\nspec:\n strategy:\n canary:\n # canaryService and stableService are references to Services which the Rollout will modify\n # to target the canary ReplicaSet and stable ReplicaSet respectively (required).\n canaryService: my-svc-canary\n stableService: my-svc-stable\n trafficRouting:\n appMesh:\n # The referenced virtual-service will be used to determine the virtual-router that is\n # manipulated to update canary weights.\n virtualService:\n # name of the virtual-service App Mesh CR\n name: my-svc\n # Optional set of routes to update. If empty, all routes associated with the virtual-service are updated.\n routes:\n - http-primary\n # virtualNodeGroup is a structure to refer App Mesh virtual-node CR corresponding to Canary and Stable versions\n virtualNodeGroup:\n # canaryVirtualNodeRef refers to virtual-node corresponding to canary version. Rollouts controller will\n # update the podSelector of this virtual-node to latest canary pod-hash generated by controller.\n canaryVirtualNodeRef:\n name: my-vn-canary\n # stableVirtualNodeRef refers to virtual-node corresponding to stable version. Rollouts controller will\n # update the podSelector of this virtual-node to latest stable pod-hash generated by controller.\n stableVirtualNodeRef:\n name: my-vn-stable\n steps:\n - setWeight: 25\n - pause: {}\n ...\n
In this guide, the two services are: my-svc-canary and my-svc-stable respectively. There are two virtual-node CRs corresponding to these services named my-vn-canary and my-vn-stable respectively. In addition, there is a virtual-service named rollout-demo-vsvc that is provided by a virtual-router CR named rollout-demo-vrouter. This virtual-router need have at least one route with action to forward traffic to the canary and stable virtual-nodes. Initially weight for canary is set to 0% while for stable it is 100%. During rollout, controller will modify the weights on route(s) based on the configuraiton defined in steps[N].setWeight.
The canary and stable services are configured to be headless. This is necessary to allow App Mesh to properly handle conneciton pooling as pods are reassigned from canary to stable.
To summarize, run the following commands to deploy a service:
- Two services (stable and canary)
- One service (for VIP and DNS lookup)
- Two App Mesh virtual-nodes (stable and canary)
- One App Mesh virtual-router with routes to virtual-nodes
- One App Mesh virtual-service corresponding to VIP service
- A rollout
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/examples/appmesh/canary-service.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/examples/appmesh/canary-rollout.yaml\n
"},{"location":"getting-started/appmesh/#2-verify-service","title":"2. Verify service","text":"First make sure that rollout is stable.
kubectl argo rollouts get rollout my-rollout -n argo-examples -w\n
Then make sure the service is functional.
kubectl -n argo-examples port-forward svc/my-svc 8181:80\n
"},{"location":"getting-started/appmesh/#3-rollout-new-version","title":"3. Rollout new version","text":"Now its time to deploy new version. Update the rollout with new image.
kubectl argo rollouts set image my-rollout demo=argoproj/rollouts-demo:green -n argo-examples\n
Rollout should deploy a new canary revision and update the weights under virtual-router.
kubectl get -n argo-examples virtualrouter my-vrouter -o json | jq \".spec.routes[0].httpRoute.action.weightedTargets\"\n[\n {\n \"virtualNodeRef\": {\n \"name\": \"my-vn-canary\"\n },\n \"weight\": 25\n },\n {\n \"virtualNodeRef\": {\n \"name\": \"my-vn-stable\"\n },\n \"weight\": 75\n }\n]\n
Now manually approve the rollout that is paused indefinitely, and continue watching the routes get updated
kubectl argo rollouts promote my-rollout -n argo-examples\n\nwatch -d 'kubectl get -n argo-examples virtualrouter my-vrouter -o json | jq \".spec.routes[0].httpRoute.action.weightedTargets\"'\n
"},{"location":"getting-started/istio/","title":"Getting Started - Istio","text":"This guide covers how Argo Rollouts integrates with the Istio Service Mesh for traffic shaping. This guide builds upon the concepts of the basic getting started guide.
"},{"location":"getting-started/istio/#requirements","title":"Requirements","text":" - Kubernetes cluster with Istio installed
Tip
See the environment setup guide for Istio on how to setup a local minikube environment with Istio
"},{"location":"getting-started/istio/#1-deploy-the-rollout-services-istio-virtualservice-and-istio-gateway","title":"1. Deploy the Rollout, Services, Istio VirtualService, and Istio Gateway","text":"When Istio is used as the traffic router, the Rollout canary strategy must define the following mandatory fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n strategy:\n canary:\n # Reference to a Service which the controller updates to point to the canary ReplicaSet\n canaryService: rollouts-demo-canary\n # Reference to a Service which the controller updates to point to the stable ReplicaSet\n stableService: rollouts-demo-stable\n trafficRouting:\n istio:\n virtualServices:\n # One or more virtualServices can be configured\n # Reference to a VirtualService which the controller updates with canary weights\n - name: rollouts-demo-vsvc1\n # Optional if there is a single HTTP route in the VirtualService, otherwise required\n routes:\n - http-primary\n # Optional if there is a single HTTPS/TLS route in the VirtualService, otherwise required\n tlsRoutes:\n # Below fields are optional but if defined, they should match exactly with at least one of the TLS route match rules in your VirtualService\n - port: 443 # Only required if you want to match any rule in your VirtualService which contains this port\n # Only required if you want to match any rule in your VirtualService which contain all these SNI hosts\n sniHosts:\n - reviews.bookinfo.com\n - localhost\n - name: rollouts-demo-vsvc2\n # Optional if there is a single HTTP route in the VirtualService, otherwise required\n routes:\n - http-secondary\n # Optional if there is a single HTTPS/TLS route in the VirtualService, otherwise required\n tlsRoutes:\n # Below fields are optional but if defined, they should match exactly with at least one of the TLS route match rules in your VirtualService\n - port: 443 # Only required if you want to match any rule in your VirtualService which contains this port\n # Only required if you want to match any rule in your VirtualService which contain all these SNI hosts\n sniHosts:\n - reviews.bookinfo.com\n - localhost\n tcpRoutes:\n # Below fields are optional but if defined, they should match exactly with at least one of the TCP route match rules in your VirtualService\n - port: 8020 # Only required if you want to match any rule in your VirtualService which contains this port\n
The VirtualService and route referenced in either trafficRouting.istio.virtualService or trafficRouting.istio.virtualServices. trafficRouting.istio.virtualServices helps in adding one or more virtualServices unlike trafficRouting.istio.virtualService where only single virtualService can be added. This is required to have either HTTP, TLS, TCP or a mixed route specs that splits between the stable and the canary services referenced in the rollout. If the route is HTTPS/TLS, we can match it based on the given port number and/or SNI hosts. Note that both of them are optional and only needed if you want to match any rule in your VirtualService which contains these.
In this guide, the two services are: rollouts-demo-stable and rollouts-demo-canary respectively. The weights for these two services should initially be set to 100% on the stable service and 0% on the canary service. During an update, these values will get modified by the controller. If there are multiple VirtualService then weight values for stable and canary service of each VirtualService will be modified by the controller simultaneously.
Note that since we have both the HTTP and HTTPS routes in our rollout spec and they match the VirtualService specs, weights will get modified for both these routes.
apiVersion: networking.istio.io/v1alpha3\nkind: VirtualService\nmetadata:\n name: rollouts-demo-vsvc1\nspec:\n gateways:\n - rollouts-demo-gateway\n hosts:\n - rollouts-demo-vsvc1.local\n http:\n - name: http-primary # Should match rollout.spec.strategy.canary.trafficRouting.istio.virtualServices.routes\n route:\n - destination:\n host: rollouts-demo-stable # Should match rollout.spec.strategy.canary.stableService\n port:\n number: 15372\n weight: 100\n - destination:\n host: rollouts-demo-canary # Should match rollout.spec.strategy.canary.canaryService\n port:\n number: 15372\n weight: 0\n tls:\n - match:\n - port: 443 # Should match the port number of the route defined in rollout.spec.strategy.canary.trafficRouting.istio.virtualServices.tlsRoutes\n sniHosts: # Should match all the SNI hosts of the route defined in rollout.spec.strategy.canary.trafficRouting.istio.virtualServices.tlsRoutes\n - reviews.bookinfo.com\n - localhost\n route:\n - destination:\n host: rollouts-demo-stable # Should match rollout.spec.strategy.canary.stableService\n weight: 100\n - destination:\n host: rollouts-demo-canary # Should match rollout.spec.strategy.canary.canaryService\n weight: 0\n tcp:\n - match:\n - port: 8020 # Should match the port number of the route defined in rollout.spec.strategy.canary.trafficRouting.istio.virtualServices.tcpRoutes\n route:\n - destination:\n host: rollouts-demo-stable # Should match rollout.spec.strategy.canary.stableService\n weight: 100\n - destination:\n host: rollouts-demo-canary # Should match rollout.spec.strategy.canary.canaryService\n weight: 0\n
apiVersion: networking.istio.io/v1alpha3\nkind: VirtualService\nmetadata:\n name: rollouts-demo-vsvc2\nspec:\n gateways:\n - rollouts-demo-gateway\n hosts:\n - rollouts-demo-vsvc2.local\n http:\n - name: http-secondary # Should match rollout.spec.strategy.canary.trafficRouting.istio.virtualServices.routes\n route:\n - destination:\n host: rollouts-demo-stable # Should match rollout.spec.strategy.canary.stableService\n port:\n number: 15373\n weight: 100\n - destination:\n host: rollouts-demo-canary # Should match rollout.spec.strategy.canary.canaryService\n port:\n number: 15373\n weight: 0\n tls:\n - match:\n - port: 443 # Should match the port number of the route defined in rollout.spec.strategy.canary.trafficRouting.istio.virtualServices.tlsRoutes\n sniHosts: # Should match all the SNI hosts of the route defined in rollout.spec.strategy.canary.trafficRouting.istio.virtualServices.tlsRoutes\n - reviews.bookinfo.com\n route:\n - destination:\n host: rollouts-demo-stable # Should match rollout.spec.strategy.canary.stableService\n weight: 100\n - destination:\n host: rollouts-demo-canary # Should match rollout.spec.strategy.canary.canaryService\n weight: 0\n tcp:\n - match:\n - port: 8020 # Should match the port number of the route defined in rollout.spec.strategy.canary.trafficRouting.istio.virtualServices.tcpRoutes\n route:\n - destination:\n host: rollouts-demo-stable # Should match rollout.spec.strategy.canary.stableService\n weight: 100\n - destination:\n host: rollouts-demo-canary # Should match rollout.spec.strategy.canary.canaryService\n weight: 0\n
Run the following commands to deploy:
- A Rollout
- Two Services (stable and canary)
- One or more Istio VirtualServices
- An Istio Gateway
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/istio/rollout.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/istio/services.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/istio/multipleVirtualsvc.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/istio/gateway.yaml\n
After applying the manifests you should see the following rollout, services, virtualservices, and gateway resources in the cluster:
$ kubectl get ro\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE\nrollouts-demo 1 1 1 1\n\n$ kubectl get svc\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nrollouts-demo-canary ClusterIP 10.103.146.137 <none> 80/TCP 37s\nrollouts-demo-stable ClusterIP 10.101.158.227 <none> 80/TCP 37s\n\n$ kubectl get virtualservice\nNAME GATEWAYS HOSTS AGE\nrollouts-demo-vsvc1 [rollouts-demo-gateway] [rollouts-demo-vsvc1.local] 54s\nrollouts-demo-vsvc2 [rollouts-demo-gateway] [rollouts-demo-vsvc2.local] 54s\n\n$ kubectl get gateway\nNAME AGE\nrollouts-demo-gateway 71s\n
kubectl argo rollouts get rollout rollouts-demo\n
"},{"location":"getting-started/istio/#2-perform-an-update","title":"2. Perform an update","text":"Update the rollout by changing the image, and wait for it to reached the paused state.
kubectl argo rollouts set image rollouts-demo rollouts-demo=argoproj/rollouts-demo:yellow\nkubectl argo rollouts get rollout rollouts-demo\n
At this point, both the canary and stable version of the Rollout are running, with 5% of the traffic directed to the canary. To understand how this works, inspect the VirtualService which the Rollout was referencing. When looking at both the VirtualService, we see that the route destination weights have been modified by the controller to reflect the current weight of the canary.
apiVersion: networking.istio.io/v1beta1\nkind: VirtualService\nmetadata:\n name: rollouts-demo-vsvc1\n namespace: default\nspec:\n gateways:\n - rollouts-demo-gateway\n hosts:\n - rollouts-demo-vsvc1.local\n http:\n - name: http-primary\n route:\n - destination:\n host: rollouts-demo-stable\n port:\n number: 15372\n weight: 95\n - destination:\n host: rollouts-demo-canary\n port:\n number: 15372\n weight: 5\n tls:\n - match:\n - port: 443\n sniHosts:\n - reviews.bookinfo.com\n - localhost\n route:\n - destination:\n host: rollouts-demo-stable\n weight: 95\n - destination:\n host: rollouts-demo-canary\n weight: 5\n tcp:\n - match:\n - port: 8020\n route:\n - destination:\n host: rollouts-demo-stable\n weight: 95\n - destination:\n host: rollouts-demo-canary\n weight: 5\n
apiVersion: networking.istio.io/v1beta1\nkind: VirtualService\nmetadata:\n name: rollouts-demo-vsvc2\n namespace: default\nspec:\n gateways:\n - rollouts-demo-gateway\n hosts:\n - rollouts-demo-vsvc2.local\n http:\n - name: http-primary\n route:\n - destination:\n host: rollouts-demo-stable\n port:\n number: 15373\n weight: 95\n - destination:\n host: rollouts-demo-canary\n port:\n number: 15373\n weight: 5\n tls:\n - match:\n - port: 443\n sniHosts:\n - reviews.bookinfo.com\n route:\n - destination:\n host: rollouts-demo-stable\n weight: 95\n - destination:\n host: rollouts-demo-canary\n weight: 5\n tcp:\n - match:\n - port: 8020\n route:\n - destination:\n host: rollouts-demo-stable\n weight: 95\n - destination:\n host: rollouts-demo-canary\n weight: 5\n
As the Rollout progresses through steps, the HTTP, TLS, and/or TCP route(s) destination weights will be adjusted to match the current setWeight of the steps.
"},{"location":"getting-started/mixed/","title":"Getting Started - Multiple Providers (Service Mesh Interface and NGiNX Ingress)","text":"Important
Available since v1.2
This guide covers how Argo Rollouts integrates with multiple TrafficRoutings, using Linkerd and NGINX Ingress Controller for traffic shaping, but you should be able to produce any other combination between the existing trafficRouting options.
This guide builds upon the concepts of the basic getting started guide, NGINX Guide, and SMI Guide.
"},{"location":"getting-started/mixed/#requirements","title":"Requirements","text":" - Kubernetes cluster with Linkerd installed
- Kubernetes cluster with NGINX ingress controller installed and part of the mesh
Tip
See the environment setup guide for linkerd on how to setup a local minikube environment with linkerd and nginx.
"},{"location":"getting-started/mixed/#1-deploy-the-rollout-services-and-ingress","title":"1. Deploy the Rollout, Services, and Ingress","text":"When SMI is used as one the traffic routers, the Rollout canary strategy must define the following mandatory fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n strategy:\n canary:\n # Reference to a Service which the controller will update to point to the canary ReplicaSet\n canaryService: rollouts-demo-canary\n # Reference to a Service which the controller will update to point to the stable ReplicaSet\n stableService: rollouts-demo-stable\n trafficRouting:\n smi: {}\n
When NGINX Ingress is used as the traffic router, the Rollout canary strategy must define the following mandatory fields: apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n strategy:\n canary:\n # Reference to a Service which the controller will update to point to the canary ReplicaSet\n canaryService: rollouts-demo-canary\n # Reference to a Service which the controller will update to point to the stable ReplicaSet\n stableService: rollouts-demo-stable\n trafficRouting:\n nginx:\n # Reference to an Ingress which has a rule pointing to the stable service (e.g. rollouts-demo-stable)\n # This ingress will be cloned with a new name, in order to achieve NGINX traffic splitting.\n stableIngress: rollouts-demo-stable\n...\n
A combination of both should have comply with each TrafficRouting requirements, in this case:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n strategy:\n canary:\n # Reference to a Service which the controller will update to point to the canary ReplicaSet\n canaryService: rollouts-demo-canary\n # Reference to a Service which the controller will update to point to the stable ReplicaSet\n stableService: rollouts-demo-stable\n trafficRouting:\n nginx:\n # Reference to an Ingress which has a rule pointing to the stable service (e.g. rollouts-demo-stable)\n # This ingress will be cloned with a new name, in order to achieve NGINX traffic splitting.\n stableIngress: rollouts-demo-stable\n smi: {}\n
The Ingress referenced in canary.trafficRouting.nginx.stableIngress is required to have a host rule which has a backend targeting the Service referenced under canary.stableService. In our example, that stable Service is named: rollouts-demo-stable:
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n name: rollouts-demo-stable\n annotations:\n kubernetes.io/ingress.class: nginx\nspec:\n rules:\n - host: rollouts-demo.local\n http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n # Reference to a Service name, also specified in the Rollout spec.strategy.canary.stableService field\n name: rollouts-demo-stable\n port:\n number: 80\n
Run the following commands to deploy:
- A Rollout with the Linkerd
linkerd.io/inject: enabled annotation - Two Services (stable and canary)
- An Ingress
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/mixed/rollout.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/mixed/services.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/mixed/ingress.yaml\n
After applying the manifests you should see the following rollout, services, and ingress resources in the cluster:
$ kubectl get ro\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE\nrollouts-demo 1 2 1 2\n\n$ kubectl get svc\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nrollouts-demo-canary ClusterIP 10.111.69.188 <none> 80/TCP 23m\nrollouts-demo-stable ClusterIP 10.109.175.248 <none> 80/TCP 23m\n\n$ kubectl get ing\nNAME CLASS HOSTS ADDRESS PORTS AGE\nrollouts-demo-stable <none> rollouts-demo.local 192.168.64.2 80 23m\n
You should also see a TrafficSplit resource which is created automatically and owned by the rollout:
$ kubectl get trafficsplit\nNAME SERVICE\nrollouts-demo rollouts-demo-stable\n
When inspecting the generated TrafficSplit resource, the weights are automatically configured to send 100% traffic to the rollouts-demo-stable service, and 0% traffic to the rollouts-demo-canary. These values will be updated during an update.
apiVersion: split.smi-spec.io/v1alpha1\nkind: TrafficSplit\nmetadata:\n name: rollouts-demo\n namespace: default\nspec:\n backends:\n - service: rollouts-demo-canary\n weight: \"0\"\n - service: rollouts-demo-stable\n weight: \"100\"\n service: rollouts-demo-stable\n
You should also notice a second ingress created by the rollouts controller, rollouts-demo-rollouts-demo-stable-canary. This ingress is the \"canary ingress\", which is a clone of the user-managed Ingress referenced under nginx.stableIngress. It is used by nginx ingress controller to achieve canary traffic splitting. The name of the generated ingress is formulated using <ROLLOUT-NAME>-<INGRESS-NAME>-canary. More details on the second Ingress are discussed in the following section.
"},{"location":"getting-started/mixed/#2-perform-an-update","title":"2. Perform an update","text":"Now perform an update the rollout by changing the image, and wait for it to reached the paused state.
kubectl argo rollouts set image rollouts-demo rollouts-demo=argoproj/rollouts-demo:yellow\nkubectl argo rollouts get rollout rollouts-demo\n
At this point, both the canary and stable version of the Rollout are running, with 5% of the traffic directed to the canary and 95% to the stable. When inspecting the TrafficSplit generated by the controller, we see that the weight has been updated to reflect the current setWeight: 5 step of the canary deploy.
apiVersion: split.smi-spec.io/v1alpha1\nkind: TrafficSplit\nmetadata:\n name: rollouts-demo\n namespace: default\nspec:\n backends:\n - service: rollouts-demo-canary\n weight: \"5\"\n - service: rollouts-demo-stable\n weight: \"95\"\n service: rollouts-demo-stable\n
When inspecting the rollout controller generated Ingress copy, we see that it has the following changes over the original ingress: - Two additional NGINX specific canary annotations are added to the annotations.
- The Ingress rules will have an rule which points the backend to the canary service.
apiVersion: extensions/v1beta1\nkind: Ingress\nmetadata:\n name: rollouts-demo-rollouts-demo-stable-canary\n annotations:\n kubernetes.io/ingress.class: nginx\n nginx.ingress.kubernetes.io/canary: \"true\"\n nginx.ingress.kubernetes.io/canary-weight: \"5\"\nspec:\n rules:\n - host: rollouts-demo.local\n http:\n paths:\n - backend:\n serviceName: rollouts-demo-canary\n servicePort: 80\n
As the Rollout progresses through steps, the weights in the TrafficSplit and Ingress resource will be adjusted to match the current setWeight of the steps.
"},{"location":"getting-started/nginx/","title":"Getting Started - NGINX Ingress","text":"This guide covers how Argo Rollouts integrates with the NGINX Ingress Controller for traffic shaping. This guide builds upon the concepts of the basic getting started guide.
"},{"location":"getting-started/nginx/#requirements","title":"Requirements","text":" - Kubernetes cluster with NGINX ingress controller installed
Tip
See the environment setup guide for NGINX on how to setup a local minikube environment with nginx.
"},{"location":"getting-started/nginx/#1-deploy-the-rollout-services-and-ingress","title":"1. Deploy the Rollout, Services, and Ingress","text":"When NGINX Ingress is used as the traffic router, the Rollout canary strategy must define the following mandatory fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n strategy:\n canary:\n # Reference to a Service which the controller will update to point to the canary ReplicaSet\n canaryService: rollouts-demo-canary\n # Reference to a Service which the controller will update to point to the stable ReplicaSet\n stableService: rollouts-demo-stable\n trafficRouting:\n nginx:\n # Reference to an Ingress which has a rule pointing to the stable service (e.g. rollouts-demo-stable)\n # This ingress will be cloned with a new name, in order to achieve NGINX traffic splitting.\n stableIngress: rollouts-demo-stable\n...\n
The Ingress referenced in canary.trafficRouting.nginx.stableIngress is required to have a host rule which has a backend targeting the Service referenced under canary.stableService. In our example, that stable Service is named: rollouts-demo-stable:
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n name: rollouts-demo-stable\n annotations:\n kubernetes.io/ingress.class: nginx\nspec:\n rules:\n - host: rollouts-demo.local\n http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n # Reference to a Service name, also specified in the Rollout spec.strategy.canary.stableService field\n name: rollouts-demo-stable\n port:\n number: 80\n
Run the following commands to deploy:
- A Rollout
- Two Services (stable and canary)
- An Ingress
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/nginx/rollout.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/nginx/services.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/nginx/ingress.yaml\n
After applying the manifests you should see the following rollout, services, and ingress resources in the cluster:
$ kubectl get ro\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE\nrollouts-demo 1 1 1 1\n\n$ kubectl get svc\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nrollouts-demo-canary ClusterIP 10.96.6.241 <none> 80/TCP 33s\nrollouts-demo-stable ClusterIP 10.102.229.83 <none> 80/TCP 33s\n\n$ kubectl get ing\nNAME CLASS HOSTS ADDRESS PORTS AGE\nrollouts-demo-stable <none> rollouts-demo.local 192.168.64.2 80 36s\nrollouts-demo-rollouts-demo-stable-canary <none> rollouts-demo.local 192.168.64.2 80 35s\n
You should also notice a second ingress created by the rollouts controller, rollouts-demo-rollouts-demo-stable-canary. This ingress is the \"canary ingress\", which is a clone of the user-managed Ingress referenced under nginx.stableIngress. It is used by nginx ingress controller to achieve canary traffic splitting. The name of the generated ingress is formulated using <ROLLOUT-NAME>-<INGRESS-NAME>-canary. More details on the second Ingress are discussed in the following section.
kubectl argo rollouts get rollout rollouts-demo\n
"},{"location":"getting-started/nginx/#2-perform-an-update","title":"2. Perform an update","text":"Update the rollout by changing the image, and wait for it to reached the paused state.
kubectl argo rollouts set image rollouts-demo rollouts-demo=argoproj/rollouts-demo:yellow\nkubectl argo rollouts get rollout rollouts-demo\n
At this point, both the canary and stable version of the Rollout are running, with 5% of the traffic directed to the canary. One thing to note, is that the rollout is able to achieve a 5% canary weight despite only running two pods. This is able to be achieved since the traffic split happens at the ingress controller (as opposed to weighted replica counts and kube-proxy in the basic guide).
When inspecting the rollout controller generated Ingress copy, we see that it has the following changes over the original ingress:
- Two additional NGINX specific canary annotations are added to the annotations.
- The Ingress rules will have an rule which points the backend to the canary service.
apiVersion: extensions/v1beta1\nkind: Ingress\nmetadata:\n name: rollouts-demo-rollouts-demo-stable-canary\n annotations:\n kubernetes.io/ingress.class: nginx\n nginx.ingress.kubernetes.io/canary: \"true\"\n nginx.ingress.kubernetes.io/canary-weight: \"5\"\nspec:\n rules:\n - host: rollouts-demo.local\n http:\n paths:\n - backend:\n serviceName: rollouts-demo-canary\n servicePort: 80\n
As the Rollout progresses through steps, the canary-weight annotation will be adjusted to match the current setWeight of the steps. The NGINX ingress controller examines the original Ingress, the canary Ingress, and the canary-weight annotation to determine what percentage of traffic to split between the two Ingresses.
"},{"location":"getting-started/setup/","title":"Environment Set Up","text":"This guide shows how to set up a local environment for development, testing, learning, or demoing purposes.
"},{"location":"getting-started/setup/#helm","title":"Helm","text":"Some dependencies are installable via the Helm stable repository:
helm repo add stable https://charts.helm.sh/stable\nhelm repo add grafana https://grafana.github.io/helm-charts\nhelm repo add prometheus-community https://prometheus-community.github.io/helm-charts\nhelm repo update\n
"},{"location":"getting-started/setup/#minikube","title":"Minikube","text":""},{"location":"getting-started/setup/#nginx-ingress-controller-setup","title":"NGINX Ingress Controller Setup","text":"The following instructions describe how to configure NGINX Ingress Controller on minikube. For basic ingress support, only the \"ingress\" addon needs to be enabled:
minikube addons enable ingress\n
Optionally, Prometheus and Grafana can be installed to utilize progressive delivery functionality:
# Install Prometheus\nkubectl create ns monitoring\nhelm install prometheus prometheus-community/prometheus -n monitoring -f docs/getting-started/setup/values-prometheus.yaml\n\n# Patch the ingress-nginx-controller pod so that it has the required\n# prometheus annotations. This allows the pod to be scraped by the\n# prometheus server.\nkubectl patch deploy ingress-nginx-controller -n ingress-nginx -p \"$(cat docs/getting-started/setup/ingress-nginx-controller-metrics-scrape.yaml)\"\n
Note
For Minikube version 1.18.1 or earlier, change the -n parameter value (namespace) to kube-system.
# Install grafana along with nginx ingress dashboards\nhelm install grafana grafana/grafana -n monitoring -f docs/getting-started/setup/values-grafana-nginx.yaml\n\n# Grafana UI can be accessed by running:\nminikube service grafana -n monitoring\n
"},{"location":"getting-started/setup/#istio-setup","title":"Istio Setup","text":"The following instructions describe how to configure Istio on minikube.
# Istio on Minikube requires additional memory and CPU\nminikube start --memory=8192mb --cpus=4\n\n# Install istio\nminikube addons enable istio-provisioner\nminikube addons enable istio\n\n# Label the default namespace to enable istio sidecar injection for the namespace\nkubectl label namespace default istio-injection=enabled\n
Istio already comes with a Prometheus database ready to use. To visualize metrics about istio services, Grafana and Istio dashboards can be installed via Helm to leverage progressive delivery functionality:
# Install Grafana and Istio dashboards\nhelm install grafana grafana/grafana -n istio-system -f docs/getting-started/setup/values-grafana-istio.yaml\n\n# Grafana UI can be accessed by running\nminikube service grafana -n istio-system\n
In order for traffic to enter the Istio mesh, the request needs to go through an Istio ingress gateway, which is simply a normal Kubernetes Deployment and Service. One convenient way to reach the gateway using minikube, is using the minikube tunnel command, which assigns Services a LoadBalancer. This command should be run in the background, usually in a separate terminal window:
minikube tunnel\n
While running minikube tunnel, the istio-ingressgateway Service will now have an external IP which can be retrieved via kubectl:
$ kubectl get svc -n istio-system istio-ingressgateway\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nistio-ingressgateway LoadBalancer 10.100.136.45 10.100.136.45 15020:31711/TCP,80:31298/TCP.... 7d22h\n
The LoadBalancer external IP (10.100.136.45 in this example) is now reachable to access services in the Istio mesh. Istio routes requests to the correct pod based on the Host HTTP header. Follow the guide on supplying host headers to learn how to configure your client environment to supply the proper request to reach the pod.
"},{"location":"getting-started/setup/#linkerd-setup","title":"Linkerd Setup","text":"Linkerd can be installed using the linkerd CLI.
brew install linkerd\nlinkerd install | kubectl apply -f -\n
Linkerd does not provide its own ingress controller, choosing instead to work alongside your ingress controller of choice. On minikube, we can use the built-in NGINX ingress addon and reconfigure it to be part of the linkerd mesh.
# Install the NGINX ingress controller addon\nminikube addons enable ingress\n\n# Patch the nginx-ingress-controller deployment to allow injection of the linkerd proxy to the\n# pod, so that it will be part of the mesh.\nkubectl patch deploy ingress-nginx-controller -n kube-system \\\n -p '{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"linkerd.io/inject\":\"enabled\"}}}}}'\n
"},{"location":"getting-started/setup/#supplying-host-headers","title":"Supplying Host Headers","text":"Most ingress controllers and service mesh implementations rely on the Host HTTP request header being supplied in the request in order to determine how to route the request to the correct pod.
"},{"location":"getting-started/setup/#determining-the-hostname-to-ip-mapping","title":"Determining the hostname to IP mapping","text":"For the Host header to be set in the request, the hostname of the service should resolve to the public IP address of the ingress or service mesh. Depending on if you are using an ingress controller or a service mesh, use one of the following techniques to determine the correct hostname to IP mapping:
"},{"location":"getting-started/setup/#ingresses","title":"Ingresses","text":"For traffic which is reaching the cluster network via a normal Kubernetes Ingress, the hostname should map to the IP of the ingress. We can retrieve the external IP of the ingress from the Ingress object itself, using kubectl:
$ kubectl get ing rollouts-demo-stable\nNAME CLASS HOSTS ADDRESS PORTS AGE\nrollouts-demo-stable <none> rollouts-demo.local 192.168.64.2 80 80m\n
In the example above, the hostname rollouts-demo.local should be configured to resolve to the IP 192.168.64.2. The next section describes various ways to configure your local system to resolve the hostname to the desired IP.
"},{"location":"getting-started/setup/#istio","title":"Istio","text":"In the case of Istio, traffic enters the mesh through an Ingress Gateway, which simply is a load balancer sitting at the edge of mesh.
To determine the correct hostname to IP mapping, it largely depends on what was configured in the VirtualService and Gateway. If you are following the Istio getting started guide, the examples use the \"default\" istio ingress gateway, which we can obtain from kubectl:
$ kubectl get svc -n istio-system istio-ingressgateway\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nistio-ingressgateway LoadBalancer 10.100.136.45 10.100.136.45 15020:31711/TCP,80:31298/TCP.... 7d22h\n
In the above example, the hostname rollouts-demo.local should be configured to resolve to the IP 10.100.136.45. The next section describes various ways to configure your local system to resolve the hostname to the desired IP.
"},{"location":"getting-started/setup/#configuring-local-hostname-resolution","title":"Configuring local hostname resolution","text":"Now that you have determined the correct hostname to IP mapping, the next step involves configuring the system so that will resolve properly. There are different techniques to do this:
"},{"location":"getting-started/setup/#dns-entry","title":"DNS Entry","text":"In real, production environments, the Host header is typically achieved by adding a DNS entry for the hostname in the DNS server. However, for local development, this is typically not an easily accessible option.
"},{"location":"getting-started/setup/#etchosts-entry","title":"/etc/hosts Entry","text":"On local workstations, a local entry to /etc/hosts can be added to map the hostname and IP address of the ingress. For example, the following is an example of an /etc/hosts file which maps rollouts-demo.local to IP 10.100.136.45.
##\n# Host Database\n#\n# localhost is used to configure the loopback interface\n# when the system is booting. Do not change this entry.\n##\n127.0.0.1 localhost\n255.255.255.255 broadcasthost\n::1 localhost\n\n10.100.136.45 rollouts-demo.local\n
The advantages of using a host entry, are that it works for all clients (CLIs, browsers). On the other hand, it is harder to maintain if the IP address changes frequently.
"},{"location":"getting-started/setup/#supply-header-in-curl","title":"Supply Header in Curl","text":"Clients such as curl, have the ability to explicitly set a header (the -H flag in curl). For example:
$ curl -I -H 'Host: rollouts-demo.local' http://10.100.136.45/color\nHTTP/1.1 200 OK\ncontent-type: text/plain; charset=utf-8\nx-content-type-options: nosniff\ndate: Wed, 24 Jun 2020 08:44:59 GMT\ncontent-length: 6\nx-envoy-upstream-service-time: 1\nserver: istio-envoy\n
Notice that the same request made without the header, fails with a 404 Not Found error.
$ curl -I http://10.100.136.45/color\nHTTP/1.1 404 Not Found\ndate: Wed, 24 Jun 2020 08:46:07 GMT\nserver: istio-envoy\ntransfer-encoding: chunked\n
"},{"location":"getting-started/setup/#browser-extension","title":"Browser Extension","text":"Similar to curl's ability to explicitly set a header, browsers can also achieve this via browser extensions. One example of a browser extension which can do this, is ModHeader.
"},{"location":"getting-started/smi/","title":"Getting Started - SMI (Service Mesh Interface)","text":"Important
Available since v0.9
This guide covers how Argo Rollouts integrates with the Service Mesh Interface (SMI), using Linkerd and NGINX Ingress Controller for traffic shaping. Since the SMI TrafficSplit resource is supported by multiple service mesh providers, the concepts taught here are applicable to other service mesh providers that support the interface. See the SMI Ecosystem for other projects that support SMI. This guide builds upon the concepts of the basic getting started guide.
"},{"location":"getting-started/smi/#requirements","title":"Requirements","text":" - Kubernetes cluster with Linkerd installed
- Kubernetes cluster with NGINX ingress controller installed and part of the mesh
Tip
See the environment setup guide for linkerd on how to setup a local minikube environment with linkerd and nginx.
"},{"location":"getting-started/smi/#1-deploy-the-rollout-services-and-ingress","title":"1. Deploy the Rollout, Services, and Ingress","text":"When SMI is used as the traffic router, the Rollout canary strategy must define the following mandatory fields:
apiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n name: rollouts-demo\nspec:\n strategy:\n canary:\n # Reference to a Service which the controller will update to point to the canary ReplicaSet\n canaryService: rollouts-demo-canary\n # Reference to a Service which the controller will update to point to the stable ReplicaSet\n stableService: rollouts-demo-stable\n trafficRouting:\n smi: {}\n
Run the following commands to deploy:
- A Rollout with the Linkerd
linkerd.io/inject: enabled annotation - Two Services (stable and canary)
- An Ingress
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/smi/rollout.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/smi/services.yaml\nkubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/smi/ingress.yaml\n
After applying the manifests you should see the following rollout, services, and ingress resources in the cluster:
$ kubectl get ro\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE\nrollouts-demo 1 2 1 2\n\n$ kubectl get svc\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nrollouts-demo-canary ClusterIP 10.111.69.188 <none> 80/TCP 23m\nrollouts-demo-stable ClusterIP 10.109.175.248 <none> 80/TCP 23m\n\n$ kubectl get ing\nNAME CLASS HOSTS ADDRESS PORTS AGE\nrollouts-demo-stable <none> rollouts-demo.local 192.168.64.2 80 23m\n
You should also see a TrafficSplit resource which is created automatically and owned by the rollout:
$ kubectl get trafficsplit\nNAME SERVICE\nrollouts-demo rollouts-demo-stable\n
When inspecting the generated TrafficSplit resource, the weights are automatically configured to send 100% traffic to the rollouts-demo-stable service, and 0% traffic to the rollouts-demo-canary. These values will be updated during an update.
apiVersion: split.smi-spec.io/v1alpha1\nkind: TrafficSplit\nmetadata:\n name: rollouts-demo\n namespace: default\nspec:\n backends:\n - service: rollouts-demo-canary\n weight: \"0\"\n - service: rollouts-demo-stable\n weight: \"100\"\n service: rollouts-demo-stable\n
"},{"location":"getting-started/smi/#2-perform-an-update","title":"2. Perform an update","text":"Now perform an update the rollout by changing the image, and wait for it to reached the paused state.
kubectl argo rollouts set image rollouts-demo rollouts-demo=argoproj/rollouts-demo:yellow\nkubectl argo rollouts get rollout rollouts-demo\n
At this point, both the canary and stable version of the Rollout are running, with 5% of the traffic directed to the canary. When inspecting the TrafficSplit generated by the controller, we see that the weight has been updated to reflect the current setWeight: 5 step of the canary deploy.
apiVersion: split.smi-spec.io/v1alpha1\nkind: TrafficSplit\nmetadata:\n name: rollouts-demo\n namespace: default\nspec:\n backends:\n - service: rollouts-demo-canary\n weight: \"5\"\n - service: rollouts-demo-stable\n weight: \"95\"\n service: rollouts-demo-stable\n
As the Rollout progresses through steps, the weights in the TrafficSplit resource will be adjusted to match the current setWeight of the steps.
"},{"location":"security/security/","title":"Security Policy for Argo Rollouts","text":""},{"location":"security/security/#reporting-a-vulnerability","title":"Reporting a Vulnerability","text":"If you find a security related bug in Argo Rollouts, we kindly ask you for responsible disclosure and for giving us appropriate time to react, analyze and develop a fix to mitigate the found security vulnerability.
Please report vulnerabilities by e-mail to the following address:
- cncf-argo-security@lists.cncf.io
All vulnerabilites and associated information will be treated with full confidentiality.
"},{"location":"security/security/#public-disclosure","title":"Public Disclosure","text":"We will publish security advisiories using the GitHub Security Advisories feature to keep our community well informed, and will credit you for your findings (unless you prefer to stay anonymous, of course).
"},{"location":"security/security/#internet-bug-bounty-collaboration","title":"Internet Bug Bounty collaboration","text":"We're happy to announce that the Argo project is collaborating with the great folks over at Hacker One and their Internet Bug Bounty program to reward the awesome people who find security vulnerabilities in the four main Argo projects (CD, Events, Rollouts and Workflows) and then work with us to fix and disclose them in a responsible manner.
If you report a vulnerability to us as outlined in this security policy, we will work together with you to find out whether your finding is eligible for claiming a bounty, and also on how to claim it.
"},{"location":"security/signed-release-assets/","title":"Verification of Argo Rollouts Artifacts","text":""},{"location":"security/signed-release-assets/#prerequisites","title":"Prerequisites","text":" - cosign
v2.0.0 or higher installation instructions - slsa-verifier installation instructions
"},{"location":"security/signed-release-assets/#release-assets","title":"Release Assets","text":"Asset Description argo-rollouts-checksums.txt Checksums of binaries argo-rollouts-cli.intoto.jsonl Attestation of CLI binaries & manifiest dashboard-install.yaml Dashboard install install.yaml Standard installation method kubectl-argo-rollouts-darwin-amd64 CLI Binary kubectl-argo-rollouts-darwin-arm64 CLI Binary kubectl-argo-rollouts-linux-amd64 CLI Binary kubectl-argo-rollouts-linux-arm64 CLI Binary kubectl-argo-rollouts-windows-amd64 CLI Binary namespace-install.yaml Namespace installation notifications-install.yaml Notification installation rollout_cr_schema.json Schema sbom.tar.gz Sbom sbom.tar.gz.pem Certificate used to sign sbom sbom.tar.gz.sig Signature of sbom"},{"location":"security/signed-release-assets/#verification-of-container-images","title":"Verification of container images","text":"Argo Rollouts container images are signed by cosign using identity-based (\"keyless\") signing and transparency. Executing the following command can be used to verify the signature of a container image:
cosign verify \\\n--certificate-identity-regexp https://github.com/argoproj/argo-rollouts/.github/workflows/image-reuse.yaml@refs/tags/v \\\n--certificate-oidc-issuer https://token.actions.githubusercontent.com \\\nquay.io/argoproj/argo-rollouts:v1.5.0 | jq\n
The command should output the following if the container image was correctly verified: The following checks were performed on each of these signatures:\n - The cosign claims were validated\n - Existence of the claims in the transparency log was verified offline\n - The code-signing certificate was verified using trusted certificate authority certificates\n[\n {\n \"critical\": {\n \"identity\": {\n \"docker-reference\": \"quay.io/argoproj/argo-rollouts\"\n },\n \"image\": {\n \"docker-manifest-digest\": \"sha256:519522f8c66c7b4c468f360ebe6c8ba07b8d64f5f948e71ae52c01b9953e1eb9\"\n },\n \"type\": \"cosign container image signature\"\n },\n \"optional\": {\n \"1.3.6.1.4.1.57264.1.1\": \"https://token.actions.githubusercontent.com\",\n \"1.3.6.1.4.1.57264.1.2\": \"push\",\n \"1.3.6.1.4.1.57264.1.3\": \"aa1afcb418fcebcc68b063377c48225f5a9d1511\",\n \"1.3.6.1.4.1.57264.1.4\": \"Release\",\n \"1.3.6.1.4.1.57264.1.5\": \"argoproj/argo-rollouts\",\n \"1.3.6.1.4.1.57264.1.6\": \"refs/tags/v1.5.0\",\n ...\n
"},{"location":"security/signed-release-assets/#verification-of-container-image-attestations","title":"Verification of container image attestations","text":"A SLSA Level 3 provenance is generated using slsa-github-generator.
The following command will verify the signature of an attestation and how it was issued. It will contain the payloadType, payload, and signature.
cosign verify-attestation --type slsaprovenance \\\n--certificate-identity-regexp https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v \\\n--certificate-oidc-issuer https://token.actions.githubusercontent.com \\\nquay.io/argoproj/argo-rollouts:v1.5.0 | jq\n
The payload is a non-falsifiable provenance which is base64 encoded and can be viewed by using the command below: cosign verify-attestation --type slsaprovenance \\\n--certificate-identity-regexp https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v \\\n--certificate-oidc-issuer https://token.actions.githubusercontent.com \\\nquay.io/argoproj/argo-rollouts:v1.5.0 | jq -r .payload | base64 -d | jq\n
Tip
cosign or slsa-verifier can both be used to verify image attestations. Check the documentation of each binary for detailed instructions.
"},{"location":"security/signed-release-assets/#verification-of-cli-artifacts-with-attestations","title":"Verification of CLI artifacts with attestations","text":"A single attestation (argo-rollouts.intoto.jsonl) from each release is provided. This can be used with slsa-verifier to verify that a CLI binary or manifest was generated using Argo Rollouts workflows on GitHub and ensures it was cryptographically signed.
slsa-verifier verify-artifact kubectl-argo-rollouts-linux-amd64 --provenance-path kubectl-argo-rollouts.intoto.jsonl --source-uri github.com/argoproj/argo-rollouts\n
"},{"location":"security/signed-release-assets/#verifying-an-artifact-and-output-the-provenance","title":"Verifying an artifact and output the provenance","text":"slsa-verifier verify-artifact kubectl-argo-rollouts-linux-amd64 --provenance-path kubectl-argo-rollouts.intoto.jsonl --source-uri github.com/argoproj/argo-rollouts --print-provenance | jq\n
"},{"location":"security/signed-release-assets/#verification-of-sbom","title":"Verification of Sbom","text":"cosign verify-blob --signature sbom.tar.gz.sig --certificate sbom.tar.gz.pem \\\n--certificate-identity-regexp ^https://github.com/argoproj/argo-rollouts/.github/workflows/release.yaml@refs/tags/v \\\n--certificate-oidc-issuer https://token.actions.githubusercontent.com \\\n ~/Downloads/sbom.tar.gz | jq\n
"},{"location":"security/signed-release-assets/#verification-on-kubernetes","title":"Verification on Kubernetes","text":""},{"location":"security/signed-release-assets/#policy-controllers","title":"Policy controllers","text":"Note
We encourage all users to verify signatures and provenances with your admission/policy controller of choice. Doing so will verify that an image was built by us before it's deployed on your Kubernetes cluster.
Cosign signatures and SLSA provenances are compatible with several types of admission controllers. Please see the cosign documentation and slsa-github-generator for supported controllers.
"}]}
\ No newline at end of file