[pull] main from open-telemetry:main

content/en/blog/2023/opamp-status/index.md

@@ -0,0 +1,152 @@
+---
+title: Open Agent Management Protocol (OpAMP) State of the Nation 2023
+linkTitle: OpAMP Status
+date: 2023-10-23
+author: >-
+  [Jacob Aronoff](https://github.com/jaronoff97) (Lightstep), [Evan
+  Bradley](https://github.com/evan-bradley) (Dynatrace), [Michael
+  Hausenblas](https://github.com/mhausenblas) (AWS), [Andy
+  Keller](https://github.com/andykellr) (observIQ, Inc), [Tigran
+  Najaryan](https://github.com/tigrannajaryan) Splunk)
+draft: false
+cSpell:ignore: Aronoff Najaryan observ Tigran
+---
+
+[Open Agent Management Protocol (OpAMP)](/docs/collector/management/) is the
+emerging open standard to manage a fleet of telemetry agents at scale. In 2022,
+Splunk donated OpAMP to the OpenTelemetry (OTel) project, with initial feedback
+from observIQ, based on observIQ’s custom protocol used in BindPlane. The
+[OpAMP specification](https://github.com/open-telemetry/opamp-spec/blob/main/specification.md)
+defines a network protocol for remote management of fleets of agents. These
+agents can really be anything, from telemetry agents such as the
+[OpenTelemetry Collector](/docs/collector/) to
+[Fluent Bit](https://fluentbit.io/) to custom agents you might use in your
+environment.
+
+In OpAMP, we distinguish between the server side, usually hosted in a control
+plane, and the client side, implemented in the respective agent you want to
+manage. For example, using OpAMP to manage a fleet of OpenTelemetry Collectors,
+may look something like shown in the following:
+
+![OpAMP high-level concept: control plane an agents](opamp-concept.svg)
+
+The collectors report their status to and receive configuration from an OpAMP
+control plane. The OpAMP protocol is vendor-agnostic and generic (not OTel
+specific), so an OpAMP server can remotely monitor and manage a fleet of
+different agents. OpAMP currently supports, amongst other things:
+
+- Agents, such as the OpenTelemetry Collector, can report their properties, for
+  example, type and version, or also the host operating system details to the
+  Server (OpAMP control plane).
+- The Server can push configurations to Agents and ensures that said
+  configurations are applied, for example through reloading the Agent.
+- You can ingest the Agent's own telemetry (logs and metrics) into an
+  [OTLP](/docs/specs/otlp/)-compliant observability backend.
+- Secure auto-updating capabilities for both upgrading and downgrading of the
+  Agents.
+- Built-in connection credentials management, including client-side TLS
+  certificate revocation and rotation.
+
+Now that we have a rough idea of what OpAMP is and what it supports, let’s have
+a look at how it is implemented in the OpenTelemetry Collector.
+
+In discussions with OTel end-users and collector contributors we found that they
+want to use OpAMP both as an collector extension, with limited functionality, as
+well as as an (collector-external) supervisor that implements a broader set of
+OpAMP capabilities.
+
+{{% alert title="Note" color="info" %}} If you want to dive deeper here, we
+suggest you read the
+[OpAMP for OpenTelemetry Collector document](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/opampsupervisor/specification)
+which describes how to implement both options while minimizing code
+duplication.{{% /alert %}}
+
+The main idea to support both modes is to implement an extension in the
+collector with a minimal set of OpAMP capabilities. This collector extension can
+be used on its own, and can also be used to create an external Supervisor that
+uses said extension as a helper and implements the remaining OpAMP capabilities
+on top of what the extension implements:
+
+![OpAMP Supervisor](opamp-supervisor.png)
+
+Let us start with a closer look at the collector OpAMP extension and then we
+move on to the OpAMP Supervisor.
+
+## OpAMP extension
+
+The
+[OpenTelemetry Collector OpAMP extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/16594)
+will implement an OpAMP client within the collector and will be used in both the
+standalone and supervised models for managing a collector instance through
+OpAMP. The functionality for the OpAMP extension while working with the
+Supervisor has been defined as part of the Supervisor's design document, where
+the extension will largely be tasked with providing bootstrapping information to
+the Supervisor and communicating the collector's effective configuration.
+
+## OpAMP supervisor
+
+The OpAMP Supervisor will exist as a separate binary that runs an OpenTelemetry
+Collector instance and implements an OpAMP client to relay configuration from an
+OpAMP server to the collector by merging remote and local configuration sources
+into a file that the collector can then use on startup. The supervised model of
+managing the collector will also allow for downloading additional binaries
+through the OpAMP protocol, allowing for downloading additional files as well as
+updating the collector.
+
+In addition, if the OpAMP Server sends the Supervisor a "bad" configuration and
+the collector fails to start, since it is running as a separate process, the
+Supervisor can communicate this with the OpAMP server to notify it of the
+situation. On top of implementing an OpAMP client, the Supervisor will also
+implement an OpAMP server where it will communicate with the OpAMP extension
+running in the collector to receive information about the Collector. We defined
+the Supervisor's functionality in a
+[design document](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/opampsupervisor/specification)
+and based on it we contributed an initial implementation to the
+[opentelemetry-collector-contrib repository](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/opampsupervisor)
+with a backlog of issues created to guide further development.
+
+## OpAMP in Kubernetes
+
+In OTel we have dedicated support for Kubernetes as a compute platform via the
+[OpenTelemetry Operator for Kubernetes](/docs/kubernetes/operator/). In this
+context, we're also working on OpAMP support in Kubernetes which will be
+available through a
+[bridge component](https://docs.google.com/document/d/1M8VLNe_sv1MIfu5bUR5OV_vrMBnAI7IJN-7-IAr37JY/)
+to be deployed by the OpenTelemetry Operator:
+
+![OpAMP bridge in the OTel Operator](opamp-kubernetes-bridge.png)
+
+{{% alert title="Note" color="info" %}} Currently, we don't have support for
+deployment via Helm chart, however, if you'd like to contribute to this, you're
+welcome to contribute.{{% /alert %}}
+
+The OpAMP bridge is a binary developed by the OTel
+[SIG Kubernetes Operator](https://docs.google.com/document/d/1Unbs2qp_j5kp8FfL_lRH-ld7i5EOQpsq0I4djkOOSL4/)
+and is responsible for maintaining pools of OpenTelemetry Collector resources in
+a Kubernetes cluster. The bridge acts as a client for an OpAMP operator server,
+reporting the effective configuration of the collector pools and enables the
+remote configuration of pools of collectors. In the future, the bridge will
+report richer information about the collector pools running in Kubernetes
+through enhanced status and improved health checks. You can also extend the
+bridge to support the remote configuration of Instrumentation resources. The
+bridge’s image is already built and released with the OpenTelemetry Operator. In
+addition, we're working on a convenience
+[Custom Resource Definition (CRD)](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1alpha1/opentelemetrycollector_types.go)
+which will enable you to easily deploy the bridge into a Kubernetes cluster.
+
+## What’s next?
+
+The community has been working on OpAMP now for more than a year and users are
+excited about the opportunities it promises to deliver. If you’re around at
+[KubeCon NA in Chicago, USA](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/)
+between Nov 6 and 9 2023, consider visiting us at the OpenTelemetry maintainers
+booth or find us at any of the many observability-related events such as the
+[Observability day](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/co-located-events/observability-day/).
+We’re super interested to learn from you about non-OTel collectors use cases and
+requirements as well. For now, if you’re a vendor and implementing the OpAMP
+spec please provide feedback and as an end-user you may wish to try out the
+reference implementation of the Server, Supervisor and a simple UI by following
+the steps in the [OpAMP entry in the OTel docs](/docs/collector/management/) or
+maybe even dive deep into the
+[Supervisor code (in Go)](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/opampsupervisor)
+in the OTel collector.

content/en/docs/demo/docker-deployment.md

@@ -2,15 +2,16 @@
 title: Docker deployment
 linkTitle: Docker
 aliases: [docker_deployment]
-cSpell:ignore: otelcollector otlphttp
+cSpell:ignore: otelcollector otlphttp spanmetrics
 ---
 
 ## Prerequisites
 
 - Docker
 - [Docker Compose](https://docs.docker.com/compose/install/#install-compose)
   v2.0.0+
-- 4 GB of RAM for the application
+- Make (optional)
+- 6 GB of RAM for the application
 
 ## Get and run the demo
 
@@ -26,21 +27,17 @@ cSpell:ignore: otelcollector otlphttp
     cd opentelemetry-demo/
     ```
 
-3.  Run docker compose[^1] to start the demo:
+3.  Use make to start the demo:
 
     ```shell
-    docker compose up --no-build
+    make start
     ```
 
     > **Notes:**
     >
-    > - The `--no-build` flag is used to fetch released docker images from
-    >   [ghcr](https://ghcr.io/open-telemetry/demo) instead of building from
-    >   source. Removing the `--no-build` command line option will rebuild all
-    >   images from source. It may take more than 20 minutes to build if the
-    >   flag is omitted.
-    > - If you're running on Apple Silicon, run `docker compose build`[^1] in
-    >   order to create local images vs. pulling them from the repository.
+    > If you do not have the make utility installed, you can also use
+    > `docker compose up --force-recreate --remove-orphans --detach`[^1] to
+    > start the demo.
 
 ## Verify the web store and Telemetry
 
@@ -78,24 +75,28 @@ with an editor.
       endpoint: <your-endpoint-url>
   ```
 
-- Then add a new pipeline with your new exporter:
+- Then override the `exporters` for telemetry pipelines that you want to use for
+  your backend.
 
   ```yaml
   service:
     pipelines:
       traces:
-        receivers: [otlp]
-        processors: [batch]
-        exporters: [otlphttp/example]
+        exporters: [spanmetrics, otlphttp/example]
   ```
 
+{{% alert title="Note" color="info" %}} When merging YAML values with the
+Collector, objects are merged and arrays are replaced. The `spanmetrics`
+exporter must be included in the array of exporters for the `traces` pipeline if
+overridden. Not including this exporter will result in an error. {{% /alert %}}
+
 Vendor backends might require you to add additional parameters for
 authentication, please check their documentation. Some backends require
 different exporters, you may find them and their documentation available at
 [opentelemetry-collector-contrib/exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter).
 
 After updating the `otelcol-config-extras.yml`, start the demo by running
-`docker compose up`[^1]. After a while, you should see the traces flowing into
-your backend as well.
+`make start`. After a while, you should see the traces flowing into your backend
+as well.
 
 [^1]: {{% _param notes.docker-compose-v2 %}}

content/en/docs/demo/kubernetes-deployment.md

@@ -2,7 +2,7 @@
 title: Kubernetes deployment
 linkTitle: Kubernetes
 aliases: [kubernetes_deployment]
-cSpell:ignore: loadgen otlphttp
+cSpell:ignore: loadgen otlphttp spanmetrics
 ---
 
 We provide a [OpenTelemetry Demo Helm chart](/docs/kubernetes/helm/demo/) to
@@ -13,8 +13,8 @@ Helm's [documentation](https://helm.sh/docs/) to get started.
 
 ## Prerequisites
 
-- Kubernetes 1.23+
-- 4 GB of free RAM for the application
+- Kubernetes 1.24+
+- 6 GB of free RAM for the application
 - Helm 3.9+ (for Helm installation method only
 
 ## Install using Helm (recommended)
@@ -64,38 +64,62 @@ To expose the frontendproxy service use the following command (replace
 kubectl port-forward svc/my-otel-demo-frontendproxy 8080:8080
 ```
 
-In order for spans from the browser to be properly collected, you will also need
-to expose the OpenTelemetry Collector's OTLP/HTTP port (replace `my-otel-demo`
-with your Helm chart release name accordingly):
-
-```shell
-kubectl port-forward svc/my-otel-demo-otelcol 4318:4318
-```
-
 > **Note**: `kubectl port-forward` will proxy the port until the process
 > terminates. You may need to create separate terminal sessions for each use of
 > `kubectl port-forward`, and use <kbd>Ctrl-C</kbd> to terminate the process
 > when done.
 
-With the frontendproxy and Collector port-forward set up, you can access:
+With the frontendproxy port-forward set up, you can access:
 
 - Web store: <http://localhost:8080/>
 - Grafana: <http://localhost:8080/grafana/>
 - Feature Flags UI: <http://localhost:8080/feature/>
 - Load Generator UI: <http://localhost:8080/loadgen/>
 - Jaeger UI: <http://localhost:8080/jaeger/ui/>
 
-### Expose services using service type configurations
+### Expose Demo components using service or ingress configurations
+
+{{% alert title="Note" color="info" %}} We recommend that you use a values file
+when installing the Helm chart in order to specify additional configuration
+options. {{% /alert %}}
+
+#### Configure ingress resources
 
 > **Note** Kubernetes clusters may not have the proper infrastructure components
 > to enable LoadBalancer service types or ingress resources. Verify your cluster
 > has the proper support before using these configuration options.
 
-Each demo service (ie: frontendproxy) offers a way to have its Kubernetes
-service type configured. By default these will be `ClusterIP` but you can change
-each one using the `serviceType` property for each service.
+Each demo component (ie: frontendproxy) offers a way to have its Kubernetes
+service type configured. By default, these will not be created, but you can
+enable and configure them through the `ingress` property of each component.
+
+To configure the frontendproxy component to use an ingress resource you would
+specify the following in your values file:
+
+```yaml
+components:
+  frontendProxy:
+    ingress:
+      enabled: true
+      annotations: {}
+      hosts:
+        - host: otel-demo.my-domain.com
+          paths:
+            - path: /
+              pathType: Prefix
+              port: 8080
+```
+
+Some ingress controllers require special annotations or service types. Refer to
+the documentation from your ingress controller for more information.
 
-To configure the frontendproxy service to use a LoadBalancer service type you
+#### Configure service types
+
+Each demo component (ie: frontendproxy) offers a way to have its Kubernetes
+service type configured. By default, these will be `ClusterIP` but you can
+change each one using the `service.type` property of each component.
+
+To configure the frontendproxy component to use a LoadBalancer service type you
 would specify the following in your values file:
 
 ```yaml
@@ -105,30 +129,24 @@ components:
       type: LoadBalancer
 ```
 
-> **Note** It is recommended to use a values file when installing the Helm chart
-> in order to specify additional configuration options.
-
-The Helm chart does not provide facilities to create ingress resources. If
-required these would need to be created manually after installing the Helm
-chart. Some Kubernetes providers require specific service types in order to be
-used by ingress resources (ie: EKS ALB ingress, requires a NodePort service
-type).
+#### Configure browser telemetry
 
 In order for spans from the browser to be properly collected, you will also need
-to expose the OpenTelemetry Collector's OTLP/HTTP port to be accessible to user
-web browsers. The location where the OpenTelemetry Collector is exposed must
-also be passed into the frontend service using the
-`PUBLIC_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` environment variable. You can do
-this using the following in your values file:
+to specify the location where the OpenTelemetry Collector is exposed. The
+frontendproxy defines a route for the collector with a path prefix of
+`/otlp-http`. You can configure the collector endpoint by setting the following
+environment variable on the frontend component:
 
 ```yaml
 components:
   frontend:
     env:
       - name: PUBLIC_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
-        value: http://otel-demo-collector.mydomain.com:4318/v1/traces
+        value: http://otel-demo.my-domain.com/otlp-http/v1/traces
 ```
 
+#### Installation with a values file
+
 To install the Helm chart with a custom `my-values-file.yaml` values file use:
 
 ```shell
@@ -165,13 +183,13 @@ opentelemetry-collector:
     service:
       pipelines:
         traces:
-          receivers: [otlp]
-          processors: [batch]
-          exporters: [otlphttp/example]
+          exporters: [spanmetrics, otlphttp/example]
 ```
 
-> **Note** When merging YAML values with Helm, objects are merged and arrays are
-> replaced.
+{{% alert title="Note" color="info" %}} When merging YAML values with Helm,
+objects are merged and arrays are replaced. The `spanmetrics` exporter must be
+included in the array of exporters for the `traces` pipeline if overridden. Not
+including this exporter will result in an error. {{% /alert %}}
 
 Vendor backends might require you to add additional parameters for
 authentication, please check their documentation. Some backends require

content/en/docs/instrumentation/python/automatic/logs-example.md

@@ -39,7 +39,6 @@ Paste the following contents into `example.py`:
 import logging
 
 from opentelemetry import trace
-import logging
 
 tracer = trace.get_tracer_provider().get_tracer(__name__)