-
Notifications
You must be signed in to change notification settings - Fork 177
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs(otel_metrics): add OTEL metrics output documentation
Signed-off-by: Leonardo Di Giovanna <[email protected]>
- Loading branch information
Showing
2 changed files
with
195 additions
and
0 deletions.
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,195 @@ | ||
# OTEL Metrics | ||
|
||
- **Category**: Metrics/Observability | ||
- **Website**: <https://opentelemetry.io/docs/concepts/signals/metrics/> | ||
|
||
## Table of content | ||
|
||
- [OTEL Metrics](#otel-metrics) | ||
- [Table of content](#table-of-content) | ||
- [Configuration](#configuration) | ||
- [Example of config.yaml](#example-of-configyaml) | ||
- [Additional info](#additional-info) | ||
- [Running a whole stack with docker-compose](#running-a-whole-stack-with-docker-compose) | ||
|
||
## Configuration | ||
|
||
| Setting | Env var | Default value | Description | | ||
|--------------------------------|--------------------------------|--------------------|-------------------------------------------------------------------------------------------------------------------------------------| | ||
| `otlp.metrics.endpoint` | `OTLP_METRICS_ENDPOINT` | | OTLP endpoint, typically in the form http{s}://{domain or ip}:4318/v1/metrics | | ||
| `otlp.metrics.protocol` | `OTLP_METRICS_PROTOCOL` | `grpc` | OTLP transport protocol to be used for metrics data; it can be `"grpc"` or `"http/protobuf"` | | ||
| `otlp.metrics.timeout` | `OTLP_METRICS_TIMEOUT` | `10000` (from SDK) | OTLP timeout for outgoing metrics in milliseconds | | ||
| `otlp.metrics.headers` | `OTLP_METRICS_HEADERS` | `""` | List of headers to apply to all outgoing metrics in the form of `some-key=some-value,other-key=other-value` | | ||
| `otlp.metrics.extraenvvars` | `OTLP_METRICS_EXTRAENVVARS` | `""` | Extra env vars (override the other settings) | | ||
| `otlp.metrics.minimumpriority` | `OTLP_METRICS_MINIMUMPRIORITY` | `""` (=`debug`) | Minimum priority of event for using this output, order is `emergency,alert,critical,error,warning,notice,informational,debug or ""` | | ||
| `otlp.metrics.checkcert` | `OTLP_METRICS_CHECKCERT` | `true` | Set to false if you want to skip TLS certificate validation (only with https) | | ||
|
||
> [!NOTE] | ||
For the extra Env Vars values see [standard `OTEL_*` environment variables](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/): | ||
|
||
## Example of config.yaml | ||
|
||
```yaml | ||
otlp: | ||
metrics: | ||
# endpoint: "" # OTLP endpoint, typically in the form http{s}://{domain or ip}:4318/v1/metrics | ||
# protocol: "" # OTLP transport protocol to be used for metrics data; it can be "grpc" or "http/protobuf" (default: "grpc") | ||
# timeout: "" # OTLP timeout for outgoing metrics in milliseconds (default: "" which uses SDK default: 10000) | ||
# headers: "" # List of headers to apply to all outgoing metrics in the form of "some-key=some-value,other-key=other-value" (default: "") | ||
# extraenvvars: # Extra env vars (override the other settings) (default: "") | ||
# OTEL_EXPORTER_OTLP_METRICS_TIMEOUT: 10000 | ||
# OTEL_EXPORTER_OTLP_TIMEOUT: 10000 | ||
# minimumpriority: "" # Minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default: "") | ||
# checkcert: true # Set to false if you want to skip TLS certificate validation (only with https) (default: true) | ||
``` | ||
|
||
## Additional info | ||
|
||
> [!NOTE] | ||
The OTLP Metrics are only available for the source: `syscalls`. | ||
|
||
## Running a whole stack with docker-compose | ||
|
||
Below `docker-compose` file runs a stack of: | ||
|
||
- `falco` | ||
- `falcosidekick` | ||
- `prometheus` as metrics backend | ||
- OTEL collector to collect OTEL metrics from `falcosidekick` and let prometheus scrape them | ||
- `events-generator` to generate arbitrary Falco events | ||
|
||
### Requirements | ||
|
||
A local Linux kernel capable of running `falco`--modern-bpf`, see <https://falco.org/blog/falco-modern-bpf/>. | ||
|
||
### Configuration files | ||
|
||
You need to create these files: | ||
|
||
- `./docker-compose.yaml`: minimal docker-compose configuration | ||
|
||
```yaml | ||
--- | ||
services: | ||
falco: | ||
image: falcosecurity/falco:0.39.0 | ||
privileged: true | ||
volumes: | ||
- /var/run/docker.sock:/host/var/run/docker.sock | ||
- /dev:/host/dev | ||
- /proc:/host/proc:ro | ||
- /boot:/host/boot:ro | ||
- /lib/modules:/host/lib/modules:ro | ||
- /usr:/host/usr:ro | ||
- /etc/falco:/host/etc:ro | ||
command: [ | ||
"/usr/bin/falco" , | ||
"-o", "json_output=true", | ||
"-o", "http_output.enabled=true", | ||
"-o", "http_output.url=http://sidekick:2801", # Set the HTTP output url to Falco sidekick endpoint | ||
"-o", "http_output.insecure=true" | ||
] | ||
|
||
sidekick: | ||
image: falcosidekick:latest | ||
ports: | ||
- "2801:2801" # Expose default port towards Falco instance | ||
environment: | ||
- OTLP_METRICS_ENDPOINT=http://otel-collector:4317 | ||
- OTLP_METRICS_CHECKCERT=false | ||
|
||
otel-collector: | ||
image: otel/opentelemetry-collector-contrib | ||
volumes: | ||
- ./config.yaml:/etc/otelcol-contrib/config.yaml | ||
ports: | ||
- "4317:4317" # Expose OTLP gRPC port | ||
|
||
prometheus: | ||
image: prom/prometheus:latest | ||
volumes: | ||
- ./prometheus.yml:/etc/prometheus/prometheus.yml | ||
ports: | ||
- "9090:9090" # Expose port to access Prometheus expression browser | ||
|
||
event-generator: | ||
image: falcosecurity/event-generator | ||
command: run | ||
restart: always | ||
trigger: | ||
image: alpine | ||
command: [ # Alternate reads to /etc/shadow with creations of symlinks from it | ||
"sh", | ||
"-c", | ||
"while true; do cat /etc/shadow > /dev/null; sleep 5; ln -s /etc/shadow shadow; rm shadow; sleep 5; done" | ||
] | ||
``` | ||
|
||
> `./docker-compose.yaml` mentions the `falcosidekick:latest` docker image, that must be locally available before | ||
> bringing up the stack. You can build it from source by cloning the repository and issuing the building commands: | ||
> ```shell | ||
> git clone https://github.com/falcosecurity/falcosidekick.git | ||
> cd falcosidekick | ||
> go build . && docker build . -t falcosidekick:latest | ||
> ``` | ||
- `./config.yaml`: minimal OTEL collector configuration | ||
```yaml | ||
--- | ||
receivers: | ||
otlp: | ||
protocols: | ||
grpc: | ||
endpoint: "0.0.0.0:4317" | ||
exporters: | ||
prometheus: | ||
endpoint: "0.0.0.0:9090" | ||
service: | ||
pipelines: | ||
metrics: | ||
receivers: [otlp] | ||
processors: [] | ||
exporters: [prometheus] | ||
``` | ||
- `./prometheus.yml`: minimal prometheus configuration | ||
|
||
```yaml | ||
global: | ||
scrape_interval: 5s | ||
|
||
scrape_configs: | ||
- job_name: 'otel-collector' | ||
static_configs: | ||
- targets: ['otel-collector:9090'] | ||
``` | ||
### Run it | ||
To bring up the stack, and see the results on prometheus expression browser: | ||
1. Bring up the stack | ||
```shell | ||
docker compose up | ||
``` | ||
|
||
2. Navigate to <http://localhost:9090/graph> to start browsing the local prometheus expression browser | ||
|
||
3. Navigate to the `Graph` tab and adjust the time interval to be comparable to the stack uptime (e.g.: 15 minutes) | ||
|
||
5. To get information regarding the `falcosecurity_falco_rules_matches_total` metric, you can enter a simple query like | ||
`falcosecurity_falco_rules_matches_total` or `sum by (rule) (falcosecurity_falco_rules_matches_total)` and press | ||
`Execute` | ||
|
||
6. Explore the obtained results | ||
![Falco metrics view](images/otlp_metrics-prom_view.png) | ||
|
||
1. Bring down the stack | ||
|
||
```shell | ||
docker compose down | ||
``` |