Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: enhancements to the troubleshooting section #3238

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
122 changes: 0 additions & 122 deletions docs/content/en/docs/troubleshooting/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,125 +5,3 @@ description: Learn how to troubleshoot Tetragon
aliases: ["/docs/tutorials/debugging-tetragon"]
---

## Automatic log and state collection

Before you report a problem, make sure to retrieve the necessary information
from your cluster.

Tetragon's bugtool captures potentially useful information about your
environment for debugging. The tool is meant to be used for debugging a single
Tetragon agent node but can be run automatically in a cluster. Note that in the
context of Kubernetes, the command needs to be run from inside the Tetragon
Pod's container.

Key information collected by bugtool:
- Tetragon configuration
- Network configuration
- Kernel configuration
- eBPF maps
- Process traces (if tracing is enabled)

### Automatic Kubernetes cluster sysdump

You can collect information in a Kubernetes cluster using the Cilium CLI:

```shell
cilium-cli sysdump
```

More details can be found in the [Cilium docs](https://docs.cilium.io/en/stable/operations/troubleshooting/#automatic-log-state-collection).
The Cilium CLI `sysdump` command will automatically run `tetra bugtool` on each
nodes where Tetragon is running.

### Manual single node sysdump

It's also possible to run the bug collection tool manually with the scope of a
single node using `tetra bugtool`.

#### Kubernetes installation

1. Identify the Tetragon Pod (`<tetragon-namespace>` is likely to be `kube-system`
with the default install):

```bash
kubectl get pods -n <tetragon-namespace> -l app.kubernetes.io/name=tetragon
```

2. Execute tetra bugtool within the Pod:

```bash
kubectl exec -n <tetragon-namespace> <tetragon-pod-name> -c tetragon -- tetra bugtool
```

3. Retrieve the created archive from the Pod's filesystem:

```bash
kubectl cp -c tetragon <tetragon-namespace>/<tetragon-pod-name>:tetragon-bugtool.tar.gz tetragon-bugtool.tar.gz
```

#### Container installation

1. Enter the Tetragon Container:

```bash
docker exec -it <tetragon-container-id> tetra bugtool
```

2. Retrieve the archive using docker cp:

```bash
docker cp <tetragon-container-id>:/tetragon-bugtool.tar.gz tetragon-bugtool.tar.gz
```

#### Systemd host installation

1. Execute tetra bugtool with Elevated Permissions:

```bash
sudo tetra bugtool
```

## Enable debug log level

When debugging, it might be useful to change the log level. The default log
level is controlled by the log-level option at startup:

* Enable debug level with `--log-level=debug`
* Enable trace level with `--log-level=trace`

### Change log level on Kubernetes

{{< warning >}}
The Pods of the Tetragon DaemonSet will be restarted automatically after
changing the debug Helm value.
{{< /warning >}}

It is possible to change the log level of Tetragon's DaemonSet Pods by setting
`tetragon.debug` to `true`.

### Change log level dynamically

It is possible to change the log level dynamically by using the `tetra loglevel`
sub-command. `tetra` needs access to Tetragon's gRPC server endpoint which can
be configured via `--server-address`.

{{< warning >}}
Keep in mind that Tetragon's gRPC server is (by default) only exposed on
`localhost`. Also, it's important to understand that this only changes the log
level of the single Tetragon instance targeted with `--server-address` and not
all Tetragon instances when it's, for example, running as DaemonSet in a
Kubernetes environment.
{{< /warning >}}

* Get the current log level:

```shell
tetra loglevel get
```

* Dynamically change the log level. Allowed values are
`[trace|debug|info|warning|error|fatal|panic]`:

```shell
tetra loglevel set debug
```
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
---
title: "BPF programs statistics"
weight: 5
description: "Monitor bpf programs statistics"
weight: 3
description: "Monitor BPF programs statistics"
aliases: ["/docs/concepts/performance-stats"]
---

This page shows you how to monitor bpf programs statistics.
This page shows you how to monitor BPF programs statistics.

## Concept

Expand Down
48 changes: 48 additions & 0 deletions docs/content/en/docs/troubleshooting/loglevel.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
title: "Log level"
weight: 2
description: "Learn how to configure log levels."
---

When debugging, it might be useful to change the log level. The default log
level is controlled by the log-level option at startup:

* Enable debug level with `--log-level=debug`
* Enable trace level with `--log-level=trace`

## Change log level on Kubernetes

{{< warning >}}
The Pods of the Tetragon DaemonSet will be restarted automatically after
changing the debug Helm value.
{{< /warning >}}

It is possible to change the log level of Tetragon's DaemonSet Pods by setting
`tetragon.debug` to `true`.

## Change log level dynamically

It is possible to change the log level dynamically by using the `tetra loglevel`
sub-command. `tetra` needs access to Tetragon's gRPC server endpoint which can
be configured via `--server-address`.

{{< warning >}}
Keep in mind that Tetragon's gRPC server is (by default) only exposed on
`localhost`. Also, it's important to understand that this only changes the log
level of the single Tetragon instance targeted with `--server-address` and not
all Tetragon instances when it's, for example, running as DaemonSet in a
Kubernetes environment.
{{< /warning >}}

* Get the current log level:

```shell
tetra loglevel get
```

* Dynamically change the log level. Allowed values are
`[trace|debug|info|warning|error|fatal|panic]`:

```shell
tetra loglevel set debug
```
100 changes: 100 additions & 0 deletions docs/content/en/docs/troubleshooting/sysdump.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
---
title: "System dump"
weight: 1
description: "Learn how to collect system dumps."
---

Before you report a problem, make sure to retrieve the necessary information
from your cluster.

Tetragon's bugtool captures potentially useful information about your
environment for debugging. The tool is meant to be used for debugging a single
Tetragon agent node but can be run automatically in a cluster. Note that in the
context of Kubernetes, the command needs to be run from inside the Tetragon
Pod's container.

Key information collected by bugtool:
- Tetragon configuration
- Network configuration
- Kernel configuration
- eBPF maps
- Process traces (if tracing is enabled)

## Automatic Kubernetes cluster sysdump

You can collect information in a Kubernetes cluster using the Cilium CLI:

```shell
cilium-cli sysdump
```

More details can be found in the [Cilium docs](https://docs.cilium.io/en/stable/operations/troubleshooting/#automatic-log-state-collection).
The Cilium CLI `sysdump` command will automatically run `tetra bugtool` on each
nodes where Tetragon is running.

## Manual single node sysdump

It's also possible to run the bug collection tool manually with the scope of a
single node using `tetra bugtool`.

### Kubernetes installation

1. Identify the Tetragon Pod (`<tetragon-namespace>` is likely to be `kube-system`
with the default install):

```bash
kubectl get pods -n <tetragon-namespace> -l app.kubernetes.io/name=tetragon
```

2. Execute tetra bugtool within the Pod:

```bash
kubectl exec -n <tetragon-namespace> <tetragon-pod-name> -c tetragon -- tetra bugtool
```

3. Retrieve the created archive from the Pod's filesystem:

```bash
kubectl cp -c tetragon <tetragon-namespace>/<tetragon-pod-name>:tetragon-bugtool.tar.gz tetragon-bugtool.tar.gz
```

### Container installation

1. Enter the Tetragon Container:

```bash
docker exec -it <tetragon-container-id> tetra bugtool
```

2. Retrieve the archive using docker cp:

```bash
docker cp <tetragon-container-id>:/tetragon-bugtool.tar.gz tetragon-bugtool.tar.gz
```

### Systemd host installation

1. Execute tetra bugtool with Elevated Permissions:

```bash
sudo tetra bugtool
```

## Enable debug log level

When debugging, it might be useful to change the log level. The default log
level is controlled by the log-level option at startup:

* Enable debug level with `--log-level=debug`
* Enable trace level with `--log-level=trace`

### Change log level on Kubernetes

{{< warning >}}
The Pods of the Tetragon DaemonSet will be restarted automatically after
changing the debug Helm value.
{{< /warning >}}

It is possible to change the log level of Tetragon's DaemonSet Pods by setting
`tetragon.debug` to `true`.

Loading