diff --git a/docs/content/en/docs/concepts/tracing-policy/_index.md b/docs/content/en/docs/concepts/tracing-policy/_index.md index 91f5b7ed18f..dc8ff0e55d3 100644 --- a/docs/content/en/docs/concepts/tracing-policy/_index.md +++ b/docs/content/en/docs/concepts/tracing-policy/_index.md @@ -30,6 +30,10 @@ Tracing Policies can be loaded and unloaded at runtime in Tetragon, or on startup using flags. - With Kubernetes, you can use `kubectl` to add and remove a `TracingPolicy`. - You can use `tetra` gRPC CLI to add and remove a `TracingPolicy`. -- You can use the `--tracing-policy` and `--tracing-policy-dir` flags, see more - in the [daemon configuration page]({{< ref "/docs/reference/tetragon-configuration#configure-tracing-policies-location" >}}). +- You can use the `--tracing-policy` and `--tracing-policy-dir` flags to statically add policies at + startup time, see more in the [daemon configuration page]({{< ref + "/docs/reference/tetragon-configuration#configure-tracing-policies-location" >}}). + +Hence, even though Tracing Policies are structured as a Kubernetes CR, they can also be used in +non-Kubernetes environments using the last two loading methods. diff --git a/docs/content/en/docs/concepts/tracing-policy/example.md b/docs/content/en/docs/concepts/tracing-policy/example.md index 07c6e1a2eef..9bc9a63996f 100644 --- a/docs/content/en/docs/concepts/tracing-policy/example.md +++ b/docs/content/en/docs/concepts/tracing-policy/example.md @@ -39,6 +39,10 @@ spec: - action: Sigkill ``` +The policy checks for file descriptors being created, and sends a `SIGKILL` signal to any process that +creates a file descriptor to a file named `/tmp/tetragon`. We discuss the policy in more detail +next. + ## Required fields ```yaml @@ -68,11 +72,11 @@ spec: type: "file" ``` -The beginning of the specification describe the hook point to use. Here we are +The beginning of the specification describes the hook point to use. Here we are using a kprobe, hooking on the kernel function `fd_install`. That's the kernel -function that gets called when a new file descriptor needs to be created. We +function that gets called when a new file descriptor is created. We indicate that it's not a syscall, but a regular kernel function. We then -specify the argument of the specified function symbol to be able to extract +specify the function arguments, so that Tetragon's BPF code will extract and optionally perform filtering on them. See the [hook points page]({{< ref "/docs/concepts/tracing-policy/hooks" >}}) @@ -92,11 +96,11 @@ for further information on the various hook points available and arguments. ``` Selectors allow you to filter on the events to extract only a subset of the -events based on different properties and optionally take an enforcement action. +events based on different properties and optionally take an action. In the example, we filter on the argument at index 1, passing a `file` struct to the function. Tetragon has the knowledge on how to apply the `Equal` -operator over a Linux kernel `file` struct and you can basically match on the +operator over a Linux kernel `file` struct and match on the path of the file. Then we add the `Sigkill` action, meaning, that any match of the selector @@ -112,33 +116,40 @@ First, let's create the `/tmp/tetragon` file with some content: echo eBPF! > /tmp/tetragon ``` -Starting Tetragon with the above `TracingPolicy`, for example putting the -policy in the `example.yaml` file, compiling the project locally and starting -Tetragon with (you can do similar things with container image releases, see the -docker run command in the [Try Tetragon on Linux guide] +You can save the policy in an `example.yaml` file, compile Tetragon locally, and start Tetragon: ```shell-session sudo ./tetragon --bpf-lib bpf/objs --tracing-policy example.yaml ``` +(See [Quick Kubernetes Install]({{< ref "/docs/getting-started/install-k8s" >}}) and [Quick Local +Docker Install]({{< ref "/docs/getting-started/install-docker" >}}) for other ways to start +Tetragon.) + + {{< note >}} Stop tetragon with Ctrl+C to disable the policy and remove the BPF programs. {{< /note >}} +Once the Tetragon starts, you can monitor events using `tetra`, the tetragon CLI: +```shell-session +./tetra tetra getevents -o compact +``` + Reading the `/tmp/tetragon` file with `cat`: ```shell-session cat /tmp/tetragon ``` -Should result in the following events: +Results in the following events: ``` 🚀 process /usr/bin/cat /tmp/tetragon 📬 open /usr/bin/cat /tmp/tetragon 💥 exit /usr/bin/cat /tmp/tetragon SIGKILL ``` -And the shell will return: +And the shell where the `cat` command was performed will return: ``` Killed ``` diff --git a/docs/content/en/docs/concepts/tracing-policy/hooks.md b/docs/content/en/docs/concepts/tracing-policy/hooks.md index 24b14024ba8..139d8fba13e 100644 --- a/docs/content/en/docs/concepts/tracing-policy/hooks.md +++ b/docs/content/en/docs/concepts/tracing-policy/hooks.md @@ -5,18 +5,16 @@ weight: 2 description: "Hook points for Tracing Policies and arguments description" --- -The `spec` or specification of a `TracingPolicy` can be composed of `kprobes`, -`tracepoints` or `uprobes` at the moment. It describes the arbitrary event in -the kernel that Tetragon will trace. These hook points have arguments and -return values that can be specified using the `args` and `returnArg` fields -detailed in the following sections. +Tetragon can hook into the kernel using `kprobes` and `tracepoints`, as well as in user-space +programs using `uprobes`. Users can configure these hook points using the correspodning sections of +the `TracingPolicy` specification (`.spec`). These hook points include arguments and return values +that can be specified using the `args` and `returnArg` fields as detailed in the following sections. ## Kprobes -Kprobes enables you to dynamically break into any kernel routine and collect -debugging and performance information non-disruptively. kprobes are highly -tight to your kernel version and might not be portable since the kernel symbols -depend on your build. +Kprobes enables you to dynamically hook into any kernel function and execute BPF code. Because +kernel functions might change across versions, kprobes are highly tied to your kernel version and, +thus, might not be portable across different kernels. Conveniently, you can list all kernel symbols reading the `/proc/kallsyms` file. For example to search for the `write` syscall kernel function, you can @@ -33,7 +31,7 @@ ffffdeb15092a740 d _eil_addr___arm64_sys_write ``` You can see that the exact name of the symbol for the write syscall on our -kernel version is `__arm64_sys_write`. Note that on `x86_64`, the prefix should +kernel version is `__arm64_sys_write`. Note that on `x86_64`, the prefix would be `__x64_` instead of `__arm64_`. {{< caution >}} @@ -52,7 +50,7 @@ In our example, we will explore a `kprobe` hooking into the kernel function. The `fd_install` kernel function is called each time a file descriptor is installed into the file descriptor table of a process, typically referenced within system calls like `open` or `openat`. Hooking `fd_install` -has its benefits and limitations, which are out of the scope of this document. +has its benefits and limitations, which are out of the scope of this guide. ```yaml spec: @@ -69,7 +67,7 @@ function use a [different ABI](https://gitlab.com/x86-psABIs/x86-64-ABI). {{< /note >}} -As usual, kprobes calls can be defined independently in different policies, +Kprobes calls can be defined independently in different policies, or together in the same Policy. For example, we can define trace multiple kprobes under the same tracing policy: ```yaml @@ -83,198 +81,11 @@ spec: # [...] ``` -### Lists - -It's possible to define list of functions and use it in the kprobe's `call` field. - -Following example traces all `sys_dup[23]` syscalls. - - -```yaml -spec: - lists: - - name: "dups" - type: "syscalls" - values: - - "sys_dup" - - "sys_dup2" - - "sys_dup3" - kprobes: - - call: "list:dups" -``` - -It is basically a shortcut for following policy: - -```yaml -spec: - kprobes: - - call: "sys_dup" - syscall: true - - call: "sys_dup2" - syscall: true - - call: "sys_dup3" - syscall: true -``` - -As shown in subsequent examples, its main benefit is allowing a single definition for -calls that have the same filters. - -The list is defined under `lists` field with arbitrary values for `name` and `values` fields. - -```yaml -spec: - lists: - - name: "dups" - type: "syscalls" - values: - - "sys_dup" - - "sys_dup2" - - "sys_dup3" - ... -``` - -It's possible to define multiple lists. - -```yaml -spec: - lists: - - name: "dups" - type: "syscalls" - values: - - "sys_dup" - - "sys_dup2" - - "sys_dup3" - name: "another" - - "sys_open" - - "sys_close" -``` - -Specific list can be referenced in kprobe's `call` field with `"list:NAME"` value. - -```yaml -spec: - lists: - - name: "dups" - ... - - kprobes: - - call: "list:dups" -``` - -The kprobe definition creates a kprobe for each item in the list and shares the rest -of the config specified for kprobe. - -List can also specify `type` field that implies extra checks on the values (like for `syscall` type) -or denote that the list is generated automatically (see below). -User must specify `syscall` type for list with syscall functions. Also `syscall` functions -can't be mixed with regular functions in the list. - -The additional selector configuration is shared with all functions in the list. -In following example we create 3 kprobes that share the same pid filter. - - -```yaml -spec: - lists: - - name: "dups" - type: "syscalls" - values: - - "sys_dup" - - "sys_dup2" - - "sys_dup3" - kprobes: - - call: "list:dups" - selectors: - - matchPIDs: - - operator: In - followForks: true - isNamespacePID: false - values: - - 12345 -``` - -It's possible to use argument filter together with the `list`. - -It's important to understand that the argument will be retrieved by using the specified -argument type for all the functions in the list. - -Following example adds argument filter for first argument on all functions in dups list to -match value 9999. - -```yaml -spec: - lists: - - name: "dups" - type: "syscalls" - values: - - "sys_dup" - - "sys_dup2" - - "sys_dup3" - kprobes: - - call: "list:dups" - args: - - index: 0 - type: int - selectors: - - matchArgs: - - index: 0 - operator: "Equal" - values: - - 9999 -``` - -There are two additional special types of generated lists. - -The `generated_syscalls` type of list that generates list with all possible -syscalls on the system. - -Following example traces all syscalls for `/usr/bin/kill` binary. - -```yaml -spec: - lists: - - name: "all-syscalls" - type: "generated_syscalls" - kprobes: - - call: "list:all-syscalls" - selectors: - - matchBinaries: - - operator: "In" - values: - - "/usr/bin/kill" -``` - -The `generated_ftrace` type of list that generates functions from ftrace `available_filter_functions` -file with specified filter. The filter is specified with `pattern` field and expects regular expression. - -Following example traces all kernel `ksys_*` functions for `/usr/bin/kill` binary. - -```yaml -spec: - lists: - - name: "ksys" - type: "generated_ftrace" - pattern: "^ksys_*" - kprobes: - - call: "list:ksys" - selectors: - - matchBinaries: - - operator: "In" - values: - - "/usr/bin/kill" -``` - -## Uprobes - -{{% pageinfo %}} -This hook point method lacks documentation, see [issue #878](https://github.com/cilium/tetragon/issues/878). -{{% /pageinfo %}} - ## Tracepoints -A tracepoint placed in the Linux kernel code provides a hook to call a function -that you can provide at runtime using Tetragon. Tracepoints have the advantage -of being stable across kernel versions and thus more portable than kprobes. + +Tracepoints are statically defined in the kernel and have the advantage of being stable across +kernel versions and thus more portable than kprobes. To see the list of tracepoints available on your kernel, you can list them using `sudo ls /sys/kernel/debug/tracing/events`, the output should be similar @@ -346,17 +157,22 @@ spec: - index: 4 type: "int64" ``` +## Uprobes + +{{% pageinfo %}} +This hook point method lacks documentation, see [issue #878](https://github.com/cilium/tetragon/issues/878). +{{% /pageinfo %}} ## Arguments -Kprobes, uprobes and tracepoints all share a needed arguments fields called -`args`. It's a list of function arguments to include in the trace output. -Indeed the BPF code that runs on the hook point requires information about the -types of arguments of the function being traced to properly read, print and -filter on its arguments. Currently, this information needs to be provided by -the user under the `args` section. For the [available +Kprobes, uprobes and tracepoints all share a needed arguments fields called `args`. It is a list of +arguments to include in the trace output. Tetragon's BPF code requires +information about the types of arguments to properly read, print and +filter on its arguments. This information needs to be provided by the user under the +`args` section. For the [available types](https://github.com/cilium/tetragon/blob/main/pkg/k8s/apis/cilium.io/client/crds/v1alpha1/cilium.io_tracingpolicies.yaml#L64-L88), -see directly in the `TracingPolicy` CRD. +check the [`TracingPolicy` +CRD](https://github.com/cilium/tetragon/blob/main/pkg/k8s/apis/cilium.io/client/crds/v1alpha1/cilium.io_tracingpolicies.yaml). Following our example, here is the part that defines the arguments: ```yaml @@ -525,3 +341,183 @@ See [`TrackSock`](#tracksock-action) and [`UntrackSock`](#untracksock-action). Socket tracking is only available on kernels >=5.3. +## Lists + +It's possible to define list of functions and use it in the kprobe's `call` field. + +Following example traces all `sys_dup[23]` syscalls. + + +```yaml +spec: + lists: + - name: "dups" + type: "syscalls" + values: + - "sys_dup" + - "sys_dup2" + - "sys_dup3" + kprobes: + - call: "list:dups" +``` + +It is basically a shortcut for following policy: + +```yaml +spec: + kprobes: + - call: "sys_dup" + syscall: true + - call: "sys_dup2" + syscall: true + - call: "sys_dup3" + syscall: true +``` + +As shown in subsequent examples, its main benefit is allowing a single definition for +calls that have the same filters. + +The list is defined under `lists` field with arbitrary values for `name` and `values` fields. + +```yaml +spec: + lists: + - name: "dups" + type: "syscalls" + values: + - "sys_dup" + - "sys_dup2" + - "sys_dup3" + ... +``` + +It's possible to define multiple lists. + +```yaml +spec: + lists: + - name: "dups" + type: "syscalls" + values: + - "sys_dup" + - "sys_dup2" + - "sys_dup3" + name: "another" + - "sys_open" + - "sys_close" +``` + +Specific list can be referenced in kprobe's `call` field with `"list:NAME"` value. + +```yaml +spec: + lists: + - name: "dups" + ... + + kprobes: + - call: "list:dups" +``` + +The kprobe definition creates a kprobe for each item in the list and shares the rest +of the config specified for kprobe. + +List can also specify `type` field that implies extra checks on the values (like for `syscall` type) +or denote that the list is generated automatically (see below). +User must specify `syscall` type for list with syscall functions. Also `syscall` functions +can't be mixed with regular functions in the list. + +The additional selector configuration is shared with all functions in the list. +In following example we create 3 kprobes that share the same pid filter. + + +```yaml +spec: + lists: + - name: "dups" + type: "syscalls" + values: + - "sys_dup" + - "sys_dup2" + - "sys_dup3" + kprobes: + - call: "list:dups" + selectors: + - matchPIDs: + - operator: In + followForks: true + isNamespacePID: false + values: + - 12345 +``` + +It's possible to use argument filter together with the `list`. + +It's important to understand that the argument will be retrieved by using the specified +argument type for all the functions in the list. + +Following example adds argument filter for first argument on all functions in dups list to +match value 9999. + +```yaml +spec: + lists: + - name: "dups" + type: "syscalls" + values: + - "sys_dup" + - "sys_dup2" + - "sys_dup3" + kprobes: + - call: "list:dups" + args: + - index: 0 + type: int + selectors: + - matchArgs: + - index: 0 + operator: "Equal" + values: + - 9999 +``` + +There are two additional special types of generated lists. + +The `generated_syscalls` type of list that generates list with all possible +syscalls on the system. + +Following example traces all syscalls for `/usr/bin/kill` binary. + +```yaml +spec: + lists: + - name: "all-syscalls" + type: "generated_syscalls" + kprobes: + - call: "list:all-syscalls" + selectors: + - matchBinaries: + - operator: "In" + values: + - "/usr/bin/kill" +``` + +The `generated_ftrace` type of list that generates functions from ftrace `available_filter_functions` +file with specified filter. The filter is specified with `pattern` field and expects regular expression. + +Following example traces all kernel `ksys_*` functions for `/usr/bin/kill` binary. + +```yaml +spec: + lists: + - name: "ksys" + type: "generated_ftrace" + pattern: "^ksys_*" + kprobes: + - call: "list:ksys" + selectors: + - matchBinaries: + - operator: "In" + values: + - "/usr/bin/kill" +```