[tcp_stats] Use kfuncs to probe the tcp_sendmsg function #2072
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: kpattaswamy <[email protected]>
Signed-off-by: kpattaswamy <[email protected]>
Signed-off-by: kpattaswamy <[email protected]>
Signed-off-by: kpattaswamy <[email protected]>
Signed-off-by: kpattaswamy <[email protected]>
Signed-off-by: kpattaswamy <[email protected]>
Signed-off-by: kpattaswamy <[email protected]>
kpattaswamy
force-pushed
the
tcp-stats-probes
branch
from
10ac4a6 to
e478ecf
Compare
Signed-off-by: kpattaswamy <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: Adds kfunc support to BCC wrapper and utilizes it to probe the
tcp_sendmsgfunction from the tcp stats connector.We had seen a kprobe attachment conflict when the socket tracer and tcp stats connectors were enabled in Stirling due to both connectors trying to deploy kprobes for
tcp_sendmsg. This PR modifies the tcp stats connector to instead deploy a kfunc probe to avoid the conflict.Relevant Issues: #2055
Type of change: /kind bug
Test Plan: Skaffolded these changes on a dev cluster with socket tracer and tcp stats enabled and verified the existing tcp stats BPF test passes on kernel version
6.8.0-1018-gcpRelevant parts of the pem logs from skaffolding:
I20250106 22:11:24.698770 18078 source_connector.cc:35] Initializing source connector: socket_tracer I20250106 22:11:24.698838 18078 socket_trace_connector.cc:475] Kernel version greater than V5.1 detected (6.1.112), raised loop limit to 882 and chunk limit to 84 I20250106 22:11:24.698889 18078 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.1.112+ I20250106 22:11:24.698920 18078 linux_headers.cc:353] Detected kernel release (uname -r): 6.1.112+ I20250106 22:11:24.699201 18078 linux_headers.cc:369] Not Found : Could not find 'source' or 'build' under /lib/modules/6.1.112+. I20250106 22:11:24.699244 18078 linux_headers.cc:215] Looking for host Linux headers at /host/lib/modules/6.1.112+/build. I20250106 22:11:24.699276 18078 linux_headers.cc:372] Not Found : Did not find the host headers at path: /host/lib/modules/6.1.112+/build, No such file or directory. I20250106 22:11:24.699309 18078 linux_headers.cc:314] Attempting to install packaged headers. W20250106 22:11:24.699962 18078 linux_headers.cc:287] Ignoring /px/linux-headers-x86_64-4.19.271.tar.gz since it does not conform to the naming convention W20250106 22:11:24.700018 18078 linux_headers.cc:287] Ignoring /px/linux-headers-x86_64-4.14.304.tar.gz since it does not conform to the naming convention I20250106 22:11:24.700129 18078 linux_headers.cc:320] Using packaged header: /px/linux-headers-x86_64-6.1.8.tar.gz I20250106 22:11:25.539538 18078 linux_headers.cc:57] Overriding linux version code to 393584 I20250106 22:11:25.540490 18078 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.1.112+ I20250106 22:11:25.540580 18078 linux_headers.cc:99] Found kernel config at: /proc/config.gz. I20250106 22:11:25.558293 18078 linux_headers.cc:345] Successfully installed packaged copy of headers at /lib/modules/6.1.112+/build I20250106 22:11:25.558351 18078 bcc_wrapper.cc:94] Resolving task_struct offsets. I20250106 22:11:25.694319 18094 bcc_wrapper.cc:166] Initializing BPF program ... I20250106 22:11:26.983464 18094 scoped_timer.h:48] Timer(init_bpf_program) : 1.29 s I20250106 22:11:27.003376 18078 bcc_wrapper.cc:166] Initializing BPF program ... I20250106 22:11:28.365095 18078 scoped_timer.h:48] Timer(init_bpf_program) : 1.36 s I20250106 22:11:28.383139 18078 bcc_wrapper.cc:97] Successfully resolved task_struct offsets: {real_start_time=1832, group_leader=1584, exit_code=1416} I20250106 22:11:28.383253 18078 bcc_wrapper.cc:166] Initializing BPF program ... I20250106 22:12:26.397286 18078 scoped_timer.h:48] Timer(init_bpf_program) : 58.01 s I20250106 22:12:29.713171 18078 socket_trace_connector.cc:500] Number of kprobes deployed = 46 I20250106 22:12:29.713227 18078 socket_trace_connector.cc:501] Probes successfully deployed. I20250106 22:12:29.713286 18078 socket_trace_connector.cc:415] Initializing perf buffers with ncpus=4 and scaling_factor=0.75 I20250106 22:12:29.713353 18078 socket_trace_connector.cc:404] Total perf buffer usage for kData buffers across all cpus: 266587120 I20250106 22:12:29.713382 18078 socket_trace_connector.cc:404] Total perf buffer usage for kControl buffers across all cpus: 6606028 I20250106 22:12:29.821144 18078 socket_trace_connector.cc:505] Number of perf buffers opened = 8 I20250106 22:12:30.109012 18078 stirling.cc:438] Adding info class: [socket_tracer/conn_stats] I20250106 22:12:30.109094 18078 stirling.cc:438] Adding info class: [socket_tracer/http_events] I20250106 22:12:30.109122 18078 stirling.cc:438] Adding info class: [socket_tracer/mysql_events] I20250106 22:12:30.109141 18078 stirling.cc:438] Adding info class: [socket_tracer/cql_events] I20250106 22:12:30.109160 18078 stirling.cc:438] Adding info class: [socket_tracer/pgsql_events] I20250106 22:12:30.109181 18078 stirling.cc:438] Adding info class: [socket_tracer/dns_events] I20250106 22:12:30.109203 18078 stirling.cc:438] Adding info class: [socket_tracer/redis_events] I20250106 22:12:30.109225 18078 stirling.cc:438] Adding info class: [socket_tracer/nats_events.beta] I20250106 22:12:30.109246 18078 stirling.cc:438] Adding info class: [socket_tracer/kafka_events.beta] I20250106 22:12:30.109282 18078 stirling.cc:438] Adding info class: [socket_tracer/mux_events] I20250106 22:12:30.109303 18078 stirling.cc:438] Adding info class: [socket_tracer/amqp_events] I20250106 22:12:30.109321 18078 stirling.cc:438] Adding info class: [socket_tracer/mongodb_events] I20250106 22:12:30.109455 18078 source_connector.cc:35] Initializing source connector: tcp_stats I20250106 22:12:30.109503 18078 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.1.112+ I20250106 22:12:30.109681 18078 linux_headers.cc:353] Detected kernel release (uname -r): 6.1.112+ I20250106 22:12:30.109760 18078 linux_headers.cc:207] Using Linux headers from: /lib/modules/6.1.112+/build. I20250106 22:12:30.109797 18078 bcc_wrapper.cc:90] Returning the previously resolved TaskStructOffsets object I20250106 22:12:30.109912 18078 bcc_wrapper.cc:166] Initializing BPF program ... I20250106 22:12:34.116115 18078 scoped_timer.h:48] Timer(init_bpf_program) : 4.01 s I20250106 22:12:34.294086 18078 tcp_stats_connector.cc:92] Successfully deployed 2 kprobes. I20250106 22:12:34.294157 18078 tcp_stats_connector.cc:93] Successfully deployed 2 kfunc probes. I20250106 22:12:34.294176 18078 stirling.cc:438] Adding info class: [tcp_stats/tcp_stats_events]