“If you cannot measure it with low overhead, you cannot optimize it. eBPF is to the Linux kernel what JavaScript was to the web browser: it allows you to safely run custom, sandboxed code at event boundaries without modifying the kernel source or rebooting the machine.”
— Linus Torvalds
In high-throughput distributed systems, performance issues rarely manifest as simple CPU spikes. Instead, they appear as p99 tail latency outliers, lock contention in Ring 0, storage queue saturation, or scheduling delays.
Historically, debugging these issues required either:
- Adding printks and recompiling the kernel (slow, disruptive).
- Using heavy debuggers like
gdb(which halt execution and crush production throughput). - Using crude sampling tools that guess what occurred.
With the advent of the Extended Berkeley Packet Filter (eBPF) and Linux performance tracing infrastructure (perf, ftrace, bpftrace), Linux engineers can safely execute sandboxed, verified code directly in the kernel upon every packet arrival, disk request, context switch, or function invocation.
1. Classic Superuser Monitoring & The Load Average Paradox
In How Linux Works, Brian Ward explains that before deploying deep dynamic kernel tracers, superusers must understand what standard system health metrics actually report—beginning with the widely misunderstood System Load Average.
The Load Average Paradox:
When you run uptime or top, you see three numbers representing the 1, 5, and 15-minute load averages:
$ uptime
01:00:00 up 14 days, 3:24, 2 users, load average: 12.45, 8.20, 4.10
- The Common Myth: “Load average measures CPU saturation.”
- The Linux Reality: On traditional Unix, load average counted tasks in
TASK_RUNNING(R). But in 1993, Linux kernel developer Matthias Urlichs modified Linux to also include tasks inTASK_UNINTERRUPTIBLE(Dstate). - Why this matters: A process in
Dstate is typically waiting for disk I/O, NVMe request completion, or an NFS lock. - Result: A machine with 4 CPU cores reporting a load average of
12.0might have 0% CPU utilization and 12 threads frozen waiting on slow or wedged storage!
+-------------------------------------------------------------------------+
| LINUX LOAD AVERAGE CALCULATION |
| |
| Tasks in TASK_RUNNING (R) Tasks in TASK_UNINTERRUPTIBLE (D) |
| (Executing on CPU or runnable) + (Blocked on Disk I/O or Locks) |
+-------------------------------------------------------------------------+
The Standard Superuser Triage Toolkit:
vmstat 1:r: Runnable tasks.b: Tasks blocked in uninterruptible sleep (Dstate).us/sy/id/wa/st: User CPU %, System (kernel) CPU %, Idle %, I/O Wait % (waiting on disk), Steal % (time stolen by VM hypervisor).
iostat -xz 1:r/sandw/s: Read and write IOPS.await: Average time in milliseconds for I/O requests to be serviced.%util: Percentage of CPU time during which I/O requests were issued to the device.
top/htop:- Press
1to view per-CPU core breakdown. - Sort by
%CPUorRES(Resident Set Size).
- Press
2. Brendan Gregg’s Performance Methodologies
Before touching tracing tools, an engineer must apply structured methodology to avoid falling into the “random metric guessing” trap.
1. The USE Method (For Physical & Virtual Resources)
For every resource (CPUs, Memory buses, NVMe drives, Network interfaces):
- Utilization: The percentage of time the resource was actively servicing work (e.g., CPU % busy, disk %util).
- Saturation: The degree to which extra work was queued waiting for the resource (e.g., CPU runqueue length, disk queue depth).
- Errors: Count of hardware and driver error events (e.g., PCIe bus errors, network CRC drops).
2. The RED Method (For Request-Driven Services)
For every software endpoint or service:
- Rate: Requests processed per second.
- Errors: Number of failing requests.
- Duration: Latency distribution (p50, p90, p99, p99.9).
3. Linux Tracing Hook Points
Linux provides four distinct categories of probe points:
+--------------------------------------------------------------------------+
| USER SPACE: glibc, Nginx, PostgreSQL, Python, Go Runtime |
+--------------------------------------------------------------------------+
| |
v (uprobes: dynamic user function hook) v (USDT: static user probe)
+--------------------------------------------------------------------------+
| KERNEL BOUNDARY: System Call Interface (sys_enter / sys_exit) |
+--------------------------------------------------------------------------+
| |
v (kprobes / kretprobes: dynamic kernel) v (Tracepoints: static API)
+--------------------------------------------------------------------------+
| KERNEL INTERNALS: Scheduler, VFS, Network Stack, MMU Paging |
+--------------------------------------------------------------------------+
|
v (perf_events: Hardware PMU counters)
+--------------------------------------------------------------------------+
| HARDWARE SILICON: L1/L3 Cache Misses, Branch Mispredictions, CPU Cycles |
+--------------------------------------------------------------------------+
- Tracepoints (Static): Hard-coded, stable instrumentation points maintained by kernel developers (located under
/sys/kernel/tracing/events/). Guaranteed not to break across kernel releases. - kprobes & kretprobes (Dynamic Kernel): Dynamically injects a breakpoint instruction (e.g.
int3on x86-64) into the prologue or epilogue of any kernel function. Enables arbitrary kernel tracing with near-zero overhead when disabled. - uprobes & uretprobes (Dynamic User Space): Dynamically hooks functions inside user-space ELF binaries (e.g., tracing
SSL_read()inlibssl.soto inspect decrypted TLS payloads without holding private keys). - PMU Hardware Counters: Physical performance monitoring units inside modern Intel/AMD/ARM CPUs that count hardware events (instructions, cycles, cache misses) without software code modifications.
4. The eBPF Architecture
eBPF is a register-based virtual machine running directly inside the Linux kernel.
+--------------------------------------------------------------------------+
| 1. C Source Code (my_probe.bpf.c) |
+--------------------------------------------------------------------------+
|
v (clang -target bpf -O2)
+--------------------------------------------------------------------------+
| 2. ELF Bytecode (.o) |
+--------------------------------------------------------------------------+
|
v (bpf() system call with BPF_PROG_LOAD)
+==========================================================================+
| KERNEL SPACE |
| |
| +------------------------------------------------------------------+ |
| | 3. IN-KERNEL VERIFIER | |
| | - Simulates all code paths via Directed Acyclic Graph (DAG) | |
| | - Proves memory safety (no out-of-bounds pointers) | |
| | - Proves termination (limits loops, prevents kernel hangs) | |
| +------------------------------------------------------------------+ |
| | |
| v |
| +------------------------------------------------------------------+ |
| | 4. JIT (JUST-IN-TIME) COMPILER | |
| | Translates BPF bytecode into native x86-64 / ARM64 assembly | |
| +------------------------------------------------------------------+ |
| | |
| v |
| +------------------------------------------------------------------+ |
| | 5. ATTACH TO EVENT (Tracepoint, kprobe, XDP, TC, cgroup) | |
| +------------------------------------------------------------------+ |
| | |
| v |
| +------------------------------------------------------------------+ |
| | 6. BPF MAPS (Ring Buffer, Hash Maps, Per-CPU Maps) | |
| +------------------------------------------------------------------+ |
+==========================================================================+
|
v (bpf_ring_buffer__poll / mmap)
+--------------------------------------------------------------------------+
| USER SPACE OBSERVABILITY AGENT |
+--------------------------------------------------------------------------+
The In-Kernel Verifier
The verifier is the security barrier that prevents eBPF programs from crashing the kernel:
- Enforces that no register is read before being initialized.
- Validates that every memory dereference stays strictly within allocated boundaries.
- Rejects unbounded loops that could monopolize a CPU core.
BPF Maps
eBPF programs cannot call standard C library functions or allocate dynamic heap memory via malloc(). To share state between the kernel and user space, eBPF uses BPF Maps:
BPF_MAP_TYPE_HASH: Key-value lookup table.BPF_MAP_TYPE_RINGBUF: Lockless high-throughput ring buffer for streaming events to user space.BPF_MAP_TYPE_PERF_EVENT_ARRAY: Per-CPU event buffer.
5. Hands-On Practical Labs
Lab 9.1: Writing a Standalone eBPF Program with libbpf in C
In this lab, we write a real eBPF kernel program that hooks into the sys_enter_execve tracepoint to capture every new command executed on the operating system in real-time.
1. The Kernel eBPF Code: exec_trace.bpf.c
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
/* Structure defining the event emitted to user space */
struct event {
u32 pid;
u32 uid;
char comm[16];
char filename[128];
};
/* BPF Ring Buffer Map */
struct {
__uint(type, BPF_MAP_TYPE_RINGBUF);
__uint(max_entries, 256 * 1024); /* 256 KB ring buffer */
} events SEC(".maps");
/* Tracepoint: sys_enter_execve */
SEC("tracepoint/syscalls/sys_enter_execve")
int trace_sys_enter_execve(struct trace_event_raw_sys_enter *ctx) {
struct event *e;
u64 id = bpf_get_current_pid_tgid();
u32 pid = id >> 32;
/* Reserve space in the ring buffer */
e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
if (!e) {
return 0; /* Ring buffer is full, drop event */
}
e->pid = pid;
e->uid = bpf_get_current_uid_gid();
bpf_get_current_comm(&e->comm, sizeof(e->comm));
/* Read the binary path argument passed to execve (arg0) */
const char *filename_ptr = (const char *)ctx->args[0];
bpf_probe_read_user_str(&e->filename, sizeof(e->filename), filename_ptr);
/* Submit event to user space */
bpf_ringbuf_submit(e, 0);
return 0;
}
char LICENSE[] SEC("license") = "GPL";
2. Compiling the eBPF Program
clang -g -O2 -target bpf -D__TARGET_ARCH_x86 -I/usr/include/x86_64-linux-gnu -c exec_trace.bpf.c -o exec_trace.bpf.o
Lab 9.2: Production bpftrace One-Liners & Latency Histograms
bpftrace provides a high-level domain-specific language for rapid ad-hoc kernel investigations.
1. Measure VFS Read Latency as an ASCII Histogram
Measures the duration (in microseconds) of every vfs_read() invocation across the entire system:
sudo bpftrace -e '
kprobe:vfs_read {
@start[tid] = nsecs;
}
kretprobe:vfs_read /@start[tid]/ {
$duration_us = (nsecs - @start[tid]) / 1000;
@vfs_read_latency_us = hist($duration_us);
delete(@start[tid]);
}
'
Output:
@vfs_read_latency_us:
[1] 45 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[2, 4) 120 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8) 75 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 16) 18 |@@@@@@@ |
[16, 32) 4 |@ |
[32, 64) 1 | |
2. Trace Failed TCP Connections
Identifies which processes are experiencing network connection failures in real-time:
sudo bpftrace -e '
kretprobe:tcp_v4_connect {
$ret = retval;
if ($ret < 0) {
printf("PID %d (%s) failed tcp_connect: error %d\n", pid, comm, $ret);
}
}
'
3. Measure Scheduler Runqueue Latency (CPU Starvation)
Identifies how long tasks sit in the runqueue before receiving a CPU core:
sudo bpftrace -e '
tracepoint:sched:sched_wakeup {
@queued[args->pid] = nsecs;
}
tracepoint:sched:sched_switch {
if (@queued[args->next_pid]) {
$delay_us = (nsecs - @queued[args->next_pid]) / 1000;
@runqueue_latency_us = hist($delay_us);
delete(@queued[args->next_pid]);
}
}
'
Lab 9.3: CPU Profiling & Flame Graph Generation with Linux perf
A Flame Graph (invented by Brendan Gregg) visualizes call stacks where the x-axis represents the proportion of CPU time spent and the y-axis shows stack depth.
Step 1: Install linux-tools
sudo apt-get install -y linux-tools-common linux-tools-generic linux-tools-$(uname -r)
Step 2: Sample Call Stacks on a Busy System
Sample all CPU cores at 99 Hertz for 30 seconds with stack traces:
sudo perf record -F 99 -a -g -- sleep 30
Step 3: Process the Profile into a Flame Graph
Clone the canonical FlameGraph toolkit:
git clone https://github.com/brendangregg/FlameGraph /tmp/FlameGraph
# Convert perf data to folded stacks
sudo perf script | /tmp/FlameGraph/stackcollapse-perf.pl > /tmp/perf_folded.txt
# Render interactive SVG Flame Graph
/tmp/FlameGraph/flamegraph.pl /tmp/perf_folded.txt > /tmp/cpu_flamegraph.svg
Open /tmp/cpu_flamegraph.svg in any browser:
- Wide plateaus indicate functions consuming the most CPU cycles.
- Click on any frame to zoom in and pinpoint exact performance bottlenecks.
Lab 9.4: Diagnosing the Load Average D-State Disk I/O Bottleneck
In this lab, we demonstrate the How Linux Works Load Average Paradox: triggering an artificial disk I/O bottleneck to observe high load average while CPU utilization remains near zero.
# 1. In Terminal 1: Monitor CPU and blocked task states every second
vmstat 1
# 2. In Terminal 2: Generate heavy synchronous disk write load
# Writes 2GB of direct synchronous data to disk, forcing processes into TASK_UNINTERRUPTIBLE (D) state:
dd if=/dev/zero of=/tmp/io_pressure.bin bs=64k count=32768 oflag=sync &
DD_PID=$!
# 3. Observe the output of vmstat:
# - Look at the 'b' column (Blocked tasks in D state): it will spike!
# - Look at the 'wa' column (I/O wait %): it will dominate while 'us' (user CPU) is low.
# 4. In Terminal 3: Check uptime and process state:
uptime
ps aux | grep -E ' D |D\+'
# 5. Clean up
kill -9 $DD_PID 2>/dev/null
rm -f /tmp/io_pressure.bin
6. Summary Checklist
- Superuser Metrics Triage: Mastered
uptime,vmstat,iostat, and whyD-state disk tasks elevate load average. - Methodology First: Applied Brendan Gregg’s USE and RED methods to systems diagnosis.
- Instrumentation Map: Mastered Tracepoints, kprobes, uprobes, and PMU counters.
- eBPF Mechanics: Understood the in-kernel verifier, JIT compilation, and BPF maps.
- Compiled eBPF in C: Built an execution tracer using
libbpfand BPF ring buffers. -
bpftraceMastery: Generated real-time latency histograms for VFS reads and scheduling runqueues. - Generated Flame Graphs: Profiled call stacks and visualized CPU execution hotspots.