logo

Linux - Tracepoints vs Kprobes vs fentry

In the Linux kernel ecosystem, Tracepoints and Kprobes are the two primary mechanisms used to hook into kernel events for debugging, performance analysis, and monitoring (via tools like perf, ftrace, and eBPF).

Tracepoints (Static Instrumentation)

Tracepoints are "markers" explicitly placed in the kernel source code by developers. They are pre-defined locations (e.g., sched_switch, net_dev_xmit) that expose specific variables to user-space.

  • Nature: Static. You can only use tracepoints where a kernel developer has already put one.
  • Stability: High. Tracepoints are considered a stable API. Even if the underlying kernel functions are refactored, the tracepoint name and the data it provides usually remain the same.
  • Overhead: Very Low. When disabled, a tracepoint is essentially a NOP (no-operation) instruction. When enabled, they are highly optimized.
  • Data Access: They provide a structured, clean format for data. You don't need to know the internal memory layout of the kernel; the tracepoint gives you exactly what you need (e.g., PID, timestamps, filenames).

Kprobes (Dynamic Instrumentation)

Kprobes (Kernel Probes) allow you to dynamically break into any kernel routine. It works by replacing the instruction at a specific address with a breakpoint.

  • Nature: Dynamic. You can probe almost any function or even a specific line of code/instruction address while the kernel is running.
  • Stability: Low. Kprobes depend on the kernel’s internal symbols and function signatures. If the kernel is updated and a function is renamed, removed, or its arguments change, your probe will break.
  • Overhead: Higher. Because Kprobes use a breakpoint trap (exception) mechanism, the CPU has to save registers, switch contexts, execute the probe, and then single-step the original instruction. (Note: Kretprobes—for return values—have even more overhead).
  • Data Access: Raw. You have access to the CPU registers and memory. To get useful data, you often have to manually cast memory addresses to kernel structures, which requires knowledge of the kernel source code.

When to Use Which?

Use Tracepoints when:

  • You are building a tool meant to work across different kernel versions (e.g., a monitoring agent).
  • The event you want to track is a common one (scheduler events, syscalls, network interrupts, disk I/O).
  • Performance is critical, and you cannot afford the overhead of context switching.

Use Kprobes when:

  • The kernel developer did not provide a tracepoint for the specific logic you are investigating.
  • You are debugging a "Heisenbug" or a specific driver issue where you need to inspect internal function variables.
  • You are doing deep security research or root-cause analysis on a specific kernel build.

The Modern Context: BPF (eBPF)

In modern Linux (Kernel 4.x+), both Tracepoints and Kprobes serve as trigger sources for BPF programs.

  • SEC("tracepoint/syscalls/sys_enter_openat"): Your BPF program runs whenever that specific stable hook is hit.
  • SEC("kprobe/do_sys_open"): Your BPF program runs whenever the internal do_sys_open function is called.

The "Golden Rule" of Linux Tracing: Always check for a Tracepoint first. If it exists, use it. If it doesn't exist and you absolutely need the data, use a Kprobe.

fentry/fexit: BPF Trampoline programs

In the context of BPF, fentry (Function Entry) is technically neither a Tracepoint nor a traditional Kprobe. It is a newer, high-performance alternative to Kprobes that was introduced in Linux Kernel 5.5.

Is it a Kprobe? (No, but it acts like one)

Like a Kprobe, fentry allows you to hook into almost any kernel function.

  • The Difference: Traditional Kprobes use a "breakpoint" (an int3 instruction) which triggers a CPU exception. This is slow because the CPU has to stop, save everything, jump to a handler, and then jump back.
  • How fentry works: It uses the same mechanism as ftrace. When the kernel is compiled, a small "nop" (no-operation) space is left at the start of every function. fentry replaces that "nop" with a direct jump to a BPF Trampoline. There is no "break" or "exception," making it significantly faster than a Kprobe.

Why use fentry instead of a Kprobe?

If you are using a modern kernel (5.5+), you should almost always prefer fentry over kprobe for two main reasons:

A. Performance

Because it uses a direct trampoline instead of a breakpoint trap, the overhead of calling the BPF program is much lower.

B. BTF (BPF Type Format) — The "Superpower"

This is the biggest advantage.

  • With Kprobes: You have to look at the kernel source code to see that the second argument of a function is a struct file *, then you have to use PT_REGS_PARM2 to get the register, and then use bpf_probe_read to get data. It's tedious and error-prone.
  • With fentry: It is BTF-aware. It knows the function signature. You can access arguments just like you are writing C code: 
    // bpftrace example using fentry
fentry:vmlinux:do_sys_open {
    printf("Opening file: %s\n", str(args->filename));
}

Comparison Table

Feature Kprobe Tracepoint fentry (BPF)
Placement Anywhere Static locations Start of functions
Overhead High (Breakpoint) Very Low Very Low (Trampoline)
Arguments Raw Registers Structured Automatic (via BTF)
Kernel Req. Very old (2.6+) Old Modern (5.5+)
logo
PRIVACY POLICYABOUT中文