eBPF for Security Observability
The kernel sees everything. Now you can, too.
TL;DR: eBPF (extended Berkeley Packet Filter) lets you run sandboxed programs inside the Linux kernel in response to events — syscalls, network packets, kprobes, tracepoints. For security observability, it replaces agent-based monitoring with a single, low-overhead pipeline that captures process execution, file access, network connections, and DNS queries at kernel level. This post covers the primitives, a working pipeline, and the trade-offs that show up when eBPF goes live in production.
What eBPF Is (and Isn’t)
eBPF started as a packet filter — hence “Berkeley Packet Filter.” Linux used it in tcpdump and iptables for decades. In kernel 4.8 (2016), the JIT compiler landed. In 5.8 (2020), BPF CO-RE (Compile Once, Run Everywhere) made it possible to write one eBPF program and run it across kernel versions without relinking.
eBPF programs:
- Run sandboxed in the kernel. A bad program can’t crash the kernel (mostly).
- Can read and write to map structures (BPF maps) — the only way eBPF programs exchange data with user space.
- Execute within a 1 million-instruction budget per invocation (to prevent infinite loops).
- Attach to hooks: tracepoints, kprobes, uprobes, LSM hooks, XDP (eXpress Data Path), TC (traffic control), and syscalls.
What eBPF is not:
- Not a replacement for an IDS. It’s a data collection primitive. You feed that data somewhere (e.g., OpenSearch, Elasticsearch, Loki).
- Not zero-overhead. A busy system with aggressive eBPF probes can add 5-15% CPU overhead. You measure before you deploy.
- Not kernel-version independent. You need BPF CO-RE + libbpf to handle version drift, but you still need kernel 4.14+ for most features.
The Primitives
BPF Maps
BPF maps are the bridge between kernel and user space. Every eBPF security program uses at least one map:
// struct used to send process info to user space
struct event {
u32 pid;
char comm[TASK_COMM_LEN];
u32 uid;
char filename[MAXFILENAME];
int retval;
};
// Ring buffer: the fastest map type for event streaming
struct {
__uint(type, BPF_MAP_TYPE_RINGBUF);
__uint(max_entries, 256 * 1024);
} events SEC(".maps");
Ring buffers are the modern choice for event streaming. Alternatives:
BPF_MAP_TYPE_PERF_EVENT_ARRAY: Older, slightly more complex, works on older kernels.BPF_MAP_TYPE_HASH: Key-value lookups for stateful tracking (e.g., active connections).BPF_MAP_TYPE_ARRAY: Fixed-size indexed storage for counters.
Attach Points for Security
| Hook Type | What It Catches | Security Use Case |
|---|---|---|
tracepoint | Kernel tracepoint events (sched, syscalls, network) | Process creation, file access, DNS queries |
kprobe | Any kernel function | Deep function-level tracing |
syscall | sys_enter/sys_exit for any syscall | File open, network connect, execve |
LSM | Linux Security Module hooks | SELinux/AppArmor audit, custom policy |
XDP | Packets at driver level | DDoS mitigation, network-level filtering |
TC | Traffic control hooks | Ingress/egress traffic analysis |
For security observability, the most useful are:
tracepoint:syscalls:sys_enter_*andsys_exit_*— every process syscalltracepoint:raw_syscalls:sys_exit— unified exit handlertracepoint:syscall:sys_enter_execve— process executiontracepoint:connect:connect_entry— outbound network connectionstracepoint:resolve:resolve_entry— DNS lookups
A Working Pipeline
Let’s build a process-execution and file-access monitor. This is the most common starting point: detect what’s running and what it touches.
Step 1: The eBPF Program
// security_monitor.c (simplified)
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#define TASK_COMM_LEN 16
#define MAXFILENAME 256
struct event {
__u32 pid;
__u32 uid;
char comm[TASK_COMM_LEN];
char filename[MAXFILENAME];
__s64 retval;
__u64 timestamp;
};
struct {
__uint(type, BPF_MAP_TYPE_RINGBUF);
__uint(max_entries, 256 * 1024);
} events SEC(".maps");
// Ring buffer sample: how much data to reserve
#define SAMPLE_LEN sizeof(struct event)
SEC("tp/sched/sched_process_exec")
int handle_exec(struct trace_event_raw_sched_process_exec *ctx) {
struct event *e;
struct task_struct *task;
char filename[MAXFILENAME] = {};
e = bpf_ringbuf_reserve(&events, SAMPLE_LEN, 0);
if (!e) return 0;
task = bpf_get_current_task();
e->pid = task->tgid;
e->uid = task->cred->uid.val;
bpf_get_current_comm(&e->comm, sizeof(e->comm));
bpf_strncpy(e->filename, MAXFILENAME,
(char *)ctx->old_filename, MAXFILENAME - 1);
e->retval = 0;
e->timestamp = bpf_ktime_get_ns();
bpf_ringbuf_submit(e, 0);
return 0;
}
char LICENSE[] SEC("license") = "GPL";
Step 2: User-Space Runner (Go)
package main
import (
"context"
"encoding/json"
"log"
"os"
"os/signal"
"syscall"
"github.com/aquasecurity/libbpfgo"
"github.com/google/uuid"
)
type Event struct {
PID uint32 `json:"pid"`
UID uint32 `json:"uid"`
Comm string `json:"comm"`
Filename string `json:"filename"`
Retval int64 `json:"retval"`
Timestamp uint64 `json:"timestamp"`
}
func main() {
libbpfgo.Initialize()
defer libbpfgo.BpfModuleCleanup()
mod, err := libbpfgo.NewModuleFromFile("security_monitor.c", []string{})
if err != nil {
log.Fatalf("Failed to load BPF module: %v", err)
}
if err := mod.BpfCompile(); err != nil {
log.Fatalf("Failed to compile BPF program: %v", err)
}
// Attach to tracepoint
tp, err := mod.TracepointOpen("sched", "sched_process_exec")
if err != nil {
log.Fatalf("Failed to open tracepoint: %v", err)
}
defer tp.Close()
tp.Attach()
log.Println("eBPF security monitor running...")
// Ring buffer setup
ringBuf, err := mod.InitRingBuf("events", nil)
if err != nil {
log.Fatalf("Failed to init ring buffer: %v", err)
}
ringBuf.Poll(100) // 100ms timeout
stop := make(chan os.Signal, 1)
signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM)
for {
select {
case <-stop:
log.Println("Shutting down...")
return
default:
ringBuf.Poll(100)
}
}
}
Step 3: What You Get
{
"pid": 12847,
"uid": 0,
"comm": "bash",
"filename": "/usr/bin/curl",
"retval": 0,
"timestamp": 1731489600000000000
}
Every execve is captured with PID, UID, command name, and full path. No agent needed — the kernel does the work.
Expanding the Pipeline
The single-exec monitor is a starting point. Real security observability needs more.
File Access Tracking
Attach to tracepoint:syscalls:sys_enter_openat and sys_exit_openat to capture every file open with its flags (read, write, create, truncate). Combine with the exec monitor:
10:23:15 PID 12847 (bash) EXEC /usr/bin/python3
10:23:16 PID 12848 (python3) READ /etc/shadow (O_RDONLY)
10:23:16 PID 12848 (python3) READ /etc/passwd (O_RDONLY)
10:23:17 PID 12848 (python3) EXEC /usr/bin/id
This is what a file access timeline looks like — and it’s cheaper than auditd because you pick exactly which syscalls to trace.
Network Connection Tracking
Attach to tracepoint:connect:connect_entry (for TCP/UDP) and tracepoint:syscalls:sys_enter_connect. You get source IP, dest IP, dest port, protocol, and the connecting PID.
10:23:17 PID 12848 (python3) CONNECT 10.0.0.5:443 TCP
10:23:18 PID 12849 (curl) CONNECT 203.0.113.42:80 TCP
Filter on the user-space side: “Which processes are connecting to external IPs?” “Which internal IPs are connecting to unusual ports?” “Are there connections from non-web-processes to port 443?”
DNS Query Tracking
Attach to tracepoint:resolve:resolve_entry for DNS lookups. This is one of the most powerful primitives for detecting C2 traffic and data exfiltration.
10:23:17 PID 12848 (python3) DNS update.malware.example.com
10:23:18 PID 12849 (curl) DNS 203.0.113.42 (PTR)
Short DNS names from unusual domains = suspicious. Fast-flux domains = suspicious. DNS queries from unexpected PIDs = suspicious. All captured at kernel level, no agents.
The Trade-offs
Performance
eBPF is efficient but not free. The 1M-instruction limit means complex programs need careful profiling. A rule of thumb:
- Light monitors (exec, connect, DNS): < 5% CPU overhead on busy servers.
- Aggressive monitors (every syscall, kprobes on hot paths): 5-15% overhead.
- Heavy monitors (string parsing in-kernel, complex map operations): 15-30% overhead.
Profile with bpfstat or bpftool prog show before and after deployment.
Kernel Version Fragmentation
BPF CO-RE and libbpf solve most version drift, but:
- Kernel 4.14+ needed for BPF maps
- Kernel 4.15+ needed for BTF
- Kernel 5.5+ needed for BPF CO-RE
- Kernel 5.8+ recommended for ring buffers
If your fleet runs kernel 4.18 on some nodes and 6.1 on others, CO-RE handles it — but test on each version.
Complexity
eBPF programs run in the kernel. A bug can cause:
- Incorrect data (most common)
- Slowdowns (instruction budget exceeded)
- Panics (rare, but possible — e.g., use-after-free in maps)
Start with tracepoints (safe, no kprobe overhead), then add kprobes for specific functions.
Tooling
The eBPF ecosystem has matured:
- eBPF programs:
bcc,libbpf,Cilium/ebpf(Go),libbpfgo - Monitoring:
Cilium Hubble(network),Falco(syscall),Aya(Rust) - Visualization:
Cilium Hubble UI, custom dashboards - Deployment:
Cilium(full mesh),Falco(runtime security),Tetragon(Cilium’s security-focused eBPF platform)
Pick based on what you need: Cilium for network + security, Falco for syscall monitoring, Tetragon for Cilium ecosystem integration.
A Practical Deployment
Phase 1: Observe Only (1-2 weeks)
Deploy eBPF programs in “observe” mode. Collect data, don’t act on it. Build baselines:
- What processes run on this server normally?
- What files does each service touch?
- What are the outbound connection patterns?
- What DNS queries are normal?
Use ring buffers to stream events to a user-space collector (Fluent Bit, Vector, or a custom Go/Python process). Don’t try to do everything in-kernel.
Phase 2: Alert on Anomalies
Add user-space rules:
- Process not in the baseline whitelist → alert
- Unusual outbound port → alert
- DNS query to newly seen domain → alert
- High file-access rate from single PID → alert
Keep alerts quiet for the first week. Tune thresholds. False positives kill adoption faster than missed detections.
Phase 3: Act
Add enforcement:
- Kill process if it matches a known-bad pattern
- Block outbound connection from a suspicious PID
- Rate-limit DNS queries from a single PID
Start with “log and alert” mode, then “block on known pattern” mode, then “block on anomaly score” mode. Each step increases risk.
What eBPF Replaces (and What It Doesn’t)
| Tool | What It Does | eBPF Replaces? |
|---|---|---|
auditd | Syscall auditing via netlink | Yes — same data, less overhead |
strace | Process-level syscall tracing | Yes — kernel-level, no fork/exec overhead |
tcpdump | Packet capture | Partial — eBPF/XDP is faster for structured filtering |
ps//proc | Process listing | Yes — eBPF captures process lifecycle in real-time |
osquery | SQL-based system inventory | Partial — eBPF captures events, not just snapshots |
IDS/IPS (Suricata, Zeek) | Network threat detection | No — eBPF feeds data to them |
SIEM | Correlation and storage | No — eBPF feeds data to them |
eBPF is a data collection layer, not a threat detection engine. You still need a backend that correlates, scores, and alerts.
Conclusion
eBPF is the best primitive for Linux security observability today. It gives you kernel-level visibility with minimal overhead, and the ecosystem (Cilium, Falco, Tetragon) has matured enough for production use.
The trade-offs are real: kernel version drift, instruction budget limits, and the complexity of going from “observe” to “act” without killing performance. But for a security team that needs to see what’s running on every Linux server — process, file, network, DNS — eBPF is the answer.
Measure before you deploy. Start with tracepoints. Collect before you act. And always keep the user-space correlation engine separate from the kernel-side collection.
Further Reading
- Linux eBPF Documentation
- Cilium Hubble — Network observability with eBPF
- Cilium Tetragon — Runtime security with eBPF
- Falco — Syscall-based runtime security
- BCC Tools — Classic eBPF toolkit
- libbpfgo — Go bindings for libbpf
- BPF CO-RE: Compile Once — Run Everywhere — Networklore blog
- eBPF Performance — Networklore blog