The Three Layers of Logging That Actually Detect Things

Most teams have logging. They just don’t have detection.

There’s a difference.

You collect logs when you want to know what happened after something broke. You design detection when you want to know what’s happening while it’s happening. The two goals need different schemas, different retention, and different query patterns. Mix them and you get all three of the worst outcomes: you drown in noise, you miss the signal, and when you need to investigate, you can’t find what you’re looking for.

I’ve seen this pattern repeat across organizations of every size. The teams that actually detect things separate their logging into three layers and treat each one differently.

Layer 1 — Audit Logs (What Changed)

Audit logs answer: who did what, when, and what was the result.

These are the logs you need for compliance, forensics, and proving to an auditor that someone changed the production database at 2 AM. They have a stable schema:

{
  "timestamp": "2026-07-24T02:14:33Z",
  "event": "user.login",
  "actor": "alice@company.com",
  "target": "db-prod-01",
  "action": "execute_query",
  "result": "success",
  "ip": "10.0.3.42",
  "user_agent": null,
  "details": {
    "rows_affected": 15420,
    "query_time_ms": 870
  }
}

The schema never changes. New event types get added, but the core fields are always timestamp, actor, event, action, result. Every security-relevant operation — login, logout, permission change, config update, data export — produces an audit event.

Detection design for audit logs:

Brute force: event=login result=failure grouped by actor + IP, threshold > 10 in 5 min
Privilege escalation: event=permission.change action=grant where target is admin
Data exfiltration: event=data.export rows_affected > 10000 in a single query
Off-hours access: timestamp.hour < 6 or timestamp.hour > 22 for non-automated actors

The key insight: audit logs are low-volume, high-signal. You should be able to query them in real time without blowing through your log budget. If you’re ingesting terabytes of audit logs daily, your granularity is wrong.

Layer 2 — Debug Logs (What’s Happening)

Debug logs answer: why did this happen.

These are the logs you generate during normal operation — request processing, cache hits/misses, queue depths, connection pool status. They’re high-volume and structured by trace context so you can follow a single request through the stack:

{
  "trace_id": "a3f7b2c1",
  "span_id": "d4e8f1a2",
  "timestamp": "2026-07-24T14:22:11Z",
  "level": "info",
  "service": "api-gateway",
  "message": "request_processed",
  "duration_ms": 234,
  "http": {
    "method": "POST",
    "path": "/api/v2/users/export",
    "status": 200,
    "content_length": 2450000
  }
}

Detection design for debug logs:

Latency spike: avg(duration_ms) by service > 2x baseline over 5 min window
Error rate increase: level=error count() by service > 50/min
Payload anomaly: content_length > 5MB on POST endpoints that usually serve < 100KB
Cascading failure: error count() rising across 3+ services in same trace window

The key insight: debug logs are high-volume, moderate-signal. You detect by aggregating over time windows and comparing against baselines. You don’t look at individual debug log entries to detect — you look at the patterns they form.

Layer 3 — Infrastructure Logs (What the Platform Sees)

Infrastructure logs answer: is the thing still alive.

These come from the OS, the container runtime, the network — things outside your application code. They include system metrics (CPU, memory, disk I/O), network flow logs, container restart counts, TLS handshake failures:

{
  "timestamp": "2026-07-24T14:22:11Z",
  "source": "host-monitor",
  "host": "api-prod-03",
  "metric": "cpu.usage.pct",
  "value": 94.2,
  "labels": {
    "service": "api-gateway",
    "namespace": "production"
  }
}

Detection design for infrastructure logs:

Resource exhaustion: cpu.usage > 95% sustained 10 min or disk.usage > 90%
Network anomaly: bytes_in spiking 5x baseline on a single external IP
Container churn: container.restart_count > 5 in 5 min on same pod
TLS failures: tls.handshake_failure count() > 100/min on a single endpoint

The key insight: infrastructure logs are continuous, always-on. They run whether your application is logging or not. When your app crashes and stops emitting debug and audit logs, infrastructure logs are the only thing that still tells you something happened.

How They Work Together

The real detection power comes from correlating across layers. A single layer misses things the others catch:

Scenario: A slow SQL injection

Audit layer: action=execute_query rows_affected > 5000 — one suspicious query
Debug layer: duration_ms spikes from 50ms to 4500ms on /api/users — the endpoint is slow
Infrastructure layer: disk.io.write spikes on the database host — the DB is dumping temp results

Individually, each is noise. Together, they form a detection: slow query with high row count and elevated disk I/O on the same host = likely SQLi enumeration.

Scenario: A misconfigured S3 bucket

Audit layer: action=put_object to s3://uploads from external IP
Debug layer: http.status=200 content_length=5MB on /upload endpoint
Infrastructure layer: bytes_out spikes from the web server — it’s proxying large objects

Scenario: A compromised service account

Audit layer: actor=svc-api-key-3 making login events at 3 AM from 203.0.113.42 (not in normal range)
Debug layer: duration_ms=2000 on export endpoint — the key is pulling data
Infrastructure layer: bytes_out from the host increases by 3x at the same time

What Most Teams Get Wrong

Mistake 1: One log format for everything

You can’t efficiently query audit trails, debug traces, and infrastructure metrics through the same index. Different fields, different volume, different retention needs. Separate them at ingestion.

Mistake 2: Debug logs that are actually debug

“Processing request with params: {"user": "alice", "password": "secret123"}” — fine for debugging, bad for production. Secrets in debug logs, excessive detail, unbounded JSON. Production debug logs should be bounded, structured, and free of sensitive values. Use sampling, not full capture.

Mistake 3: Detecting only on audit, ignoring infrastructure

You can’t detect a crashed process if your app is the one that’s crashed. Infrastructure monitoring is the safety net when everything else stops working. Don’t treat it as “ops only” — treat it as your always-on detection layer.

Mistake 4: Baselines from one week

Your “normal” latency yesterday might be yesterday’s spike today. Baselines should roll over — use a 7-day moving average, not a snapshot. Re-evaluate thresholds after significant changes (new deployment, traffic event, config update).

The Detection Matrix

Map each layer to what it’s best at:

Layer	Best at	Weak at
Audit	Who changed what, compliance, forensics	Timing, volume spikes, infra crashes
Debug	Why something happened, request tracing	Always-on visibility, infra issues
Infrastructure	Uptime, resource exhaustion, network	Business logic, auth events, data changes

If you can only detect things using one layer, you’re missing everything the other two see. The teams that actually find problems use all three.

A Practical Starting Point

If you’re building detection from scratch, don’t try to build everything at once. Start with three rules — one per layer — that cover your most likely attack path:

Audit: Brute force login detection (threshold-based, per-actor)
Debug: Latency spike on your most-used endpoint (rolling baseline)
Infrastructure: CPU > 90% for 10 minutes on any production host (simple threshold)

These three catch the most common attack patterns: credential stuffing, resource exhaustion, and service degradation. Get them right, get them alerting, then add the next three.

Conclusion

Logging is infrastructure. Detection is a product built on top of that infrastructure. The teams that confuse the two end up with expensive log aggregators and no idea what’s happening until an incident wakes them up at 2 AM.

Three layers. Different schemas. Different detection strategies. Correlate them. That’s what detection looks like.