Roka: a context-pruning layer for log-aware AI agents

The problem: "context bloat" in log-aware agents

Coding and DevOps agents are increasingly asked to reason over raw logs — Kubernetes crash dumps, CI pipeline output, web-server access logs, mobile crash traces. The bottleneck isn't the model. It's the input.

Modern LLMs still have effective context windows measured in tens of thousands of tokens, but a single production log file can easily reach hundreds of thousands of lines. Most of that content is repetitive noise: heartbeats, health-checks, retry loops, monitoring counters. The one line that actually explains a failure — Bind to port 22 ... Address already in use — is buried 62% of the way through the file, deep in the "lost in the middle" zone where transformer attention degrades sharply.

The current developer workaround is manual: hand-written regexes, ad-hoc grep pipelines, prompt-engineered "please ignore X" instructions. This is fragile, doesn't generalize across systems, and burns tokens on noise before the model has even started thinking.

Our approach

Roka is an MCP-native middleware layer. It exposes tools like prune_file, prune_logs, and prune_tail that any agent can call before the raw log content ever enters the model's context.

The pipeline is deliberately boring — the goal is reliability, not clever ML. We chose transparent, deterministic stages because agents cannot debug what they cannot understand.

Three deterministic stages

• Collapse. Adjacent duplicate or near-duplicate lines are merged into a single line plus a count. Heartbeats and retries stop dominating the file.
• Rank. Lines matching ERROR|FATAL|CRITICAL|Exception|Traceback|panic are force-kept. Other lines are scored by rarity and by textual anomaly against the surrounding baseline.
• Fit. Ranked lines are packed into a caller-supplied character budget (default: 4,000 chars ≈ ~1,000 tokens), always preserving forced-keep lines first.

The whole pipeline runs in-memory in tens of milliseconds for a 50K-line file. No model calls, no external services, no fine-tuned classifier that needs retraining.

The benchmark

To measure this honestly, we ran the exact prune_file tool shipped in Roka's MCP server against LogHub — the standard academic corpus used in log-parsing research. 15 real system and application log sources, ranging from supercomputer job logs (BGL, Thunderbird) to mobile OS logs (Android, HealthApp) to cloud infrastructure logs (OpenStack, HDFS).

• Corpus: LogHub — 15 real production log sources, ~2,000 raw lines each (30,001 lines / 4.15M characters total).
• Tool: The same prune_file shipped in the public MCP server. No per-file tuning.
• Budget: A single 4,000-character output budget across every source (≈ 1,000 tokens).
• Rules: Collapse consecutive duplicate lines; force-keep lines matching the error regex above.
• Verification: Every source containing a real error was hand-checked to confirm at least one verbatim error line survived into the final 4,000-character output.

The results

Across all 15 LogHub sources, Roka reduced input size dramatically while keeping every real error we tested for intact. The chart below shows the percentage of lines removed after dedup + budget-fit, sorted from lowest to highest compression.

Sources with heavy templated output (OpenStack, Windows Update, HealthApp, Android) compress the hardest — often over 98% of lines are collapsible duplicates. Sources with high per-line entropy (BGL, HPC, Apache) still hit 70%+ compression because the ranker aggressively drops "expected" chatter.

Example: Thunderbird supercomputer log

To show what this looks like in practice, we picked one file: Testing/Thunderbird_2k.log — 2,000 lines, 323,193 characters. The real failure — sshd: Bind to port 22 ... Address already in use — sits 62% into the file, squarely in the "lost in the middle" zone. We gave two strategies the same 4,000-character budget.

Same budget. Same token cost. Different content. The naive tail spends its entire 4,000-character window on ntpd: synchronized and gmetad: data_thread() monitoring chatter. Roka spends most of its budget on the same chatter — collapsed into counted summaries — and reclaims exactly enough room for the two sshd error lines that explain the failure.

What an agent sees

When wired into an MCP-compatible agent (Claude Desktop, Cursor, etc.), the flow looks like this:

# Agent sees a 323 KB log and asks Roka to compress it
tool_call: prune_file(
    path="/var/log/thunderbird.log",
    budget=4000,
    force_keep=r"ERROR|FATAL|CRITICAL|Exception|Traceback|panic"
)

# Roka returns 28 lines · 4,000 chars · error lines preserved verbatim
tool_result:
    kept_lines: 28
    dropped_lines: 1972
    forced_lines: 2   # the two sshd errors
    compression_ratio: 80.8x

The agent then reasons over 28 clean lines instead of 2,000 noisy ones — with the actual failure in the middle of its context, not lost.

What we do not claim

We are an MVP. We want to be direct about the shape of this evidence:

• This benchmark measures retention, not downstream task success. Whether Claude / GPT-5 actually diagnoses the root cause faster with pruned context is our next study.
• LogHub is real production data, but it is a fixed public corpus. Results may differ on your specific stack, especially for structured-JSON or highly-domain-specific logs.
• The ranker today is regex + dedup + rarity. We have not shipped a semantic re-ranker in the default MCP tool yet; when we do, we will re-benchmark.
• Everything above was run on-device against local files with the public prune_file tool. No cherry-picked slices, no per-file tuning — but also no adversarial stress-testing against evasive log formats.

Conclusion

For AI agents to be genuinely useful on real infrastructure, they need something in front of the raw log stream — a layer that collapses noise, force-keeps the two lines that actually explain the failure, and fits both into whatever budget the caller sets. That is Roka.

On the LogHub corpus, our MVP shipped tool removes an average of 92.2% of log lines while retaining every real error we tested for, at a fixed 4,000-character output budget. It runs in tens of milliseconds, has no external dependencies, and is already available as an MCP server that any compatible agent can install with one command.