CaveBench Wrap benchmark

Result

On six deterministic, agent-shaped tool-output workloads, Caveman-wrapped Claude Code used 33.2% fewer provider-reported input tokens than direct Claude Code: 591,673 vs 885,793 tokens across 18 paired runs. Caveman passed all 18/18 exact-answer checks. The case-clustered 95% interval was 14.6% to 48.5%.

Claim basis: benchmark_counterfactual. This is controlled benchmark evidence, not production traffic, customer spend, a provider invoice, or Caveman verified_savings.

Caveman won 15/18 pairs. Headroom's three YAML runs failed the exact-answer gate and remain visible rather than counting toward savings at held quality.

Per-case results

Each case ran three times per arm.

Unsupported and no-op inputs stay in the aggregate. HTML regressed because no compression transform applied while full Caveman skill overhead remained counted.

Method

• Six immutable MCP fixtures, each 60–95 KB: logs; deployment JSON; fraud CSV; test output; configuration YAML; dashboard HTML.
• Three rotated repetitions for direct Claude Code, Caveman, and Headroom: 54 total agent runs and 18 direct/Caveman pairs.
• Claude Code 2.1.223, model claude-sonnet-5.
• Headroom 0.33.0, pinned at upstream commit e0ce4b1d.
• Every arm called the same fixture exactly once and returned a structured answer checked by an exact semantic JSON oracle.
• Every arm used Claude Code's modelUsage counters. Primary input metric: input_tokens + cache_read_input_tokens + cache_creation_input_tokens.
• Cache buckets were summed without price weighting. Wrapper-native tokenizer estimates did not drive the comparison.
• Recovery was available only when required answer data was absent from visible compressed content. Recovery calls and follow-up provider input remained counted.
• Full Caveman skill prompt overhead remained counted from the first request.
• Aggregate reduction compares summed Caveman input with summed paired direct input. The interval uses a deterministic 10,000-resample percentile bootstrap clustered by case.

Verification and provenance

• Generated: 2026-08-06T14:31:43Z
• Publication gate: passed
• Same provider-usage source: 54/54 runs
• Fixture called exactly once: 54/54 runs
• Caveman skill installed, loaded, and applied: 18/18 runs
• Positive proxy compression observed: 15/18 Caveman runs; no-op runs remained included
• Permission denials: 0
• Corpus SHA-256: 9a400a6dc38591dc3ce59bc2e3fa6fc59d99e211dc9185b430979de78991760a
• Caveman skill SHA-256: 5e30bb56afbd0b01bd736f2da84180e76f18db4a64de8e124525d5c8dc2e8605
• Harness source SHA-256: e6322a3a55cfdfa6c7942022a1be9adb49fa30c380352c819c99bbd4bff30cc4
• Fixture MCP binary SHA-256: 5c71768780582708c00fa1d6862a6a5b93fc33fc23655ff430b0edb8fee1e790
• Claude binary SHA-256: 4163c57c719e27680336f323ebdcd2ba8aa48a683fdfa427240ec4b506a21e45
• Harness Git commit: 630e157246b68b63559fb8baab29b87042db996b
• Dirty worktree at execution: false

Reproduction availability

This repository contains published report and provenance hashes, but not raw harness or run artifacts for this result. It cannot be independently reproduced from this checkout. Treat result as pinned report, not reproducible public benchmark, until harness and raw artifacts are published here.

Publication requires at least six cases and three repetitions; exact direct and Caveman quality on every run; one fixture call per run; the same provider usage source; active Caveman skill; at least one proven compression; positive aggregate reduction; and a 95% interval entirely above zero. Negative and no-op cases may not be removed.

Boundaries

• Workloads are deterministic large tool outputs, not open-ended coding tasks or customer traffic.
• Result applies to this pinned suite and runtime. It is not a universal savings promise.
• Local host isolation reset a dedicated Claude configuration before every arm. This was not a sealed container, and no exact filesystem or egress isolation is claimed.
• Cost, latency, and output-token deltas are secondary because agent trajectories can differ. Primary claim is paired provider input tokens at held quality.

Per-file compression

Measured 2026-10-03, outside any agent run. Each of the six corpus files was regenerated byte for byte (corpus SHA-256 matches the one above) and passed once through engine.Simulate in compress mode, with the case question as the query. That is the path the proxy's query-aware CompressSegmentQuery takes. Tokens are counted with the engine's o200k_base counter. No model call, no recovery.

The session results above ran on the older engine. Today's engine is a little less aggressive on CSV and logs and far more aggressive on JSON (three JSON elision fixes landed 2026-09-30). The session benchmark has not been re-run on it. Without a query, the older engine gave 161 tokens for test output and 119 for YAML; today's engine gives the same either way.

File reduction is not session reduction. Every request also carries the system prompt, tool definitions, conversation, and caveman skill, and any recovery read counts. All five compressed files use S4 transforms, which need the local recovery store.

Reproduce: generate the payloads with CorpusCases() from Caveman-Cloud internal/wrapbench/corpus.go, then call engine.New(nil, nil).Simulate(payload, engine.Options{Mode: engine.ModeCompress, Query: question}).