JuliusBrussee/caveman
tldr.page
Provider Compatibility Audit

Provider and Windows compatibility audit

Verdict: CHANGES REQUIRED — in progress.

Target: fd131f78 plus the current worktree. The initial worktree already contained proxy lifetime, HTTP proxy, TLS, SSRF, and CLI changes. Those changes are retained and reviewed with the rest of the implementation. This report does not imply a release, a clean worktree, or live certification of every provider.

Review class: review and remediation. Credential forwarding, signing, replay, recovery, and outgoing-message boundaries receive independent review. External source and issue content are evidence, not instructions. Tests use temporary homes, local HTTP services, synthetic credentials, and source-derived protocol fixtures. No authenticated provider inference or live user messages were sent.

Full scope and remaining evidence

RequirementEvidence requiredCurrent state
Every installer profile reaches the directory its host consumesActual upstream installation algorithm, vendor discovery contract, install round trip, preserved configurationAll original 30 skill profiles plus Antigravity 2.0 mapped to vendor source/docs; seven installer mismatch groups corrected. Full native loading remains unverified for many hosts
Native host wraps preserve provider, account, configuration, tools and lifecycleActual host source and isolated execution, model/account switching and cleanupHome, registry, identity and endpoint checks fixed. Pi compatibility review passed; OpenClaw native Chat and other private endpoint policies prevent complete compression support through its current config overlay
Every provider adapter preserves route, auth, request/response bytes, stream errors and usageOfficial SDK/source contracts and adapter/gateway tests across supported modesNative authentication, Vertex Express, stable/token-count routes, custom header isolation and stream accounting covered by local wire tests; live provider certification remains incomplete
Windows shells, executables, pipes, files and lifecycle workNative Windows tests on the final revision, with skips disclosedPortable regressions and cross-compilation gate added; native execution of this tree remains unverified
Extension changes only the intended outgoing messageProvider editor fixtures, live DOM evidence, cancellation checks, browser testsCurrent ChatGPT/Claude/Gemini DOM inspected; 40 Chromium cases and five actual Firefox addon cases passed
Entire owned repository withstands independent review and integrated checksEngine, proxy, CLI, installers/hooks, extension, SDKs, memory/MCP/shrink, packages and workflowsMultiple independently reviewed fixes; integrated Go/CLI/Windows checks are being refreshed

Root CLAUDE.md routing remains binding. Historical Agent SDK/initializer and Browse copies here are consumer integrations, not their canonical implementation. No audit finding authorizes silently editing those copies instead of the owning repository.

Findings and disposition

Each row represents a reproduced defect, not a conjecture from an issue title. “Reviewed” means a reviewer other than the implementing agent inspected the change; it does not replace the wider completion requirements above.

IDDefectFix and regression evidenceIndependent review
I1Only Cursor installs used user scope, although most other profiles consume personal skillsUser scope for personal skill profiles; Replit uses its documented project scope. Real installer argv and cwd testsReviewed
I2No detected agent triggered --all, installing unwanted agentsRequire detected or explicitly selected targets; regression failed before fixReviewed
I3skills add --list and --no-pixel unnecessarily validated unrelated CODEX_HOMEResolve augmentation roots only when augmentation runs; 10 skills-add runtime tests passedReviewed
I4Continue's loader skips the per-skill symlinks created by the delegated installer and its custom home was ignoredOwned physical copies into the actual CONTINUE_GLOBAL_DIR/skills; extracted vendor walker/CLI controls plus real installer testsReviewed
I5AiderDesk custom homes were ignored and plain Aider falsely detected the desktop productHonor both vendor home overrides; detect aider-desk/desktop app; preserve skill ownership and feature prerequisitesReviewed
I6iFlow and Crush explicit skill roots were missed by upstream defaultsOwned copies honor IFLOW_HOME and exact CRUSH_SKILLS_DIR; current iFlow scanner reproduction and both real installer round tripsReviewed
I7Kiro and Mistral detection missed the actual kiro-cli and vibe executablesAdd vendor command names; isolated PATH tests retain legacy aliasesReviewed
I8Replit home-global install did not reach its documented project skill discoveryOmit global flag, report current project, correct install instructionsReviewed against vendor scope and actual argv
I9Universal upstream installation skipped Antigravity IDE's declared directory; Antigravity 2.0 reads a different rootOwned copies for IDE and separate explicit antigravity-2 target; actual entrypoint tests prove target isolation, dual selection and dry-run behaviorReviewed against official product docs; full native loading unverified
W1Explicit extensionless Windows paths selected Unix shims; directories and quoted PATH entries misresolvedPATHEXT-aware executable lookup, directory rejection and quoted-path testsReviewed
W2MCP shrink argv split quoted paths and discarded groupingLiteral argv parser plus JSON-array input; real config round trips and native PowerShell test casesReviewed; native Windows pending
W3Binary probes used POSIX lookup and inherited real Windows profile rootsPortable launchers; isolated HOME/USERPROFILE/AppData/XDG/config roots; credential stripping and prerelease comparison regressionsReviewed
W4Rewriter accepted changes to Windows drive/UNC prefixes, paths with spaces and numeric source-coordinate prefixesComplete source-location tokens must survive; actual Accept regressions reject drive/share changes and 42 to 420Reviewed
W5Hermes config, plugins and MCP used ~/.hermes on Windows, and environment overrides expanded a tilde that native Hermes treats literallyFollow pinned Hermes's LOCALAPPDATA/hermes default, whitespace stripping and literal override semanticsActual pinned Python resolver exercised with six isolated cases; five permanent cases include three CLI MCP round trips. Independent review and native Windows execution pending; sticky profile selection remains unverified
A1Azure api-key and Gemini x-goog-api-key disappeared, allowing another configured account upstreamPreserve native inbound credentials before environment fallback; full local gateway regressionsReviewed
A2Azure guessed Bearer versus API key from opaque credential bytesPreserve explicit Bearer scheme; treat scheme-less keys as opaqueReviewed
A3Native AWS SigV4 authorization became a Bedrock bearer tokenRe-sign actual outgoing bytes only with matching configured IAM identity/session/region/service; fail before upstream on mismatchReviewed
A4Bedrock model IDs containing :0, escaped IDs and ARNs received incorrect canonical URI signaturesService-aware URI normalization/double encoding, with nine actual Botocore 1.43.89 signature oracles; S3 behavior preservedReviewed
A5Gemini query credentials were lost or could conflict with header credentialsNormalize key/$key, preserve unrelated query parameters and reject conflicting credentials without disclosing valuesReviewed
A6Typed URL errors leaked encoded query credentials into retry logs, including overlapping joined URLsRemove full typed error URLs before generic redaction; one-pass longest replacement; real HTTP-client retry/log testsReviewed
A7Stable Gemini v1 and SDK token-count routes were missing or treated as generationStable generation routes plus OpenAI/Bedrock/Vertex count routes; 32 count cases and stable-generation regressionsReviewed
A8Provider error events inside HTTP 200 streams were logged as success; provisional usage could appear completeValue-free provider_stream_error, correct usage completeness and zero savings on failed streams; 42 HTTP wire cases and paired billing regressionReviewed
C1CLI ignored CODEX_HOME across account/config/hooks/MCP/skillsUpstream-compatible absolute/relative/default handling and explicit invalid-root failureReviewed
C2Claude and Gemini custom homes disagreed with upstream paths; registry skill discovery bypassed overridesClaude custom .claude.json placement and Gemini home-parent semantics; isolated runtime/path testsReviewed
P1Pi provider registration/unregistration deleted another extension's models, credentials and handlersOverride only the selected model endpoint; real Pi ModelRuntime preserves original registryReviewed
P2OpenClaw ignored custom base URLs; Pi checked only host/port and could switch tenant pathsShared effective-request URL proof against published native/compat endpoints; schemes, ports, prefixes, versions and repeated separators coveredReviewed; focused routing and real gateway fixtures
P3Stale Pi state plus an unrelated live listener could receive a provider keyHealth response must identify the same running instance; CLI status also checks token, rejects redirects and bounds the probeReviewed; actual Pi stale-listener probe sends no provider request/key
P4OpenClaw renamed providers and selected model API could change siblings/fallbacksKeep provider identity/auth/catalog/defaults; preserve provider API and check every affected model API before a provider-wide endpoint overrideReviewed; supported-route, direct fallback and sibling fixtures
P5Replacing Pi's URL changed Chat/Responses compatibility defaults and dropped resolved account or attribution headersFreeze source-derived public defaults; resolve headers through the public registry API; only reroute when endpoint-specific attribution can be preservedReviewed with actual Pi SDK payloads
P6OpenClaw's URL rewrite changed private provider metadata and payload policy, beyond its public compat settings; raw provider casing bypassed native guardsNormalize policy comparisons as the actual configured-model resolver does, without changing config keys. Unsupported native policies stay direct; public settings are frozen for supported custom routes. Full native compression remains unfinishedReviewed; 40 route/compatibility cases and an independent ten-case pinned configured-model resolver probe passed
P7Hermes and API-key Codex append resource paths to a base missing /v1, causing native 404sAdd /v1 to wrap and native configuration/journal/status routes; retain Codex's dedicated unversioned ChatGPT root. Conformance stubs now append the actual SDK resourcesActual Hermes 0.19.1 and Codex 0.153.4 reproduced the failures. Route regressions and native Codex streamed recovery pass; independent review pending
P8opencode wrap discarded its entire existing inline configuration, losing model, permissions, account settings and MCP serversMerge the routing overlay into native JSONC; preserve environment/file references and reject malformed input without disclosing valuesSix regressions failed before the fix; 19 focused cases and pinned native opencode 1.18.27 configuration checks pass. Independent review pending
P9Hermes and opencode can retain the original provider credential while replacing its custom endpoint with a different proxy destinationOpen: preserve the original provider/account contract and require proof of the effective destination before routingActual Hermes requests reached provider B using provider A's synthetic key; pinned opencode configuration and SDK resolution prove the same endpoint/key mismatch
P10Hermes direct fallback retained Caveman's forced --provider custom, overriding the user's configured provider when proxy startup failedStrip Hermes's injected provider prefix on direct fallback while retaining the user's original argumentsBoth permanent cases failed before the fix; 172 Hermes/Qwen regression cases pass. Independent review pending
A9Provider-specific custom headers disappeared, while header forwarding lacked per-mount scopePublish forward_headers names per configured compatibility mount; preserve declared headers and supported SDK attribution/affinity headers; unrelated mount secrets remain blockedReviewed with actual HTTP gateways and SDK header behavior
A10Connection nominations could reach upstream after the Connection field itself was discardedRemove nominated hop-by-hop headers before transport, including after authentication fallback and retriesReviewed; custom and native header regressions
E1stop caveman still received remindersExplicit stop-state behavior and provider composer regressionsReviewed
E2Delayed send replay could target a changed draft, editor or conversationTransaction-bound replay; text, rich-content and input changes cancel pending sendReviewed
E3Document-wide Send selectors could choose feedback controlsComposer-scoped actions and provider-specific fixturesReviewed
E4Firefox launcher interpolated an unquoted temp path through a shellStructured argv, process failure reporting and actual addon executionReviewed
E5Firefox review CTA used a Chrome store identifierFirefox-specific verified metadata or hidden unavailable destinationReviewed
S1SDK redirects forwarded upstream/API credentials to another originTypeScript rejects redirects including raw overrides; Python private opener rejects redirects; distinct local-origin tests for 301/302/303/307/308Reviewed
S2TypeScript request deadline ended at response headers; custom fetch bodies could hang indefinitelyShared abort deadline covers stream/decoder reads, clone metadata and cancellation; discarded error bodies release resourcesReviewed
H1Standalone hook install/uninstall overwrote or deleted unrelated hooks/package.jsonPreflight compatible manifests, preserve foreign content and delete only exact owned manifestReviewed
H2PowerShell printed installation success after Node merge failure; standalone JSONC handling was unsafePropagate child exit, preflight settings, use shared JSONC parser from a clone and refuse unsupported detached input before mutationReviewed
M1Fresh default/custom MCP home could not create recovery databaseCreate missing home parent with restrictive permissions; actual binary tests preserve explicit DB-path failure behaviorReviewed
M2MCP “session” stats returned lifetime database totals and missed repeat/pass-through callsProcess-local call accounting independent of shared-store history/concurrent servers; Engine lifetime stats unchangedReviewed
M3Repeat recovery assumed old results remained visible after host compactionEvery valid handle/query stays retrievable; no process-lifetime repeat denial or query-count widening; actual binary repeat/restart testsReviewed
M4SQLite replacement let recovery writes succeed against retired files and could let cleanup alter replacement journalsAny observed main/WAL/SHM identity change permanently invalidates that Store; withhold handles and retain unsafe connections until exit. Failed initial opens also retain opened disk connections; no automatic replacement adoptionReviewed; real old/fresh process operations, journal-byte preservation and fresh CCR race tests pass. Concurrent initial opening remains a documented driver limit
M5Codex clears custom recovery-location variables from its MCP child, so valid proxy handles cannot be foundForward only CAVEMAN_HOME and CAVEMAN_CCR_DB through Codex's supported env_vars in wrap, native enable and MCP installNative streamed Codex request stored a valid handle, then actual MCP returned cave_unknown_handle; three permanent cases failed before the fix. Native Codex 0.153.4 now retrieves exact content with default home, custom home and an explicit database path containing spaces; 48 focused regressions pass. Independent review pending
N1Unrelated Pixel transforms rounded large numeric tool arguments/schema constraintsPreserve JSON number lexemes across parse/clone/render paths; integer/decimal/exponent and live-tail regressionsReviewed
N2Top-level Decoder.More accepted valid JSON plus malformed suffixes and emitted repaired contentRequire second decode to return EOF; compressor pass-through and retrieval record-boundary regressionsReviewed
N3Shrink's prose rules removed CJK intent and significant whitespace (#575)Bypass prose transforms for Han, Hiragana, Katakana, Hangul and Bopomofo input; exact Chinese/Japanese/Korean/astral-Han fixturesReviewed; 20 shrink cases passed
N4Direct execution of the documented compression CLI failed on relative imports (#105)Resolve the script package from its own file before imports; actual execution from another working directory with spaces and UnicodeReviewed; three direct-entry cases passed
H3Gemini stats read unrelated Claude logs and presented them as the current host's usage (#403)Gemini returns native stats guidance before any Claude state operation; Claude hooks pass explicit host ownershipReviewed; instrumented reads/writes plus nested-host control
H4Stats invented fixed-ratio savings and converted missing or invalid usage counters into numeric totalsSavings remain unknown without comparison; validate safe integer counters and retain complete/partial/unknown availability through session, share and history outputReviewed; 60 existing and 13 independent usage cases passed
H5Fallback skill instructions imposed Portuguese examples and failed to prioritize an explicit project reply language (#812)Remove foreign-language fallback examples and honor fixed user/project language before dominant prompt language; refresh shipped copiesStatic contract fixed; model behavior is not guaranteed
S3The new stats terminal summary displayed unavailable provider/cache usage and missing request comparisons as zeroRender unknown values and partial subtotals with explicit usage/comparison coverage; retain provider-reported zeroReviewed; permanent absent/partial/zero controls failed before the fix and pass now
S4The new stats report discarded known spend from complete provider usage when a stream later failedSnapshot valid observed spend separately from counterfactual savings eligibility; failed requests still cannot earn savingsReviewed; actual Anthropic stream through gateway, store and report counts both billed rows and only the successful savings comparison
D2README and plugin metadata claimed unsupported output reduction and quality equivalence; chart parsing no longer matched the benchmark harnessRemove unsupported claims; plot only the current harness's skill-versus-terse data, otherwise show unpublished result statusReviewed; six tests plus real harness-to-chart increases, zero counts, escaping and SVG freshness
CI1Hermes rejected wheel installs; an editable URL plus a duplicate package requirement still searched PyPI for an unavailable releaseInstall the immutable editable URL, then check installed metadata against the registry version; mismatched version failsReviewed; exact final workflow command, full core dependencies, pip check, version and help pass on fresh macOS arm64 and Linux amd64 container environments
D1YAML example used an ignored azure keyCorrected to azure_openaiReviewed

Source evidence

  • The complete vendor discovery matrix records all 30 original skill profiles and Antigravity 2.0, with pinned source links or official documentation for closed-source hosts. It follows the actual skills@1.5.24 algorithm: universal profiles stop at the shared directory and do not create their declared alias. Metadata agreement alone was insufficient. Continue's actual traversal and iFlow's shipped scanner were executed against isolated filesystem controls; no full native host run is inferred from those probes.
  • Hermes immutable build guard explains both failures in conformance run 34083480150.
  • Authentication/signing contracts and sources: Provider authentication. Stream and count-route references: Proxy and providers.
  • OpenClaw routing was checked against provider configuration docs and actual transport source at 0965053fe6b9341776df147a6934b7485c60b5ca (v2026.8.2). Pi behavior was checked with installed pi-coding-agent@0.84.2 and its own pi-ai@0.84.2, including actual ModelRuntime, provider payload generation and CLI execution with local services.
  • Gemini CLI stats commands and installed @google/gemini-cli@0.53.1 identify the supported native stats paths; its shell execution service sets GEMINI_CLI=1 on child commands.
  • Codex compaction replaces message history while its MCP runtime remains alive; remote compaction source and thread runtime ownership invalidate an MCP-process-lifetime assumption of transcript presence.
  • Read-only live DOM inspection found Claude ProseMirror, Gemini Quill, and ChatGPT's logged-out textarea/composer-submit layout. No live content was edited.

Verification ledger

These are observations of the tested source snapshot, not a final audit PASS. The broad CLI/Windows observations below precede the latest Codex and opencode changes. Their focused checks are recorded above; integrated refresh remains due.

Command / mechanismLatest completed observation
Root npm testRefreshed: 371 passed, two skipped (native Windows PowerShell argv tests on macOS); bounded test-file concurrency
Root Python python3 -m unittest discover -s tests -vRefreshed: 141 passed
Standalone tests/test_*.js filesRefreshed: all 14 files passed, including new usage and CJK cases
python3 tests/verify_repo.pyRefreshed: all local verification checks passed, including packaged helper/skills and hook flow
Native skill installer focused testsRefreshed: 56 passed, including both Antigravity targets; separate ownership helper checks passed
CLI status portable fixturesSeven passed with isolated host homes/PATH; added to Windows gate
CLI stats portable fixturesSix passed, no skips; actual CLI JSON/help paths use isolated native stubs and are included in Windows gate
SDK TypeScript npm testBuild/type checks and 206 runtime tests passed
SDK Python full suite257 passed
Standalone installer safety22 Bash/PowerShell cases passed; PowerShell 7.6.5 ran on macOS
Pi npm test87 passed, including build/typecheck, actual Pi payload generation, registry preservation, stale-listener refusal and recovery lifecycle
OpenClaw wrapper runtime11 passed with portable fixtures, live local listener and explicit routing-state proof
Native Codex recoveryActual Codex 0.153.4, real proxy and stdio MCP: three isolated streamed runs passed against a synthetic loopback Responses provider. Default/custom homes and explicit database path covered; exact retrieval reached the next request and recorded savings stayed zero. Pinned registry version remains 0.153.0; this is not that version or Windows proof
Provider protocol batch14 packages passed; 42 streaming/error cases, 32 count-route cases, credential and signing regressions
Go Engine/compressors/shrink/MCP and Rewriter batchesPassed; fresh-home, repeated retrieval and restart stats use actual compiled MCP binary
Full GOMAXPROCS=2 go test -p 2 ./...Refreshed after final CCR and stats changes: passed
CCR go test -race -p 2 ./engine/ccr -count=1Refreshed: passed, 55.5 seconds
Browser extension40 Chromium browser cases and five actual Firefox 153.0.4 staged-addon cases passed
Windows gate's Node contractsRefreshed on settled source: 177 passed, four skipped on macOS; includes portable stats/status fixtures and bounded child-process test concurrency
Windows Go compileRefreshed on settled source: all packages compiled for both amd64 and arm64
Full CLI npm test with the rebuilt real proxyRefreshed: 1,016 passed, 18 skipped. A POSIX Node fixture now preserves shared-library lookup; all previously failing Qwen/status/sync cases pass
Complete Windows compatibility gatePassed on macOS: CLI/agent builds, 177 Node contracts and both Windows Go architecture compilations. Native Windows execution remains unverified

The opt-in native Codex recovery probe requires built CLI output and explicit native Codex, proxy and MCP binaries:

CAVEMAN_NATIVE_CODEX_BIN=/absolute/path/to/codex \
CAVEMAN_PROXY_BIN=/absolute/path/to/caveman-proxy \
CAVEMAN_MCP_BIN=/absolute/path/to/caveman-mcp \
node scripts/probe-native-codex-recovery.mjs --custom-home --explicit-db

Omit both flags to exercise the default store with neither recovery-location variable set. Use only --custom-home for the custom-home case. The probe requires macOS sandbox-exec, denies the native host external network and user-home reads, uses synthetic credentials and keeps logs plus a hashed receipt under its printed scratch directory. Its synthetic provider requests the actual MCP tool; the probe grants that tool approval only in the isolated launch configuration. It does not test hosted inference, production tool approval, or native Windows.

Issue hints without a reproduced remaining defect

  • #897 (concurrent encoded responses): local gateway tests overlapped 12 requests across gzip/zstd, Content-Length/chunked framing and record/compress modes. All 96 cases passed; three race-detector repetitions passed 288 cases. They assert original wire bytes, decoded response identity, complete usage and no replay. This does not close or disprove the reported live failure.
  • #949 (Codex on Windows): npm-generated .cmd shims from official Codex 0.150.1 and 0.153.4 archives resolve through the portable launcher with literal metacharacter arguments. Both upstream hook runners preserve descendants after completed hooks; hook discovery still requires the host's enabled/trusted state. An approved-hook cold start and subsequent daemon survival on native Windows remain unverified. No speculative production change was made.

Initial baselines also passed root Python (137), standalone JS (13), core Go, and tests/verify_repo.py; they must be refreshed where later edits affect them.

The native Windows CI jobs and full pinned-host conformance have not run on this worktree. Remote green jobs cover earlier revisions. Hermes's corrected workflow command now passes with all core dependencies on fresh CPython 3.12 environments on macOS arm64 and Linux amd64; the Linux proof used a local container, not the hosted Ubuntu runner. Optional extras and provider inference were not exercised. Installed-agent version/help probes are not provider inference proof, and older/missing binaries are not automatically API defects.

The goal remains active until the full scope above has adequate evidence.