Whitepaper
Outline. PIN-0001 is green: GDN standing-place state now stores to disk and restores across a full vLLM restart with the exact answer intact. The full note is being written from that run.
Full PDF: in writing. Headline result below; sections are the table of contents.
PIN-0001 result — measured 2026-08-21
Qwen3-class 27B, vLLM + LMCache multiprocess, TP=2, speculative decoding on, CPU L1 + disk L2. A random 24-hex key is buried inside a long document; the context is stored, the engine is fully restarted, then the same prefix is restored from disk and the key is asked for.
| Context | Restore wall | Cold wall | E2E × | Restore TTFT | Cold TTFT | TTFT × | Hit |
|---|---|---|---|---|---|---|---|
| 5k | 2.84 s | 6.91 s | 2.4× | 1.76 s | 6.20 s | 3.5× | 97.0% |
| 13k | 4.11 s | 16.29 s | 4.0× | 3.19 s | 15.83 s | 5.0× | 98.8% |
| 26k | 6.99 s | 33.92 s | 4.9× | 6.19 s | 33.48 s | 5.4× | 98.7% |
- Correctness: exact buried key, character-for-character, on every restore run.
- No cross-contamination: a never-stored context gets 0% hit and a clean cold prefill.
- Stability: 12/12 pass — multi-length restore, warm re-ask idempotence, cold control, and two independent repeatability cycles.
- TP-agnostic: the same patch verified at TP=2 (25:1 page geometry) and TP=1 on a different hybrid (17:1) — exact key, 95.4% hit, 0.72 s vs 1.53 s. The fix is geometry-driven, not configuration-driven.
E2E figures are total wall including answer generation. TTFT is the prefill the restore actually replaces, and is the fairer measure of what was eliminated.
Two root causes were in LMCache 0.5.3, not vLLM: a layout misread that
persisted only 1 of 25 kernel pages per chunk, and async host-buffer
corruption from mutating shared metadata. A third failure mode — a
variable-shadowing TypeError that disguised failures as
worker-timeout hangs — lived in our own diagnostic layer, not upstream,
and hid the real bugs until it was removed. Upstream ships an intended
fix for the layout bug, but it never activates on vLLM ≥ 0.26's KV
layout — verified broken through the current development branch. As of
2026-08-21 the 4-file patch — verified byte-for-byte against a pristine
wheel — is the only verified working fix on any LMCache version.
- Problem. Hosted prefix caches are best-effort and evaporate. Hybrid GDN models do not have a growing token-indexed KV for most layers — they have a recurrent state. CacheBlend-style recombination does not apply to that state.
- Standing place. After a prefix is read, the useful object to persist is the GDN recurrent snapshot plus the remaining full-attention KV — one place you stand, not a trail of footprints.
- Product order. First: lossless exact-prefix warm of hot docs (PIN-0002), keyed by token hash, salted per project. Second: LinearKV for shuffled RAG on hybrids (PIN-0003). Not CacheBlend on GDN pages.
- Isolation. Dedicated hardware. Verified today: a never-stored context gets 0% hit and a clean cold prefill — no cross-contamination. Planned for PIN-0002: a per-tenant salt in every cache key. Knowledge stays in the customer’s Git. Flaiwheel is a source of curated chunks, not a writer of vLLM KV.
- What we will measure. Exact-answer restore vs cold prefill, latency after disk restore, survival across a vLLM restart, and failure modes (NaN logits when MTP and LMCache disagree). Done for PIN-0001 — numbers above. Still to measure: PIN-0002 hot-doc packs under concurrency, and PIN-0003 LinearKV quality on shuffled RAG. No number on this site until it is measured on the pin hardware.
Working notes live in the GitHub repo as they land.