Half the Memory, None of the Answers Lost
Small AI models are cheap to run — until you ask them about something long. To answer questions about a document or a support chat, a model has to keep the whole thing in memory, and that bill climbs with every extra line of history. On a small model — on a phone or a laptop — remembering the conversation can cost more than the model itself. So we set a simple challenge: take a small model that is already good at its job, leave it untouched, and shrink what it has to remember — without changing a single answer. One idea flopped. The other cut the memory in half for free, and to a fifth with a little tuning. We are sharing both, because the flop taught us as much as the win.
01 — the problem
On a small model, the conversation costs more than the model
A model's size is fixed — the same however you use it. The memory it needs to follow a conversation is not: every extra line of history adds to it, growing in a straight line. Past a few thousand words it overtakes the model itself — and, unlike the model, it is paid again on every request.
We wanted to attack that one cost and nothing else — no retraining, no new hardware, no change to how the model is served. Leave the model alone, shrink what it remembers, keep the answers identical.
02 — the setup
A reader that genuinely needs the history
We built a deterministic troubleshooting simulator — a device-support world of routers, printers, power and error codes — that generates session histories and, from an audited event log, the ground-truth answers to eight families of questions. Splits are group-disjoint, so nothing leaks between them.
The reader is Qwen3-1.7B, LoRA-fine-tuned to competence and then frozen. It is genuinely good — and, crucially, the task really needs the history. Everything below compresses what the frozen reader stores; none of it makes the reader smarter.
03 — what didn't work
Injecting memory the reader was never trained to read
The seductive idea first. Instead of storing the history, train a small encoder to squeeze it into a handful of continuous "memory vectors," and feed those to the frozen reader as if they were input embeddings — a long history replaced by a few dozen vectors.
It didn't work. We tried four variants, and in every one the reader's answers did not depend on the memory: swap in memory computed from a different history and nothing changed — a correct-vs-shuffled advantage of about zero. We did fix a real bug along the way (the encoder's vectors were ~65× the magnitude of the reader's own embeddings and pointed off its input manifold), which made the output coherent — but it still carried nothing from the specific history. Our honest conclusion: the configurations we tested opened no channel, so we stopped rather than scale a method with no signal.
04 — what worked
Compress the reader's own cache, read through its own attention
The reframe was simple: stop inventing vectors the reader was never trained to read, and compress the real key/value tensors it already produces — then feed them back through its normal attention. This is the Eigen-Attention / Palu lineage: low-rank compression of the KV cache, calibrated per layer and head.
Each token's key and value live in a 128-dimensional space per head. We fit a projection per (layer, head) that squeezes it to r < 128 dimensions, store the small r-dim codes, and reconstruct on the fly — keys compressed after RoPE, positions and masks preserved. Plumbing check first: routing the uncompressed cache through the path reproduces the model's output token-for-token.
05 — the free 2×
Values compress; keys are the binding constraint
With plain SVD projections — just the top-r directions of each head's statistics, no training at all — the cache compresses to half its size at native quality: 98.4% accuracy against 99.9% with the full cache, a 1.5-point drop confirmed on fresh histories.
Two things fell out of the sweep. Values are far more compressible than keys — they survive compression to 25% of their size losslessly, while keys are the binding constraint. And the two interact nonlinearly: each is tolerated far better alone than together, because over-compressed keys reshape the attention weights, and the compressed values are then read through those shifted weights.
06 — past 5× with a little learning
Where generic SVD dies, learned projections hold
Below half payload, generic SVD falls apart — down to 3.5% accuracy at 5.3× compression. But let the projections learn — freeze the reader, train only the per-(layer, head) maps on the domain's questions, with a distillation objective matching the frozen reader's own next-token distribution — and the frontier extends dramatically.
07 — the ablations
Three ingredients, three different jobs
We pulled the learned recipe apart. The parts turn out to do genuinely different things — and only one of them is really about accuracy.
Schedule drives accuracy
The learning-rate schedule — warmup plus cosine — buys the accuracy. Remove it and accuracy drops about four points, hitting the hardest question families first.
Distillation drives fidelity
A weaker distillation term leaves greedy accuracy essentially unchanged, but nearly doubles the divergence from the native output distribution (top-token agreement falls from 99% to 82%). It is what makes the compressed model behave like the original under sampling.
Balanced sampling did little
Rebalancing the training data barely moved the needle — the question families were already close to balanced.
A satisfying detail explains why accuracy holds: on the answer tokens, the compressed model's top prediction matches the native model's ~100% of the time on questions it gets right; the disagreement concentrates on the ones it gets wrong. That is also why we report distribution fidelity separately — "same answer" and "same distribution" are different claims.
08 — at serving time
Memory always; speed only when the context is long
The compression is algebraically equivalent to a form that never reconstructs the full tensors — you can attend directly against the small codes. Measured at the reader's real geometry, the durable per-request cache is halved at every context length, for a small fixed overhead of shared projection matrices.
Latency shows a crossover. The extra projection maths makes the compressed path slightly slower at short context, but about 2× faster beyond ~8K tokens, where attention becomes memory-bandwidth-bound and there is simply half as much cache to read. This domain's histories are short, so here the win is memory, not speed — the speed win is a long-context phenomenon.
09 — the takeaway
A specialised model's cache is more compressible than it looks
You can halve a small frozen model's KV cache for free with calibration-only SVD at native answer quality, and compress it past 5× with a small per-domain learned projection — all through the model's native attention, with no retraining of the model itself.
The two results reinforce each other. Injecting foreign representations the model was never trained to read went nowhere. Compressing the model's own representations, read back through its own attention, worked — and a little domain-specific learning bought a lot of extra compression, precisely because the KV of a specialised model, on its domain, is genuinely low-rank.
We are careful about the claims. This is task-quality-preserving, not lossless — the tensors are approximate; what is preserved is the answers. The memory saving is real and unconditional; the latency saving only at long context. And this is a clean single-setup study — one synthetic domain, one model — a strong proof of concept, not a generalisation claim. A general method needs a second domain, a second model scale, and a real long-context workload where the serving win is felt. Those are the next experiments.