# Fourier Magnitude KV Cache Quantization **URL:** \ **Category:** Research\ **Created:** [August 18, 2026, 1:16pm UTC](https://discuss.huggingface.co/t/fourier-magnitude-kv-cache-quantization/178815 "2026-08-18T13:16:10Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![ntrillard](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@ntrillard](https://discuss.huggingface.co/u/ntrillard)\ **Post date:** [August 18, 2026, 1:16pm UTC](https://discuss.huggingface.co/t/fourier-magnitude-kv-cache-quantization/178815/1 "2026-08-18T13:16:10Z")
Hi, I’ve been running some tests on KV quantization and have found that Fourier Magnitude preserves phase remarkably well. Files here [GitHub - ntrillard/kv-findings · GitHub](https://github.com/ntrillard/kv-findings/) Key results: - **Fmag4+phase8b (12 total bits): 95.8% match, 62% savings** - Fmag4+phase6b (10 total bits): 95.7% match, 69% savings - Fmag4 (4-bit mag only): 96.9% on 40 prompts - Fmag3: 78.4% - Fmag2: 64.9% - Std4: 54.9% Cross-Model Fmag4 Results Fmag4 works on 3/4 model families. Gemma, Qwen, and SmolLM2 all show good results (81-100%). Pythia fails. The failure is not from higher Fmag4 error. The reconstruction error is similar across all models (rel error 0.035-0.050). The issue is architectural: Pythia uses LayerNorm (not RMSNorm) learned absolute positional embeddings (not RoPE). These differences make the K quantization error have a larger downstream impact. > ## Finding > > Quantizing the **Fourier magnitude spectrum** of K and V cache values at 4-bit, while preserving the **phase at full precision** , achieves **96.9% token match** with the fp16 reference. This is a **5-10x improvement** over standard min-max quantization at the same bit width (54.9%). > > ## Method > > ```auto > K → FFT → quantize|magnitude|@4bit → combine|with phase| → IFFT → K' > > ``` > > The pipeline: > > 1. Compute FFT of K along the head\_dim dimension > 2. Quantize the magnitude spectrum to 4-bit (16 levels) using standard min-max > 3. Keep the phase (angle) at full bfloat16 precision > 4. Reconstruct via IFFT: `K' = IFFT(mag_q · cos(angle) + j · mag_q · sin(angle))` > > ## Why It Works > > The Fourier transform separates the K signal into two components: > > - **Phase** : determines the positions of features in the K vector — this is the critical structural information > - **Magnitude** : determines the energy distribution across frequencies — this is smooth and compressible > > The QK dot product is robust to magnitude scaling (softmax normalizes), so the 4-bit magnitude quantization introduces minimal error in the attention output. The phase is preserved at full precision, maintaining the positional structure. > > ## Results (40 prompts, Gemma-3-1B) > | Method | Bits | Token Match | 100% Prompts | Savings vs bf16 | > | --- | --- | --- | --- | --- | > | **Fmag4** | **4** | **96.9%** | **34/40** | **62%** | > | Fmag3 | 3 | 78.4% | 23/40 | 69% | > | Fmag2 | 2 | 70.3% | 15/40 | 75% | > | Std 4b | 4 | 54.9% | 11/40 | 62% | > | Std 3b | 3 | 38.7% | 3/40 | 69% | > | Std 2b | 2 | 13.7% | 0/40 | 75% | > ## Key Insights > > 1. **Phase is the primary carrier of structural information.** The phase determines where features are positioned in the K vector. The magnitude only determines their relative strength. > > 2. **The magnitude spectrum is smooth.** K values along head\_dim have a concentrated energy distribution. The 4-bit quantization (16 levels) is sufficient to capture this. > > 3. **Log1p compression doesn’t help at 4-bit.** At low bit widths (2-bit), log1p compression helps redistribute quantization levels. At 4-bit, there are enough levels already. > > 4. **Fmag outperforms standard quantization at every bit width.** Fmag2 (70.3%) beats Std4 (54.9%) despite using half the bits. > > ## Practical Impact > > For Qwen2.5-7B @ 4-bit NF4 on a 10GB 3080 Ti: > | KV Config | Max Context | Total Memory | Fits 10GB? | > | --- | --- | --- | --- | > | bf16 | 68K | 9.50 GB | ✅ | > | **Fmag4+int8** | **137K** | **9.50 GB** | **✅** | > | bf16 @ 96K | — | 11.07 GB | ❌ | > | **Fmag4 @ 96K** | **96K** | **8.31 GB** | **✅** | > Fmag4 doubles the maximum context length at the same total memory budget. > > ## Prior Work > | Paper | Year | Approach | Difference from Fmag | > | --- | --- | --- | --- | > | **SPECTRA** (arXiv:2608.07915) | 2026 | PCA-based coordinate transform + bit allocation | Data-dependent transform, not Fourier | > | **Codec-Gauge** (arXiv:2607.20538) | 2026 | Learned orthogonal transforms (DCT) + quantization | Learned transform, not fixed Fourier | > | **eOptShrinkQ** (arXiv:2605.02905) | 2026 | SVD denoising + TurboQuant | SVD-based, not frequency-domain | > | **Quantize What Counts** (arXiv:2502.15075) | 2025 | More bits for keys, fewer for values | Asymmetric allocation, not Fourier | > **Fmag4 is novel** in using the Fourier transform specifically for KV cache quantization. The closest prior work (Codec-Gauge) uses DCT with learned transforms, while Fmag4 uses the standard FFT with no learning required. The insight that the **phase is more important than the magnitude** for K cache quantization is a new contribution. > > ## Limitations > > - Tested on Gemma-3-1B and Qwen2.5-7B only. Generalization to other architectures (LLaMA, Mistral) unverified. > - 4-bit magnitude quantization is the sweet spot. 3-bit shows degradation (78.4%), 2-bit loses coherence (70.3%). > - Requires FFT computation per token, adding ~0.1% compute overhead vs standard quantization. > - The phase must be stored at full precision (16-bit), which limits the maximum compression ratio. > ## Key Files > > ### 1. Main Proof File: `algebraic_kv_tests.py` > > The primary reproduction script. Tests Fourier magnitude quantization at 4/3/2-bit against standard quantization across 40 prompts. Run: > > ```auto > HF_TOKEN="your_token" python3 algebraic_kv_tests.py > > ``` > > Key results (lines 255-260 in the output): > > - Fmag4: 94.8% token match, 34/40 prompts identical > - Fmag3: 78.4%, 23/40 > - Fmag2: 64.9%, 12/40 > - Std4: 54.9%, 11/40 > - Std3: 38.7%, 3/40 > - Std2: 13.7%, 0/40 > > ### 2. FMAG\_KV\_FINDINGS.md > > Full write-up with method, results tables, practical impact, and prior work comparison. > > ## Supporting Files > > - `max_context.py` — Original KV cache quantizer (baseline comparison) > - `gpt2_asymmetric_test.py` — Early Fmag4 prototype on GPT-2 > - `cross_model_experiment.py` — Tests across Gemma, Qwen, and Gemma-4B > - `real_asymmetric_cache.py` — Int8 storage implementation > - `scientific_experiment.py` — Rigorous 10-prompt logprob evaluation > - `kv_sweep.py` — Systematic 64-config bit-width sweep > - `generation_experiment.py` — Generation quality with token match metrics > - `FMAG_APPLICABILITY.md` — Analysis of Fmag across all LLM signals > > ## Quick Start > > ```auto > # The core Fmag4 function > def fmag4(t): > tf = torch.fft.fft(t.float(), dim=-1) > mag = quant_pt(tf.abs(), 4) # 4-bit magnitude > return torch.fft.ifft(torch.complex( > mag * torch.cos(tf.angle()), > mag * torch.sin(tf.angle()) > ), dim=-1).real.to(t.dtype) > > ``` ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [August 19, 2026, 12:34am UTC](https://discuss.huggingface.co/t/fourier-magnitude-kv-cache-quantization/178815/2 "2026-08-19T00:34:06Z")
I did a quick experiment in Colab and was able to reproduce a similar phenomenon: * * * My main takeaway is that the observation looks worth separating into a few smaller questions rather than treating it as one all-or-nothing claim about “Fourier phase.” In a small K-only probe, I also found conditions where a simple raw 4-bit K quantizer immediately changed the generation trajectory, while a Fourier-domain perturbation that quantized magnitude and preserved phase kept the reference trajectory. So at least qualitatively, I do see the same kind of robustness effect. What I am less sure about is **which part of the construction is responsible for it**. There seem to be several effects mixed together: ```plaintext observation | +-- Fourier basis itself? +-- transform/preconditioning in general? +-- magnitude vs phase/direction precision? +-- polar vs Cartesian representation? +-- where the fixed bit budget is allocated? +-- what error is actually visible through attention? +-- interaction with RoPE? ``` I think the highest-information next step is therefore not a much larger benchmark yet. A few small controls can distinguish most of these branches. ### The three controls I would try first **1. Hold the total rate fixed while moving bits between magnitude and phase.** For example, for a direct-angle representation: ```plaintext mag4 + phase8 mag5 + phase7 mag6 + phase6 mag7 + phase5 mag8 + phase4 ``` This separates two claims that are easy to accidentally merge: - phase precision matters; - most of the available bits should go to phase. In my small probe, lowering phase precision too far definitely hurt, so I think the first statement has some empirical support. But at a fixed total rate, `4+8` was **not** consistently the best allocation: `5+7` or `6+6` often did better depending on the model and metric. So I would currently phrase the interesting hypothesis more like: > **Magnitude and phase appear to have asymmetric sensitivity, but the optimal rate allocation is still an empirical question.** This also connects fairly naturally to recent KV-cache work on non-uniform bit allocation. [RateQuant](https://arxiv.org/abs/2605.06675) explicitly treats the distortion-vs-rate curve as quantizer-dependent, and the very recent [AATC](https://arxiv.org/abs/2608.14191) treats KV compression as an attention-aware transform-coding/rate-allocation problem. **2. Add one transform/representation control at roughly matched rate.** I would not make this a large ablation grid. Even something like ```plaintext FFT polar FFT Cartesian (real/imag) Hadamard ``` would tell a lot. The reason is that there is now a fairly strong adjacent literature showing that **the coordinate system presented to the quantizer is itself a design variable**. [Codec-Gauge](https://arxiv.org/abs/2607.20538) directly compares raw, random, Hadamard, DCT and learned cache-coordinate transforms; [NOVA-KV](https://arxiv.org/abs/2608.04074) derives transforms from attention-product distortion rather than treating the original KV coordinates as privileged. My own tiny probe pointed in the same direction: DCT/Hadamard/random-orthogonal controls were surprisingly competitive, and even with the FFT held fixed, changing the coefficient representation from polar-style to real/imag changed the error substantially. That would give a clean branch: ```plaintext If FFT stays clearly better at matched rate: -> a specifically Fourier-domain effect becomes more plausible. If several transforms behave similarly: -> the larger finding may be transform/preconditioning robustness. If FFT is fixed but polar vs Cartesian differs strongly: -> representation/codebook geometry is an important part of the effect. ``` None of those outcomes makes the original observation less interesting; they just locate it more precisely. **3. Add one error metric that attention can actually “see.”** K reconstruction error alone may not rank these methods correctly. Even one of the following would be enough for a first pass: ```plaintext QK-logit error attention KL / JS top-attended-token flips attention-output error fixed-sequence NLL ``` Recent work increasingly makes this distinction explicit. [NOVA-KV](https://arxiv.org/abs/2608.04074) defines distortion in terms of attention products, [Block-GTQ](https://arxiv.org/abs/2606.24033) allocates key bits based on their contribution to future RoPE query-key logits, and [AATC](https://arxiv.org/abs/2608.14191) derives an attention-aware distortion objective. In my probe I saw several cases with quite similar K-space reconstruction error but very different attention/final-logit perturbation. So this seems especially relevant here. I would do those three before asking the method to survive a much bigger LongBench/RULER-style evaluation. They are cheap, and almost every possible result tells you _why_ the method is working. > **What I actually tested** > > This was deliberately a small sanity check, not a reproduction of the 40-prompt table. > > The main run used: > > - T4; > - FP32 for the final diagnostic run; > - SmolLM2-135M and Pythia-160M; > - cached/post-RoPE **K-only** interventions; > - V left unchanged; > - fake quantization rather than packed storage; > - a few fixed texts; > - one-step attention/logit diagnostics; > - one short 16-token greedy-generation smoke test on SmolLM2. > > For the generation smoke, I got: > > | K intervention | Matching reference tokens | > | --- | --- | > | baseline | 16 / 16 | > | simple raw 4-bit | 0 / 16 | > | full FFT + mag4 + exact phase | 16 / 16 | > | full FFT + mag4 + quantized cos/sin at 8 bits each | 12 / 16 | > | rFFT + mag4 + direct 8-bit angle | 16 / 16 | > | Hadamard + 6-bit coefficients | 12 / 16 | > > I would not read much into the exact ranking from one 16-token sample. The useful part for me was simply that the qualitative phenomenon survived an independent small probe: **the raw low-bit perturbation could be destructive while transform-domain variants remained much more stable.** > > That is why I think this is worth mechanism controls rather than dismissal. > **Fixed-rate magnitude/phase allocation was more informative than just sweeping phase bits** > > One thing I found particularly useful was keeping the rate fixed. > > For an `rFFT + magnitude + direct angle` control, I held the spectral payload at 12 bits per unique complex coefficient: > > | Allocation | SmolLM2 K NRMSE | SmolLM2 attention JS | SmolLM2 final-logit rel. RMSE | > | --- | --- | --- | --- | > | mag4 + phase8 | 0.0344 | 0.00091 | 0.0182 | > | mag5 + phase7 | **0.0215** | **0.00037** | 0.0130 | > | mag6 + phase6 | 0.0294 | 0.00059 | **0.0100** | > > That is a very small sample, but it illustrates why I think a matched-rate sweep is valuable. > > It confirms the intuitive part — phase cannot simply be made arbitrarily coarse — while also showing that: > > ```plaintext > phase is important > > ``` > > does not automatically imply > > ```plaintext > 4 bits magnitude + 8 bits phase is the optimal allocation > > ``` > > Pythia made this even clearer: the best allocation depended on whether I ranked methods by K reconstruction, attention distortion, or final-logit distortion. > > That seems consistent with a wider pattern in KV quantization: sensitivity is heterogeneous across model, layer, head, K/V side, and quantizer. [KVTuner](https://arxiv.org/abs/2502.04420) uses layer-wise sensitivity, [RateQuant](https://arxiv.org/abs/2605.06675) shows that even the distortion-rate curve depends on the quantizer, and [TurboAngle](https://arxiv.org/abs/2603.27467) uses asymmetric norm/angle precision and layer-specific allocation. > > There are also two different works called PolarQuant that are relevant search neighbors, though neither is the same construction as Fmag: > > - [PolarQuant: Quantizing KV Caches with Polar Transformation](https://arxiv.org/abs/2502.02617) uses random preconditioning followed by polar/angular quantization. > - [PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Acceleration](https://arxiv.org/abs/2502.00527) groups RoPE-coupled key dimensions into 2-D radius/angle pairs. > > So I think “angular/directional information deserves different treatment from radial information” has useful neighboring precedent, without implying that the exact Fourier magnitude/phase decomposition here has already been done. > **Why I would control the basis and the complex representation separately** > > There are actually two different questions here: > > ```plaintext > Which transform? > > ``` > > and > > ```plaintext > How are transformed coefficients represented/quantized? > > ``` > > They can be separated. > > For the first: > > ```plaintext > raw > FFT > DCT > Hadamard > fixed random orthogonal > > ``` > > is enough to test whether FFT is exceptional or whether transforming the coordinates generally makes the quantizer happier. > > [Codec-Gauge](https://arxiv.org/abs/2607.20538) is a useful connection here because it explicitly treats KV channel basis as a compression variable and evaluates random/Hadamard/DCT/PCA-type controls around fixed compression backends. > > For the second, hold FFT fixed and compare something like: > > ```plaintext > magnitude + phase > vs > real + imaginary > > ``` > > In my small probe, this second distinction was surprisingly large. A same-payload full-FFT Cartesian control reconstructed the cache much more accurately than the literal magnitude/cos/sin representation. > > That does **not** mean Cartesian is necessarily the better practical codec. It only means that part of the observed rate-distortion behavior may come from the coefficient representation/codebook rather than Fourier phase alone. > > This is also why I would be cautious about importing the classic image-processing intuition that “Fourier phase carries structure” too literally. In an image, the transformed axes have a natural spatial ordering. A Transformer head dimension does not obviously have the same semantics. > > I tried random channel permutations as a sanity check. Across multiple permutation seeds I did **not** get evidence that the native channel order was uniquely favorable to the FFT. What did remain true was that similar K reconstruction errors could produce quite different model-visible errors. > > So at this point I would treat “the native channel ordering contains a special Fourier geometry” as an interesting possible outcome of a control, rather than an assumption needed to explain the current result. > **The finite-phase implementation and the physical codec may be worth separating** > > There is one representation detail that I think is easiest to handle by separating the experimental question from the storage question. > > The exact-phase experiment is conceptually clean: > > ```python > z = fft(K) > mag = quantize(abs(z), 4) > z_hat = mag * exp(1j * angle(z)) > K_hat = ifft(z_hat) > > ``` > > That tests the basic observation very directly. > > For finite phase, the README-style version quantizes something equivalent to: > > ```python > cos_q = quantize(cos(angle), p) > sin_q = quantize(sin(angle), p) > > ``` > > separately. > > That is a perfectly reasonable fake-quant experiment, but it has two interpretation consequences: > > 1. `cos_q` and `sin_q` are two stored quantities if taken literally as a codec; > 2. independent quantization generally moves the point off the unit circle, so it perturbs effective magnitude as well as angle. > > So I would distinguish: > > ```plaintext > "p-bit phase precision" as an experimental idea > > ``` > > from > > ```plaintext > "p physical bits for phase" as a storage representation > > ``` > > rather than making one stand in for the other. > > A particularly clean physical-codec candidate seems to be: > > ```plaintext > real K > -> rFFT > -> 4-bit magnitude index > -> one 8-bit phase-angle index > -> irFFT > > ``` > > Because K is real-valued, the FFT is Hermitian-symmetric. PyTorch’s [`torch.fft.rfft`](https://docs.pytorch.org/docs/main/generated/torch.fft.rfft.html) explicitly stores only the non-redundant one-sided spectrum. > > For `head_dim = 64`, that gives 33 unique bins. Ignoring metadata for a moment: > > ```plaintext > (4 magnitude bits + 8 phase bits) * 33 / 64 > = 6.1875 payload bits per original K scalar > > ``` > > Relative to BF16’s 16 bits, that is about a 61% payload reduction. As the head dimension grows, the one-sided-spectrum overhead approaches one half, so the asymptote is about: > > ```plaintext > 6 bits / original scalar > => 62.5% below BF16 > > ``` > > So I do **not** think the ~62% number needs to be discarded. Rather, it becomes much easier to interpret if the intended storage format is stated explicitly. > > For example: > > ```plaintext > scientific probe: > full FFT + magnitude quantization + exact phase > > finite-phase ablation: > explicitly defined fake-quant representation > > physical codec: > rFFT + packed magnitude index + packed angle index + metadata > > ``` > > Those can all support the same research direction without needing to be the same representation. > > One more scope detail: if this is currently a **K-only** codec, that percentage is a K-cache figure. A full KV-cache memory figure additionally needs a V representation. If Fmag is intended for both K and V, then specifying the V path separately would make the whole-cache number unambiguous. > **Why attention-visible distortion seems especially useful here** > > I would avoid relying too heavily on raw K MSE/NRMSE as the mechanism test. > > Attention only observes K through its interaction with Q, and different error directions can have very different effects even at nearly identical Euclidean error. > > A cheap diagnostic ladder would be: > > ```plaintext > K reconstruction error > | > v > QK-logit error > | > v > softmax / attention-distribution error > | > v > attention-output error > | > v > fixed-sequence NLL / final logits > > ``` > > You do not need all of these. Even one extra level is informative. > > This also avoids having to lean too hard on the explanation that softmax “normalizes” magnitude scaling. Multiplying K changes the scale of the QK logits, and multiplicative logit scaling generally changes the softmax distribution. So I would leave the softmax mechanism open and measure the downstream distortion directly. > > [NOVA-KV](https://arxiv.org/abs/2608.04074) is particularly relevant: it formulates KV quantization as transform coding where the distortion is error in the attention products rather than ordinary cache reconstruction error. > > [AATC](https://arxiv.org/abs/2608.14191) similarly derives an attention-aware distortion and allocates rate around that objective. > > [Block-GTQ](https://arxiv.org/abs/2606.24033) is another useful comparison because under RoPE it explicitly decomposes a key’s future logit contribution into 2-D frequency blocks and spends more bits on the blocks that matter more to QK accuracy. > > My small experiments made this distinction fairly concrete. I had cases where K reconstruction errors were very similar while attention and final-logit errors were materially different. Pythia was an extreme example: relatively small perturbations could produce large changes in individual attention distributions while the final-logit difference remained much smaller. > > That seems like useful evidence that the “right” distortion measure is itself part of the research question. > **Pythia may be a more interesting architecture control than it first appears** > > One architecture detail may be worth updating before drawing a rule from the Pythia result. > > The published [`EleutherAI/pythia-160m` config](https://huggingface.co/EleutherAI/pythia-160m/blob/main/config.json) has: > > ```json > { > "rotary_pct": 0.25, > "use_parallel_residual": true > } > > ``` > > So Pythia-160M is not a clean “learned absolute position embeddings instead of RoPE” control; it uses **partial RoPE**. > > I actually think that makes the failure case more useful, because the architecture branch can become: > > ```plaintext > full RoPE > vs > partial RoPE > > and/or > > pre-RoPE K > vs > post-RoPE/cached K > > ``` > > rather than simply: > > ```plaintext > RoPE > vs > no RoPE > > ``` > > There is precedent for treating this boundary as important. [KVQuant](https://arxiv.org/abs/2401.18079) explicitly uses **pre-RoPE key quantization** to avoid some of the quantization difficulty introduced after rotation. > > And [Block-GTQ](https://arxiv.org/abs/2606.24033) goes further by treating the native 2-D RoPE blocks themselves as non-uniformly sensitive quantization units. > > That gives another possible low-cost control: > > ```plaintext > FFT/global head-dim representation > vs > native RoPE 2-D pairs/blocks > > ``` > > I would not infer from one Pythia result that partial RoPE, LayerNorm, parallel residuals, or any other single architecture feature is the cause. There are too many family differences at once. > > But Pythia looks like a useful **partial-RoPE branch** for narrowing that question. > **K-only diagnostics and a full KV codec are different stages** > > I do not think a K-only experiment is a problem by itself. > > For mechanism isolation it can actually be useful, because it asks a simpler question: > > > How much can I perturb cached keys before the attention routing changes? > > What I would keep separate is the later claim about an end-to-end KV-cache codec. > > There is a long-standing reason to be cautious about assuming that the same rule should apply to K and V. [KIVI](https://arxiv.org/abs/2402.02750) found different distributional behavior and used per-channel quantization for K but per-token quantization for V. More recent methods also often find K/V asymmetry or layer-specific sensitivity. > > So a clean progression could be: > > ```plaintext > Stage 1: > K-only mechanistic probe > > Stage 2: > K codec + independent V baseline/control > > Stage 3: > joint K+V memory/quality accounting > > Stage 4: > packed serving implementation > > ``` > > There is no need to do all four before the first result is interesting. > **A small reproducibility note** > > I may simply be following a different experiment path/version, so I would treat this as a navigation note rather than a result issue. > > The [`algebraic_kv_tests.py`](https://github.com/ntrillard/kv-findings/blob/main/algebraic_kv_tests.py) path I followed behaves like a small fixed-prompt, K-only diagnostic, while the forum post reports the larger 40-prompt comparison and the finite-phase tables. > > If the 40-prompt / Std4-3-2 / phase-6-8 results came from another runner or commit, pinning that exact script plus its raw output would make the result substantially easier for other people to reproduce or extend. > > Something as simple as: > > ```plaintext > experiments/ > reproduce_hf_table.py > prompts.json > results.csv > > ``` > > plus the model revision / Transformers version would probably be enough. > > Again, it is entirely possible that this already exists somewhere I missed; I am mostly mentioning it because the result seems interesting enough that a single canonical reproduction path would be useful. > **Related work I found nearby** > > I did not find the exact construction > > ```plaintext > head_dim FFT > -> low-bit Fourier magnitude > -> preserved/high-precision Fourier phase > > ``` > > in the papers I checked. > > I did find several nearby branches that seem useful for positioning/searching: > > ### Polar / angular KV representations > > - [PolarQuant: Quantizing KV Caches with Polar Transformation](https://arxiv.org/abs/2502.02617) — random preconditioning followed by polar/angular quantization. > - [PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Acceleration](https://arxiv.org/abs/2502.00527) — radius/angle representation of 2-D RoPE-coupled key dimensions. > - [TurboAngle](https://arxiv.org/abs/2603.27467) — angle quantization in a randomized Hadamard domain with asymmetric norm precision and layer-sensitive allocation. > > These are not Fourier-phase methods, but they are close to the broader idea that radial and angular information need not receive the same representation or precision. > > ### Basis / transform choice > > - [Codec-Gauge](https://arxiv.org/abs/2607.20538) — treats the cache coordinate basis itself as a post-training compression variable. > - [NOVA-KV](https://arxiv.org/abs/2608.04074) — attention-preserving transform coding. > - [AATC](https://arxiv.org/abs/2608.14191) — attention-aware transform coding and rate allocation. > > ### RoPE-aware allocation > > - [Block-GTQ](https://arxiv.org/abs/2606.24033) — allocates key bits over native 2-D RoPE frequency blocks according to attention-logit sensitivity. > - [KVQuant](https://arxiv.org/abs/2401.18079) — among other things, explicitly separates pre-RoPE key quantization. > > ### FFT already appearing in adjacent KV quantization work > > - [VidKV](https://arxiv.org/abs/2503.16257) is a VideoLLM method and a rather different setting, but it does combine FFT with ultra-low-bit key-cache quantization for selected channels. > > Because of that, I would probably avoid a broad claim such as “the first use of FFT/Fourier transforms in KV-cache quantization.” > > But I have **not** found the exact Fmag magnitude/phase factorization above, so there may still be a much narrower novelty claim there. I would just phrase it as an exact-method literature-search question rather than a settled novelty judgment. > **I would leave kernel work until after the representation question is settled** > > If the goal eventually becomes a production cache, I would separate three numbers: > > ```plaintext > 1. algorithmic/fake-quant quality > 2. physical packed bytes > 3. end-to-end latency / throughput > > ``` > > They do not automatically move together. > > The current Hugging Face [KV-cache documentation](https://huggingface.co/docs/transformers/kv_cache) explicitly notes that a quantized cache can reduce memory while _hurting latency_ at short context lengths when enough VRAM is available. > > For Fmag there is an additional transform/decode cost, so eventually the useful systems question becomes something like: > > ```plaintext > saved memory bandwidth > versus > FFT / inverse-FFT + unpack/dequant overhead > > ``` > > possibly with fusion. > > But I would not make that the next blocker. First I would establish which representation and which distortion objective are actually worth implementing. A CUDA/Triton kernel for the wrong codec would be much more expensive than another tiny matched-rate control. So if I had to reduce all of this to one default next route, it would be: ```plaintext 1. Keep the basic Fmag observation. 2. Compare magnitude/phase allocations at fixed total rate. 3. Add one non-Fourier transform and one FFT Cartesian control. 4. Measure one attention-visible distortion metric. 5. If the effect survives those controls, then choose the physical rFFT/phase representation. 6. Only then spend effort on K+V packing and kernels. ``` That route seems relatively cheap, and every branch gives useful information: - if FFT remains special, the Fourier interpretation gets stronger; - if several transforms work, the finding broadens into a transform/preconditioning result; - if angle/direction consistently deserves more rate, the phase story gets stronger; - if optimal allocation varies by model/layer, that points toward adaptive precision; - if K-space error and attention-space error rank methods differently, the attention geometry becomes part of the mechanism; - if the compact `rFFT + magnitude + angle` version retains the effect, the observation also has a fairly clean path toward a real codec. That seems like a good position to be in: the initial observation does not need to be weakened in order to test the alternatives — the controls mostly tell you **what kind of finding it is**. ---
**Author:** ![ntrillard](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@ntrillard](https://discuss.huggingface.co/u/ntrillard)\ **Post date:** [August 19, 2026, 3:59pm UTC](https://discuss.huggingface.co/t/fourier-magnitude-kv-cache-quantization/178815/3 "2026-08-19T15:59:29Z")
Hi, some more experimenting: > Update on the Fmag4 replication / ablation > We fixed a token-comparison bug in the evaluation scripts. The old code stripped the prompt from decoded text with text[len(prompt):] and re-tokenized, which is fragile because tokenizers can add/remove whitespace. The corrected code compares generated token IDs directly (gen\_ids[prompt\_len:]). All numbers below are post-fix. > All mechanism/isolation tests below come from experiments/mechanism\_controls.py and are K-only by default (V is left untouched) unless explicitly labeled K+V or V-only: > MODEL\_ID=“google/gemma-3-1b-it” MAX\_NEW=60 python3 experiments/mechanism\_controls.py > > 1. Is it really about preserving exact phase? > Not on this prompt set. Quantizing phase at 7 bits beats keeping it exact at the same magnitude budget: > Method Match% > rFFT mag5+phase7 (quantized phase) 95.3% > rFFT mag5+exact phase (global) 85.3% > rFFT mag5+exact phase (per-frequency) 85.2% > So the “phase must stay exact” story is not supported here. > 2. Is the Fourier transform special, or is any orthogonal preconditioning enough? > Other fixed transforms are competitive; FFT is not uniquely superior: > Method Match% > rFFT mag5+phase7 95.3% > raw 6-bit 94.7% > DCT coefficients 6-bit 85.3% > full FFT mag4+cos/sin8 85.2% > Hadamard coefficients 6-bit 80.5% > FFT Cartesian real/imag 6-bit 80.5% > DCT and raw 6-bit are in the same ballpark as the exact-phase Fmag conditions, so the effect looks more like “quantization in a good basis” than something Fourier-specific. > 3. What is the right magnitude/phase rate split? > The original 4+8 split is not clearly optimal on this prompt set, but the differences are small and the sample is only 20 prompts, so treat the ordering as suggestive rather than definitive: > Split Match% > rFFT mag5+phase7 95.3% > rFFT mag4+phase8 (K-only) 85.2% > rFFT mag6+phase6 80.7% > rFFT mag7+phase5 80.2% > rFFT mag8+phase4 70.7% > 5+7 is the best split among those tested; whether it would stay best on a larger prompt set is unclear. > 4. Do learned/data-dependent transforms help? > They underperform the fixed transforms on the held-out test prompts: > Method Match% > attention-aware per-layer 6-bit 80.7% > learned per-layer 6-bit 80.7% > learned per-head 6-bit 70.8% > Note the low K-NRMSE does not predict good token match — K-space reconstruction error is misleading for ranking methods. > 5. Is the robustness in K or V? > V is extremely robust. Quantizing only V barely changes generation: > Method Match% > V-only rFFT mag4+phase8 90.2% > K-only rFFT mag4+phase8 85.2% > K+V rFFT mag4+phase8 75.7% > The action is in K; V can tolerate strong quantization. > 6. Pure-Fmag ablation and long-generation check > From experiments/fmag\_ablation.py (60 tokens) and experiments/fmag\_longgen\_check.py (150 tokens): > Method Match% > rFFT Fmag6 (per-token), 60 tokens 95.2% > rFFT Fmag6 (global), 60 tokens 92.8% > rFFT Fmag6 (global), 150 tokens 92.6% > rFFT Fmag4 (per-token), 60 tokens 85.3% > Commands: > MODEL\_ID=“google/gemma-3-1b-it” MAX\_NEW=60 python3 experiments/fmag\_ablation.py > MODEL\_ID=“google/gemma-3-1b-it” MAX\_NEW=150 python3 experiments/fmag\_longgen\_check.py > The 150-token result for Fmag6 global (92.6%) is close to its 60-token result (92.8%), so the result is not a short-generation saturation artifact. > Bottom line > The original 96.9% is not reproduced by the literal 4-bit exact-phase Fmag4 recipe on our 20-prompt set. The closest conditions are: > > - rFFT mag5+phase7 at 95.3% (quantized phase, not exact phase) > - raw 6-bit K quantization at 94.7% > - Fmag6 per-token at 95.2% > This weakens the two core claims we tested: (1) that phase must be kept exact, and (2) that the Fourier transform is uniquely responsible for the robustness. The effect looks more like “a well-scaled 5–6 bit polar/orthogonal representation” plus likely prompt-set dependence. > Caveats: 20 prompts is small; fine-grained rankings (e.g. 94.7% vs 95.3%) can flip with a few token differences or a different prompt set. > All scripts, raw JSON outputs, and READMEs: ntrillard/kv-findings. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [August 20, 2026, 11:59pm UTC](https://discuss.huggingface.co/t/fourier-magnitude-kv-cache-quantization/178815/4 "2026-08-20T23:59:53Z")
After running a few additional experiments, my best read is probably: * * * The token-ID comparison fix looks right to me, and I think it is worth keeping that separate from the issue below. The old decoded-text slicing/re-tokenization problem is fixed; what I ran into next is a different question about the generation/cache harness used by the mechanism controls. My short answers to the six questions would now be: 1. **Does phase need to stay exact?** I would put this back in the **open** category. I would not currently treat `mag5+phase7 > exact phase` as established. 2. **Is Fourier special?** Probably **not uniquely** , but I also would not jump from that to “the basis barely matters.” A more interesting interpretation may be: **the coordinate basis / preconditioning matters, but Fourier is only one useful basis**. 3. **What is the right magnitude/phase split?** I would treat `5+7` as provisional until the generation invariant below passes. The cheap rerun is only `4+8 / 5+7 / 6+6 / 7+5 / 8+4`, plus one matched exact-phase anchor. 4. **Do learned/data-dependent transforms help?** I think this should be split into two questions. The ordinary learned-transform ranking needs the corrected generation path; the attention-aware/adaptive calibration paths also seem to have a separate calibration-state issue, so I would not read their current underperformance as evidence against learned transforms in general. 5. **Is the sensitivity mainly in K or V?** K being the more delicate side still looks plausible, but I do not think the current numbers cleanly establish the size of the effect. One corrected `K-only / V-only / K+V` test with a representative codec would probably answer most of this cheaply. 6. **Does the effect survive longer generation?** I would no longer use the current `92.8% -> 92.6%` result as evidence that error does not accumulate. A corrected run showed meaningful length sensitivity. I also would not claim the opposite extreme (“errors necessarily compound monotonically”) from one small run. The main reason is that the current custom generation loop appears to mix **the codec intervention** with **a different cache lifecycle**. I think separating those two things makes the controls more useful rather than less useful. ### The smallest route I would try first I would make one invariant non-negotiable: ```python assert identity_custom_generation == model.generate(do_sample=False) ``` Then define the intervention contract explicitly: ```plaintext full prompt ↓ prefill exactly once choose generated token #0 from the ordinary prefill logits ↓ apply the intended codec to the stored prompt KV ↓ feed only newly generated/unprocessed tokens ↓ preserve cache history according to the model's cache semantics ↓ quantize newly appended KV according to the codec's online update rule ``` After that, I would initially rerun only: ```plaintext identity raw6 DCT6 rFFT mag5+phase7 rFFT mag5+exact-phase with matched magnitude scaling Fmag6 ``` If the qualitative ordering survives, then the rate split, K/V split, learned transforms, and longer runs are worth expanding. If it changes substantially, I would update the mechanism interpretation before paying for the full grid. I would also separate four layers in the experiment description: | Layer | What is being specified | | --- | --- | | **Representation / transform** | raw, FFT-polar, DCT, Hadamard, PCA/KLT, learned rotation, etc. | | **Quantizer / rate** | bit allocation, scale granularity, metadata | | **Online cache policy** | what gets quantized when new KV arrives; whether old KV is requantized | | **Evaluation** | identity invariant, fixed-prefix distortion, autoregressive token match, long-run quality | That separation is the part I think makes the mechanism story much easier to interpret. > **Why I think the cache lifecycle is the first thing to isolate** > > Looking at the current `97e6d06` generation paths, I see two operations that are independent of the quantizer. > > First, the full prompt has already populated the cache, but intervention generation starts by feeding the final prompt token again: > > ```python > out = model(ids, use_cache=True) > cache = build_cache_from(out.past_key_values) > > # final prompt token is already in that cache > nid = ids[:, -1:] > out = model(nid, past_key_values=cache, use_cache=True) > > ``` > > Second, after decoding, the scripts rebuild a fresh cache containing only the newest K/V slice: > > ```python > pk = list(out.past_key_values) > cache = DynamicCache() > > for li in range(n_layers): > k_new = pk[li][0][..., -1:, :] > v_new = pk[li][1][..., -1:, :] > cache.update(k_new, v_new, li) > > ``` > > So the previous prompt/generated history is no longer represented in the next cache object. > > That differs from the ordinary custom-generation contract described in the [Transformers caching documentation](https://huggingface.co/docs/transformers/cache_explanation): cached past is reused, newly computed K/V is added, and after a prefilled cache the model should receive tokens that have not already been processed. `cache_position`/attention-mask state also needs to describe the actual accumulated context. > > There is a Gemma 3-specific reason not to reduce this to “all tensors must simply grow forever.” Gemma 3 mixes full and sliding attention, so the correct invariant is: > > > **preserve history according to each layer’s cache semantics** > > rather than requiring every physical cache tensor to have the same monotonic length. > > Current Transformers also creates a config-aware cache for Gemma 3: > > ```python > if use_cache and past_key_values is None: > past_key_values = DynamicCache(config=self.config) > > ``` > > and derives subsequent positions from the cache’s logical sequence state. See the [Gemma 3 implementation](https://github.com/huggingface/transformers/blob/main/src/transformers/models/gemma3/modular_gemma3.py) and [cache implementation](https://github.com/huggingface/transformers/blob/main/src/transformers/cache_utils.py). > > So I would preferably retain the cache object produced by the model and modify only the K/V entries the codec intends to modify, instead of rebuilding all layers as a generic unconfigured cache. > > Importantly, I do **not** think this means all of the mechanism work should be discarded. > > The transform definitions, fake-quantization experiments, reconstruction measurements, and choice of controls are separate. The narrower implication is that **generation-dependent quantities** — token-match percentages, first divergence, and the 60/150-token comparison — should be rerun after an identity loop reproduces ordinary generation. > **Small identity/runtime check I ran** > > I tried to answer one question before touching any codec: > > > If the quantizer is the identity, does the custom loop reproduce ordinary greedy generation? > > I used `unsloth/gemma-3-1b-it`, loaded directly with standard Transformers rather than an Unsloth-patched runtime. This was Transformers 5.14.1 on a T4/FP16, so I would not use fine-grained percentages as replacements for your BF16 results. > > The useful part is the causal isolation. > > I got: > > ```plaintext > model.generate(do_sample=False) > == > full-prompt-prefill identity custom loop > == > prefix-prefill identity custom loop > > ``` > > token-for-token on the identity probes. > > Then I separated the two cache changes: > > ```plaintext > repeat the already-cached final prompt token only > -> can change generation > > discard prior cache history only > -> changed generation on all 4 identity probes > > repo-style combination of both > -> changed generation on all 4 identity probes > > ``` > > No quantizer was involved. > > That is the result I find most informative, because the harness difference appears **before** asking whether Fmag, DCT, phase quantization, etc. are good or bad. > > For the corrected codec run I also imposed: > > ```plaintext > 1. full prompt is prefetched exactly once > 2. generated token #0 is selected before any KV intervention > 3. token #0 must equal the baseline token > 4. stored prompt KV is then transformed/quantized > 5. subsequent forwards receive only the new token > 6. prior cache history is retained > > ``` > > All tested codec conditions passed the token-0 guard on all 20 prompts. > **What happened in a small corrected rerun** > > I would treat these as a **directional runtime control, not replacement numbers for your table**. > > The environment differs from yours (including T4/FP16 versus the requested BF16 path), so the important observation is whether the qualitative conclusions are invariant to fixing the cache loop. > > For 20 prompts × 60 generated tokens I got: > > | Condition | Token match | > | --- | --- | > | identity | **100.0%** | > | DCT6 | **91.9%** | > | Fmag6 per-token | **88.3%** | > | Fmag6 global, append/new-only interpretation | **86.6%** | > | Fmag6 global, full-history requantization | **81.2%** | > | mag5 + exact phase, matched per-token magnitude scaling | **69.0%** | > | Fmag4 per-token | **58.0%** | > | mag5 + phase7 | **54.5%** | > | raw6 | **51.9%** | > > Again, I would not call these the “correct numbers.” > > What seems useful is: > > - the old qualitative ordering was **not invariant** to fixing the loop; > - I did not reproduce `mag5+phase7 > exact phase` under the matched-scale control; > - I did not reproduce raw6 staying near the strongest transformed conditions; > - DCT remained strong. > > So I currently read this less as: > > ```plaintext > Fourier has unique magic > > ``` > > or: > > ```plaintext > the basis barely matters > > ``` > > and more as: > > ```plaintext > a useful coordinate basis / preconditioner may matter, > while Fourier itself may not be unique > > ``` > > That is broadly compatible with other quantization work where rotations/preconditioning make low-bit representations easier. For example, [QuaRot](https://proceedings.neurips.cc/paper_files/paper/2024/hash/b5b939436789f76f08b9d0da5e81af7c-Abstract-Conference.html) uses rotations to reduce quantization difficulty, while [SpinQuant](https://proceedings.iclr.cc/paper_files/paper/2025/file/e5b1c0d4866f72393c522c8a00eed4eb-Paper-Conference.pdf) reports that different rotations can produce meaningfully different quantized performance and learns the rotation. > > Those papers do not prove anything about FFT vs DCT here; they just make “basis choice is a real variable” a plausible direction to preserve. > **How I would revisit the six mechanism questions** > > ### 1. Exact phase > > I think two questions should be separated: > > ```plaintext > mechanism: > what happens if phase error is removed? > > codec: > at a finite total budget, how should bits be split? > > ``` > > An exact-phase condition is useful as a mechanism upper bound, but it is not a physically matched low-bit codec unless the cost of storing exact phase is included. > > The cheapest clean comparison seems to be: > > ```plaintext > mag5 + exact phase > mag5 + phase7 > > ``` > > with **identical magnitude scaling/granularity**. > > My matched-scale corrected check went in the exact-phase direction, but that is not directly the same condition as the current exact/global or exact/per-frequency rows. So I would stop at: > > > **phase precision is still an open mechanism question.** > > One conceptual detail may also help: the current FFT is applied with `dim=-1`, i.e. over the **head/channel dimension** , not the token/time axis. I would therefore describe this as a structured channel-basis representation rather than reading FFT phase literally as temporal/context phase. > > That makes FFT, DCT, Hadamard, PCA/KLT, and learned orthogonal transforms much easier to compare as one family. > > * * * > > ### 2. Is Fourier special? > > I think the broader conclusion “FFT is probably not uniquely special” remains plausible. > > What I would hold back is the stronger implication: > > > raw quantization is almost as good, therefore the basis is mostly irrelevant. > > The corrected probe did not preserve that. > > A potentially stronger mechanism question is: > > > **Which coordinate systems make K easier to quantize while preserving what attention actually consumes?** > > That retains the useful DCT/learned-transform controls and connects naturally to rotation/preconditioning approaches such as QuaRot/SpinQuant. > > * * * > > ### 3. Magnitude/phase rate split > > I would not run a large sweep yet. > > Once the identity invariant is fixed: > > ```plaintext > 4+8 > 5+7 > 6+6 > 7+5 > 8+4 > + exact-phase mechanism anchor > > ``` > > should be enough to recover the shape. > > There is also a useful distinction between **nominal mechanism rate** and **deployable codec bitrate**. > > Gemma 3 1B has `head_dim=256`; an `rfft` has 129 unique bins. So `5+7` is roughly: > > ```plaintext > 12 × 129 / 256 ≈ 6.05 > > ``` > > coefficient-payload bits per original K scalar, before scale/metadata overhead. That makes raw6 vs 5+7 a sensible payload-matched mechanism comparison. > > Exact phase is different: it is better described as an upper-bound reference than as a 5-bit codec. > > If this eventually becomes a concrete memory-saving claim, scale/zero-point/phase/residual-window overhead can then be counted separately. I would not complicate the current mechanism table with that accounting yet. > > * * * > > ### 4. Learned/data-dependent transforms > > I would split this into: > > ```plaintext > A. ordinary PCA/KLT/learned transforms > B. attention-aware/adaptive calibration > > ``` > > For A, the current result supports: > > > these learned objectives did not win in this implementation. > > I would not yet generalize that to: > > > learned transforms do not help. > > The generation ranking needs the same corrected cache path, and work such as SpinQuant gives independent reason to expect the choice of learned rotation/objective to matter. > > For B, there seems to be another issue: caches from multiple independent calibration prompts are concatenated along the sequence dimension and then evaluated as though they were one attention history, using a token from one prompt. > > Those KV states were produced under separate contexts/positions, so I would instead evaluate the candidate transform **per calibration prompt** and average the objective: > > ```plaintext > for each calibration prompt: > build its valid cache/context > evaluate candidate transform > > average loss across prompts > > ``` > > That turns this from “attention-aware transforms seem weak” into a cleaner question: > > > **what attention-visible objective should a learned transform preserve?** > > This may also fit your observation that raw K-NRMSE is not a very good predictor of generation behavior. Recent attention-aware transform work such as [OSCAR](https://arxiv.org/abs/2605.17757) similarly motivates optimizing structures closer to what attention consumes rather than raw cache reconstruction alone. > > * * * > > ### 5. K vs V > > I would keep the K-sensitivity hypothesis. > > [KIVI](https://arxiv.org/abs/2402.02750) is one useful reference point: it found different K/V distribution behavior and uses different quantization granularities for K and V. > > But I would redo only one corrected three-way comparison: > > ```plaintext > representative codec: > K only > V only > K + V > > ``` > > rather than duplicating it across all transforms. > > There is also a separate diagnostic detail: the current one-step metric call effectively evaluates quantized K with reference V. So a V-only method can show zero one-step V-induced distortion simply because quantized V is not supplied to that metric path. > > That does not invalidate the generation experiment; it just means I would not use the zero V diagnostic as independent evidence. > > * * * > > ### 6. Longer generation > > On the same 5 prompts, one corrected sentinel gave: > > | Condition | 60 tokens | 150 tokens | > | --- | --- | --- | > | Fmag6 per-token | **87.7%** | **70.9%** | > | Fmag6 global, append/new-only | **67.0%** | **63.6%** | > | Fmag6 global, full-history requantization | **50.0%** | **32.1%** | > > Again: directional, not replacement numbers. > > I think this is enough to say: > > > **length sensitivity exists and should be measured again with the corrected cache lifecycle.** > > It is not enough to claim a universal monotonic accumulation law; the different online policies already behave differently. > > A cheap next check is therefore just: > > ```plaintext > identity > one representative Fmag condition > one strong non-Fourier condition > > 60 vs 150 > > ``` > > If that establishes the trend, I would not spend time running every method to 150 tokens. > **One extra design axis: define what 'global' means online** > > I found this surprisingly important. > > Once generation is autoregressive, these are different codecs: > > ```plaintext > A. fixed/global calibration > derive a scale and keep the rule fixed > > B. append-only online > preserve old quantized history > quantize only newly appended KV > > C. full-history requantization > recompute a global scale > requantize old + new KV every step > > D. residual/recent-window policy > keep recent KV at higher precision > quantize older history later > > ``` > > B and C produced meaningfully different behavior in the small probe. > > So I would make the **online cache update policy part of the codec specification** , rather than treating it as an implementation detail. > > This is also why existing implementations often specify a residual/cache policy explicitly; the Hugging Face [KV-cache quantization overview](https://huggingface.co/blog/kv-cache-quantization) is a useful example. > **One evaluation change that may make the mechanism results easier to interpret** > > I think exact token match has two distinct jobs. > > ### Harness correctness > > For identity: > > ```plaintext > custom loop == model.generate() > > ``` > > should be exact. > > This is an excellent regression test. > > ### Lossy-codec quality > > Once greedy generation first diverges, later tokens are conditioned on **different prefixes**. > > So later token mismatch contains both: > > ```plaintext > local codec distortion > + > autoregressive trajectory divergence > > ``` > > Token match is still useful — trajectory stability is a real end-to-end property — but I would not make it the only mechanism metric. > > A cheap three-level setup would be: > > ```plaintext > 1. Harness correctness > exact identity equality > > 2. Fixed-trajectory fidelity > first divergence > next-token logit KL / CE on the same reference prefix > attention JS / attention-output distortion > > 3. End-to-end behavior > autoregressive token match > optionally perplexity/task quality later > > ``` > > For #2, a reference trajectory can be generated once, then both reference and quantized-cache paths receive the **same reference next token** at every step. > > That gives a much cleaner measure of local codec distortion without letting an early greedy branch make all later comparisons apples-to-oranges. > > Since you already have attention/logit diagnostics, this does not need to become a large new evaluation framework. > **Two smaller diagnostic notes** > > These seem secondary, so I would not let them block the main corrected rerun. > > ### V-only diagnostic > > As above, the current metric path does not appear to feed the quantized V into `compute_metrics()`. That makes the V-only zero-distortion result difficult to interpret. > > ### `qk_mse` > > The current proxy applies something like: > > ```python > torch.log_softmax(attention_probabilities) > > ``` > > to values that are already post-softmax probabilities. > > That is not equivalent to recovering the original QK logits up to an additive constant. > > I see three reasonable choices: > > ```plaintext > cheapest: > keep attention JS and remove/rename qk_mse > > still cheap: > compare centered log(attention.clamp_min(eps)) > > most direct: > capture actual pre-softmax attention scores > > ``` > > I would probably take the first or second unless pre-softmax QK distortion becomes central. > > The underlying idea — measure distortion where attention sees it rather than only raw K-space error — still seems useful. ### What I think is still worth preserving Even after all of this, I do **not** think the useful conclusion is “the new controls are invalid.” I would keep pursuing: - **Raw K reconstruction error does not seem sufficient to predict model-visible quality.** - **Fourier does not look uniquely privileged.** - **But the choice of basis/preconditioner may still matter substantially.** - **K/V asymmetry remains a plausible mechanism question.** - **Learned/data-aware transforms remain worth testing once their objective is separated cleanly.** - **Long-generation behavior should be treated as its own axis because it can interact with the online cache policy.** So if I wanted the lowest-cost path from the current repo, I would do: ```plaintext 1. Add identity == generate() as a permanent regression test. 2. Lock down the online cache contract. 3. Rerun only: raw6 DCT6 mag5+phase7 matched mag5+exact Fmag6 4. Add one fixed-prefix next-token/logit metric. 5. If the qualitative ordering survives: do the rate split do one K/V three-way test do 60 vs 150 6. Treat attention-aware/learned calibration as a separate branch. ``` If step 3 preserves the original ordering, most of the current controls can probably stay with updated numbers. If it changes the ordering, I would update the mechanism interpretation before expanding the grid. Either way, I think the mechanism-control direction remains useful. The main change I would make is to turn **cache lifecycle** into an explicit experimental boundary, so the transform/quantizer questions are not carrying generation semantics along with them. ---
**Author:** ![ntrillard](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@ntrillard](https://discuss.huggingface.co/u/ntrillard)\ **Post date:** [August 22, 2026, 3:52am UTC](https://discuss.huggingface.co/t/fourier-magnitude-kv-cache-quantization/178815/5 "2026-08-22T03:52:13Z")
More Tests: > Update: new harness (no custom cache loop), and a sub-2-bit result > > Thanks @John6666 — your two reviews reshaped how we ran everything since. Three things worth reporting. > > ### 1. New harness sidesteps the cache-lifecycle critique by construction > > Instead of patching the custom generation loop in `experiments/mechanism_controls.py`, we rebuilt evaluation around **`model.generate()` itself**, with quantization applied via forward hooks (`k_proj`/`v_proj`) or a patched `apply_rotary_pos_emb` for true post-RoPE cache tests. Baseline and all conditions go through the identical `generate()` path, so the identity invariant you proposed holds by construction rather than by assertion. The harness is `rapid_lab.py`: ~150 registered micro-tests, each hard-capped at ≤10s (later 30s), model loaded once, every run logged to JSONL with per-prompt vectors. > > We also adopted your other suggestions: held-out prompt sets separate from tuning sets, prefix-match alongside exact-match (first-divergence — turned out nearly identical to exact here, so no realignment inflation), effective- bits accounting including anchor overhead with automatic degeneracy flags, and comma-filtered reruns for cheap replication. > > ### 2. Main finding: selective-layer decode anchoring makes sub-int8-bit KV fp16-faithful > > The fragility in low-bit KV is not primarily in prompt tokens or attention sinks — it’s an **autoregressive error snowball starting at the first decoded token**. Protecting the first D _generated_ tokens’ KV in high precision, only on probe-selected sensitive layers, changes the picture: > | Exact match vs fp16 greedy | holdout | hard | ~830-tok ctx | > | --- | --- | --- | --- | > | **Ternary KV {−s,0,+s} g8/g4 (1.58 b total) + anchors** | 100% | 100% | 100% | > | **Sorted-group int2 g8/g4 (2 b total) + anchors** | 100% | 100% | 100% | > | **NF4-K + V-int4-g64 (4.25 b) + anchors** | 100% | 100% | 100% | > | int8 KV reference | 93% | 72% | 71% | > On Qwen2.5-1.5B the same recipe works **once anchors use Qwen’s own sensitivity profile** (a 0.5s logit-drift probe; its profile is disjoint from Gemma’s — layer 0 dominates at ~7× drift, then scattered mid/late layers): 4.25b → 100%, 2 b → 99.7%, 1.58 b → 98.7%, vs int8’s 41.3%. Our initial “sub-2-bit doesn’t transfer to Qwen” was an artifact of reusing Gemma’s layer map. > > Mechanism ablations: protection must target early _decode_ steps (prompt-only protection scores worse than nothing); K/V anchor synergy (each alone ~42%, both 90%); monotone D-response; and a **D-scaling rule** — anchors must cover the generation horizon (D=96 holds 100% at horizon 100 while D=64 gives 97%). > > Long-context check (needle-in-haystack, diverse-sentence haystacks, needles at 15/55/85% depth): at 16K tokens, anchored ternary/sorted-2-bit retrieve at the **fp16 ceiling** , while unanchored variants collapse to 0–1 of 3. Caveat: the 16K mid-depth needle is missed by fp16 itself (base-model limit), and 32K+ exceeds this GPU’s SDPA mask memory. > > ### 3. What this means for the Fourier question > > Consistent with your read: the basis is a real variable but not uniquely Fourier. In the new harness, Hadamard rotation helps uniform int2 (22.8→43.3% at 4/4) but destroys range-mapped codebooks (NF4 92.7→43.3%) — rotation and codebook geometry interact. The strongest surviving claim from the original thread is narrower: rFFT mag5+phase7 remains competitive at matched payload, and magnitude/phase asymmetry is real, but “phase must be exact” and “Fourier-specific magic” are retired — including by our own audit. > > Also retired by us, with numbers in the repo: sliding-window sink protection on short prompts (backfires — it quantizes exactly the fragile early-decoded tokens), k-means-fitted codebooks (lose to fixed NF4 levels; tail preservation is what matters), full-prefill anchoring at short contexts (anchor overhead exceeds savings — now auto-flagged as DEGEN). > > Prior-art note, incorporating your pointers (KIVI, KVQuant pre-RoPE/Nu, KVSink/PFN, RotateKV, KVmix, InnerQ, IntactKV): we found no published KV result below ~2 bits average — the 1.58-bit operating point may be open, but our eval (greedy self-match, small models) can’t establish superiority over those systems yet; an in-harness KIVI proxy is marked inconclusive-by- construction in FINDINGS.md. > > ### Links & repro > > - Harness: [`rapid_lab.py`](https://github.com/ntrillard/kv-findings/blob/main/rapid_lab.py) > - Long-context validator: [`niah_lab.py`](https://github.com/ntrillard/kv-findings/blob/main/niah_lab.py) > - Survived-vs-retired claims: [`FINDINGS.md`](https://github.com/ntrillard/kv-findings/blob/main/FINDINGS.md) > - Evidence trail: 50+ runs in `rapid_lab_outputs/history.jsonl` > - Quick start: `python3 rapid_lab.py --prompts holdout --only both2_dp32_sens,kv_k8_v8` > > Remaining known gaps: fake-quant simulation (bf16 storage, no packed kernels), greedy self-match metric rather than PPL/NIAH-standard suites, 1B-scale models, and the 32K SDPA wall on the 10GB test card. The D-scaling rule and the probe-based layer selection are specified precisely enough to implement in a serving kernel if anyone wants to try. ---
**Author:** ![ntrillard](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@ntrillard](https://discuss.huggingface.co/u/ntrillard)\ **Post date:** [August 22, 2026, 9:19am UTC](https://discuss.huggingface.co/t/fourier-magnitude-kv-cache-quantization/178815/6 "2026-08-22T09:19:10Z")
Some more tests: > `sink_runner`’s `layer_pred` filtered which layers received QUANTIZATION hooks, not which layers received ANCHORS. Every “\_sens/\_l0/\_s0/\_qsens” result therefore left non-anchor layers COMPLETELY fp16 - the selective- layer anchoring claims, depth-redundancy conclusion, minimal-recipe, scale-validation 100%s and their NLL verifications were measuring near-no-op interventions. All such rows are RETRACTED. > > Corrected semantics (quantize ALL layers, anchors only protect) on Gemma holdout: > > - NF4/g64, no anchors: 39.8% @ 4.25b > - NF4/g64 + dp32 anchors ALL layers: 90.3% @ 13.47 eff bits > - NF4/g64 + L0-only anchors: 56.8% @ 4.69 eff bits > - 2-bit sorted + d48 anchors: 25.0% | ternary 6.0% | sign 2.3% > - sliding window s64 (all layers): 100% @ 13.21 eff bits > > Surviving truths: anchoring works (39.8-\>90.3) but only applied to ALL layers; its dp-mode cost SCALES WITH PREFILL (A=L+D), so effective bits -\>16 at long prefill - dp-mode is memory-theater for long prompts. Fixed- size windows (s64) do amortize (eff → ~4.25) and passed 16K retrieval, but their short-ctx fidelity was window-coverage, not compression. Sub-int8-bit KV with int8-level fidelity at true sub-int8 memory is NOT achieved in this repo. int8 KV (93%) remains the honest baseline. > > Systematic discovery campaign using `rapid_lab.py`: ~150 tests, every test ≤10s, 40+ logged runs, audited metrics (effective-bits accounting, prefix-match, held-out prompts, degeneracy flags). > > ## Minimal recipe (post-debunk): anchor layer 0 only > > Sweeping nested anchor subsets revealed a single critical layer: fp16 anchors on **layer 0 alone** (D=48) + binary sign KV {−s,+s} everywhere else gives **100% exact-match** on holdout AND hard sets, fp16-ceiling retrieval at 16K, at **~1.6 effective bits** (90.5% real savings vs bf16 KV). Qwen: same single-layer recipe = 98.0% (vs int8 41.3%). Layer 0 is the highest-drift layer on both models (Gemma 0.08+, Qwen 0.985 - 7x its runner-up), consistent with pivot-token/attention-sink massive activations living in the first layer. Effective-bit floor: ~1.6 at short ctx, → ~1.1 long ctx (single-layer prompt protection amortizes to ~0.57 bits). > > ## Milestone: 100% exact-match to fp16 at sub-4-bit nominal > > `{quant} + sens-layer decode anchoring D=48` scores **100.0% exact-match vs fp16 on every prompt set** (holdout, hard, long-context), including sets where **int8 KV only reaches 72%** : > | Quant (nominal) | holdout | hard | longctx | Eff bits @142142142142 tok | > | --- | --- | --- | --- | --- | > | ternary {−s,0,+s} g8/g4 ( **1.58b total** ) | 100% | 100% | 100% | ~4.5 | > | sorted-group int2 g8/g4 ( **2-bit total** ) | 100% | 100% | 100% | ~4.6 | > | NF4-K + int4-g64 V ( **4.25b** ) | 100% | 100% | 100% | ~5.9 | > | int8 KV reference | 93% | 72% | 71% | 8.0 | > Debunk audit (sign\_d48): quantization verified real (K/V rel-err 0.54-0.63 on non-anchor layers); result reproduced through clean path; all negative controls behaved (no-anchor 3%, references match history). Mechanism: greedy trajectory is set by fragile layers; other 22 layers are depth-redundant for token choice. CORRECTION: dp-mode keeps prompt fp16 on anchor layers permanently → effective-bit floor ~4.2 regardless of nominal bits; nominal “1-bit” applies only as T-\>infinity with prompt-protection removed. Honest headline: ~4.2-4.6 eff bits beating int8 (8.0) on fidelity. > > Horizon boundary: at 100 generated tokens (vs 50) the 2-bit config holds 92.3%, ternary 75.2% — anchor depth D must scale with generation length; exact-100% claims are for horizons ≤ D. > > ## Best results (Gemma-3-1B, held-out prompts, honest effective bits) > | Recipe | Match | Nominal | Eff. bits | Savings | > | --- | --- | --- | --- | --- | > | **2-bit total + selective-layer decode anchors** (`both2_dp32_sens`) | **96.7%** | 2.0 | ~4.35 | 73% | > | Same, long context (~830 tok) | **100%** | 2.0 | ~4.64 | 71% | > | Ternary KV (1.58b) + int8 anchors (`ternboth_a8`) | 92.7% | 1.58 | 2.79 | 83% | > | 1-bit KV (sign) + int8 anchors (`signboth_a8`) | 91.3% | 1.0 | 2.32 | 86% | > | 4-bit total + anchors (`nfv4g64_dp32_sens`) | 91.3–100% | 4.25 | ~5.5 | 59% | > | int8 KV reference | 93.0% | 8.0 | 8.0 | 50% | > Qwen2.5-1.5B with its **own probe-derived anchor layers** {0,5,9,13,15,18} (layer 0 dominates with ~7x the drift of any other): NF4/int4-g64 **100%** , sorted-2-bit **99.7%** , ternary **98.7%** — vs int8’s 41.3%. The initial sub-2-bit transfer failure was an artifact of reusing Gemma’s layer set; the 0.5s sensitivity probe is what makes the recipe model-general. > > ## The winning recipe: Selective-Layer Decode Anchoring > > > Quantize K/V aggressively (sorted-group 2-bit, ternary, or NF4/int4-g64), but keep the KV of the **first N decoded tokens** in high precision **only on the quantization-sensitive layers** (here layers 0–3, 6–7, identified by a 0.5s logit-drift probe). > > Components, each discovered via ≤10s tests: > > 1. **Magnitude-sorted grouping** (g=8 for K, g=4 for V): sort each row descending, quantize groups; outliers share one wide group. Rotation-immune. > 2. **Decode anchoring** : protection must target early _generated_ tokens. Monotone in N (dp2→dp32: 61→90%); protecting the prompt alone _hurts_ (42%) vs protecting prompt+early decode (90%). > 3. **Layer-selective anchors** : fp16 anchors on 6/28 layers cut anchor overhead ~4.7x at equal quality. > 4. **Anchor-precision dial** : fp16→int8→int4 anchors trade 100→81→71% quality against ~1.3 effective bits per step. > > At 1.58-bit nominal (ternary) and 1.0-bit nominal (sign), this is — per our prior-art search — below the published floor for KV quantization (RotateKV/KIVI at 2-bit, KVmix at ~2.2–2.4 avg). > > ## Mechanism findings > > - **Error snowball, not attention sinks** : the fragility lives in early _autoregressive_ steps (closest to token decision boundaries), not in prompt tokens. Sliding-window sink protection on short prompts is counterproductive (quantizes exactly the fragile tokens). > - **Prompt-length ranking instability** : method rankings flip between short and long prompts (sinks hurt NF4 on short prompts 92.7→44%, help on long prompts 33→74%). KV-quant papers evaluating only on short prompts risk inverted conclusions. > - **Tails are sacred** : four independent confirmations that outlier preservation dominates — clipping, k-means-fitted codebooks, scale shrinkage, and error diffusion all collapse; fixed nonuniform codebooks (NF4) with intact range win. > - **Model specificity** : Gemma’s QK-norm pipeline makes K uniquely forgiving. Qwen collapses under most K-quantization without anchors. > - **int8 KV is not lossless** under exact-match on hard prompts (fails on code/arithmetic continuations). ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [August 23, 2026, 10:51am UTC](https://discuss.huggingface.co/t/fourier-magnitude-kv-cache-quantization/178815/7 "2026-08-23T10:51:15Z")
It looks like you’ve pushed the exploration quite a bit further: * * * My short version is: **I think the correction is substantive, but I do not think it removes the interesting signal. It narrows it into something more testable.** What I would carry forward from the current state is less: > Fourier / phase is uniquely special > selective layers are redundant > early decode anchors give a sub-int8 recipe and more: > **the exact K representation being quantized matters a lot, and low-bit perturbation of the K that is actually stored/reused after normalization + RoPE can be genuinely fragile.** I would probably make that tensor/locus distinction the organizing boundary from here. For current Transformers Gemma 3, the K path is essentially: ```plaintext hidden state ↓ k_proj ↓ k_norm ↓ RoPE ↓ past_key_values.update(K, V, ...) ↓ attention ``` You can see that ordering directly in the [Gemma 3 implementation in Transformers](https://github.com/huggingface/transformers/blob/v5.15.0/src/transformers/models/gemma3/modular_gemma3.py). That means a hook on `k_proj` is testing a different object from the post-KNorm/post-RoPE K entering the cache. Once I separated those, the picture became much easier to interpret. My default route from here would be: ```plaintext intervention/storage locus ↓ storage-only mechanism ↓ teacher-forced distributional validation ↓ better K quantizer geometry ↓ joint K/V rate accounting ↓ only then packed bytes / kernels / serving claims ``` The part I found most convincing is that the effect still survives a **post-forward storage-only control** : keeping the forward that creates a K state completely full precision, then demoting only the persistent cached copy, can still alter later logits and later generation. So there does seem to be a real cache-reuse phenomenon left after the retractions. > **What seems to survive the selective-layer correction** > > I am treating the retraction at the start of [post 6](https://discuss.huggingface.co/t/fourier-magnitude-kv-cache-quantization/178815/6) and the current [`kv-findings` repository](https://github.com/ntrillard/kv-findings) as the current state. > > The selective-layer result looks important to keep retracted because `layer_pred` was selecting where quantization happened, rather than selecting where an otherwise all-layer quantization run received anchors. > > So I would separate the old and surviving claims roughly like this: > > | Claim | Where I would put it now | > | --- | --- | > | Fourier is uniquely responsible for the robustness | no longer established | > | phase must be preserved exactly | no longer established | > | selective-layer anchoring gives the sub-int8 result | retracted | > | one/few layers are enough because of depth redundancy | retracted with that experiment | > | old selective-layer NLL/KL verifies the recipe | should be rerun under corrected semantics | > | basis / codebook / representation geometry matters | still plausible | > | K and V behave differently | still well motivated | > | quantization locus matters | strongly supported | > | post-KNorm/post-RoPE K is fragile in this Gemma setup | strongly supported | > | future reuse of quantized cached K can alter generation | survives a storage-only control | > > So I do not read the correction as “the whole thing was an artifact.” I read it as a fairly large change in **what the actual result is**. > **The intervention point now looks like the first thing to name explicitly** > > There is already a useful three-locus clue in the current experiments. > > On the matched Gemma holdout, the rough pattern I get from the existing runs is: > > ```plaintext > NF4-style K at k_proj / pre-KNorm ~44% token-position match > post-KNorm / pre-RoPE ~5% > post-RoPE / cache-entry ~6% > > ``` > > There is an earlier run with different absolute values but the same qualitative split: pre-KNorm is much less destructive than post-KNorm/post-RoPE. > > That makes me reluctant to describe the `k_proj` result as direct evidence for a stored-K codec. > > It also suggests that the big boundary is already present by the time KNorm has been applied; RoPE is not required to produce the collapse. > > I would still stop short of saying: > > > “KNorm is the cause.” > > A safer interpretation is: > > > **KNorm is a plausible error-healing / re-normalization boundary for a perturbation injected before it, and this deserves to be distinguished from perturbing the representation that actually reaches cache storage.** > > That distinction also connects naturally to existing KV-quantization work. [KVQuant](https://arxiv.org/abs/2401.18079), for example, explicitly treats **pre-RoPE Key quantization** as a design choice rather than assuming that all K representations are equivalent for quantization. > **A storage-only control makes the surviving effect much cleaner** > > There was one confound I thought was worth separating. > > If the new K is quantized _inside_ `Cache.update()`, the returned cache tensor is immediately used by the **same forward’s attention**. So that experiment mixes: > > ```plaintext > A. immediate same-forward K perturbation > B. future reuse of that low-bit K as past cache > > ``` > > I therefore tried the stricter version: > > ```plaintext > forward that creates K_t: > K_t stays full precision > Cache.update runs normally > current attention receives an FP copy > > after the update: > only the persistent cached copy of K_t is NF4-reconstructed > > future forwards: > quantized K_t appears only as past cache > > ``` > > The boundary controls were clean. > > Across 294 single-token/all-layer canaries: > > ```plaintext > same-forward logits exact: 294 / 294 > same-forward KL: 0 > same-forward top1 flips: 0 > > ``` > > But once that demoted K became _past_ cache, nonzero effects appeared: > > | First future observation | observations | FP-vs-test top1 flips | mean KL | > | --- | --- | --- | --- | > | +1 | 288 | 16 (5.6%) | ~0.033 | > | +4 | 270 | 12 (4.4%) | ~0.066 | > | +8 | 246 | 13 (5.3%) | ~0.161 | > > These are small-probe numbers, not benchmark claims, but the causal control is the useful part: > > > **future reuse of a low-bit stored K can matter even when the K-creation forward itself is unchanged.** > > The free-generation ordering also behaved as expected: after demoting K at decoded position `t`, divergence never appeared before `t+2`. A canary at `t=48` in a 50-token generation was a useful negative control because there was no later forward left to reuse it, and those outputs remained exact. > > That is why I think there is still a real storage/cache mechanism here after separating the earlier experimental issues. > > The cache implementation details are also worth keeping in mind: Transformers’ [cache implementation](https://github.com/huggingface/transformers/blob/v5.15.0/src/transformers/cache_utils.py) makes a distinction between current/original-precision states and older quantized storage in its quantized-cache designs. That is closer to the storage-only question than quantizing the just-created K before the same forward consumes it. > **I would soften the “early-token snowball” explanation** > > I think there is still something useful in the “autoregressive snowball” picture, but I would phrase it differently now. > > A decode-prefix protection sweep can look very striking: > > ```plaintext > protect no decoded K -> very low match > protect first 16 -> large recovery > protect first 32 -> larger recovery > protect everything relevant -> identity > > ``` > > But increasing the protected prefix changes several things at once: > > ```plaintext > fewer K states are perturbed > + > the first perturbation moves later > + > there is less remaining autoregressive horizon in which it can matter > > ``` > > I tried fixing the perturbation dose instead: exactly one decoded K token, on every layer, with every other K and all V full precision. > > Free generation still became much safer as the canary moved later, but the **one-step local damage on a fixed reference trajectory was not monotonic in position**. There were large sensitivity spikes at later positions too. > > So I would currently say: > > > **early perturbations are dangerous partly because they have more future trajectory to act on, while local K sensitivity itself is strongly state-dependent.** > > That seems more defensible than: > > > “the first N decoded K states are intrinsically special.” > > The long-lag results also look somewhat heavy-tailed rather than like a smooth deterministic error accumulation process: many effects remain tiny, but occasionally a future state/decision is very sensitive. > > So “snowball” can still be a useful intuition for trajectory divergence, but I would not make it a literal monotonic per-step error-growth model yet. > **The decision-margin signal looks real, but narrower than a cache-retention oracle** > > One result I did find quite interesting was next-token decision margin. > > For the full-precision reference trajectory I used: > > ```plaintext > margin = top1_logit - top2_logit > > ``` > > and then injected a single-token K perturbation. > > In the same-forward experiment, low margin predicted an argmax flip quite strongly: > > ```plaintext > AUC using -raw margin ~0.89 > AUC using -normalized margin ~0.90 > > ``` > > Within each prompt, the smallest-margin quartile had roughly a 30% flip rate, while the largest-margin quartile had no flips in that small sample. > > However, the storage-only version changed the interpretation. > > For predicting a **future** storage-induced flip: > > ```plaintext > margin when K_t was created: > AUC ~0.68 > > margin at the later decision actually receiving the perturbation: > AUC ~0.94 > > ``` > > So I would not yet turn creation-time margin into: > > > “retain this K in high precision if its current decision margin is small.” > > The stronger interpretation seems to be: > > > **decision margin is a downstream susceptibility variable: once some perturbation reaches a decision, a narrow top1/top2 gap makes an argmax crossing much more likely.** > > That is still useful mechanistically, but it is different from having an online K-retention oracle. > > It also explains why margin can predict top1 flips very strongly without necessarily predicting the overall norm of the logit-vector distortion equally well. > **A true moving recent-K residual was less encouraging on the long prompt** > > I also wanted to distinguish the earlier `s64`-style behavior from a true moving residual cache. > > The current `sN` implementation is effectively **fixed-prefix protection** : positions near the beginning of the sequence stay protected. It is not the same as: > > ```plaintext > always keep the newest W cached K states in FP > demote older K as they age out > > ``` > > So I tried the latter with storage-only demotion: > > ```plaintext > same-forward K = FP > > persistent K: > newest W -> FP > older -> NF4-reconstructed > > V = FP > > W = 0, 8, 16, 32, 64 > > ``` > > The aggregate token-position matches were: > > | recent FP K window | match | > | --- | --- | > | 0 | 8% | > | 8 | 13% | > | 16 | 25% | > | 32 | 37% | > | 64 | 84% | > > At first glance W64 looks quite good. > > But the per-prompt breakdown changes the picture. > > For two short prompts, the complete cached trajectory was only 62 or 64 K positions, so W64 was literally a **no-demotion control**. > > Several other short prompts had 80–85% of their entire K history still in FP at W64. > > The more informative case was the ~443-token prompt. Its final cache length was about 492, so W64 retained only ~13% of K in FP: > > | W | token-position match | prefix | FP K fraction | > | --- | --- | --- | --- | > | 0 | 12% | 2% | 0% | > | 8 | 4% | 2% | 1.6% | > | 16 | 4% | 2% | 3.3% | > | 32 | 4% | 2% | 6.5% | > | 64 | 6% | 2% | 13.0% | > > So in that example: > > > **keeping only the latest 64 K states full precision did not preserve the information needed from the older prompt K.** > > That makes me hesitant to pursue W128/W256/etc. as the immediate next sweep. If the only way to recover fidelity is to leave most of the history full precision, the residual is becoming a memory workaround rather than a low-bit codec. > > There is also a simple rate warning here. > > With V still at FP16: > > ```plaintext > K4 + V16 > > ``` > > already has a nominal average payload floor of 10 bits/value before metadata. > > The W64 run was around 12.7 nominal K+V bits/value when weighted by final cached-token counts. > > For comparison, the same small runtime control gave an int8 K/V baseline of 8 nominal bits/value, and that baseline was exact on the long prompt that the W64 NF4-K residual case failed. > > Those numbers are not a fair “winner/loser benchmark” between optimized codecs—the quantizers are different—but they are enough that I would not spend much time packaging this particular NF4+FP-residual path as a memory result yet. > **At this point I would change the next axis from anchor width to K quantizer geometry** > > The moving-window result does **not** tell me that low-bit K is impossible. > > It tells me that this particular K quantizer/locus combination is a hard case. > > That distinction matters because mature KV-quantization work has repeatedly found K geometry to be first-order. > > [KIVI](https://arxiv.org/abs/2402.02750) reports a strong K/V asymmetry: > > ```plaintext > K -> per-channel quantization > V -> per-token quantization > > ``` > > rather than treating both tensors with the same grouping direction. > > [KVQuant](https://arxiv.org/abs/2401.18079) goes further and combines several K-specific choices: > > - per-channel Key quantization, > - pre-RoPE Key quantization, > - non-uniform datatypes, > - separate outlier handling. > > That seems especially relevant here because the local Gemma result already says: > > ```plaintext > perturb before the normalization / positional path > != > perturb the representation actually entering the cache > > ``` > > So before increasing the FP anchor or residual budget, I would put the next experimental budget into things like: > > ```plaintext > quantization axis > grouping direction > pre- vs post-RoPE representation > outlier handling > codebook geometry > K/V asymmetry > > ``` > > The broader “representation matters” idea is therefore still interesting even if the original Fourier-specific explanation gets narrowed. > > I would just keep two questions separate: > > ```plaintext > Can a representation be quantized with low model error? > > and > > Can that representation be stored/reused as a practical KV cache format? > > ``` > > Those are not automatically the same result. > **The next validation I would trust more than another greedy sweep** > > At this point I would probably stop adding generation recipes for one round and rebuild the distributional validation under the corrected semantics. > > A small matrix would be enough: > > ```plaintext > FP reference > > true-cache int8 K/V > > storage-only moving: > W0 > W16 > W32 > W64 > > ``` > > using the same reference tokens and measuring: > > ```plaintext > ΔNLL > KL(FP || test) > true FP-vs-test top1 flip > > ``` > > The reason is visible in the long-prompt moving-window result: > > ```plaintext > W0 -> 12% greedy token match > W8 -> 4% > W16 -> 4% > W32 -> 4% > W64 -> 6% > > ``` > > I would not interpret that as “more FP K made the underlying distribution worse.” > > Greedy generation is a trajectory-level measurement: once one argmax changes, every later input can change. > > Teacher forcing answers a different and useful question: > > > given the _same_ prefix, how much did the model’s predictive distribution move? > > That would also give a much cleaner basis for comparing quantizer designs. > > I would regenerate these numbers rather than reuse the earlier selective-layer NLL/KL values, because the old validator path inherited the same layer-selection semantics that were later retracted. > **A few naming/accounting details that may make the evidence trail easier to extend** > > These are mostly bookkeeping, but I think they would help future readers reproduce the experiments. > > ### 1. `exact` vs token-position match > > A metric such as: > > ```python > sum(a == b for a, b in zip(reference, test)) / n > > ``` > > is useful, but below 100% I would call it something like: > > ```plaintext > position_match > token_position_match > > ``` > > and reserve “exact” for: > > ```plaintext > the whole generated token sequence is identical > > ``` > > That also makes tables with “100% prompts” less ambiguous. > > ### 2. `s64` / `sN` > > If `sN` protects the first N sequence positions, I would call it: > > ```plaintext > fixed-prefix protection > > ``` > > rather than a sliding window. > > A true recent residual has different behavior and different long-context scaling. > > ### 3. “effective bits” > > For fake quantization experiments I would use something like: > > ```plaintext > nominal payload bits > anchor-adjusted nominal bits > > ``` > > until there is an actual packed representation. > > Real storage accounting eventually needs: > > - quantized indices, > - scale/min/max or zero-point metadata, > - grouping metadata, > - residual entries, > - padding/alignment, > - any permutation/index information, > - actual allocated bytes. > > ### 4. Reproducibility metadata > > For cache-locus experiments I would log at least: > > ```plaintext > model ID + revision > Transformers version > PyTorch version > GPU > dtype > cache class/config > intervention locus > quantized layers > protected region/window > quantizer parameters > > ``` > > The exact Transformers version matters here because the cache API and model attention implementation define what tensor the hook actually sees. > > ### 5. Keep model artifacts separate > > If a public mirror is used for an independent reproduction because the gated Google artifact is unavailable, I would record that explicitly rather than merge the absolute numbers into the Google-artifact table. > > The qualitative controls can still be useful, but artifact identity should remain visible. > **Related work that now seems especially useful for comparison** > > I would probably keep the literature list relatively short and use it to illuminate the current branches rather than to establish novelty. > > ### KIVI > > [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://arxiv.org/abs/2402.02750) > > The especially relevant observation here is the asymmetric quantization geometry: > > ```plaintext > Key -> per-channel > Value -> per-token > > ``` > > That is a good reminder that “4-bit K” is not a single intervention; grouping orientation matters. > > ### KVQuant > > [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://arxiv.org/abs/2401.18079) > > Relevant pieces: > > - per-channel Key quantization, > - pre-RoPE Key quantization, > - non-uniform datatypes, > - dense-and-sparse/outlier handling. > > The pre-RoPE result is particularly relevant to the large pre/post transformation difference showing up here. > > ### Transformers cache implementation > > For the exact software-side semantics, the two references I would keep nearby are: > > - [Gemma 3 attention implementation](https://github.com/huggingface/transformers/blob/v5.15.0/src/transformers/models/gemma3/modular_gemma3.py) > - [Transformers cache implementation](https://github.com/huggingface/transformers/blob/v5.15.0/src/transformers/cache_utils.py) > > Those answer a surprisingly large fraction of the “what object are we actually perturbing?” questions. If I were choosing the next path by goal, I would use something like this: ```plaintext If the goal is to understand the mechanism: corrected storage-only teacher-forced KL/NLL ↓ locate which older prompt regions / K states matter ↓ test whether sensitivity tracks age, content, attention, or quantizer error If the goal is a practical low-bit cache: stop increasing the FP window for now ↓ improve K quantizer geometry ↓ compare against a strong int8 / KIVI-like baseline ↓ reintroduce a residual only if it buys a real rate-quality improvement If the goal is a deployable codec: first get a fidelity-successful K/V policy ↓ define an actual packed representation ↓ count metadata and real bytes ↓ then measure latency / throughput / context scaling ``` So, from my side, I would not read the corrections as the exploration collapsing. The interesting part seems to have **narrowed from a fairly broad Fourier/anchor story into a cleaner question about K representation, cache locus, and future error propagation**. That is a smaller claim, but it also looks much easier to falsify, compare against existing KV methods, and turn into a reproducible next experiment.