# KVQuant attention-aware extensions to KV cache vector quantization (paper + code)

**URL:** <https://discuss.huggingface.co/t/kvquant-attention-aware-extensions-to-kv-cache-vector-quantization-paper-code/176091>\
**Category:** Research\
**Created:** [May 19, 2026, 9:16am UTC](https://discuss.huggingface.co/t/kvquant-attention-aware-extensions-to-kv-cache-vector-quantization-paper-code/176091 "2026-05-19T09:16:42Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![syedMohib44](https://avatars.discourse-cdn.com/v4/letter/s/9f8e36/32.png) [@syedMohib44](https://discuss.huggingface.co/u/syedMohib44)\
**Post date:** [May 19, 2026, 9:16am UTC](https://discuss.huggingface.co/t/kvquant-attention-aware-extensions-to-kv-cache-vector-quantization-paper-code/176091/1 "2026-05-19T09:16:42Z")

</div>

Hi everyone,

Sharing my paper KVQuant five structure-aware extensions to KV cache

compression built on top of TurboQuant (Zandieh et al., 2025).

TurboQuant rotates KV vectors then applies Lloyd-Max quantization

near-optimal MSE with provable bounds. But it treats every token the same

and ignores that quantization error has exploitable structure.

**\*\*Five extensions:\*\***

- Attention-weighted bit assignment 47–70% lower weighted distortion

- Delta compression 1.1–2.2x lower MSE on correlated streams

- Adaptive bit allocation EMA tracker, promotes/demotes during generation

- Low-rank error correction rank-4 SVD recovers 96% of 2-bit PPL loss

- Product quantization 2-bit storage matching 3-bit scalar quality

**\*\*Key result:\*\*** 2-bit + rank-4 correction on gpt2-medium drops dPPL from

+173 to +5.95. PQ (M=16, b=8) produces coherent generation where 2-bit

scalar completely collapses.

Paper: [OSF](https://doi.org/10.17605/OSF.IO/9WSKZ)

Code: [GitHub - syedMohib44/kvquant: Attention-aware KV cache quantization for LLM inference · GitHub](https://github.com/syedMohib44/kvquant)