# \[Research/Code\] Verified 384x KV-Cache Compression with Linear Complexity using Finite Scalar Quantization (UL-SMF) **URL:** \ **Category:** Models\ **Created:** [August 13, 2026, 9:48am UTC](https://discuss.huggingface.co/t/research-code-verified-384x-kv-cache-compression-with-linear-complexity-using-finite-scalar-quantization-ul-smf/178623 "2026-08-13T09:48:47Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![liventruth](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/liventruth/32/62296_2.png) [@liventruth](https://discuss.huggingface.co/u/liventruth)\ **Post date:** [August 13, 2026, 9:48am UTC](https://discuss.huggingface.co/t/research-code-verified-384x-kv-cache-compression-with-linear-complexity-using-finite-scalar-quantization-ul-smf/178623/1 "2026-08-13T09:48:47Z")
Hey everyone, As context windows scale to 128k+ tokens, KV cache VRAM consumption becomes a massive bottleneck for real-time inference. I’ve open-sourced the Unified Latent-State Memory Fabric (UL-SMF), a hardware-software co-designed architecture that compresses standard FP32 KV caches into a 16-dimensional discrete latent space using Finite Scalar Quantization (FSQ). Live Benchmark Results (CUDA T4 / FP32): • Compression Ratio: 384x reduction (Reduces a 48 MB FP32 block to 0.12 MB) • Semantic Retention: 94.15% - 95.84% Cosine Similarity on real LLM attention tensors • Pipeline Latency: ~14.1 ms to 19.6 ms end-to-end CUDA event verified latency The self-contained Python package allows you to monkey-patch standard Transformer attention layers with two lines of code. GitHub Repository & Code: [GitHub - liventruth/UL-SMF-Cache-Compression: The Unified Latent-State Memory Fabric (UL-SMF) is a hardware-software co-designed memory compression fabric that solves the memory bottleneck in long-context Transformer inference. By combining FSQ with dynamic 16-dimensional latent mapping, UL-SMF compresses Key-Value (KV) cache tensors by up to 384x while maintaining \>94% semantic retention. · GitHub](https://github.com/liventruth/UL-SMF-Cache-Compression) Technical Specification (Zenodo DOI): [Live Verification: The Unified Latent-State Fabric: Resolving the Inference Memory Wall via GLRP v2.0 and Aegis-KV | Zenodo](https://zenodo.org/records/21896117) Feedback, testing, and benchmarks on longer context models are welcome! Dual-licensed under AGPLv3 with commercial options available. ![1000028149](https://us1.discourse-cdn.com/hellohellohello/original/3X/4/3/43633e5b96ab3647b6d722eae5647e7ff5a4b703.jpeg) ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [August 14, 2026, 2:33am UTC](https://discuss.huggingface.co/t/research-code-verified-384x-kv-cache-compression-with-linear-complexity-using-finite-scalar-quantization-ul-smf/178623/2 "2026-08-14T02:33:34Z")
Hmm… for now, I think I was able to reproduce the codec part: * * * I went through the public [UL-SMF repository](https://github.com/liventruth/UL-SMF-Cache-Compression), the linked [Zenodo verification material](https://zenodo.org/records/21896117), and the trained verification oracle, and then tried to reproduce the parts that I could without inventing an integration path that is not public. The short version is: - the **3072 → 16 → 3072 codec path does appear to work** ; - I reproduced a mean reconstruction cosine of about **0.94145** , essentially matching the reported ~94.15%; - on a T4, the stable 4096-row measurement I got for the trained oracle was about **19.5 ms** , again very close to the reported ~19.6 ms; - the 16-D output was genuinely discrete/integer-valued in the run I checked; - but I think there are two different things worth separating now: 1. **codec-level compression** , which I could reproduce reasonably well; 2. **an actual autoregressive compressed K/V cache** , whose read/write path I could not yet reproduce from the public integration. So I would personally treat the codec result as a useful first result, and then make the live-cache path the next boundary to pin down before doing the longer-context benchmark you asked for. The rough sequence I would use is: ```plaintext codec validation ↓ physical packed representation ↓ actual autoregressive K/V cache read/write path ↓ real resident/peak memory + decode latency ↓ real 4K/8K context smoke test ↓ RULER / LongBench or a similar longer-context evaluation ``` I did **not** count the 4096-row codec test as a long-context-model test, because that would overstate what I actually reproduced. > **What I was able to reproduce** > > ### Codec reconstruction > > Using the trained verification oracle, I reproduced approximately: > > ```plaintext > input: 3072-D FP32 > latent/code: 16-D > reconstruction: 3072-D FP32 > > mean cosine ≈ 0.94145 > > ``` > > That is close enough to the reported ~94.15% that, at least from my run, the reconstruction number does not look like the interesting ambiguity anymore. > > The returned 16-D values were also integer-valued even though the Tensor storing them was floating-point, so there does appear to be a genuine discrete bottleneck rather than just a continuous 16-D autoencoder output. > > This is also conceptually consistent with [Finite Scalar Quantization](https://arxiv.org/abs/2309.15505): the logical discrete representation and the temporary PyTorch dtype used to carry those values do not have to be the same thing. > > ### T4 timing > > For the 4096-row case, a stable repeat measurement on T4 gave me approximately: > > ```plaintext > trained oracle forward ≈ 19.5 ms > > ``` > > which is very close to the ~19.6 ms figure in the verification material. > > I did see noisier timing at some smaller sizes and an outlier during an earlier 4096 measurement, so I would not infer a complete scaling curve from one timing run. But the **~20 ms order of magnitude for the reported 4096-row codec operation itself was reproducible** for me. > > ### Important scope note > > What this reproduces is the codec operation: > > ```plaintext > 3072-D state > ↓ > encoder > ↓ > 16-D discrete state > ↓ > decoder > ↓ > 3072-D reconstruction > > ``` > > It does not, by itself, reproduce: > > ```plaintext > token t > ↓ > write compressed K/V cache > ↓ > token t+1 > ↓ > read compressed K/V cache > ↓ > attention > ↓ > update cache again > > ``` > > I think keeping those two layers separate makes the current result easier to interpret. > **How I am currently reading the 384× figure** > > This was the part I initially found confusing, but I think the numbers make sense if I separate **logical/packed compression** from the Tensor representation currently returned by the verification oracle. > > For one 3072-D FP32 block: > > ```plaintext > 3072 × 4 bytes = 12,288 bytes > > ``` > > The trained oracle returned 16 values as `torch.float32`, so the actual returned Tensor occupies: > > ```plaintext > 16 × 4 bytes = 64 bytes > > 12,288 / 64 = 192× > > ``` > > For the 4096-row verification tensor, the same accounting is: > > ```plaintext > raw FP32 input: 48.0 MiB > returned FP32 codes: 0.25 MiB > actual Tensor ratio: 192× > > ``` > > The reported 384× follows exactly if the 16 discrete values are stored at **2 bytes each** : > > ```plaintext > 16 × 2 bytes = 32 bytes > > 12,288 / 32 = 384× > > ``` > > or, for 4096 rows: > > ```plaintext > 48.0 MiB / 0.125 MiB = 384× > > ``` > > So my current interpretation would be: > > | Quantity | What it means | What I observed | > | --- | --- | --- | > | Logical reduction | 3072 values → 16 discrete values | reproduced | > | Current oracle Tensor payload | 16 FP32 values | 192× | > | 2-byte packed-code payload | 16 × 2-byte values | 384× | > | Complete live K+V-cache reduction | all layers, K+V, runtime buffers, etc. | not yet reproduced | > > I **would not take this as evidence that 384× is an implausible target**. The code values I saw were discrete, so packing them into a narrower physical representation looks perfectly reasonable in principle. > > There are also other extreme-compression results in the literature. For example, [SVDq](https://arxiv.org/abs/2502.15304) reports up to **410× key-cache compression** under its own method and accounting. What I find useful about that comparison is mostly the labeling: it makes very explicit that the headline ratio is for the **key cache**. > > So if I am reading UL-SMF correctly, perhaps the clean distinction is simply: > > ```plaintext > 384× = packed discrete-code payload target > > ``` > > versus > > ```plaintext > 192× = physical Tensor payload returned by the current verification oracle > > ``` > > until an actual packed storage path is included. > > If there is already an INT16/packed representation elsewhere in the project, then this distinction may already be resolved and I simply have not reached that path. > **What the 3072-D verification block appears to represent** > > I was also able to reconstruct where the `3072` comes from in the GPT-2 verification path. > > For GPT-2 small, each layer’s key state contributes 768 features per token: > > ```plaintext > 12 heads × 64 head dimensions = 768 > > ``` > > The verification construction I reproduced takes the keys from the first four layers and concatenates them: > > ```plaintext > layer 0 K: 768 > layer 1 K: 768 > layer 2 K: 768 > layer 3 K: 768 > ↓ > 3072-D > > ``` > > So the codec input is a **four-layer K-derived representation**. > > The verification example also starts from a short GPT-2 sample and repeats the resulting rows until the tensor reaches 4096 rows. > > I therefore think the safest description of that particular experiment is something like a: > > > **4096-row GPT-2-derived codec stress/throughput test** > > rather than a complete 4096-token autoregressive KV-cache run. > > That distinction does not diminish the codec result; it just tells us what the test establishes. > > In particular, I do not think **K-only** is inherently suspicious. Several KV-cache methods deliberately treat K and V differently. For example, [KIVI](https://arxiv.org/abs/2402.02750) uses different quantization granularities for keys and values, and [SVDq](https://arxiv.org/abs/2502.15304) specifically targets the key cache. > > I just could not tell from the public verification whether UL-SMF has: > > ```plaintext > K → this 16-D codec > V → some separate path > > ``` > > or whether the current public test is intentionally only validating the K side so far. > > That distinction matters mainly when comparing the 384× number against methods reporting the memory footprint of the **complete live K+V cache**. ### The part I would most like to try next The remaining piece for me is not really another codec test. It is the transition from: > **compressed representation** to: > **compressed autoregressive cache**. If there is already a minimal read/write path for the 16-D state somewhere, even a very small generation example would probably be enough to make the longer-context test possible without guessing at the architecture. Something as small as this conceptual loop would be enough to establish the contract: ```plaintext token t produces K/V ↓ UL-SMF state is written ↓ token t+1 reads that state ↓ K/V are reconstructed OR attention consumes the latent directly ↓ attention result ↓ new compressed state is appended ``` There seem to be several possible implementations, and I did not want to arbitrarily choose one and then call the result a reproduction of UL-SMF. > **Why I think the read path is the important next boundary** > > There are at least three reasonable architectures: > > ### A. Reconstruct full K/V before attention > > ```plaintext > compressed latent > ↓ > decoder > ↓ > full K/V > ↓ > ordinary attention > > ``` > > This is close in spirit to work such as [Palu](https://arxiv.org/abs/2407.21118), which caches reduced intermediate states and reconstructs full keys/values on the fly. > > In that case, the storage saving can still be real, but there is a separate **reconstruction/bandwidth tax** to measure. > > Palu therefore also spends effort on GPU kernels and operator fusion rather than treating low-dimensional storage as the end of the systems problem. > > ### B. Fuse/absorb decoding into attention > > ```plaintext > compressed latent > ↓ > attention-compatible transformation > ↓ > attention > > ``` > > This is a more interesting systems path if full K/V never need to be materialized in HBM. > > [CommVQ](https://arxiv.org/abs/2506.18879), for example, does not stop at creating a vector-quantized KV representation: it explicitly designs the codebook so that it behaves compatibly with RoPE and can integrate decoding into self-attention efficiently. > > There are also newer approaches that explicitly optimize around avoiding the cost of rebuilding full K during attention. > > ### C. Reconstruct only what is needed > > A third possibility is to use the compressed representation for lookup/selection and only reconstruct some of the cache. > > Again, all three seem plausible; I just do not know which of them corresponds to the intended UL-SMF runtime. > > That is why I stopped short of writing my own integration. If I choose A/B/C myself, any LongBench/RULER result would primarily measure **my UL-SMF-inspired implementation choice** , not necessarily this project. > **What happened when I tried the public wrapper against the current Transformers cache interface** > > I also did one small integration sanity check. > > On the Transformers installation in my test environment, I wrapped a tiny Llama attention module with the public UL-SMF interceptor and compared its actual returned cache with the unmodified baseline. > > Generation continued to work, including a second autoregressive step, but the actual stored K/V cache shape and byte count remained the same as the baseline. > > So, on that particular current Transformers path, I did **not** observe the wrapper replacing the real stored K/V cache. > > I would interpret that narrowly: > > > the public wrapper did not intercept the active cache-storage contract in the Transformers path I tested. > > I would **not** infer from that that an UL-SMF cache integration cannot work, or that another Transformers version/custom attention implementation would behave the same way. > > The current Hugging Face caching interface makes this distinction fairly explicit. The [Transformers cache documentation](https://huggingface.co/docs/transformers/main/cache_explanation) describes generation roughly as: > > ```python > new_K, new_V = cache.update(k_t, v_t, layer_idx) > attn_output = attention(q_t, new_K, new_V) > > ``` > > and the cache stores K and V per layer. > > That API boundary has also changed over time. [NVIDIA KVPress](https://github.com/NVIDIA/kvpress), which integrates multiple compression methods with Transformers, has had to make breaking compatibility changes when the Transformers KV-cache machinery changed. > > So a **single known-good combination** such as: > > ```plaintext > Transformers version X > model Y > attention backend Z > minimal generation example > > ``` > > would probably be more useful for reproducibility than trying to claim generic integration across every recent version immediately. > > If the currently published interceptor is intended mainly as a reference/architecture example rather than the full production cache adapter, that would also explain what I observed. ### About the longer-context testing This is the part you explicitly asked for, and I think it is worth doing — but I have **not** done a real UL-SMF long-context model test yet. The 4096-row experiment above tests the codec on a large tensor; it is not the same thing as having a model consume a natural 4K context, store the compressed cache, and continue decoding from it. I deliberately stopped there because implementing the missing cache lifecycle myself would make the experiment ambiguous. If there is already a minimal actual-cache path, I think a relatively small validation ladder would give a lot of information before doing a huge benchmark: ```plaintext 1. 2–16 autoregressive decode steps └─ confirm the compressed state is really the state reused later 2. measure actual stored bytes └─ K + V + code/metadata, not just nominal bits 3. measure peak VRAM and decode latency └─ catch temporary reconstruction/workspace costs 4. real 4K-context smoke test 5. real 8K-context smoke test 6. small RULER / LongBench subset 7. only then scale farther ``` That ordering is partly just to avoid spending GPU time on a benchmark before knowing what implementation is being benchmarked. There is some precedent for doing exactly this kind of small-first validation. The [longctx\_bench repository](https://github.com/henryzhongsc/longctx_bench) from the EMNLP 2024 KV-cache compression benchmark provides quick-start subsets specifically to verify an implementation/environment before committing to the full long-context experiments. And [RULER](https://github.com/NVIDIA/RULER) is useful beyond a single Needle-in-a-Haystack score because it covers several types of long-context behavior rather than only simple retrieval. > **Metrics I would keep separate once the cache path is running** > > I think separating the measurements would make the result much easier to compare with other KV-cache work: > > ### Codec metrics > > ```plaintext > logical code size > physical code dtype/packing > reconstruction cosine/error > encode latency > decode latency > > ``` > > ### Cache/system metrics > > ```plaintext > actual resident K+V/cache bytes > temporary reconstruction workspace > peak allocated VRAM > prefill memory > decode-step memory > > ``` > > ### Serving metrics > > ```plaintext > TTFT > inter-token latency > tokens/sec > end-to-end generation time > > ``` > > ### Quality metrics > > ```plaintext > perplexity > QK / attention distortion if useful > retrieval accuracy > RULER / LongBench > long-generation behavior > > ``` > > The distinction matters because a very strong **payload compression ratio** does not automatically imply the same reduction in **peak memory** or the same gain in **throughput**. > > For example, the [Transformers cache documentation](https://huggingface.co/docs/transformers/en/kv_cache) treats memory-saving cache strategies and generation-speed trade-offs separately. > > [NVIDIA KVPress](https://github.com/NVIDIA/kvpress) likewise exposes separate accuracy and speed/memory evaluations, and its own implementations include cases where temporary prefilling state can increase peak memory despite eventual cache compression. > > The systems study [Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving](https://arxiv.org/abs/2503.24000) makes the same broader point: reducing KV-cache storage does not necessarily translate directly into end-to-end throughput if the attention/runtime implementation cannot efficiently consume the compressed representation, and quality changes can alter total generated length as well. > > So I think UL-SMF would actually benefit from keeping these numbers separate rather than trying to collapse everything into one compression figure. > **One terminology point I was not completely sure how to read** > > I was also not sure what scope you intend for **“linear complexity.”** > > If this refers to the UL-SMF **encoding/storage/routing** path growing linearly with the number of cached states, that interpretation seems straightforward to me. > > I would only distinguish that from the complexity of the attention operation itself. > > The current [Transformers cache explanation](https://huggingface.co/docs/transformers/main/cache_explanation) already describes ordinary autoregressive KV-cache storage as growing linearly with context length, while cached autoregressive attention has per-step work proportional to the amount of previous context. > > So I would personally read the claim as something like: > > ```plaintext > O(N) compressed-state encoding/storage/routing > > ``` > > unless there is also a different compressed-attention execution path in UL-SMF that changes the attention computation itself. > > If there is, that would be another particularly interesting piece to see. Overall, I think I came away **more convinced by the codec part than I expected** : the trained 3072→16→3072 path ran, the ~0.9415 cosine was reproducible, and the T4 timing was in essentially the reported range. The remaining ambiguity for me is mostly architectural/accounting rather than whether the codec exists: ```plaintext codec: reproducible from what I tested packed physical code: plausible, but not what the current oracle Tensor stores full live K+V cache: not yet reproduced long-context model: not yet tested by me ``` If an actual compressed cache read/write path already exists somewhere, **that would be the most useful thing for me to try next** , because then I could test the longer-context behavior you originally asked about without making up the missing integration myself. If that path is not public yet, I think the current material still gives a fairly clear and reproducible **codec-stage result** ; I would just keep the packed-payload number and the eventual full live-cache number as two separate milestones for now. ---
**Author:** ![liventruth](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/liventruth/32/62296_2.png) [@liventruth](https://discuss.huggingface.co/u/liventruth)\ **Post date:** [August 14, 2026, 4:35am UTC](https://discuss.huggingface.co/t/research-code-verified-384x-kv-cache-compression-with-linear-complexity-using-finite-scalar-quantization-ul-smf/178623/3 "2026-08-14T04:35:19Z")
John, I appreciate you taking the time to run through the material and independently test the replication. Your breakdown of the validation ladder is spot-on, and it accurately separates the standalone codec validation from the live autoregressive state-transition loop. As you noted, the public repository intentionally establishes the core codec prior art and the 16-D FSQ geometry (~0.9415 cosine at 19.5 ms on T4). The specific autoregressive read/write lifecycle—handling how token t+1 reads the compressed latent state directly—is governed by the dynamic routing and virtual-to-physical memory mapping logic housed within the aegis\_kv\_oracle\_core.pt binary. Because that binary acts as the proprietary integration layer for hardware and enterprise deployment, it is restricted to secured diligence environments. However, your proposed validation sequence (stepping from 2-16 decode steps through to a RULER/LongBench subset) is precisely the validation roadmap currently being executed with hardware integration partners. Thank you for engaging deeply with the architecture and providing such a rigorous technical assessment. Best regards, Charles Clark Lawrence Principal Architect | Lawrence Architectures ---
**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\ **Post date:** [August 15, 2026, 6:16pm UTC](https://discuss.huggingface.co/t/research-code-verified-384x-kv-cache-compression-with-linear-complexity-using-finite-scalar-quantization-ul-smf/178623/4 "2026-08-15T18:16:49Z")
Closed on August 15, 2026, 6:16pm UTC This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.