broken link? https://huggingface.co/spaces/neullabs/grite

Good catch, that path is dead. I folded the four standalone demos into one Space to stay under HF's 3-slot cpu-basic quota, so grite moved.

Live here, running over local git data (the grite tab):
https://huggingface.co/spaces/neullabs/agent-infra

Thanks for flagging it. Did it 404 for you, or hang on a cold boot?

Hey Dipankar, noticed you liked my Agent Failure Atlas benchmark recently. Really appreciated that. I’m working on a related independent research submission and wanted to ask you something about arXiv. Is there a good way to reach you privately?

You do not need a private channel for the arXiv part.

Endorsement runs on arXiv's side. They hand the submitter a code and a link, the endorser acts there, and nobody needs your email. Whether I am even an endorser in the category they slot you into is arXiv's call rather than mine, so I would not want you counting on it either way.

What I would rather do is read the paper.

And before you write it up, one thing from re-pulling the atlas smoke set tonight, because a reviewer will find it too.

I checked this set two days ago, hashed all 20 traces, got 20 unique sha256, and moved on. That check was blind. Every trace embeds its own filename in the session id, so 20 unique hashes are guaranteed no matter what the payload is.

Drop that one field and the 20 traces are 8 scenarios. Twelve files are byte-identical to another file. synthetic-01, -09, -17 and -20 are the same 662 bytes of "Safe bounded read". 02/10/19 are one scenario, 03/11/18 are one. Zero records differ besides the id, and labels agree inside every group, so the collapse is clean.

That lands in synthetic_metrics.json. true_positives 20 is 9 distinct detections. scope_violation reads as the best-covered category at tp 7 and is 3 scenarios. approval_bypass reads 3 and is one scenario copied three times. Copy count also becomes weight: a detector failing only "Safe bounded read" loses 20% of the headline, one failing only "Runaway repeated search" loses 10%, for the same single defect.

Separate one. taxonomy.yaml declares 8 categories, per_category has rows for 6. unauthorized_tool and unhandled_tool_failure have no fixture and no row, so a detector that does not exist yet cannot lose a point on this set.

I liked the dataset for the limitation paragraph, which is rarer than it should be. This is that paragraph with numbers on it.

Happy to paste the per-scenario table. Is the duplication deliberate, or did the generator emit 20 from an 8-item template?

Table first, then one thing on TSC.

Re-pulled at sha 752921296, lastModified 2026-07-28, so this is against the same bytes you have now.

One correction to my own wording last night. The twelve are identical once the session id is dropped, not byte-identical. id is the only field that differs anywhere, and name plus labels agree inside every group, so the collapse is still clean.

per_category tp is counting files:

approval_bypass 3 becomes 1 scenario. prompt_injection 2 becomes 1. runaway_loop 2 becomes 1. silent_tool_failure 2 becomes 1. scope_violation 7 becomes 3, at scenarios 2, 4 and 8. secret_exposure 4 becomes 2, at scenarios 4 and 7. So tp 20 is 9 distinct detections.

Now the part I would fix before the weighting.

Your negative class is one scenario. fp 0 and precision 1.0, in every category row, rest on "Safe bounded read" copied four times. Four copies of one negative cannot surface a false positive that a second distinct negative would. Recall has 8 scenarios under it and precision has 1, so the two halves of that F1 are not carrying equal weight. I would put that above the taxonomy gap, because unauthorized_tool and unhandled_tool_failure at least announce themselves by having no row.

On TSC.

The ceiling check is right. 32/8 = 4.0 and 32/6 = 5.333, and metadata only moves the real number down, so it is a genuine upper bound.

But one artifact cannot explain both overshoots. K4V4 at 4.1 is 2.5% over. K4V2 at 6.1 to 6.2 is 14 to 16% over. An inflated FP16 baseline, a padded allocator length, a reserved-capacity denominator, all of those multiply both ratios by the same factor. These are not the same factor.

In bits it is sharper. 4.1 implies 7.805 bits where 8 are nominal, so 0.195 bits are unaccounted for. 6.1 to 6.2 implies 5.16 to 5.25 where 6 are nominal, so 0.75 to 0.84 bits. Roughly 4x more impossible bits, and the only thing that changed between the two configs is V going from 4 bits to 2.

A constant metadata or alignment error leaves a constant bit gap. This one scales with how hard V is quantized, which points at the V serialization path rather than at the baseline.

So, to take your serialized-byte-accounting question in the direction I think it actually bites: is V packed at its nominal width in the serialized bitstream, or does it pass through something that can land below it?

Before you trace V, one thing that will save you half the trace: the impossible bits are not in either ratio. They are in the difference between them.

Any per-unit term that is the same in both configs cancels when you subtract. Metadata, alignment padding, a wrong FP32 baseline, a reserved-capacity denominator: all of them drop out. So the number that has to be explained is the saving from V4 to V2, and it is bigger than V is.

Taking your ratios as quoted to two figures, so 4.05-4.15 and 6.05-6.25:

K4V4  4.1x   -> 7.71 to 7.90 bits per unit   (nominal 8)
K4V2  6.1x   -> 5.12 to 5.29 bits per unit   (nominal 6)
saved by dropping V from 4 bits to 2         2.42 to 2.78
nominal saving available                     2.00
excess                                       0.42 to 0.78

That excess is the whole problem, and it is config-dependent by construction. Whatever produces it has to be present at V4 and absent at V2, or sized differently in the two.

I tried to give you a data-side answer and it does not survive

My first guess was that something downstream is entropy coded, zstd or similar in the serializer. A 2-bit code stream from absmax quantization is very peaky, so it compresses well below nominal, and that would look exactly like sub-nominal packing.

I measured it instead of asserting it. 2M elements, per-group absmax uniform quantization, three distributions, group sizes 32/64/128, then both the Shannon limit of the code stream and what zlib and xz actually get on the packed bytes.

dist         group  b=2 H   b=4 H   delta(4->2)   zlib delta
gaussian      32    1.571   3.720     2.149         2.135
gaussian      64    1.453   3.597     2.144         2.102
gaussian     128    1.357   3.483     2.125         2.065
laplace       64    1.313   3.173     1.860         1.858
student-t3    64    1.253   2.960     1.707         1.748
                                      needed: 2.42 to 2.78

It fails, and it fails in the direction that rules it out rather than leaving it open. The 4-bit stream compresses harder in absolute bits than the 2-bit stream does, so entropy coding shrinks the delta. Best case anywhere in the sweep is 2.15, and the heavier-tailed distributions, which is what KV activations actually look like, come in at 1.7 to 1.9. The mechanism moves the number the wrong way.

Second leg, independent of the first. If an entropy step were on at all, it would have to be on at K4V4 too. At 4 bits it already sheds 0.28 to 1.35 bits per element on V alone in this sweep, and your entire K4V4 gap is 0.195 bits for K and V together. So K4V4 puts a ceiling on how much compression can be happening anywhere in the path, and that ceiling is below what K4V2 needs.

Both legs point away from the bit packing you were going to trace. V is probably stored at exactly 2 bits.

Where I would look instead

A term of roughly 0.5 bits per element that the V4 accounting counts and the V2 accounting does not. Group scales are the obvious shape, because they are the one thing in a quantized KV cache whose cost per element depends on a config field rather than on the data:

one fp16 scale per group of 32     0.500 bits/elem   -> predicts K4V2 6.03x
one fp16 scale+zero per group 64   0.500 bits/elem   -> predicts K4V2 6.03x
one fp16 scale per group of 16     1.000 bits/elem   -> predicts K4V2 6.66x
one fp16 scale per group of 64     0.250 bits/elem   -> predicts K4V2 5.76x

You measured 6.1 to 6.2. The 0.5 row lands at 6.03 and nothing else is close.

So the two questions I would put to the byte counter are whether the V2 path uses the same group size as V4, and whether the scale tensor is inside the counted buffer in both.

The check that decides it in one run

Feed the exact serializer a V tensor of uniform random codes over the 2^b symbols, so the payload is incompressible by construction, and re-measure.

If the ratio stays at 6.1x, nothing in the path is data dependent, and the excess is accounting. If it collapses toward 5.3x, something is compressing after all and my sweep used the wrong distribution.

Either answer is one run, and it tells you whether to keep reading the packer or go read the counter.

Everything above the last section is synthetic. Different distributions, not your tensors. What is not synthetic is the constraint, because the 0.42 to 0.78 excess comes out of your two published ratios and nothing else.

Does the V2 config share a group size with V4, or did it get finer when you dropped to 2 bits?

Before you trace the group sizes, I have to walk back my own last message. Group size cannot be the answer, and the table says so in one look.

I sent you after a ~0.5 bit/element term that the V4 accounting counts and the V2 accounting does not. That explains the delta. It does not survive the levels, and I never checked the levels.

Both configs are sub-nominal, not just K4V2

Your ratios against a 16-bit-per-element baseline, so 32 bits per K/V pair:

config   actual b/pair    nominal   counted overhead
K4V4     7.711 - 7.901       8      -0.289 to -0.099
K4V2     5.120 - 5.289       6      -0.880 to -0.711

Both negative. K4V4 is not the well-behaved config with a small gap I called it. It is the same defect, one order smaller. And every counted overhead you could find, group scales included, is positive, so it pushes both rows further from nominal rather than closer.

Which is why group size moves the wrong way

Solve each ratio for the baseline it implies, at each group size, fp16 scale counted on both K and V:

group   s b/elem   baseline from K4V4   baseline from K4V2
none      0.000    16.20 - 16.60        18.15 - 18.75
128       0.125    16.71 - 17.12        18.91 - 19.53
64        0.250    17.21 - 17.64        19.66 - 20.31
32        0.500    18.22 - 18.68        21.18 - 21.88
16        1.000    20.25 - 20.75        24.20 - 25.00

The two columns never overlap on any row. Nearest approach is 16.60 against 18.15, a 9.3% gap, at the row where nothing is counted at all. The gap then grows with s, +2.05 b/elem at no scale to +4.10 at group 16, so finer grouping is further from an answer than coarser.

Solving the pair of equations for one common overhead returns s between -0.82 and -1.16 bits per element in every corner of your quoted ranges. Negative storage. That is the system saying the model is wrong, not the measurement.

What is actually left

Two candidates. Either the two ratios are not against the same baseline, or a payload is not the width its config name claims.

The second is louder. K pinned at exactly 4 bits, fp16 baseline:

V4 measures   3.711 - 3.901 b/elem   nominal 4   short  2.5% -  7.2%
V2 measures   1.120 - 1.289 b/elem   nominal 2   short 35.5% - 44.0%

A 3% shortfall on V4 is what a padded last group or an off-by-one in a shape looks like. A 40% shortfall on V2 is not that.

The one run

Do not trace two config paths. Print three integers: baseline cache bytes, K4V4 cache bytes, K4V2 cache bytes, same fixed sequence length and batch, same measurement call.

If the baseline integer is not identical across the two runs, it is the denominator, and the packer is fine. If it is identical, then V2 is storing 1.12 to 1.29 bits where it should store 2, and that is a question about what got written, not about how it was counted.

All of the above comes out of the two ratios you published and nothing else. Tighter figures shrink the intervals but not the disjointness, since the columns miss each other by 9.3% at their closest.

Were both ratios measured in one run, or are they two separate benchmark invocations?

Your denominator answer closes the last cheap explanation, and it moves the problem off V and onto K.

Thank you for going back to the archived runs instead of answering from memory. One shared reference, one pkv, one invocation per model is exactly what was missing, because it lets me subtract your two rows against each other. That difference is the strongest object in the thread. It cancels the baseline, it cancels K, and it cancels any per-group scale or zero-point metadata, since that is additive and identical in both configs.

The K4V2 row alone is impossible, in four numbers

Qwen2.5-32B, your integers:

fp16 reference        128.0 MB
honest 4-bit K        base/8   = 16.0 MB
honest 2-bit V        base/16  =  8.0 MB
                      required   24.0 MB
K4V2 measured                    20.7 MB

An honest 4-bit K on those PKVs is 16.0 MB. Your K4V2 total is 20.7, which leaves 4.7 MB for V. An honest 2-bit V is 8.0 MB. Scale metadata only pushes the requirement further up.

Read it the other way. Give V its full 8.0 and K is capped at 12.7 MB against a nominal 16.0.

The 72B says the same thing on its own numbers:

                      32B        72B
reference           128.0 MB   160.0 MB
K4V2 measured        20.7       26.0
V honest at 2 bits    8.0       10.0
K implied          <= 12.7    <= 16.0
K nominal            16.0       20.0
K short              20.6%      20.0%

Two models, different shapes, same 20%. That is not rounding.

The difference says the opposite of the sum

Only V changed between your rows, so the delta is pure V payload:

        K4V4 - K4V2   nominal V 4->2 saving   excess
32B       10.2 MB          8.0 MB             +27.5%
72B       12.8 MB         10.0 MB             +28.0%

Your figures are three significant digits. Carrying that through, the excess band is +26.2% to +29.0%. It never reaches zero.

Now put the two facts together. Suppose every payload is off by one common factor f:

f implied by the difference    1.275 (32B)   1.280 (72B)
f allowed by the total        <= 0.966      <= 0.970

Disjoint, and not narrowly. A packer that under-stores everything makes the difference smaller, not larger. So one bug cannot produce both rows. There are at least two effects and they point opposite ways.

What each side probably is

V, over by about 27%. Group metadata cancels in the delta only if the granularity matches, so the clean candidate is that your V4 and V2 paths do not share a group size. With an fp16 scale plus an int8 zero, V4 at group 32 and V2 at group 128 gives +0.5625 bits per pair, which lands inside both models' bands. No other power-of-two pair fits. I swept six group sizes on each side, 16 to 512, against three metadata layouts (fp16 scale alone, fp16 scale plus fp16 zero, fp16 scale plus int8 zero). 108 combinations, one fit. If that is what the code does, the V story is a labelling issue and not a correctness one.

K, short by 20%, and this one survives every V explanation, because it falls out of the K4V2 row with V granted its full nominal payload and nothing else assumed. You mentioned the attention importances are precomputed once. If K is 4 bits on a kept fraction and dropped elsewhere, your two models imply a keep rate of at most 0.794 and 0.800. A round 0.80 sits right at the top of both bands.

If that is the design rather than a bug, then 6.2x is real but the label is not. It would be K at 4 bits on 80% density plus V, reported as K4V2.

The one run, four integers

Not two config traces. Print K bytes and V bytes separately off the same enc object, both configs, same fixed sequence length and batch:

K4V4  ->  K bytes, V bytes
K4V2  ->  K bytes, V bytes

Two predictions, so this is falsifiable rather than a fishing trip:

• K bytes are identical across the two configs, since only V changed.
• that shared K number is near 12.7 MB on the 32B and 16.0 MB on the 72B, not 16.0 and 20.0.

If K comes back at 16.0 and 20.0, my arithmetic is wrong and I will say so plainly. If it comes back near 12.7 and 16.0, then enc.memory_bytes() is fine and the open question is what the K path is not writing.

Does the K quantizer use those precomputed importances to drop anything, or is every K element written?