JevK5: please rerun its 15,317 refused requests (questions with more than 16 options)

#13
by alibiserikbay - opened

Hi! JevK5 refused every question with more than 16 options, because it answers with the letters A-P. That is 15,317 of the 121,057 requests:

Benchmark Refused requests Options
CLINC150+OOS 5,500 151
ChessBench ~4,216 varies
BANKING77 3,080 77
POP909 2,000 129
API-Bank 508 53
HLE ~13 varies

The four exact counts are from answer_gaps in your methodology.json, which lists JevK5's top four reasons (11,088 requests). We derived the ChessBench and HLE counts from JevK5's coverage there (0.1574 of 5,000 and 0.9741 of 513), so each may be off by a few. SemIf, which has the same 16-option limit, has the same coverage on both, and its 15,317 unanswered requests all carry one reason: "supports only 2–16 answer options". BBH's coverage gap (0.9682 of 5,507) looks like the same cause, but BBH was added in 0.2 and sits outside the 121,057.

jevk5 0.2.2 answers these with the same weights, and I'd like to ask for a lift like the ones Jobe and Winnow got.

What changed (runtime only, same weights at alibiserikbay/JevK5)

  • Questions with 16 options or fewer take one pass, exactly as before. On JevBench's 231 public items, 0.2.2 returns bit-identical probabilities to 0.2.1 (max |dp| 0) through both the CUDA runtime and the llama.cpp client. The check is bench/parity.py, and you're welcome to rerun it on in-range index rows.

  • With more than 16 options, probabilities() / decide() returns a probability for every option, summing to 1:

    1. The options are split, in the given order, into groups of at most 16, and each group is read whole.
    2. A final of 16 is read: the top options of every group, with the free places going to the next most likely options.
    3. Finalists keep the final's distribution, times the chance the answer is among them. Every other option gets its group's share of the final times its in-group probability.
    4. A second temperature (0.77) sharpens the combined distribution. It never changes the answer.

    Every option is offered and scored by the model, and nothing is pruned before being read. That is ceil(n/16)+1 forward passes: 6 for BANKING77, 11 for CLINC150.

  • The rule and the 0.77 were chosen and fitted on MASSIVE en-US's train split, which is not on the index. Choosing the knockout over a tree readout was done on BANKING77 and CLINC150 train splits, never their test splits.

How to run it

  • pip install "jevk5 @ git+https://github.com/allebee/jevk5@v0.2.2" (commit 0571ef3).
  • Then the same call as before: JevK5("alibiserikbay/JevK5").probabilities(state, question). The wrapper's "more than 16 options" refusal needs to go.
  • input_tokens now sums every pass. If you apply a context limit, the longest single pass is in model.last_pass_tokens.
  • Expected cost on an H100: about 115 ms per BANKING77 row and 200 ms per CLINC150 row (against ~13 ms for short in-range rows).

What we measured (train splits, every option offered, your request shape, 500 items each)

Accuracy Macro-F1 ECE
BANKING77 train 0.690 0.674 0.039
CLINC150+OOS train 0.666 0.720 0.039

Out of scope is the weak spot: recall 0.36 on CLINC150. We haven't measured API-Bank, POP909, ChessBench or HLE.

Declared data overlap

  • JevK5 v0.2 trained on 237 BANKING77 train-split items (shown with 4-8 of the 77 intents). It has no CLINC150, API-Bank, POP909 or ChessBench data.
  • 47 BANKING77 test items were in v0.2's held-out dev set. They were never trained on, and the temperature is fitted on other dev rows.
  • Separately, and already flagged by the index: v0.1 and v0.2 trained on MMLU-Pro test items (940 in v0.2). The count, the item ids and a memorization measurement are in our CHANGELOG.

Thanks for running the index!

Hi, I have re-run JevK5 with the new setup! Excluding the contaminated bits that you reported. Cheers!

multimodalart changed discussion status to closed

Thank you for the quick rerun! One small thing: in index.json (generated 18:11 UTC) POP909-CL still shows coverage 0 and ChessBench 0.157 for JevK5, although the methodology notes 15,317 rows answered by the lift. Maybe those two haven't been merged yet? No rush at all.

Sign up or log in to comment