Spaces:

---

OpenEvals

/

README

like 4

Running

Not sure what are the specific requirements for benchmarks to be included, but we would like to have this functionality on these language specific benchmarks that we've built. They're quite recent so we don't have much sources yet beyond our own benchmarking efforts and EuroEval.

Manually translated and culturally adapted IFEval for Estonian.
https://huggingface.co/datasets/tartuNLP/ifeval_et

Manually translated and culturally adapted WinoGrande for Estonian.
https://huggingface.co/datasets/tartuNLP/winogrande_et

I'm not completely sure yet how to port the configs from LM Evaluation Harness to eval.yaml though.

Hi, we maintain Encyclo-K, a benchmark for evaluating LLMs with dynamically composed knowledge statements.

Dataset: https://huggingface.co/datasets/m-a-p/Encyclo-K
Paper: https://arxiv.org/abs/2512.24867
Leaderboard: https://encyclo-k.github.io/

We've added the eval.yaml file and would like to be included in the shortlist.

hey @yimingliang ! everything look great, we will add you to the shortlist and all should be set very impressive work on the evals, do you think it could be possible to open PRs on the models you evaluated with the results from your leaderboard ?

hey @adorkin ! Thanks for reaching out. IFEval would require custom code to run; this feature is not available yet, but will be in the future. For winogrande, you could absolutely make a eval.yaml file and turn it into a benchmark. You would need a small modification, though: The answer field should be either A or B instead of 1 or 2, and instead of having two columns for the choices, it would be easier to use one column with a list of choices. Then, your benchmark would simply be a multichoice benchmark :)

@SaylorTwift I see, thanks! Is the yaml expected to contain the prompt itself? I mean it works well as a multiple choice problem, but nonetheless the formulation is a bit non-standard, because you're filling the gap rather than answering a question.

@adorkin yes you can set th eprompt in the yaml file like so https://huggingface.co/datasets/cais/hle/blob/main/eval.yaml. Using the multiple_choice solver instead of the system prompt. Here are the docs from inspect.

@SaylorTwift I've added the eval.yaml and a custom dataset config to work with it. The dataset viewer seems to be stuck now which may or may not be related.
https://huggingface.co/datasets/tartuNLP/winogrande_et/blob/main/eval.yaml

📋 New Benchmark: FINAL Bench — Functional Metacognitive Reasoning

Dataset: https://huggingface.co/datasets/FINAL-Bench/Metacognitive

Paper: FINAL Bench: Measuring Functional Metacognitive Reasoning in Large Language Models
(Taebong Kim, Minsik Kim, Sunyoung Choi, Jaewon Jang — currently under review)

Blog: https://huggingface.co/blog/FINAL-Bench/metacognitive

Leaderboard: https://huggingface.co/spaces/FINAL-Bench/Leaderboard

What it measures

FINAL Bench is the first benchmark for evaluating functional metacognition in LLMs — the ability to detect and correct one's own reasoning errors. Unlike MMLU/GPQA that measure final-answer accuracy, FINAL Bench asks: "What did you do when you got it wrong?"

Key specs

• 100 tasks | 15 domains | 8 TICOS metacognitive types | 3 difficulty grades
• 5-axis rubric: MA (Metacognitive Accuracy), ER (Error Recovery), FA (Factual Accuracy), CO (Coherence), SP (Specificity)
• Hidden cognitive traps (confirmation bias, anchoring, base-rate neglect) embedded in every task
• 9 SOTA models evaluated: GPT-5.2, Claude Opus 4.6, Gemini 3 Pro, DeepSeek-V3.2, Kimi K2.5, etc.
• DOI: 10.57967/hf/7873

eval.yaml

eval.yaml has been added to the dataset repo.

We would love to be included in the benchmark shortlist! 🚀

Hey @SeaWolf-AI ! Sorry, I missed your message. We are limiting the number of benchmarks on the hub for now so that we can grow the ones we already have before adding more. However, we just noticed your "All bench leaderboard," and it's great! This is exactly what we have in mind when pushing leaderboards on the hub. Would you be up for a quick chat ?

Hey @SeaWolf-AI ! Sorry, I missed your message. We are limiting the number of benchmarks on the hub for now so that we can grow the ones we already have before adding more. However, we just noticed your "All bench leaderboard," and it's great! This is exactly what we have in mind when pushing leaderboards on the hub. Would you be up for a quick chat ?

Hi @SaylorTwift ,

Thank you for the kind message. I’d be very happy to have a quick chat.

Just to clarify our setup: FINAL Bench is our standalone benchmark for functional metacognitive reasoning, while ALL Bench is our unified leaderboard that brings FINAL Bench together with other major benchmarks in one comparable view.

I’m glad to hear that ALL Bench resonates with your vision for leaderboards on the Hub. I’d love to discuss how it could fit with the OpenEvals / community evals direction, and also whether FINAL Bench itself might eventually be considered for the shortlist as the ecosystem expands.

Happy to coordinate here or by email, whichever is easier for you.

Email is best! What email can i reach you at ?

Email is best! What email can i reach you at ?

kimminsik1116@gmail.com

Hi @SaylorTwift and the OpenEvals team,
I'd like to request that pdf-parse-bench be added to the official benchmark allowlist.

What it benchmarks: PDF parsing quality for mathematical formula and table extraction, evaluated via LLM-as-a-Judge on synthetically generated PDFs with automatic ground truth from LaTeX source.

• Dataset: https://huggingface.co/datasets/piushorn/pdf-parse-bench
• GitHub: https://github.com/phorn1/pdf-parse-bench

Why LLM-as-a-Judge? Rule-based metrics correlate poorly with human judgment. We validated this in two dedicated human annotation studies:

• Formula extraction (750 ratings): best rule-based metric r = 0.31, LLM judge r = 0.77 (https://arxiv.org/abs/2512.09874)
• Table extraction (1,500+ ratings): rule-based TEDS/GriTS top at r = 0.70, LLM judge r = 0.93 (https://arxiv.org/abs/2603.18652)

Current state:

• 22 models benchmarked on OCR parsing
• eval.yaml already present in the dataset repo
• pip-installable evaluation package: pip install pdf-parse-bench

I'm happy to submit a PR to huggingface.js to register pdf-parse-bench as a framework identifier. Please let me know if there's anything else needed.

Thanks!

Hi, we'd like to register OrgForge EpistemicBench as an official benchmark. Dataset: aeriesec/orgforge & Leaderboard. The eval.yaml is in the repo root. We also request orgforge-epistemicbench be added to the evaluation_framework enumerable in eval.ts.

What it measures

EpistemicBench evaluates agentic reasoning over a causally grounded enterprise corpus, not what a model knows, but how it reasons under constrained information access. Three tracks:

• PERSPECTIVE — Can a model stay within an actor's visibility cone and knowledge horizon while answering correctly? Out-of-cone tool calls are penalized even when they produce the right answer.
• COUNTERFACTUAL — Can a model identify the correct causal mechanism and traverse a cause-effect chain in the correct order?
• SILENCE — Can a model prove something didn't happen by searching the right artifact space before concluding absence? A correct "no" without evidence of search scores zero on trajectory regardless of answer correctness.

Why the scoring design is intentionally different

The primary metric is violation_adjusted_combined_score = combined_score × (1 - violation_rate)². Trajectory quality is weighted at 60–70% on two of the three tracks. A model cannot overcome epistemic gate violations through high answer accuracy alone. Answers receive no partial credit, and zero-shot answers are verified via NLI to confirm corpus grounding rather than fluent hallucination. This was a deliberate design decision: outcome-only scoring is a structural weakness of existing benchmarks and we didn't want to replicate it.

Three evaluation conditions are defined in eval.yaml: gated (primary), ungated (establishes the Epistemic Tax ceiling), and zero-shot (establishes the hallucination floor). The delta between ungated and gated combined_score is the Epistemic Tax, a derived metric with no equivalent in current benchmarks.

Results across six models (n=78)

The results demonstrate exactly the kind of differentiation the benchmark was designed to surface. Zero-shot scores cluster tightly across five of six models while gated scores diverge substantially, parametric knowledge explains little of the variance in agentic performance. Gated outperforms ungated for every model except Qwen, showing that well-designed permission constraints complete models rather than restrict them. The SILENCE track produces the sharpest differentiation: zero-shot search space coverage is zero across all models, gated coverage ranges from 82.7% to 94.4%.

Hi team — I'd like to register a benchmark dataset for inclusion in the Community Evals shortlist.

Dataset: https://huggingface.co/datasets/lianghsun/tw-legal-benchmark-v1
Domain / Language: Taiwan law, Traditional Chinese (zh-Hant)
Format: 209 multiple-choice questions
Eval framework: inspect-ai (eval.yaml is already in the repo root, following the docs)

The shortlist currently covers strong general-knowledge and reasoning benchmarks (MMLU-Pro, GPQA, HLE, AIME, MuSR), but doesn't yet include a Traditional Chinese benchmark or a domain-specific legal benchmark. Given Hugging Face's stated goal of expanding to "the most relevant benchmarks" and "new tasks and domains that challenge SOTA models," I think this is a useful gap-filler:

• Most existing Chinese benchmarks (C-Eval, CMMLU) are Simplified Chinese and PRC-centric. Taiwan legal QA exercises a distinct legal system, terminology, and writing convention.
• Legal reasoning is a domain where current frontier models still show meaningful variance — useful signal beyond saturated general benchmarks.

Happy to:

• Adjust the eval.yaml / dataset card to whatever spec you'd like.
• Provide reference results across a few open models (e.g., Qwen, Llama, our Formosa-1) so you can sanity-check the benchmark behaves as expected.
• Open a PR to add zh-Hant or a legal tag to the framework enum if needed.

For reference, I also opened https://github.com/huggingface/hub-docs/issues/2311 a few weeks ago before realising this discussion is the right venue — happy to consolidate everything here. Let me know what would be most helpful on your end. Thanks!

Vinayak Multistep Recursive Reasoning Benchmark (VMRRB)

Benchmark for evaluating advanced reasoning, recursive dependency resolution, and robustness capabilities of large language models in dynamic, noisy, and structurally challenging environments.

It uses dynamically created database for each run, so each run is unique. (No Static Database)

GitHub Repo:
https://github.com/vbepipe/vmrrb-benchmark

Hello Team Huggingface,

We at Team Vaani would like to join the Hugging Face Community Eval initiative and register our ASR evaluation dataset for Indic languages: ARTPARK-IISc/Vaani-Benchmark-V1.0.

The Vaani Benchmark V1.0 is a curated ASR evaluation set derived from the Project Vaani featuring 5,343 audio segments (~11.7 hours) from 1,103 unique speakers across 104 districts in India. A key feature is that each segment includes three independent human transcriptions to better account for annotator variance during evaluation. Currently, the eval set supports Hindi, but we are planning to add other Indic languages included in the Vaani Dataset.

I have already added the eval.yaml file, scoring WER and CER against all three transcription versions.
Please let us know if you require any further information or adjustments to our configuration.

Thank you for your time. Dataset link: https://huggingface.co/datasets/ARTPARK-IISc/Vaani-Benchmark-V1.0.

📋 New Benchmark: Metacognition-Bench — Functional Metacognition at Scale (corrected)

Correction to my earlier comment above — updating the numbers after a direct re-check against the leaderboard JSON. The framing is unchanged.

Hi @SaylorTwift and the OpenEvals team,

We'd like to register Metacognition-Bench for the community-evals shortlist.

• Dataset: https://huggingface.co/datasets/ginigen-ai/Metacognition-Bench
• Leaderboard: https://huggingface.co/spaces/ginigen-ai/Metacognition-Leaderboard-Space
• Adapters (reproducibility artifacts): https://huggingface.co/collections/FINAL-Bench/metacognition-adapters-6a42c032e6beb803dd032961
• Paper: "Does Your LLM Know When It's About to Be Wrong?" — https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6280258 (under review)
• License: Apache-2.0
• eval.yaml: placed at the repo root, inspect_ai schema

What it measures

Metacognition-Bench measures functional metacognition — what a model does when it is headed the wrong way. This is deliberately an axis that MMLU-Pro, GPQA, HLE, AIME, and MuSR do not cover: those score the final answer; ours scores the trajectory of self-correction.

Every task embeds a hidden cognitive trap (confirmation bias, anchoring, base-rate neglect, framing, sunk-cost). A response earns credit only when the model recognizes the trap and substantively overrides it — arriving at the correct final claim without that step is scored INCORRECT. Fluent hallucination is not rewarded.

• 300 tasks (META-001 to META-300)
• 121 domains — mathematics, physics, biology, law, medicine, economics, statistics, ethics, computer science, linguistics, music theory, chemistry, and more
• 8 TICOS metacognitive behavior types: E_SelfCorrecting / A_TrapEscape / G_PivotDetection / B_ContradictionResolution / C_ProgressiveDiscovery / D_MultiConstraint / F_ExpertPanel / H_DecisionUnderUncertainty
• Grade A/B/C difficulty tiers (frontier / expert / core)
• Format: free-form generation, model-graded with a rubric that rejects "right answer, wrong reasoning"
• Two reporting axes:
  1. trap_rate — fraction of tasks the trap succeeded against the base model (lower is better)
  2. probe_vs_base — Δ(probe_trap_rate − base_trap_rate) when the base is equipped with a released metacognition adapter (more negative = larger improvement)

Sample task columns: task_id, domain, grade, ticos_type, difficulty (0–1), prompt, expected_behavior, hidden_trap, ticos_required.

Why this fills a gap on the current shortlist

1. Distinct signal. Final-answer benchmarks saturate around final correctness. In our current data the rank orders on Metacognition-Bench do not correlate above chance with rank orders on MMLU-Pro / GPQA / HLE.
2. 121 domains enables generalization studies. Per-model behavior can be decomposed into "metacognition skill" vs "domain knowledge" — a decomposition the current shortlist cannot support at the domain granularity.
3. Actionable output. Because we also score probe_vs_base, the benchmark points to what fixes the failure, not only that the failure exists.

Reproducibility — the adapter collection

Alongside the dataset and leaderboard, we have released per-base-model metacognition adapters as an open, downloadable collection so third parties can reproduce our reported adapter gains without retraining from scratch:

https://huggingface.co/collections/FINAL-Bench/metacognition-adapters-6a42c032e6beb803dd032961

Currently covers 11 base models across four families: Darwin (Darwin-28B-Opus, Darwin-9B-NEG), Qwen3.5 (9B, 27B, 35B-A3B, Qwen-AgentWorld-35B-A3B), Gemma-4 (12B-it, 26B-A4B-it), and others (FastContext-1.0-4B-SFT, Ornith-1.0-9B, JGOS-31B-Citizen).

Each adapter is a drop-in module; the reported adapter-vs-base numbers on the leaderboard are reproducible by loading base + adapter and re-running the eval. All updated within the last week.

Current benchmarking coverage

The live leaderboard currently ranks 33 models across open and frontier tiers. Representative rows:

Lowest base trap_rate (best metacognition without adapter):

• JGOS-Model/JGOS-31B-Citizen — trap_rate 0.005
• FINAL-Bench/Darwin-31B-Opus — trap_rate 0.008
• Qwen/Qwen3.5-27B — trap_rate 0.010

Largest adapter improvement (probe_vs_base, most negative = best):

• FINAL-Bench/Darwin-31B-Opus — probe_vs_base −0.832
• OrionLLM/GRM-2.6-Plus — probe_vs_base −0.338
• Qwen/Qwen3.5-27B — probe_vs_base −0.331

(Frontier API models — GPT-5.2, Claude Opus 4.6, Gemini 3 Pro, DeepSeek-V3.2, Kimi K2.5 — are being added.)

Infrastructure

• eval.yaml: placed at repo root (https://huggingface.co/datasets/ginigen-ai/Metacognition-Bench/blob/main/eval.yaml), inspect_ai schema, generate solver + model_graded_qa scorer with a rubric that separates "correct final answer" from "correct metacognitive reasoning". Happy to align exact field names or grader model to whatever schema OpenEvals prefers.
• Dataset viewer: functional.
• Dataset format: 300 rows in metacog_bench.jsonl, 9-field schema (task_id, domain, grade, ticos_type, difficulty, prompt, expected_behavior, hidden_trap, ticos_required).
• License: Apache-2.0 (dataset and adapters).

What we can commit to

• Open PRs on the pages of every model we evaluate, adding the Metacognition-Bench score (as you requested from @yimingliang for Encyclo-K).
• Grow benchmarked-model coverage from the current 33 to 60+ by end of Q3 2026.
• Refine eval.yaml to whatever exact schema you prefer.
• Contribute to any unified metacognition-domain tag PR you'd like to open on huggingface.js.

About the account

ginigen-ai is our neutral operator account for benchmarks and leaderboards. We intentionally publish base models under separate accounts/orgs (FINAL-Bench for adapters, VIDraft for base models), apart from the operator account (this one), so the operator does not have an incentive to make its own participants win. Happy to explain the governance model over email if useful.

Thank you for the openevals infrastructure — it is genuinely raising the bar for benchmark discoverability on the hub.

— @ginigen-ai

📋 New Benchmark: REFUTE — Scientific epistemics & calibration on recent paper summaries

Hi @SaylorTwift and the OpenEvals team,

We'd like to register REFUTE (Reasoning Over Evidence — Falsification, Uncertainty, Truth-grounding & Epistemics) for the community-evals shortlist.

• Dataset: https://huggingface.co/datasets/BGPT-OFFICIAL/refute
• Leaderboard: https://huggingface.co/spaces/BGPT-OFFICIAL/refute-leaderboard
• License: Apache-2.0
• eval.yaml: at repo root (evaluation_framework: inspect-ai, 8 tasks — judge-free MCQ axes + generative critique splits)
• inspect_evals PR: https://github.com/UKGovernmentBEIS/inspect_evals/pull/1894
• Paper: technical report on dataset card; arXiv submission pending endorsement

REFUTE tests whether LLMs critique recent science summaries with calibrated judgment: falsification, overclaim detection, missing-evidence abstention, planted-flaw detection, and Brier-scored calibration. v2 adds judge-free axes so leaderboard scores are fully reproducible without a judge model.

Happy to open model PRs with .eval_results/*.yaml once registered. Thanks!