|
|
--- |
|
|
title: SearchAgent_Leaderboard |
|
|
emoji: 🥇 |
|
|
colorFrom: green |
|
|
colorTo: indigo |
|
|
sdk: gradio |
|
|
app_file: app.py |
|
|
pinned: true |
|
|
license: apache-2.0 |
|
|
short_description: A standardized leaderboard for search agents |
|
|
sdk_version: 4.23.0 |
|
|
tags: |
|
|
- leaderboard |
|
|
--- |
|
|
|
|
|
# Overview |
|
|
|
|
|
SearchAgent Leaderboard provides a simple, standardized way to compare search-augmented QA agents across: |
|
|
- General QA: NQ, TriviaQA, PopQA |
|
|
- Multi-hop QA: HotpotQA, 2wiki, Musique, Bamboogle |
|
|
- Novel closed-world: FictionalHot |
|
|
|
|
|
We display a minimal set of columns for clarity: |
|
|
- Rank, Model, Average, per-dataset scores, Model Size (3B/7B) |
|
|
|
|
|
|
|
|
# Data format (results) |
|
|
|
|
|
Place model result files in `eval-results/` as JSON. Scores are decimals in [0,1] (the UI multiplies by 100). |
|
|
|
|
|
```json |
|
|
{ |
|
|
"config": { |
|
|
"model_dtype": "torch.float16", |
|
|
"model_name": "YourMethod-Qwen2.5-7b-Instruct", |
|
|
"model_sha": "main" |
|
|
}, |
|
|
"results": { |
|
|
"nq": { "exact_match": 0.469 }, |
|
|
"triviaqa": { "exact_match": 0.640 }, |
|
|
"popqa": { "exact_match": 0.501 }, |
|
|
"hotpotqa": { "exact_match": 0.389 }, |
|
|
"2wiki": { "exact_match": 0.382 }, |
|
|
"musique": { "exact_match": 0.185 }, |
|
|
"bamboogle": { "exact_match": 0.392 }, |
|
|
"fictionalhot": { "exact_match": 0.061 } |
|
|
} |
|
|
} |
|
|
``` |
|
|
|
|
|
Notes: |
|
|
- `model_name` uses the format `Method-Qwen2.5-{3b|7b}-Instruct` (no org prefix required) |
|
|
- Tasks: `nq`, `triviaqa`, `popqa`, `hotpotqa`, `2wiki`, `musique`, `bamboogle`, `fictionalhot` |
|
|
- Metric key: `exact_match` |
|
|
|
|
|
|
|
|
# Submission (via Community) |
|
|
|
|
|
We accept submissions via the Space Community (Discussions): |
|
|
1) Open the Space page and go to Community: `https://huggingface.co/spaces/TencentBAC/SearchAgent_Leaderboard` |
|
|
2) Create a discussion with title `Submission: <YourMethod>-<model_name>-<model_size>` |
|
|
3) Include: |
|
|
- Model weights link (HF or GitHub) |
|
|
- Short method description |
|
|
- Evaluation JSON (inline or attached) |
|
|
|
|
|
|
|
|
# Local development |
|
|
|
|
|
Run locally (example): |
|
|
```bash |
|
|
python app.py |
|
|
``` |
|
|
|
|
|
The app reads local data only (no remote download) from: |
|
|
- Results: `./eval-results` |
|
|
- (Optional) Requests: `./eval-queue` (not required for the simplified table) |
|
|
|
|
|
If you see missing dependencies, install minimally: |
|
|
```bash |
|
|
pip install gradio gradio_leaderboard pandas huggingface_hub apscheduler |
|
|
``` |
|
|
|
|
|
|
|
|
# Customize |
|
|
|
|
|
- Tasks and page texts: `src/about.py` |
|
|
- Displayed columns: `src/display/utils.py` (we keep Rank, Model, Average, per-dataset, Model Size) |
|
|
- Custom model links (name→URL mapping): `src/display/formatting.py` (`custom_links` dict) |
|
|
- Data loading and ranking: `src/leaderboard/read_evals.py`, `src/populate.py` |
|
|
|
|
|
Restart the app after changes. |
