kimsan0622 commited on
Commit
b87c952
·
verified ·
1 Parent(s): 209d029

Publish camera-ready metadata reproduction and validation scope

Browse files
Files changed (2) hide show
  1. README.md +5 -1
  2. manifest.json +4 -3
README.md CHANGED
@@ -105,4 +105,8 @@ The API returns a wrapper object with `id`, `object`, `created`, `model`, and `r
105
 
106
  ## Diagnose your own model
107
 
108
- Use the [complete KoEVD own-model workflow](https://github.com/KETI-NLP/KoEVD/blob/main/docs/OWN_MODEL_GUIDE.md) to generate your model’s responses, measure Task 1–5 behavior, and produce HTML/JSON/CSV reports of metadata-specific strengths, weaknesses, strategies and source-matched tool-selection gaps. The current GitHub source adds this pipeline; the original v0.1.0 code tag does not include it. A subsequent [live B200 pipeline test](https://github.com/KETI-NLP/KoEVD/blob/main/docs/PIPELINE_VALIDATION_RESULTS.md) completed 40/959 actual generation requests with unmodified code. It also found batch-dependent strategy outputs and concerning safety judgments; execution success does not establish measurement accuracy.
 
 
 
 
 
105
 
106
  ## Diagnose your own model
107
 
108
+ Use the [complete KoEVD own-model workflow](https://github.com/KETI-NLP/KoEVD/blob/main/docs/OWN_MODEL_GUIDE.md) to generate your model’s responses, measure Task 1–5 behavior, and produce HTML/JSON/CSV reports of metadata-specific strengths, weaknesses, strategies and source-matched tool-selection gaps. The current GitHub source adds this pipeline; the original v0.1.0 code tag does not include it. A subsequent [live B200 pipeline test](https://github.com/KETI-NLP/KoEVD/blob/main/docs/PIPELINE_VALIDATION_RESULTS.md) completed 40 smoke requests and 959 expanded generation requests with unmodified code. It also found batch-dependent strategy outputs and concerning safety judgments; execution success does not establish measurement accuracy.
109
+
110
+ ## Later reproduction scope — 2026-09-15
111
+
112
+ A separate B200 check reproduced the assistant-safety classifier’s historical raw outputs and labels on all 532 original validation inputs at batches 2, 4, and 32, retaining macro/micro F1 0.9981/0.9981 and the same single error. It did not rerun response-strategy validation F1. On 191 new generated responses, suspicious all-safe safety outputs and 5/191 batch-dependent strategy label sets remain unresolved; no new human gold or new-response F1 was established. See the [reproduction scope](https://github.com/KETI-NLP/KoEVD/blob/main/docs/REPRODUCIBILITY.md). Earlier operational-test statements above describe their original 12/96-candidate sample, not this subsequent check.
manifest.json CHANGED
@@ -24,8 +24,8 @@
24
  },
25
  {
26
  "path": "README.md",
27
- "bytes": 7451,
28
- "sha256": "6fd787f402dfe55f16daa7aa3175959affcf23fec3720da23f7ef3290dd6949e"
29
  },
30
  {
31
  "path": "adapter_config.json",
@@ -62,5 +62,6 @@
62
  "bytes": 1099,
63
  "sha256": "5c5d35fa571bff9b1687a906a1f96756f44b8aae45381b5be94a2d66f6836546"
64
  }
65
- ]
 
66
  }
 
24
  },
25
  {
26
  "path": "README.md",
27
+ "bytes": 8201,
28
+ "sha256": "e34f8817001e00e8fcc5d6c93e4fd036918bc9267080ca5f039a11cc698b4ff2"
29
  },
30
  {
31
  "path": "adapter_config.json",
 
62
  "bytes": 1099,
63
  "sha256": "5c5d35fa571bff9b1687a906a1f96756f44b8aae45381b5be94a2d66f6836546"
64
  }
65
+ ],
66
+ "documentation_update": "2026-09-15 camera-ready clarification; canonical data and model weights unchanged"
67
  }