Roflimjonny commited on
Commit
52ecbed
·
verified ·
1 Parent(s): 4e13ca2

Add held-out eval + honest Clark-vs-heuristic comparison + CP-SAT bound

Browse files
Files changed (1) hide show
  1. README.md +18 -0
README.md CHANGED
@@ -55,6 +55,24 @@ Validated on the predecessor "[Jack](https://github.com/jarmstrong158/Jack)" sin
55
 
56
  v2.11 additionally makes the per-worker schedule realistic — task switching drops from ~29/worker/day (a flip every ~10 min) to ~14 (every ~38 min) via a structural minimum-dwell mask, with grades held (A+B ~90%, 100% completion on the validation facility). Full methodology, the honest F-rate read, and the v2.5→v2.11 iteration log are in the [repo README](https://github.com/jarmstrong158/Clark#readme) and [CHANGELOG](https://github.com/jarmstrong158/Clark/blob/master/CHANGELOG.md).
57
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
58
  ## Limitations
59
 
60
  - Numbers above are on synthetic facilities + the Jack-translated config; real facilities will vary, and a per-facility fine-tune is recommended.
 
55
 
56
  v2.11 additionally makes the per-worker schedule realistic — task switching drops from ~29/worker/day (a flip every ~10 min) to ~14 (every ~38 min) via a structural minimum-dwell mask, with grades held (A+B ~90%, 100% completion on the validation facility). Full methodology, the honest F-rate read, and the v2.5→v2.11 iteration log are in the [repo README](https://github.com/jarmstrong158/Clark#readme) and [CHANGELOG](https://github.com/jarmstrong158/Clark/blob/master/CHANGELOG.md).
57
 
58
+ ### Held-out generalization
59
+
60
+ On **20 freshly-sampled stage-3 facilities the model never trained on** (hardest tier — up to 50 workers, deliberate-overload days), each a full simulated work-year scored by the in-env production grader: median **A+B 97.5%** (p10–p90 65.1–100), **A 76.5%**, **F 0.5%**, order completion **100%**. Reproduce with `clark eval --n-per-stage 20 --stages 3`.
61
+
62
+ ### Vs. a strong classical baseline (honest comparison)
63
+
64
+ Clark is benchmarked against a deliberately *strong* rule-based **heuristic scheduler** (bottleneck-aware dispatching, same action masks, same grader, same 20 held-out facilities). The honest result:
65
+
66
+ | Metric (median, 20 held-out facilities) | Heuristic scheduler | **Clark** |
67
+ |---|---|---|
68
+ | A + B grade days | 98.3% | 97.5% |
69
+ | Order completion | 100% | 100% |
70
+ | **A days (no overtime)** | 43.3% | **76.5%** |
71
+ | **F days** | 1.7% | **0.5%** |
72
+ | A+B worst-case (p10) | 56.9 | **65.1** |
73
+
74
+ A well-engineered heuristic **ties Clark on throughput** (A+B, completion). Clark's edge is specific: it finishes **without overtime ~33 pp more often** (less paid OT), has **~3× fewer catastrophic days**, and generalizes across facilities with **zero per-site tuning** (the heuristic's constants are hand-fit to one distribution). A separate CP-SAT (constraint-programming) bound confirms throughput is never the binding constraint — the difference is in balancing the *soft* quality objectives. Write-up: [ENGINEERING_NOTES §9–§10](https://github.com/jarmstrong158/Clark/blob/master/docs/ENGINEERING_NOTES.md).
75
+
76
  ## Limitations
77
 
78
  - Numbers above are on synthetic facilities + the Jack-translated config; real facilities will vary, and a per-facility fine-tune is recommended.