SearchScore Labs / EXP-0002
Founding recommendation-quality baseline (benchmark v1)
Hypothesis
A frozen 1001-query benchmark can establish a reproducible baseline for recommendation quality and confidence calibration that every future release must beat.
Method
Run the reasoner over benchmark-v1 (sha256-sealed); compute confusion matrix, precision/recall/F1, out-of-scope P/R, golden pattern accuracy, and calibration (ECE, Brier, MCE, reliability curve). Store immutably.
Dataset
bench/benchmark-v1.jsonl — 1001 labelled queries, 18 categories, 10 golden pattern labels.
Metrics
| precision / recall / F1 | record |
| ECE / Brier | record |
| pattern accuracy | record |
Result
{
"answerPrecision": 0.75,
"answerRecall": 0.944,
"oosRecall": 0.747,
"patternAccuracyGolden": 0.9,
"ece": 0.138,
"brier": 0.169,
"mce": 0.579,
"note": "Well-calibrated in-domain (80–90% bin: 84%→91%) but overconfident in the 70–80% band (77%→54%) — homograph-driven."
}
Decision
BASELINE-RECORDED. Frozen as the yardstick. No release ships if worse than this on any headline metric.
Lessons learned
- The reliability curve localised the miscalibration to the exact homograph confidence band — a precise target for embeddings.
- Freezing the benchmark makes "better" a testable claim, not an opinion.
Confidence
HIGH
Evidence
eval report —bench/history/eval-v1-*.jsonprogramme — docs/kos-intelligence-programme.md