No video this session — Meet's recording didn't run. The session notes below were compiled from Granola, and the full notes doc is in the Drive folder.
Want the build, not just the story? The companion note has the one prompt every bot got, the wiring between Todoist and Notion, the rubric, the dry-run scores, and the screens.
#56 - Build The Eval Bench: Prompts, Runs & Screens
The live rubric, transcripts, and Evals database are linked from there.
The bench after both rounds — six prefixed tasks in Todoist, all worked and graded
The bench after both rounds — six prefixed tasks in Todoist, all worked and graded
Two graders on the same answer — agreement on mechanics, divergence on judgment: Audience fit 1.25 vs 1.2, Format 2.0 vs 1.7
The Evals database after the dry run — twelve rows, every judgment logged with scores and feedback
The Evals database after the dry run — twelve rows, every judgment logged with scores and feedback
Office hours 56 took on the question every AI builder eventually hits: how do you know your AI is any good — and how do you know it's still good next month? The answer is evals: grade AI output against a rubric, with a different model as the judge. Rahul and Jabari Grubb each built a working eval pipeline live, from opposite ends of the cost spectrum.
Why evals, and the one rule that matters
An eval is three things: a task, a rubric, and a golden set of expected behavior. You run it to catch model drift — prompts change, models upgrade, tools get added, and quality moves without anyone noticing — and to grade the intermediate steps of agentic pipelines, not just the final answer. The one rule: never let a model grade its own work. Claude always gives itself an A. Cross-model grading — ChatGPT judging Claude, Grok judging ChatGPT — is what makes the scores mean something.
Rahul's build: three bots, one rubric, Todoist as dispatcher
Claude, ChatGPT, and Grok each summarized the same transcript into an action plan. A four-criteria rubric — completeness, fidelity, format, audience fit, scored 0–2 — was itself generated by asking Claude what matters for the task type. Todoist dispatched the work and collected each bot's answer as comments on the task; results landed in Notion, where the rubric stays live and editable so the whole test reruns after any change. The scores had personality: ChatGPT graded Grok harshly and itself generously, Grok was the most discerning judge, and Claude won both rounds.
Jabari's build: the zero-dollar pipeline
The same discipline, no API bill: Gemma 3 generating answers, Mistral judging them, Python glue, and Notion holding the golden set (12 cases), the results, and a run dashboard. Notion AI triages the failures by error type. The punchline came from a prompt A/B test: pass rates moved dramatically, not marginally — and the best run still only hit ~58%, which the dashboard rightly flagged as barely better than a coin flip. Without the eval, nobody would have known.
Choosing your layer