<aside> ⚖️

Evals with Claude, ChatGPT and GrokBot for Managers and Operators. Rahul's portion: the talk, the demo and everything needed to run it again. 45 minutes plus 15 of Q&A.

</aside>

One place for everything

officehours 56 Eval Bench

Office Hours 56: Evals with Claude, ChatGPT, and Grok for Managers and Operators

What Where
Slides (13 diagrams, one page) officehours 56 Deck
Follow-along notes (draft for the site post) Office Hours 56: Evals with Claude, ChatGPT, and Grok for Managers and Operators
Rubric ‣
Transcripts A and B ‣
Evals database ‣
Todoist Inbox section officehours 56 Eval Bench, six tasks, labels round-1, round-2, worker-done, graded
Cover art officehours-56-rubric-cover.png (judge robot, rubric, grading bench)
Follow-up session Decision models as judges

Office Hours 56 Deck

The claim

An eval is a task, a rubric and a grader. You can run one with Todoist and Notion and three chat assistants, and the same shape scales to a real harness. The eval is the exam, the judge is the examiner, the harness is the exam hall. Build one without the other two and you cannot tell whether your agent got better or just different.

Talk outline (45 min + 15 Q&A)

Time Section Slide
0:00 Why grade AI: demos pass, production drifts 2
5:00 Where evals came from: HumanEval 2021, SWE-bench 2023, harness tooling 2024, decision models 2026 3
11:00 What an eval is, per Anthropic's guide: criteria first, cases second, grade with the fastest reliable method 4
15:00 Three words, one system: eval, judge, harness 5
19:00 The demo stack: Todoist carries messages, Notion holds knowledge 6
21:00 The sequence: three things you type 7
23:00 Round 1 work, round 1 grade, read the Disagreements view 8, 9, 10
35:00 Round 2 regression under rubric v2 11
38:00 Where this goes: swap yourself for a schedule 12
43:00 Homework 13
45:00 Q&A

If timing slips, cut round 2 first.

Citation for section 3: Define success criteria and build evaluations, Anthropic.

How the intelligences coordinate

Four intelligences, one of them human, never talk to each other directly. Todoist is the communications layer: who has which task, what the answer was, what the grades were, what state each task is in. Notion is the knowledge layer: the rubric that defines good, the transcripts that are the test cases, the Evals table that accumulates every judgment. Communications are ephemeral and get reset; knowledge is durable and versioned.

The task

Meeting transcript in, sponsor status update out. Under 150 words, three sections, owner and date on every action item, nothing not stated in the transcript. Transcript A plants three decisions, two owned action items, one open question, one withdrawn proposal and one figure stated two ways. Transcript B is the regression case: one action item has no stated owner, and rubric v2 says the right answer is Unassigned.

The one prompt every bot gets

Pasted verbatim into Claude, ChatGPT and GrokBot. Each works out its own name from the app it runs in.

You are one of three assistants in an evaluation exercise: Claude, ChatGPT, or GrokBot. Work out which one you are from the application you are running in. State your name in one line before doing anything else.

Phase 1, work. In Todoist, open the Inbox section "officehours 56 Eval Bench". Find the open task labelled round-1 whose title begins with your name followed by a colon. Read the description and complete the task exactly as stated. Post your answer as a comment on that task starting with "ANSWER:". Add the label worker-done. Ignore tasks whose prefix is not your name.

Phase 2, grade. Wait until you are told to grade. Then find the tasks in that section labelled worker-done and round-1 whose prefix is not your name. For each one, read the ANSWER comment and the rubric linked in the description. Score each of the four criteria 0, 1 or 2 with one sentence of justification per criterion. Post a comment on the task starting with "GRADE from <your name>:". Then add one row to the Notion database "Evals" under the page "officehours 56 Eval Bench" with: Task (the task title), Round, Worker (from the task prefix), Grader (your name), Self-identified as (your name as you stated it), Rubric version, the four scores, one paragraph of feedback, and the link to the ANSWER comment. When both grades are posted, add the label graded to each task.

Never grade your own task. Never edit another assistant's comments. For round 2, repeat both phases with the round-2 label and rubric version v2.