Bad Customer Data Poisons Your Eval Harness
An eval harness is only as honest as the tasks it reflects. If the tasks were reverse-engineered from what founders hoped customers needed rather than what those customers actually do, the harness measures fitness for a fantasy. Fitzpatrick's core argument is that compliments tell you nothing; only concrete past behavior tells you something. Before you write a single eval case, you need ground truth from real workflows your customers already run, not approval for the workflow you imagine they run.
Commitment Is the Signal Your Agent Design Needs
Fitzpatrick draws a hard line between vague enthusiasm and real commitment — time given, money placed, reputation staked. That same test applies when scoping an agent. If no customer will hand over actual data, grant actual permissions, or change an actual process to use your agent, the task you are automating is not load-bearing. Agents, workflows, and evals built around low-commitment tasks produce polished demos that never reach production. Design around the tasks people will fight to keep.
Human Judgment Starts With Knowing What to Ask
An AI-native company routes decisions to humans when the stakes or ambiguity are high enough to warrant it. But humans can only judge what they understand, and they only understand what was surfaced honestly. Fitzpatrick's method — anchoring every conversation in specific past behavior and concrete consequences — is how you learn where the real stakes live. That knowledge tells you which agent outputs need a human in the loop, which evals need the hardest red cases, and which automations are genuinely safe to run unsupervised.
