If you're delegating real work to AI agents, you need evaluation methods that actually predict performance, not just ones that feel rigorous. The same bias that makes human hiring unreliable makes ad-hoc agent testing unreliable too.
The Farnam Street essay on job interviews is a quiet demolition of professional intuition. The core argument is that unstructured conversations feel revealing precisely because humans are good at constructing coherent narratives from thin evidence. We come away from a thirty-minute chat convinced we know someone. We don't. What predicts actual job performance is structured assessment: standardized questions, work samples, and sober attention to base rates. Confidence in a candidate correlates poorly with competence in a role.
The translation is uncomfortable for anyone who has approved an agent workflow because a demo felt smooth. A fluent output in a controlled test is the agent equivalent of a candidate who interviews well — it activates the same false confidence. The real questions are structural: does the agent perform consistently across varied inputs, edge cases, and degraded conditions? What does the base rate of failure look like across a hundred runs, not three? Founders who build evaluation frameworks before they build trust into production will catch what a good demo hides.
- Treat agent evaluation like structured hiring, not a vibe check
- run enough volume to see base-rate failure before you trust a workflow with real stakes
- the smoothness of a demo is a liability if you let it substitute for evidence.
