If you're delegating hiring-adjacent judgment to agents or letting agents screen candidates, the same biases that wreck human interviews will wreck your pipeline — just faster and at scale.
The Farnam Street essay on job interviews lands a finding that most hiring managers quietly ignore: unstructured conversations are nearly useless as predictors of job performance. Humans are pattern-matching machines who confuse fluency with competence, and liking someone with fit. The essay argues that work samples, structured questions asked identically across candidates, and honest reckoning with base rates — how often does anyone succeed in this role — produce dramatically better signal than a confident gut read after ninety minutes of rapport-building.
When agents run evaluation loops — screening applicants, scoring submissions, triaging inbound — they inherit whatever structure you give them, and no more. An agent handed a vague rubric will find confident-sounding proxies the same way a tired hiring manager does: surface polish, keyword density, response length. The discipline the essay demands of humans applies just as hard here. Define the work sample, lock the criteria before the first input arrives, and audit outputs against base rates rather than against the most recent result that felt right.
- structure your agent's evaluation criteria before it sees any data, not after
- work samples beat impressions whether the rater is human or automated
- base rates are a check on both human enthusiasm and agent drift.
