Interviewing developers when agents write the code: what to test now

Most developer interview loops still measure the thing a coding agent now does for free. The candidate inverts a tree on a whiteboard, or spends a weekend on a take-home that Claude Code finishes before lunch. If your loop scores how fast a person turns a clean spec into working code, it scores a skill whose price has collapsed. What we test instead, after rebuilding our own loop and several clients’ loops: how a candidate specifies work for an agent, how they verify what comes back, how they debug what the agent cannot reach, and what they refuse to delegate.

The live delegation exercise

Give the candidate a real ticket from a sandbox repository: context files in place, a documented test command, a coding agent installed and logged in. Forty-five minutes, screen shared. Strong candidates read the context file, run the tests once to see them green, and ask two questions about scope before they type a prompt. Then they write a task with a goal, boundaries (which files, which layer) and a definition of done that names the tests. When the agent returns, they run the suite themselves before reading the summary, and when the agent misunderstood something they correct the framing as well as the output.

Weak candidates paste the ticket verbatim, watch the tokens stream, and accept the first diff that compiles. We keep the ticket small enough that a strong candidate finishes early, with time to discuss what they would have done with more context.

Reviewing a diff with a planted bug

Hand them a pull request an agent wrote, around 150 lines, with one bug we put there: an off-by-one in a pagination cursor, a retry with no backoff, a test that asserts the mock instead of the behavior. Ask for the review they would give as the merge gate. We score three things: whether they find the bug, how long it takes, and what else they flag on the way (naming, a missing edge case, a needless dependency).

This exercise maps onto the job more closely than any other, because review has become the bottleneck on teams that use agents. We also ask what they would want in the task description to prevent the bug next time, which separates people who review from people who fix the process.

Debugging what the agent cannot see, and designing with it in the room

The third exercise is a failure the agent has no access to: an integration test that fails only against the real staging database, or a timezone bug from a server in another region. The candidate gets logs and a shell, and the agent gets nothing useful. We watch for a hypothesis, a way to confirm it, and a willingness to say "I would bisect this" instead of guessing.

Then the design conversation, with one addition. After the candidate sketches the feature, ask where the agent sits: which pieces they would delegate, which they would write by hand, and what has to be in CI before they trust the delegated pieces. People who have shipped with agents answer with specifics: the tests they would write first, the module boundaries they would set, the migration they would keep for themselves. People who have not talk about AI in general. Judgment about what to keep away from the agent (authentication, billing, anything touching live customer data) counts as much as the rest, and it is the line that separates engineering from vibe coding.

Keeping the loop fair across tools, and what to do with take-homes

Candidates arrive with different tools. Let them bring their own (Claude Code, Cursor, Copilot) or use ours with its context file, and score the process rather than the tool. For a candidate who has never used an agent, run the same exercises with the agent as an option they may ignore. A strong engineer learns the tool in two weeks; the judgment behind the signals below took years, and that is what you are buying.

Take-homes are finished as a measure of coding ability, since any agent completes them. Either drop them or invert them: send the candidate an agent-generated solution to your old take-home and ask for a written review with the changes they would require before merge. It costs the candidate an hour instead of a weekend and gives every interviewer the same artifact to score.

The three signals that predict a strong hire

Across the exercises the scoring grid has three columns. Specification quality: can the candidate turn a two-line ticket into a task with boundaries and a definition of done. Verification habit: do they run the tests before they read the summary, every time, without being asked. Taste: do they notice when code is correct and ugly, and do they care enough to say so. Years of experience predict these less than you would expect. Juniors who learned on agents often show the verification habit from day one, and the skeptical seniors usually score highest on taste.

Rebuilding a loop takes a few days: one sandbox repository, one planted-bug pull request, one scoring grid every interviewer uses. We build it with clients as a side-stream of an agentic engineering engagement, once the team has used the tools long enough to know what it is hiring for.