Verifier Engineering: How to Build the Checks That Make AI Agents Trustworthy

Published by

on

Glowing gateway with AI shield directing green progress paths and orange warning paths

I built a documentation agent to cover every endpoint in a backend API. It ran for 40 minutes, produced a clean 3,000-word spec, and confidently reported it had covered all 47 endpoints. I shipped it.

Two weeks later, a colleague opened the spec to check a specific endpoint. It wasn’t there. We dug further. Twelve of the 47 endpoints were missing. The agent had written thorough documentation for the 35 it found — then verified its own completeness by rereading what it had written. It concluded the docs were done. They weren’t.

The loop ran perfectly. The agent worked hard. The output looked complete. And it was wrong in a way that took two weeks to surface.

That’s the verifier problem. An agent without an independent check is a system that knows how to look done without being done. Verifier engineering is designing the check that catches the difference.

Related: This is part of a series on engineering AI agents. See Loop Engineering for autonomous cycle design, and Harness Engineering for the operational wrapper. Verifier engineering cuts across both — every loop needs a gate, every harness needs a check.


Where Verification Fits in the Stack

Verification isn’t a new concept — every engineering discipline has gatekeeping mechanisms. In AI agent development it became the critical layer once loops became autonomous enough to run unattended.

LayerEraWhat You ControlBottleneck It Solved
Prompt Engineering2022–2024The words you sendGetting the model to understand you
Context Engineering2025Everything the model seesGiving the model the right information
Harness EngineeringEarly 2026The environment around the modelTools, guardrails, error recovery
Loop EngineeringMid 2026The autonomous cycle that drives the agentRemoving the human from the iteration loop
Verifier EngineeringCuts across all layersThe independent check on every outputCatching what the agent gets wrong

Verifier engineering isn’t a fifth layer above loop engineering — it’s a discipline that cuts vertically through all the others. You need verifiers at the harness level (did the tool call return valid output?), at the loop level (is the task actually complete?), and at the prompt level (did the model follow the instruction?). Every layer needs a check.


The Separation Principle

Before patterns and code: one rule that overrides everything else.

Never let the model that did the work verify the work.

This sounds obvious. In practice, it isn’t. When you’re building quickly, the easiest path is a final prompt: “Review your output. Does it meet the requirements? Answer yes or no.” The model almost always says yes. It isn’t lying — it genuinely believes the output is correct, because it built a reasoning path that produced that output. It can’t see its own blind spots.

The Separation Principle requires a verifier that is structurally independent from the generator:

  • A different model instance with no access to the generation context
  • A deterministic check (test suite, linter, type checker) that doesn’t reason at all
  • A human with fresh eyes
  • An external system (CI pipeline, integration test)

The verifier must be able to catch an error the generator can’t see. If the generator and verifier share the same blind spot, the check is theatre.


Anatomy of a Verifier

Every verifier has four parts. Miss one and the check either never catches failures or never lets anything through.

1. The Check

What you’re actually testing. Is it binary (pass/fail) or scored (0–1)? Deterministic (same input, same result) or probabilistic (a model evaluates quality)? The best checks are deterministic and binary — but not all tasks allow that.

2. The Signal

How the verifier communicates its result. An exit code (0 = pass, non-zero = fail) is the most reliable signal. Structured JSON is second. Free-text from a model is the weakest — you need to parse it, and the model might hedge.

3. The Feedback

What gets sent back to the agent when the check fails. Bad feedback: “Verification failed.” Good feedback: “Tests failed. 3 assertions in test_inventory.py. Output:\n[full test output].” The richer the feedback, the better the agent’s next attempt.

4. The Threshold

When does the verifier say “good enough”? For tests, it’s exit code 0. For a model quality score, you decide: is 0.85 acceptable? 0.95? Set the threshold before the loop runs — not adjusted mid-run because the agent is struggling.


Verifier Patterns: Five Ways to Check Agent Work

Pattern 1: The Hard Gate

A deterministic check with binary output. Either the work passes or it doesn’t. No negotiation.

import subprocess
def hard_gate() -> tuple[bool, str]:
"""Run the test suite. Pass or fail, no middle ground."""
result = subprocess.run(
["pytest", "--tb=short", "-q"],
capture_output=True, text=True
)
return result.returncode == 0, result.stdout + result.stderr
def type_check_gate() -> tuple[bool, str]:
result = subprocess.run(
["npx", "tsc", "--noEmit"],
capture_output=True, text=True
)
return result.returncode == 0, result.stdout + result.stderr
def lint_gate() -> tuple[bool, str]:
result = subprocess.run(
["ruff", "check", "src/"],
capture_output=True, text=True
)
return result.returncode == 0, result.stdout

Best for: Code generation, any task where correctness is binary and testable. This is the gold standard — if your task allows a hard gate, use it as the primary verifier.

Failure mode: The test suite itself is incomplete or wrong. The agent writes code that passes tests but doesn’t meet requirements. Tests are a floor, not a ceiling.


Pattern 2: The Soft Gate

A separate model reads the output and evaluates it against criteria. Useful when “correct” isn’t binary — quality, completeness, tone, coverage.

import anthropic
import json
client = anthropic.Anthropic()
def soft_gate(output: str, criteria: str) -> tuple[bool, str]:
"""Use a separate small model to evaluate output quality."""
response = client.messages.create(
model="claude-haiku-4-5-20251001", # Cheap, fast evaluator
max_tokens=512,
system=(
"You are a strict evaluator. Assess whether output meets the specified "
"criteria. Never give benefit of the doubt. Return only valid JSON: "
"{\"pass\": true/false, \"score\": 0.0-1.0, \"issues\": [\"...\"]}"
),
messages=[{
"role": "user",
"content": f"Criteria:\n{criteria}\n\nOutput to evaluate:\n{output}\n\nEvaluate strictly."
}]
)
result = json.loads(response.content[0].text)
passed = result["pass"] and result["score"] >= 0.9
feedback = f"Score: {result['score']}. Issues: {', '.join(result['issues']) or 'none'}"
return passed, feedback

Best for: Documentation completeness, content quality, whether a plan covers all requirements.

The counterintuitive insight: Use a smaller model (Haiku) as the evaluator, not your most capable model. The evaluator’s job is pattern-matching against criteria — Haiku is fast and cheap enough for that. More importantly, a smaller model is less likely to share the same systematic biases as the larger model that generated the output.


Pattern 3: The Multi-Layer Gate

Chain hard and soft gates. Hard gates catch correctness failures first. Soft gates catch completeness and quality failures. Both must pass.

def multi_layer_gate(task_type: str, output: str) -> tuple[bool, str]:
"""
Hard gates run first — fast and binary.
Soft gate runs only if hard gates pass.
"""
# Layer 1: Tests must pass
tests_passed, test_output = hard_gate()
if not tests_passed:
return False, f"Tests failed:\n{test_output}"
# Layer 2: Type check must pass
types_passed, type_output = type_check_gate()
if not types_passed:
return False, f"Type errors:\n{type_output}"
# Layer 3: Quality check — only if hard gates pass
if task_type == "documentation":
criteria = (
"The documentation must: cover all public functions, "
"include at least one example per function, "
"have no broken internal links, contain no TODOs."
)
quality_passed, quality_feedback = soft_gate(output, criteria)
if not quality_passed:
return False, f"Quality check failed: {quality_feedback}"
return True, "All gates passed."

Best for: Production loops where you need both correctness and quality. The layering keeps costs low — cheap deterministic checks run first and filter most failures before the model evaluator runs.


Pattern 4: The Sampling Gate

When you can’t verify every item, verify a random sample. Useful for batch operations where 100% verification is impractical.

import random
from typing import Callable
def sampling_gate(
items: list[str],
verify_fn: Callable,
sample_rate: float = 0.2
) -> tuple[bool, str]:
"""
Verify a random sample of items.
Fail the whole batch if any sampled item fails.
"""
sample_size = max(1, int(len(items) * sample_rate))
sample = random.sample(items, sample_size)
failures = []
for item in sample:
passed, feedback = verify_fn(item)
if not passed:
failures.append(f"Item '{item[:50]}...': {feedback}")
if failures:
return False, f"{len(failures)}/{sample_size} sampled items failed:\n" + "\n".join(failures)
return True, f"Sampling gate passed: {sample_size}/{len(items)} items checked."

Best for: Bulk content generation, large dataset transformations, batch API calls.

The math: A 20% sample rate catches any error present in 20%+ of items with ~64% probability. For high-stakes tasks, use 50%+ or switch to full verification.


Pattern 5: The Escalation Gate

When the automated verifier can’t reach a confident decision, escalate to a human rather than forcing a binary verdict. The key is designing escalation in from the start — not as a fallback for everything, but as an explicit choice for specific uncertainty conditions.

def escalation_gate(
output: str,
criteria: str,
confidence_threshold: float = 0.7
) -> tuple[bool, str]:
"""
Run soft gate. If confidence is low, escalate to human review
rather than forcing a pass/fail decision.
"""
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=512,
system=(
"Evaluate output strictly. Return JSON: "
"{\"pass\": bool, \"score\": float, \"confidence\": float, \"issues\": []} "
"confidence reflects how certain you are of your assessment (0.0-1.0)."
),
messages=[{"role": "user", "content": f"Criteria: {criteria}\n\nOutput: {output}"}]
)
result = json.loads(response.content[0].text)
if result["confidence"] < confidence_threshold:
notify_human_reviewer(output, criteria, result)
return False, f"Low confidence ({result['confidence']:.2f}). Flagged for human review."
passed = result["pass"] and result["score"] >= 0.85
return passed, f"Score: {result['score']}, Issues: {result['issues']}"
def notify_human_reviewer(output: str, criteria: str, evaluation: dict):
"""Route to Slack, email, or ticket system — whatever fits your workflow."""
print(f"[HUMAN REVIEW NEEDED] Confidence: {evaluation['confidence']:.2f}")
print(f"Issues flagged: {evaluation['issues']}")

Best for: High-stakes decisions, compliance tasks, anything where a wrong automated verdict is worse than a delay. This isn’t a failure mode — it’s a deliberate design choice that says “this category of decision requires a human.”


The Verifier Strength Hierarchy

When choosing your verifier, use the strongest option your task allows:

Verifier TypeStrengthSpeedCostExample
Self-assessmentWeakestFastFree“Agent, are you done?”
Separate model (same tier)WeakFastLowSonnet evaluates Sonnet’s output
Separate model (smaller)MediumFastVery lowHaiku evaluates Sonnet’s output
Deterministic checkStrongVery fastFreepytest, tsc, ruff
External validationStrongestSlowVariesCI pipeline, integration tests
Human reviewAuthoritativeSlowestHighCode review, sign-off

In practice: combine them. Run deterministic checks as the primary gate. Add a soft gate for tasks where “correct” isn’t binary. Escalate to human review when confidence is low or stakes are high.


What Didn’t Work: Three Lessons

The Yes-Saying Evaluator

I asked the generator to evaluate its own documentation: “Read the docs you just wrote and the original API spec. Does the documentation cover all endpoints? Yes or no.”

It said yes. Every single time. Even when endpoints were missing. The model wasn’t lying — it had just spent 40 minutes building a mental model of the docs and was primed to see them as complete. Confirmation bias is a model failure mode, not just a human one.

Fix: Structurally separate the evaluator. A fresh API call with no context from the generation step, explicitly instructed to be skeptical.

The Test Suite That Lied

I used test-suite passage as my sole hard gate. The tests passed. The code was wrong. The test suite had been written alongside the original buggy implementation and was testing the wrong behavior — verifying against a flawed specification.

Fix: Periodically audit your verifiers. For new features, write tests before the agent implements — don’t let the agent write both the code and the tests in the same pass. The tests are only as trustworthy as the requirements they encode.

The Verifier That Was Too Strict

I set my quality threshold at 0.95 for a content generation loop. The soft gate evaluator — a different model with different calibration — consistently scored acceptable outputs at 0.87–0.91. The loop ran 20 iterations and never passed. The outputs were good. The threshold was wrong.

Fix: Calibrate thresholds before the loop runs. Generate 10 examples manually, score them with your verifier, and set the threshold at the 80th percentile of acceptable outputs. Don’t pick thresholds from intuition.


Verifier Engineering in Claude Code

Claude Code’s Stop hooks are native verifier engineering primitives. Every time Claude tries to finish a task, the hook runs before the session closes. If the hook exits non-zero, Claude doesn’t stop — it sees the error output as context and keeps working.

{
"hooks": {
"Stop": [
{
"type": "command",
"command": "pytest --tb=short -q && npx tsc --noEmit && ruff check src/",
"timeout": 60000
}
]
}
}

This is the Separation Principle in the tooling itself: Claude cannot override the hook result. The verifier is structurally independent.

For soft gates, write a hook that calls a model:

#!/bin/bash
# verify-quality.sh — runs as a Stop hook
TRANSCRIPT=$(cat .claude/session.log | tail -100)
RESULT=$(claude --model claude-haiku-4-5-20251001 -p "
Read this work summary. Evaluate whether the task is truly complete.
Criteria: all public functions documented, examples included, no TODOs remaining.
Return PASS or FAIL followed by a one-line reason.
Summary:
$TRANSCRIPT
")
if echo "$RESULT" | grep -q "^PASS"; then
exit 0 # Allow Claude to stop
else
echo "Quality gate: $RESULT"
exit 1 # Force Claude to continue
fi

The /goal command is a higher-level wrapper: it defines a success condition that Claude keeps working toward, with Haiku checking every turn. Under the hood, it’s a Stop hook with a model-based verifier — the same pattern, packaged.


When to Invest in Verifier Engineering

Not every task needs a custom verifier. Start simple and add sophistication where failures are expensive.

Use a hard gate when:

  • The task produces testable output (code, structured data, schemas)
  • Correctness is binary and you can define it precisely
  • Speed matters — deterministic checks are fast and free

Use a soft gate when:

  • The task produces qualitative output (documentation, content, plans)
  • “Complete” is a matter of degree, not binary
  • You can write clear evaluation criteria in advance

Use escalation when:

  • The cost of a wrong automated verdict exceeds the cost of delay
  • The task involves compliance, safety, or legal implications
  • The domain is new enough that your criteria aren’t calibrated yet

Skip the elaborate verifier when:

  • A human reviews the output before it matters
  • The task is low-stakes and runs rarely
  • A simple check (file exists, JSON parses, exit code 0) covers 95% of failures

My rule from Loop Engineering still holds: if you can’t write the verifier, you can’t write the loop. Design the check first. The loop follows from that.


Final Thoughts: The Check Is the Contract

Writing a verifier forces a discipline that prompting doesn’t: you have to state what “done” means precisely enough for a machine to evaluate. That specificity is itself valuable. Vague goals produce vague outputs. A well-designed verifier is a precise specification of success.

Three principles I apply to every verification layer I build:

  1. Separation over self-assessment. The generator cannot grade its own work. Ever. Not as a shortcut, not as a fallback. A separate check is non-negotiable.
  2. Hard gates first, soft gates second. Deterministic checks are cheap, fast, and unfakeable. Run them first. Escalate to model-based evaluation only when deterministic checks can’t cover the task.
  3. Calibrate before you automate. Set thresholds against real examples before the loop runs. A miscalibrated verifier that’s too strict loops forever. One that’s too lenient passes bad work. The threshold is a design decision, not a default.

The agent does the work. The verifier decides if the work is real. Both are engineering problems. Only one of them gets written twice.


FAQ

What is verifier engineering? Verifier engineering is designing independent checks that evaluate whether an AI agent’s output is correct and complete — without relying on the agent to assess its own work.

Why can’t the AI verify its own output? The model that generates output built a reasoning path that made the output look correct. It lacks the outside view needed to catch its own systematic errors or omissions. An independent check doesn’t share those blind spots.

What’s the difference between a hard gate and a soft gate? A hard gate is deterministic — a test suite, type checker, or linter returning a binary pass/fail. A soft gate uses a separate model to evaluate quality, completeness, or criteria that can’t be reduced to a binary check.

Why use a smaller model (Haiku) as the evaluator? Haiku is cheaper and faster for pattern-matching tasks. More importantly, it’s less likely to share systematic biases with the larger model that generated the output — independent failure modes make the check more meaningful.

How do Stop hooks work in Claude Code? Stop hooks are shell commands that run whenever Claude tries to finish a task. A non-zero exit code prevents Claude from stopping and sends the hook’s output to Claude as context. Claude cannot override or fake a hook result — that’s what makes them a genuine Separation Principle implementation.


Part of the AI Engineering series. See also: Loop Engineering, Harness Engineering, Building Multi-Agent AI Systems.

Leave a Reply