Champion–Challenger for LLMs: A Practical Guide with Spring Boot

A few days ago, I got a request at work: “GPT-4 is reaching its end-of-life, and we need to replace it. There are various options, but which one should we choose, and how do we decide?”

That question is exactly why Champion-Challenger testing is a must-have pattern in any serious AI developer’s toolkit. It is how you move from guessing to making data-driven decisions.

This guide walks through how to implement this pattern using Spring Boot. It includes real code snippets from a translation service that runs a “champion” model (our trusted production version) and a “challenger” model (the new experiment) in the background.

What We Will Cover

What is Champion-Challenger testing and why LLMs need it.
Shadow mode: running a challenger model invisibly in production.
Parallel execution for zero added latency.
Logging metrics: cost, latency, tokens, and timestamps.
Measuring output quality with a similarity score.
A real-time comparison API for data-driven decisions.

What Is Champion-Challenger in AI and LLM Systems?

In simple terms, Champion-Challenger is a structured evaluation framework for your AI models that runs safely in production.

The Champion is the current, trusted production model. It is the one the users interact with.
The Challenger is the new experiment. It could be a different model, a tweaked prompt, or new parameters.

Both models get the exact same input in real-time. The key is that only the Champion’s output is shown to the user. The Challenger runs in the background (often called “shadow mode”), and its performance is logged for you to analyze later.

This is a game-changer for LLM-powered features because, as we all know, even tiny changes can have huge, unexpected consequences.

Why Do LLM Applications Need This?

Unlike traditional software, LLMs are not deterministic. A single word change in a prompt can drastically alter the output’s quality, tone, cost, and latency. Without a proper testing framework, you are flying blind.

Champion-Challenger testing helps you avoid common disasters:

Quality Regressions: Your new, “smarter” prompt accidentally makes the AI sound rude or gives less accurate translations.
Cost Explosions: The new model is 5x more expensive because it uses way more tokens for the same task.
Latency Spikes: The new model is better but so slow it ruins the user experience.

This pattern lets you catch these issues before they impact a single user.

Architecture: How It Works in Spring Boot

Here is the full request lifecycle in our Spring Boot translation service:

The flow is straightforward: the Champion’s response goes to the user, while both models’ performance metrics are sent to a log store. The comparison API aggregates those metrics for data-driven model selection, and the promote API lets you swap models at runtime.

Implementing the Pattern: Code Walkthrough

Let’s walk through the real code from a working Spring Boot project.

The Use Case: A user invokes a Translation Service to translate text to their preferred language. The service internally calls two AI models in parallel — a Champion (production) and a Challenger (experimental) — logging their results for analysis. Only the Champion’s translation reaches the user.

1. The Controller: True Shadow Mode

The controller is where shadow mode is enforced. The internal TranslationResponse carries data from both models, but the user-facing TranslationApiResponse exposes only the champion translation:

			
@PostMapping
public ResponseEntity<TranslationApiResponse> translate(
        @Validated @RequestBody TranslationRequest request) {
    TranslationResponse internal = translationService.translate(request);
    TranslationApiResponse apiResponse = new TranslationApiResponse(
        internal.getChampionTranslation(),
        internal.getChampionLog(),
        internal.getSimilarityScore()
    );
    return ResponseEntity.ok(apiResponse);
}

		

This is a crucial design choice. The challenger never leaks to the user. They get the champion translation, the champion’s performance log, and a similarity score that tells them how close the two models agreed — without ever seeing the challenger’s output.

2. The DTOs: What We Track

Each model call produces a ModelLog with everything you need for analysis:

			
public class ModelLog {
    private String modelName;      // which model produced this
    private int totalTokens;
    private BigDecimal cost;       // BigDecimal avoids scientific notation like 6.8E-4
    private long latencyMs;
    private Instant timestamp;     // when the call happened
}

		

Having the modelName and timestamp on every log entry makes it possible to build time-series dashboards and trace exactly which model produced which result — especially important after a promotion when roles change.

3. The Service: Parallel Execution

This is the heart of the implementation. Both models are called concurrently using an ExecutorService, so the challenger adds zero extra latency to the user’s request:

			
@Value("${app.llm.champion.model}")
private volatile String championModel;    // volatile for runtime promotion
@Value("${app.llm.challenger.model}")
private volatile String challengerModel;
private final ExecutorService executor = Executors.newFixedThreadPool(2);
@Override
public TranslationResponse translate(TranslationRequest request) {
    String text = request.getText();
    String language = request.getPreferredLanguage();
    // Snapshot model names for thread safety during promotion
    String champModel = this.championModel;
    String challModel = this.challengerModel;
    // Run both models in parallel
    Future<TranslationResult> championFuture =
        executor.submit(() -> callLlm(text, language, champModel));
    Future<TranslationResult> challengerFuture =
        executor.submit(() -> callLlm(text, language, challModel));
    TranslationResult championResult = awaitResult(championFuture);
    TranslationResult challengerResult = awaitResult(challengerFuture);
    // Measure output similarity
    double similarity = computeSimilarity(championResult, challengerResult);
    // Log everything
    ModelLog championLog = buildLog(championResult, champModel);
    ModelLog challengerLog = buildLog(challengerResult, challModel);
    // ...
}

		

Note the volatile keyword on the model fields and the local snapshots (champModel, challModel). This ensures that if a promotion happens mid-request, the current request completes with a consistent pair of models.

4. Output Quality: Similarity Scoring

Cost and latency are easy to measure, but how do you compare output quality between two models? We use a Levenshtein similarity score — a normalized edit distance that produces a value between 0.0 (completely different) and 1.0 (identical):

			
public static double levenshteinSimilarity(String a, String b) {
    if (a == null || b == null) return 0.0;
    if (a.equals(b)) return 1.0;
    int maxLen = Math.max(a.length(), b.length());
    if (maxLen == 0) return 1.0;
    int distance = levenshteinDistance(a, b);
    return 1.0 - ((double) distance / maxLen);
}

		

This is computed per request and stored alongside the logs. Error responses are excluded from the similarity calculation:

			
private double computeSimilarity(TranslationResult champion, TranslationResult challenger) {
    String a = champion.getTranslation();
    String b = challenger.getTranslation();
    if (a == null || b == null || a.startsWith("[error]") || b.startsWith("[error]")) {
        return 0.0;
    }
    return SimilarityUtil.levenshteinSimilarity(a, b);
}

		

A high average similarity (e.g., 0.95+) means the challenger produces nearly identical output — so the decision comes down to cost and speed. A low similarity is a red flag that warrants closer inspection before any promotion.

5. Configurable Cost Estimation

Cost rates are defined per model in application.yml and stored in a ConcurrentHashMap keyed by model name. This means cost tracking survives model promotions correctly:

			
app:
  llm:
    champion:
      model: gpt-4o-mini
      costPer1kTokens: 0.02
    challenger:
      model: gpt-5
      costPer1kTokens: 0.01

		

			
private final ConcurrentHashMap<String, BigDecimal> costRates = new ConcurrentHashMap<>();
@PostConstruct
public void initCostRates() {
    costRates.put(championModel, championCostRate);
    costRates.put(challengerModel, challengerCostRate);
}
private BigDecimal estimateCost(int tokens, String model) {
    BigDecimal rate = costRates.getOrDefault(model, BigDecimal.valueOf(0.01));
    return BigDecimal.valueOf(tokens)
            .multiply(rate)
            .divide(BigDecimal.valueOf(1000), 6, RoundingMode.HALF_UP);
}

		

By keying cost rates on the model name (not the champion/challenger role), the rates stay accurate even after a promotion swaps which model is champion.

6. The API Response: What the Caller Sees

When you call POST /api/translate, the user sees only the champion’s result:

			
{
  "translation": "Bonjour, comment allez-vous ?",
  "championLog": {
    "modelName": "gpt-4o-mini",
    "totalTokens": 34,
    "cost": 0.000680,
    "latencyMs": 475,
    "timestamp": "2026-04-17T10:23:45.123Z"
  },
  "similarityScore": 0.92
}

		

The challenger’s translation is nowhere in the response. But notice the similarityScore of 0.92 — this tells you the two models agreed on 92% of the output. That is a strong signal, but you still want to look at cost and speed before deciding.

7. The Comparison API: Data-Driven Decisions

This is the punchline. GET /api/compare returns aggregated metrics for both models side by side:

			
{
  "champion": {
    "modelName": "gpt-4o-mini",
    "avgLatencyMs": 482.5,
    "avgCost": 0.000720,
    "avgTokens": 36.0,
    "totalCalls": 150
  },
  "challenger": {
    "modelName": "gpt-5",
    "avgLatencyMs": 3842.0,
    "avgCost": 0.002340,
    "avgTokens": 234.0,
    "totalCalls": 150
  },
  "avgSimilarityScore": 0.94
}

		

Even with a 94% similarity score, you can immediately see the challenger is 8x slower and 3x more expensive. That is a data-driven reason to keep your champion — or a clear signal to try a different challenger. The implementation aggregates from the in-memory log store:

			
@Override
public ComparisonResponse getComparison() {
    ModelStats champStats = computeStats(championLogs, championModel);
    ModelStats challStats = computeStats(challengerLogs, challengerModel);
    double avgSimilarity = similarityScores.isEmpty() ? 0.0 :
        similarityScores.stream()
            .mapToDouble(Double::doubleValue)
            .average()
            .orElse(0.0);
    return new ComparisonResponse(champStats, challStats, avgSimilarity);
}

		

This lets you A/B test prompts the same way you A/B test models — measure the impact on quality, cost, and latency, then decide with data.

But Can’t I Just Test This in a Lower Environment?

This is a common question: “Why do this in production? Can’t we just test in dev or staging?”

You can, and you should. But it is not enough. Lower environments are great for catching functional bugs and running static tests. However, they cannot replicate the sheer variety and unpredictability of real-world user inputs.

Only by testing with live production traffic (in a safe, shadow mode) can you be confident that your new model or prompt will perform well across all the edge cases your users will throw at it.

Key Benefits of This Approach

Data-Driven Decisions: Stop guessing which model or prompt is better. You will have hard numbers on cost, latency, token usage, and output quality.
True Shadow Testing: The challenger is completely invisible to users. Zero risk to production.
Zero Added Latency: Parallel execution means the challenger does not slow down the user’s request.
Quality Measurement: Similarity scoring gives you an objective signal on output consistency between models.
Live Model Promotion: Swap models at runtime without redeployment or downtime.
Prompt Experimentation: Test different system prompts per model with the same metrics framework.
Continuous Improvement: Create a flywheel where you are constantly challenging your production model to be better, faster, and cheaper.

What Didn’t Work / Honest Limitations

The Levenshtein similarity score works well for short, structured outputs like translations, but it is not a good fit for longer, free-form text where two semantically identical answers can have very different wording. For those cases, you would need embedding-based similarity or LLM-as-judge evaluation.

The in-memory log store is fine for prototyping but will not survive restarts. In production, you need to persist metrics to a database or time-series store.

Cost estimation based on token counts is an approximation. Actual billing from providers may differ slightly due to rounding, caching, or pricing tier changes.

Final Thoughts: Test Models the Way You Test Code

Champion-Challenger is more than a testing pattern — it is a core practice for building robust, production-grade AI systems. With Spring Boot, setting up a shadow testing pipeline is surprisingly straightforward. It empowers you to innovate quickly while keeping your application stable and your costs in check.

From here, you could persist metrics to a time-series store, build dashboards to visualize trends, add automated promotion rules, or integrate with Spring Boot Actuator to expose model metrics as Micrometer gauges. The pattern scales with your needs.

The principle is simple: never swap a production model based on vibes. Run the challenger in shadow mode, collect the data, and let the numbers tell you when it is time to promote.