Adaptation & Evaluation

How to Evaluate an LLM System

Combine deterministic checks, calibrated model judgments, human review and versioned release gates without treating one score as truth.

Reference architecture - illustrative, not a client deployment

Combine deterministic checks, calibrated judges and human review.

Complexity
Advanced
Drawing
Supplied
  • LLM Evaluation
  • Quality
  • Safety
  • Observability

Overview

This reference architecture is illustrative. It is not a record of a client deployment.

An LLM evaluation is a versioned experiment, not a single benchmark score. Run a specific system version—model, prompt, retrieval, tools and configuration—against held-out cases with explicit rubrics. Use deterministic checks for properties that code can verify, such as JSON validity, citation resolution or a forbidden action. Use a model judge for criteria that require interpretation, but calibrate it against labelled human decisions and retain its uncertainty. A judge is evidence, not ground truth.

The report keeps quality, safety, latency and cost distinct so a strong average cannot hide a critical failure. A release gate compares the candidate with defined thresholds and the currently approved version. Failed candidates return for revision; they do not silently replace production. Monitored outcomes and reviewed incidents can become future test cases, but the current held-out set must not become training data.

A concrete example

A claims assistant must produce valid structured output, quote the supplied evidence and avoid approving a claim. Code checks the schema and prohibited tool calls. A calibrated judge applies a documented evidence-faithfulness rubric. Reviewers inspect disagreements and a stratified sample. The candidate is rejected if a critical safety case fails even when its aggregate quality score improves.

Common misconception

"An LLM judge removes the need for people." Model graders can scale rubric application, but their decisions vary with prompt, model and case distribution. Human labels are still needed to calibrate judges and resolve important disagreements.

Diagram & Walkthrough

Choose a declared scenario and step, or select a component by pointer, touch or keyboard. Playback starts only when you press Play and pauses when this panel leaves the screen.

100%
How to Evaluate an LLM System An evaluation architecture combining held-out cases, deterministic and evidence checks, a calibrated model judge, human review, versioned reporting, a regression release gate, and curated production feedback. How to Evaluate an LLM System Combine deterministic checks, calibrated judges, and human review. TEST INPUTS EVALUATION a judge is not ground truth RELEASE AND FEEDBACK Held-out test inputs Outputs + retrieved evidence Outputs + rubric Labelled calibration cases Disagreements + samples Check results Judge results + uncertainty Reviewed judgments Versioned evaluation report Pass defined thresholds Reject; revise candidate Monitored outcomes Curated future cases · never mutate the current held-out set DATA · VERSIONED Held-out cases + rubrics final evaluation only MODEL · CANDIDATE Versioned LLM system prompt · retrieval · tools · model DETERMINISTIC Deterministic + evidence checks MODEL · CALIBRATED Calibrated model judge score + uncertainty NOT GROUND TRUTH HUMAN Human review + calibration DATA · REPORT Quality, safety, cost, latency judge uncertainty retained versions and thresholds recorded DETERMINISTIC Regression release gate defined thresholds · no silent swap PASS → PROMOTE EXTERNAL · APPROVED Approved release DATA · FEEDBACK Production feedback curate before future evaluation Flow legend Evaluation data Release control Feedback Rejected release Calibration examples remain separate from final held-out cases. Production feedback creates a future test version.

Main walkthrough, step 1 of 6: Version the tests and system. idle.

Main walkthrough

The primary path through this reference architecture.

Active parts: Held-out cases + rubrics, Versioned LLM system

Text walkthrough

This vertical reference remains readable without the interactive controls.

  1. 01Version the tests and system

    Run a pinned system configuration against a held-out, versioned set of cases and rubrics.

    Active parts: Held-out cases + rubrics, Versioned LLM system

  2. 02Check objective constraints and evidence

    Apply deterministic assertions and send their case-level results to the report.

    Active parts: Versioned LLM system, Deterministic + evidence checks, Quality, safety, cost, latency

  3. 03Use a calibrated judge

    Calibrate the model grader on human-labelled examples, then score outputs under the documented rubric.

    Active parts: Human review + calibration, Versioned LLM system, Calibrated model judge, Quality, safety, cost, latency

  4. 04Review disagreements

    Escalate uncertain or disagreeing cases and preserve reviewed judgments in the report.

    Active parts: Calibrated model judge, Human review + calibration, Quality, safety, cost, latency

  5. 05Apply a release gate

    Compare the versioned report with predefined thresholds and promote only a passing candidate.

    Active parts: Quality, safety, cost, latency, Regression release gate, Approved release

  6. 06Curate feedback without contaminating held-out tests

    Monitor approved behavior and curate reviewed outcomes into a future test-set version.

    Active parts: Approved release, Production feedback, Held-out cases + rubrics

Connections

  • Held-out test inputsHeld-out cases + rubrics to Versioned LLM systemdata
  • Outputs + retrieved evidenceVersioned LLM system to Deterministic + evidence checksdata
  • Outputs + rubricVersioned LLM system to Calibrated model judgedata
  • Labelled calibration casesHuman review + calibration to Calibrated model judgecontrol
  • Disagreements + samplesCalibrated model judge to Human review + calibrationcontrol
  • Check resultsDeterministic + evidence checks to Quality, safety, cost, latencydata
  • Judge results + uncertaintyCalibrated model judge to Quality, safety, cost, latencydata
  • Reviewed judgmentsHuman review + calibration to Quality, safety, cost, latencydata
  • Versioned evaluation reportQuality, safety, cost, latency to Regression release gatecontrol
  • Pass defined thresholdsRegression release gate to Approved releasecontrol
  • Reject; revise candidateRegression release gate to Versioned LLM systemcontrol
  • Monitored outcomesApproved release to Production feedbackfeedback
  • Curated future casesProduction feedback to Held-out cases + rubricsfeedback

Components

Held-out cases + rubricsdata

Stores versioned representative inputs, expected constraints and scoring rubrics kept outside optimization.

Responsibilities

  • Define coverage and success criteria without leaking final tests into training

Inputs

  • Task specification
  • Curated failure modes
  • Rubrics

Outputs

  • Versioned held-out evaluation set

Risks

  • Test contamination
  • Missing rare or high-impact cases
Versioned LLM systemmodel

Runs the exact candidate configuration, including model, prompts, retrieval, tools and policies.

Responsibilities

  • Produce traceable outputs for each held-out case

Inputs

  • Held-out inputs
  • Pinned system configuration

Outputs

  • Outputs
  • Evidence and traces

Risks

  • Unrecorded configuration drift
  • Nondeterministic variation
Deterministic + evidence checksdeterministic

Tests objective properties such as schema validity, prohibited actions and evidence-reference resolution.

Responsibilities

  • Apply reproducible assertions where code can decide

Inputs

  • System outputs
  • Retrieved evidence
  • Expected constraints

Outputs

  • Pass/fail check results

Risks

  • Shallow proxies for quality
  • Incomplete assertions
Calibrated model judgemodel

Applies an explicit rubric to subjective criteria and reports judgments with uncertainty.

Responsibilities

  • Score outputs consistently against a versioned rubric

Inputs

  • Outputs
  • Rubric
  • Labelled calibration cases

Outputs

  • Judge results
  • Rationales and uncertainty

Risks

  • Systematic judge bias
  • Sensitivity to prompt or model changes
Human review + calibrationhuman

Labels calibration cases, reviews high-impact samples and resolves important disagreements.

Responsibilities

  • Provide domain-grounded reference judgments and escalation decisions

Inputs

  • Cases
  • Outputs
  • Judge disagreements

Outputs

  • Labelled cases
  • Reviewed judgments

Risks

  • Reviewer inconsistency
  • Fatigue and sampling bias
Quality, safety, cost, latencydata

Keeps distinct evaluation dimensions and traceable case-level results in one versioned report.

Responsibilities

  • Aggregate results without hiding critical failures in one score

Inputs

  • Check results
  • Judge results
  • Human judgments
  • Operational measures

Outputs

  • Versioned evaluation report

Risks

  • Misleading averages
  • Missing subgroup analysis
Regression release gatedeterministic

Compares the report with predefined thresholds and the currently approved system.

Responsibilities

  • Approve or reject promotion under explicit release policy

Inputs

  • Evaluation report
  • Versioned thresholds
  • Current baseline

Outputs

  • Promotion or rejection decision

Risks

  • Threshold gaming
  • Approving on incomplete evidence
Approved releaseexternal

Represents the promoted system version that passed the defined gate.

Responsibilities

  • Expose the approved version and support rollback

Inputs

  • Approved candidate

Outputs

  • Production behavior

Risks

  • Production distribution shift
  • Rollback gaps
Production feedbackdata

Collects monitored outcomes, incidents and reviewed user feedback for future evaluation versions.

Responsibilities

  • Curate new failure modes without altering the current held-out set

Inputs

  • Redacted production outcomes
  • Reviewed incidents

Outputs

  • Candidate future cases

Risks

  • Privacy leakage
  • Feedback biased toward vocal users

Scenario reference

Release rejected

Failure blocks promotion; the previous approved release is not silently replaced.

  1. 01Reject and revise the candidate

    The report reaches the gate, which sends a failing candidate back for a new version rather than promoting it.

    Active parts: Quality, safety, cost, latency, Regression release gate, Versioned LLM system

Design Decisions

  • Version cases, rubrics and the complete evaluated system.
  • Match each requirement to code-based, model-based or human grading.
  • Set release thresholds before seeing candidate results.
  • Separate regression suites from production monitoring and future-case curation.

Risks & Trade-offs

Test leakage rewards memorization. Vague rubrics create unstable scores. Aggregate metrics conceal subgroup failures, and repeated judge calls add cost and latency. Production feedback is valuable but biased toward observed users and reported problems.

When to Use / Not Use

Use this pattern before model, prompt, retrieval or tool changes and continue monitoring after release. Do not rely on it when success has not been defined, when a deterministic business rule should decide the outcome directly, or when a high-impact release lacks qualified human review.

Sources

Related