Insights

Why AI Architecture Starts with Evaluation

A component built on a language model has no testable specification until its evaluation exists. Writing the cases and the acceptance bar first turns model, retrieval and prompt choices into measured architecture decisions.

Published
Reading time
3 min read
  • Evaluation
  • Architecture
  • Generative AI

Most architecture work starts from requirements that can be tested: a response-time target, a recovery objective, an access rule. A component built on a language model breaks that habit. The same input can produce different outputs, and "the answer looks right" is not something a reviewer can verify twice in the same way. Until an evaluation exists, the system has no specification for its most important behavior.

How to Evaluate an LLM System An evaluation architecture combining held-out cases, deterministic and evidence checks, a calibrated model judge, human review, versioned reporting, a regression release gate, and curated production feedback. How to Evaluate an LLM System Combine deterministic checks, calibrated judges, and human review. TEST INPUTS EVALUATION a judge is not ground truth RELEASE AND FEEDBACK Held-out test inputs Outputs + retrieved evidence Outputs + rubric Labelled calibration cases Disagreements + samples Check results Judge results + uncertainty Reviewed judgments Versioned evaluation report Pass defined thresholds Reject; revise candidate Monitored outcomes Curated future cases · never mutate the current held-out set DATA · VERSIONED Held-out cases + rubrics final evaluation only MODEL · CANDIDATE Versioned LLM system prompt · retrieval · tools · model DETERMINISTIC Deterministic + evidence checks MODEL · CALIBRATED Calibrated model judge score + uncertainty NOT GROUND TRUTH HUMAN Human review + calibration DATA · REPORT Quality, safety, cost, latency judge uncertainty retained versions and thresholds recorded DETERMINISTIC Regression release gate defined thresholds · no silent swap PASS → PROMOTE EXTERNAL · APPROVED Approved release DATA · FEEDBACK Production feedback curate before future evaluation Flow legend Evaluation data Release control Feedback Rejected release Calibration examples remain separate from final held-out cases. Production feedback creates a future test version.
Architecture LabHow to Evaluate an LLM SystemReference architecture - illustrative, not a client deployment.Open the study and its walkthrough

Evaluation is the specification

OpenAI's evaluation guide starts from that nondeterminism: because a model can return different outputs for the same input, conventional software tests are not enough. Its workflow defines the objective and collects a dataset before it defines metrics, compares runs and evaluates continuously. Anthropic's documentation puts the order plainly: building an LLM application "starts with clearly defining your success criteria and then designing evaluations to measure performance against them."

The idea predates generative AI. Google's Rules of Machine Learning lists "First, design and implement metrics" as its second rule, before any rule about models. NIST's AI Risk Management Framework states that AI systems "should be tested before their deployment and regularly while in operation."

What changes when the evaluation comes first

An evaluation set written early does three architectural jobs.

  • It makes options comparable. Prompting, retrieval, fine-tuning, a smaller model or a larger one can run against the same cases. The choice becomes a measured trade-off instead of a preference.
  • It defines the interfaces. To grade an answer, the team must decide what the system returns: citations, a refusal reason, a structured field, a confidence signal. Those decisions shape the components behind it.
  • It sets the release gate. A change to a prompt, a model version or an index ships when it meets the agreed bar on the same cases, not when a demonstration looks better.

What a first evaluation needs

A useful first set is small and specific. It covers the main task, the known edge cases and the requests the system must refuse. Each case states what a correct outcome is, in terms a reviewer can check. OpenAI's guide lists "vibe-based evals" and datasets that do not reflect production traffic among the patterns to avoid.

Automated grading helps a set grow, within limits. The MT-Bench study found that strong model judges reached over 80% agreement with human preferences, about the level at which humans agree with each other. The same study documented position, verbosity and self-enhancement biases. An automated grader should be calibrated against human judgments before it is trusted, with a human-reviewed sample kept after that.

The cost of starting elsewhere

Without an evaluation, the first design is usually chosen by demonstration. Every later change, such as a cheaper model, a new chunking strategy or an extra tool, has to be argued again, because there is no shared measure of better or worse. Evaluation then arrives late, after the interfaces are fixed, and the team discovers that the system never returned what a reviewer would need to check it.

EVALUATION FIRST STEP 1 Success criteria A correct outcome,checkable bya reviewer STEP 2 Evaluation set Main task,edge cases,must-refuse requestsAgreed acceptance bar STEP 3 Options comparedon the same cases Prompting, retrieval,fine-tuning, a smalleror larger model STEP 4 Release gate A change shipswhen it meetsthe bar evaluate continuously in production Defines the interfaces: citations, refusal reason, structured fields, confidence signal DEMONSTRATION FIRST STEP 1 Demo picks thefirst design Without anevaluation STEP 2 Interfaces fixed What the systemreturns is settled STEP 3 Evaluationarrives late The system never returnedwhat a reviewer needsto check it STEP 4 Every changeargued again Cheaper model,new chunking,extra tool no shared measure of better or worse
Evaluation first gives every later change a test to pass; demonstration first leaves it to argument.

Starting with evaluation does not slow the design down. It gives every later decision a test to pass. The Architecture Lab study How to Evaluate an LLM System walks through the components of such a loop.

Sources

  1. Evaluation best practices

    OpenAIdevelopers.openai.comChecked

  2. Define success criteria and build evaluations

    Anthropicplatform.claude.comChecked

  3. Rules of Machine Learning: Best Practices for ML Engineering

    Martin Zinkevich, Google for Developersdevelopers.google.comChecked

  4. AI Risk Management Framework 1.0, section 5.3 (Measure)

    NISTairc.nist.govChecked

  5. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    Zheng et al., NeurIPS 2023 Datasets and Benchmarksarxiv.orgChecked

Related