ML & MLOps

What MLOps Looks Like in the Real World

Follow a model from versioned data and leakage-safe evaluation through a gated release, production monitoring and a deliberate retrain or rollback decision.

Reference architecture - illustrative, not a client deployment

Separate training, release decisions, serving and gated improvement.

Complexity
Intermediate
Drawing
Supplied
  • MLOps
  • Model Lifecycle
  • Monitoring
  • Release Engineering
  • Machine Learning

Overview

This reference architecture is illustrative. It is not a record of a client deployment.

MLOps is the controlled path around a model, not a synonym for automated retraining. A versioned data snapshot first passes schema and quality checks and is split without leaking future or held-out information into training. Experiments produce candidate artifacts, but only a candidate that passes a held-out evaluation can be packaged with its preprocessing and lineage.

Release remains a separate decision. The registry provides a versioned package; a gate checks compatibility and acceptance criteria before a staged rollout sends traffic to the new inference version. Production monitoring then combines operational signals with predictions and delayed outcomes. A drift alert is evidence to investigate, not proof that a new training run will help. A person or governed review process can approve retraining, restore a compatible approved version, or leave the service unchanged.

A concrete example

A weekly demand model sees a category-code change upstream. Data validation stops the run before training. After the mapping is corrected, a candidate is evaluated against a time-ordered holdout. It passes the agreed criteria, is registered with its preprocessing package and receives a staged rollout. Later, outcome quality degrades while latency remains healthy. Reviewers inspect the signal and choose rollback because the previous package is still compatible; they do not automatically retrain on unverified data.

Common misconception

"Continuous training means every drift alert deploys a new model." Google and Microsoft architecture guidance separates data validation, model validation, release and monitoring. Automation can execute those controls, but it should not erase them.

Diagram & Walkthrough

Choose a declared scenario and step, or select a component by pointer, touch or keyboard. Playback starts only when you press Play and pauses when this panel leaves the screen.

100%
What MLOps Looks Like in the Real World Versioned data is validated and split safely before training and held-out evaluation. Passing candidates are registered and move through a release gate into production. Monitoring informs a human retrain or rollback review; drift alone does not trigger deployment. What MLOps Looks Like in the Real World Training, release decisions, serving and gated improvement are separate concerns. TRAINING AND VALIDATION · OFFLINE RELEASE AND SERVING FEEDBACK · SIGNALS REQUIRE REVIEW Versionedsnapshot Train split +preprocessing Candidateartifact Passingcandidate Versioned package Approved release Predictions+ outcomes Quality or drift signal Approved retraining Restore approvedversion Evaluation failed Versioned data DATA · SNAPSHOT Validation +leakage-safe split DETERMINISTIC Training +experiments MODEL Held-out evaluation DETERMINISTIC · GATE ✓ Model + preprocessingregistry DATA · VERSIONED Release gate +staged rollout GATED ✓ Production inference MODEL · SERVING Quality + driftmonitoring DETERMINISTIC Retrain or rollbackreview HUMAN DECISION Flow legend Data · main path Control Feedback Rejected candidate A drift signal is evidence to investigate, not proof that retraining will improve the system.

Main walkthrough, step 1 of 6: Version and validate the data. idle.

Main walkthrough

The primary path through this reference architecture.

Active parts: Versioned data, Validation + leakage-safe split

Text walkthrough

This vertical reference remains readable without the interactive controls.

  1. 01Version and validate the data

    An identifiable snapshot is checked and prepared for a reproducible run.

    Active parts: Versioned data, Validation + leakage-safe split

  2. 02Train without leakage

    Training receives only the appropriate partition and its preprocessing definition.

    Active parts: Validation + leakage-safe split, Training + experiments

  3. 03Evaluate a candidate

    The candidate is tested against held-out evidence and declared acceptance criteria.

    Active parts: Training + experiments, Held-out evaluation

  4. 04Register an approved package

    A passing candidate is registered with preprocessing and then presented to the release gate.

    Active parts: Held-out evaluation, Model + preprocessing registry, Release gate + staged rollout

  5. 05Roll out and serve

    The approved package is rolled out, and its predictions and outcomes become observable by version.

    Active parts: Release gate + staged rollout, Production inference, Quality + drift monitoring

  6. 06Review improvement signals

    A reviewer interprets the signal and explicitly authorizes a new training cycle when justified.

    Active parts: Quality + drift monitoring, Retrain or rollback review, Validation + leakage-safe split

Connections

  • Versioned snapshotVersioned data to Validation + leakage-safe splitdata
  • Train split + preprocessingValidation + leakage-safe split to Training + experimentsdata
  • Candidate artifactTraining + experiments to Held-out evaluationdata
  • Passing candidateHeld-out evaluation to Model + preprocessing registrycontrol
  • Versioned packageModel + preprocessing registry to Release gate + staged rolloutcontrol
  • Approved releaseRelease gate + staged rollout to Production inferencecontrol
  • Predictions + outcomesProduction inference to Quality + drift monitoringfeedback
  • Quality or drift signalQuality + drift monitoring to Retrain or rollback reviewfeedback
  • Approved retrainingRetrain or rollback review to Validation + leakage-safe splitcontrol
  • Restore approved versionRetrain or rollback review to Release gate + staged rolloutcontrol
  • Evaluation failedHeld-out evaluation to Retrain or rollback reviewcontrol

Components

Versioned datadata

Keeps an identifiable training snapshot and the lineage needed to reproduce it.

Responsibilities

  • Version training inputs and labels
  • Retain lineage and access boundaries

Inputs

  • Authorized source data
  • Observed labels

Outputs

  • Versioned snapshot

Risks

  • Stale or incorrectly joined labels
  • A schema change hidden inside the same version
Validation + leakage-safe splitdeterministic

Validates the snapshot and creates training and held-out partitions without fitting transformations on future data.

Responsibilities

  • Check schema and data quality
  • Create leakage-safe splits and preprocessing inputs

Inputs

  • Versioned snapshot

Outputs

  • Training split
  • Held-out split
  • Preprocessing definition

Risks

  • Target or temporal leakage
  • Preparation logic that differs from serving
Training + experimentsmodel

Fits candidate models under recorded code, configuration and data versions.

Responsibilities

  • Run reproducible experiments
  • Produce candidate artifacts and training metrics

Inputs

  • Training split and preprocessing
  • Versioned training code and configuration

Outputs

  • Candidate model artifact
  • Experiment metadata

Risks

  • Irreproducible environments
  • Optimizing repeatedly against the holdout set
Held-out evaluationdeterministic

Tests the candidate against declared quality, safety and compatibility criteria on held-out data.

Responsibilities

  • Compare against baselines and acceptance criteria
  • Reject candidates that do not pass

Inputs

  • Candidate model artifact
  • Held-out split

Outputs

  • Passing candidate
  • Or evaluation failure

Risks

  • A metric that does not represent the business outcome
  • A contaminated holdout set
Model + preprocessing registrydata

Stores the approved model together with compatible preprocessing and traceable metadata.

Responsibilities

  • Keep immutable package versions
  • Link artifacts to their data, code and evaluation

Inputs

  • Passing candidate
  • Preprocessing package and lineage

Outputs

  • Versioned release package

Risks

  • Model and preprocessing versions become mismatched
  • Missing lineage prevents reproduction
Release gate + staged rolloutdeterministic

Checks the registered package, records approval and controls staged promotion or restoration.

Responsibilities

  • Enforce release criteria
  • Support a controlled rollout and rollback

Inputs

  • Versioned release package
  • Release approval or rollback choice

Outputs

  • Approved release

Risks

  • A bypassed release gate
  • Rollback to an incompatible feature contract
Production inferencemodel

Serves predictions from the approved model and preprocessing version.

Responsibilities

  • Apply the released package consistently
  • Attach model version to predictions

Inputs

  • Approved release
  • Production feature values

Outputs

  • Predictions with model version

Risks

  • Training-serving skew
  • Latency or availability outside the decision budget
Quality + drift monitoringdeterministic

Combines service telemetry, feature distributions, predictions and available outcomes by version.

Responsibilities

  • Detect operational and statistical changes
  • Produce reviewable signals rather than automatic conclusions

Inputs

  • Predictions
  • Outcomes
  • Service telemetry

Outputs

  • Quality or drift signal

Risks

  • Delayed labels hide degradation
  • Noisy thresholds create alert fatigue
Retrain or rollback reviewhuman

Interprets evidence and authorizes retraining, rollback or no change.

Responsibilities

  • Investigate evaluation and monitoring signals
  • Choose a controlled corrective action

Inputs

  • Quality or drift signal
  • Evaluation failure

Outputs

  • Approved retraining
  • Rollback decision
  • Or no change

Risks

  • Treating drift as proof that retraining helps
  • Restoring an obsolete package without compatibility checks

Scenario reference

Candidate rejected

Keep the previously approved model available.

  1. 01Assess the candidate

    The candidate reaches held-out evaluation but does not satisfy the release criteria.

    Active parts: Training + experiments, Held-out evaluation

  2. 02Keep the candidate out of production

    The failed result goes to review while the approved production version remains available.

    Active parts: Held-out evaluation, Retrain or rollback review

Production rollback

Restore a compatible approved model and preprocessing package.

  1. 01Observe a production problem

    Versioned predictions and outcomes reveal a quality or operational concern.

    Active parts: Production inference, Quality + drift monitoring

  2. 02Review the signal

    The signal is investigated before a compatible approved package is selected.

    Active parts: Quality + drift monitoring, Retrain or rollback review

  3. 03Restore the approved version

    The selected package returns through the release gate and is rolled out under control.

    Active parts: Retrain or rollback review, Release gate + staged rollout, Production inference

Design Decisions

  • Version data, code, features, preprocessing and model artifacts together.
  • Fit preprocessing on training data only and preserve a genuinely held-out evaluation set.
  • Promote through an explicit gate and retain a tested rollback package.
  • Monitor data, service behavior and real-world quality by model version.

Risks & Trade-offs

Leakage can make offline metrics misleading. Training-serving transformations can diverge. Delayed or biased labels can trigger noisy alerts, and a technically compatible rollback may still restore stale behavior. More automation improves repeatability but adds pipeline, lineage and operational maintenance.

When to Use / Not Use

Use this pattern for models that are retrained or released repeatedly and influence production decisions. A small, rarely changed offline model may need versioning and review without a full continuous-delivery platform. Do not automate releases when valid evaluation data or rollback compatibility is unavailable.

Sources