Architecture Decisions: Balancing Cost, Latency and Control

Every AI system makes the same three-way trade. Cost is what each request and each month of operation consumes. Latency is how long a user or a downstream process waits, including for the slowest requests. Control is how much of the behavior, the data handling and the change schedule the organization decides for itself. Improving one usually costs another, so the useful output of an architecture review is not a perfect design. It is a recorded choice.

Write the trade-off down

Michael Nygard's architecture decision record keeps a decision small: its context, the decision, its status and its consequences. He notes that a decision "may have positive, negative, and neutral consequences", and a record for an AI component needs all three. "We use a smaller model for classification" is incomplete without the quality it was measured at, the cost it saves and the condition that would reverse it. Microsoft's Well-Architected guidance gives the matching warning: choices that focus only on minimizing spending can undermine a workload's business goals.

Common levers and what they cost

• Route by difficulty. FrugalGPT studied cascades that send a query to cheaper models first and escalate only when the answer is not reliable enough. On its benchmarks, the authors report matching the best individual model with up to 98% lower cost. The price is an extra decision point, a scorer that must itself be evaluated, and slower answers for the requests that escalate.
• Move work off the request path. OpenAI's Batch API offers a 50% discount compared with its synchronous APIs, and each batch completes within 24 hours. That suits evaluations, re-indexing and bulk classification, not a person waiting for an answer.
• Reuse stable prompt prefixes. Prompt caching lowers the price of repeated input and the time spent processing it, but cache reuse requires the entire rendered prefix to match. Prompt layout becomes an architectural constraint: stable instructions and shared reference material first, variable content last.
• Watch the tail, not the average. Dean and Barroso showed in The Tail at Scale that rare slow responses "may come to dominate overall service performance at large scale." A design that chains several model and tool calls compounds the effect, so latency targets should be stated as percentiles for the whole path.

Control is a cost you choose

A managed API moves operational work to the provider, and with it part of the change schedule. OpenAI's deprecations page states that it regularly retires older models and that every deprecated model receives a shutdown date. Teams that need pinned behavior, data residency or offline operation can host open-weight models instead, and then own capacity, patching, monitoring and evaluation themselves. Neither option is free. The decision record should say which costs the organization has chosen to carry.

A decision is only as good as its trigger

Each choice should name the measurement that reopens it: an evaluation score below the agreed threshold, monthly cost above budget, a latency percentile missed for a week, or a provider's shutdown notice. That turns the record into an operating rule rather than a memory of a meeting. The Architecture Lab study Serving Multiple Models Safely shows one way to place routing and fallbacks in a serving layer.