Model deployment strategies

• Shadow deployment — the new model receives a copy of live production traffic and produces predictions, but those predictions are logged only, never shown to users or acted on (the old model continues to serve real responses). Answers: “how does the new model actually behave on real production traffic, at real production input distribution and volume, with zero risk to users” — the safest possible way to observe real-world behavior before it can affect anyone.
• Canary rollout — a small percentage of real, user-facing traffic is routed to the new model, gradually increased if metrics hold up. Answers: “does the new model perform acceptably when its output genuinely reaches users and drives real outcomes,” which shadow deployment cannot answer since shadow predictions are never actually acted upon (a subtly different thing from “does it produce good predictions” — acted-upon predictions can reveal downstream effects, like a recommendation model’s engagement impact, that log-only comparison never surfaces).
• A/B testing — traffic is split (often 50/50 or another fixed ratio) between old and new for a defined period, with statistical rigor around the comparison. Answers: “which model produces better outcomes on the actual business metric,” specifically designed to detect a real difference with statistical confidence rather than just eyeballing whether the canary “looks fine.”

They compose in sequence: shadow first (near-zero risk), canary next (small real risk, watch for outright breakage), A/B once both look healthy (rigorous comparison for the actual rollout decision).

Offline evaluation runs against a fixed, historical dataset and metric, which can diverge from production reality in several concrete ways: the live input distribution shifts over time relative to whatever the offline eval set was sampled from (see the monitoring/drift subtopic); offline metrics often measure a proxy (prediction accuracy) rather than the actual downstream business outcome the model is meant to drive (a recommendation model can improve offline click-prediction accuracy while making the actual user experience or long-term engagement worse, if the offline metric does not capture what matters); and offline evaluation cannot observe second-order effects — a change in one model’s outputs can shift downstream user behavior in ways that only show up once the new model is genuinely influencing real decisions, which is exactly the gap shadow/canary/A-B deployment stages exist to close before a full, irreversible rollout.

Fast, safe rollback requires: the previous model version stays deployed and ready to serve (not deleted or torn down the moment the new one goes live), the routing/traffic-split mechanism can be flipped back quickly without a full redeploy cycle, and — critically — the new model’s side effects are compatible with a rollback, meaning nothing about serving the new model irreversibly changed state in a way the old model cannot handle (a new model that wrote data in a format the old model cannot read, or that updated a cache/feature-store schema incompatibly, breaks rollback even if the model artifact itself is trivially swappable). What commonly breaks this in practice: schema or feature changes shipped alongside the model (the new model needs a new feature the old model’s serving path never populated, so rolling back the model without also rolling back the feature pipeline leaves the system in a broken intermediate state) and stateful online-learning models that have continued updating since the new version went live, making a clean rollback to the “old” model’s prior state no longer well-defined. A genuinely safe deployment plan treats “can we roll back cleanly in under N minutes” as a requirement checked before rollout, not discovered during an incident.

Blue-green keeps two full production environments (blue = current, green = new), routes all traffic to blue while green is validated, then cuts traffic over atomically once green is confirmed healthy, keeping blue on standby for instant rollback. For a model, this means both the old and new model artifacts are fully deployed and warm simultaneously, and the cutover is a routing change rather than a model swap mid-flight — avoiding any window where requests are served by a partially-updated or inconsistent serving stack. It differs from a typical stateless service mainly in the cost and warm-up time of “green”: spinning up a second full copy of a large model (GPU memory, model-loading time, potentially a KV-cache warm-up for an LLM serving stack) is far more expensive and slower than starting a second copy of a lightweight stateless application, so blue-green for models is often reserved for genuinely high-stakes rollouts where that doubled resource cost is justified by needing an instantaneous, fully-tested cutover — canary/gradual rollout is the more commonly used default specifically because it avoids paying for two fully duplicated serving fleets simultaneously.

1. Shadow first, for a meaningful volume and duration — compare the new model’s decisions against the current model’s and against eventual ground truth (confirmed fraud/chargeback outcomes, which arrive with a lag) on real traffic, with zero user impact, specifically because both error types here are costly enough that even canary-stage real exposure should not happen until shadow results look solid.
2. Canary at low traffic percentage, with tight, automated guardrails — define explicit stop conditions up front (false-positive rate or false-negative rate — once confirmed — exceeding a threshold triggers automatic rollback, not a manual decision made under pressure after the fact).
3. Gradual ramp, extending canary duration long enough to capture delayed ground truth — fraud outcomes are not known instantly (a chargeback can arrive weeks later), so “the canary looked fine after a day” is not sufficient evidence here the way it might be for a metric available immediately; the ramp schedule must account for that lag before concluding the new model is safe.
4. A/B comparison on the actual business cost metric (a combined cost function weighting false positives and false negatives by their real dollar impact, not raw accuracy), since the two error types are not symmetric in cost and an aggregate accuracy number would hide that asymmetry.
5. Full rollout with rollback plan validated in advance — confirmed the old model stays warm and routable, and that no incompatible state changes (new features, new logging schema) block a clean revert if delayed ground truth eventually reveals a problem the earlier stages missed.

Beyond the general on-call/incident-response argument that applies to any deployment, ML rollouts have a specific extra reason: many of the metrics that reveal a problem are not instantly available — model behavior on the true production distribution, downstream business-metric impact, and especially any outcome with a natural reporting lag (fraud confirmed later, conversion attributed days later) take real time to accumulate enough signal to judge. A model that looks fine in the first hour after a Friday-afternoon deploy can reveal a real regression only once enough of the slower-arriving signal comes in — over a weekend, with reduced on-call attention and a smaller volume of people watching dashboards, which is exactly the wrong time for a regression to be quietly accumulating before anyone notices. This is a genuine, ML-specific amplification of the general “risky deploy timing” argument, not just a restatement of it — the detection lag is structurally longer for many ML metrics than for typical application error-rate/latency metrics that a normal service rollback would catch quickly.