Experiment tracking and model registries

A notebook captures the code as it looked when last saved — it does not capture the exact data version it was run against (a source table that has since changed or grown), the exact dependency versions (a library upgrade can silently change numerical results), the random seeds and hardware/execution non-determinism (GPU floating-point non-associativity, same as the LLM topic’s determinism discussion), the actual hyperparameters used in that specific run (if they were tweaked interactively and not all saved back to the file), or the order cells were actually executed in, which can differ from top-to-bottom re-execution if the author re-ran cells out of order while iterating. Reproducibility requires all of these to be captured at the moment of the run, automatically, not reconstructed after the fact from memory or a possibly-stale file — which is precisely the gap experiment tracking tools exist to close.

• Code version — the exact commit hash (and whether the working tree was clean or had uncommitted changes) so the code can be reconstructed exactly, not approximately.
• Data version — a reference to the exact dataset snapshot used (a hash, a versioned dataset ID, a query with a fixed timestamp/snapshot) — without this, “same code, same hyperparameters” can still produce different results silently if the underlying data changed between runs.
• Hyperparameters — every configurable value that affected the run, logged automatically rather than manually transcribed, so nothing is missed or mistyped.
• Environment — dependency versions (a lockfile, a container image digest), hardware type, and relevant library/driver versions where numerics can be hardware-sensitive.
• Full metric history, not just the final number — a loss curve reveals instability, overfitting onset, or a bad learning-rate schedule that a single final metric hides entirely.
• Artifacts — the trained model weights themselves, logged and linked to the run that produced them, plus any evaluation outputs (confusion matrices, sample predictions) needed to actually judge quality beyond one aggregate number.
• Random seeds — logged even though they only provide partial reproducibility given hardware non-determinism, since they at least remove one source of run-to-run variance when comparing otherwise-identical configurations.

Experiment tracking answers “what happened during training runs” — a log of many attempts, most of which are exploratory and never deployed. A model registry is a separate, curated layer that answers “which specific model artifacts are candidates for or currently in production” — it tracks a smaller set of promoted models, their stage (staging, production, archived), their lineage back to the exact experiment run and data version that produced them, and often approval/sign-off metadata for who promoted a model and why. The distinction matters operationally: experiment tracking is write-heavy and exploratory (hundreds of runs during iteration), while the registry is the authoritative, comparatively stable answer to “what is actually running in production right now, and what would we roll back to” — conflating the two (treating every tracked experiment as a deployment candidate) makes it hard to answer that reliably during an incident.

Lineage is the complete, traceable chain from a production model back through: the exact training run and code commit that produced it, the exact dataset version(s) it was trained on, the exact feature engineering/transformation logic applied, and any upstream models or labels it depended on (for a model trained on another model’s outputs, or on human-labeled data with its own versioning). During an incident — a production model is behaving unexpectedly — lineage is what lets you answer “was this model actually trained on the data we think it was” and “what exactly changed between the last-known-good model and this one” precisely, rather than reconstructing an approximate guess from memory or scattered Slack messages. Without recorded lineage, debugging a production model regression degenerates into archaeology — trying to reconstruct after the fact what should have been captured automatically at training time, often with key details (the exact data snapshot, the exact hyperparameters) no longer even recoverable.

This directly exercises whether the tracking system captured enough (per the concept above): check out the exact recorded code commit, not the current codebase; restore the exact recorded environment (rebuild from the pinned lockfile/dependency versions or, more reliably, re-run inside the exact recorded container image rather than a freshly-built one, since a lockfile alone does not pin OS-level or hardware-driver differences); and use the exact recorded data snapshot/version, not “the current version of the same table,” since the live table has almost certainly changed in six months even if its name and schema look the same. If any of these three were not actually captured at training time — often the data version, in practice, since it is the easiest one to skip — full reproduction becomes approximate at best, which is exactly the argument for why data versioning belongs in the tracking system’s required fields rather than being treated as optional metadata. Bit-for-bit numerical reproduction may still be unattainable even with all three restored, for the same hardware/floating-point non-determinism reasons covered in the LLM topic’s determinism discussion — the realistic bar is “reproduces the same model behavior and metrics within expected tolerance,” not literal bit-identical weights.

It works, barely, for one person running a handful of models with perfect memory and discipline. It breaks as soon as: multiple people run experiments concurrently and the spreadsheet becomes a manually-maintained, conflict-prone shared document nobody trusts fully; the number of runs grows past what anyone can manually curate, so genuinely useful comparisons (which hyperparameter actually mattered across 50 runs) become impractical to reconstruct by hand; someone needs to reproduce or debug a run from months ago and the spreadsheet’s “accuracy: 0.87” row has no linked commit, data version, or full config — exactly the reproducibility gap from the earlier concepts, now institutionalized as the default process rather than an occasional gap; and auditability/compliance needs (which model was in production on a given date, trained on what data, approved by whom) have no reliable source of truth at all. None of this is really an “if you scale” problem so much as a “the cost of not having it compounds with every additional run and every additional person,” which is why the fix (adopting a real tracking/registry tool) gets more expensive to retrofit the longer a team waits, not cheaper.