Testing & test doubles

The core problem: code that directly calls datetime.now() or random.random() internally is inherently non-deterministic — running the exact same test twice can produce different results, and you can’t easily assert “the output should be X” when X depends on the literal current moment or an unpredictable random draw. The heavy-handed fix — threading an explicit clock/RNG object as a parameter through every function that might eventually need it — quickly pollutes business logic with plumbing that has nothing to do with the actual problem being solved. Practical alternatives: unittest.mock.patch to directly replace datetime.now (or the RNG call) with a fixed, controlled value for the duration of a specific test, without changing the production code’s signature at all; dependency injection at boundaries — rather than threading a clock through every function, only inject it at the specific outermost layer where time actually matters, keeping the bulk of business logic pure and time-independent; or a purpose-built library like freezegun, which lets a test declare “as far as any code is concerned during this test, the current time is frozen at this exact instant” globally, without needing to patch every individual call site by hand. A good answer weighs the real tradeoff: patching is quick but couples the test to how the code happens to be implemented (importing datetime a certain way); real dependency injection is more work upfront but produces more testable, more explicitly-designed code.

(The fuller taxonomy, including dummy and spy, follows below — the short version:) A stub returns fixed, canned responses and is used to check the result of code that used it (“state verification”). A mock is specifically used to verify how it was called — that certain methods were invoked, with certain arguments, possibly in a certain order (“behavior verification”). A fake is a real, working, simplified implementation (like an in-memory dictionary standing in for a real database) that preserves genuine behavior across multiple calls, rather than just returning a fixed answer. A concrete illustration of how over-mocking actually hurts in practice: heavy reliance on mock-based behavior assertions couples your tests tightly to your code’s internal implementation details (exactly how many times something was called, in what order) rather than to its externally observable behavior — meaning a purely internal refactor, which shouldn’t require touching any tests at all if they were testing behavior correctly, instead breaks a pile of tests that were really asserting on implementation, forcing engineers to “fix the tests” on every refactor in a way that erodes trust in the test suite and discourages people from refactoring at all.

This taxonomy (from Gerard Meszaros’s widely-cited work on testing patterns) distinguishes five genuinely different kinds of test doubles: a dummy is passed around purely to satisfy a required parameter but is never actually meaningfully used by the code under test at all; a stub returns fixed, canned answers with no real underlying logic, and — critically — is used specifically for state verification, meaning you check the result/output of code that used it, not how it was called; a fake is a genuine, working, simplified implementation (an in-memory dict standing in for a real database, preserving actual real behavior across multiple calls rather than just one fixed answer); a spy records information about how it was called, often while still delegating through to real underlying behavior; a mock is pre-programmed with explicit expectations and is used specifically for behavior verification — checking that specific calls genuinely happened, with specific arguments, sometimes in a specific order. Stub and mock sit cleanly on opposite ends of that state-vs-behavior axis — a distinction that’s easy to blur together casually.

MagicMock pre-implements Python’s “magic”/dunder methods (__len__, __iter__, __enter__/__exit__, __getitem__, and others), each configured by default to return further mock objects when invoked — a bare Mock has none of this pre-configured, so code under test that does something like len(mocked_thing) or uses the mock in a with block genuinely raises TypeError against a plain Mock (since it lacks the required dunder methods entirely) while working perfectly fine against a MagicMock. A real bug this causes: mocking a dependency that’s used somewhere as a context manager (with self.some_dependency:) using a plain Mock instead of MagicMock, producing a confusing test failure that has nothing whatsoever to do with the actual business logic supposedly being tested.

A bare, unconstrained mock silently accepts any attribute access and any call signature whatsoever, always just returning another mock object — meaning if the real dependency’s actual method signature later changes (an argument gets renamed, removed, or reordered) or a method is renamed/removed entirely, your existing mock-based tests keep passing regardless, since the mock never actually validates the call against anything real — you only discover the real integration is broken once it hits actual production. autospec/spec constrains the mock’s allowed interface to genuinely match the real object’s actual interface — calling it with a signature that doesn’t match the real thing raises TypeError immediately, right there in the test. This is genuinely one of the highest-value mocking habits to internalize, and a great many people skip it in practice.

You patch where the name is looked up at call time — that is, in the namespace of the module that actually uses it (module_under_test.dependency), not the dependency’s own original, defining module. The mechanical reason: a from x import dependency statement binds a brand-new name directly inside the importing module’s own namespace, at the moment that import runs — patching x.dependency afterward, in the original defining module, does absolutely nothing to that already-separately-bound reference sitting in the importing module’s namespace, since the two are now genuinely separate name bindings pointing (at least initially) at the same object. This trips up an enormous number of people the first several times they use patch.

A stub returns the exact same canned response regardless of what it’s actually called with, and fundamentally doesn’t model any real state changes across multiple calls — a concrete scenario: a stub set up for a “create a user, then fetch that user” sequence that simply returns a fixed, hardcoded user object for any fetch call whatsoever, even one asking for a user that was genuinely never “created” in the first place — such a stub-based test would keep passing regardless, while a fake (a genuine, working, simplified in-memory implementation actually preserving real create/fetch semantics and consistency between calls) would have correctly caught this exact class of integration bug. The extra engineering cost of writing a proper fake is worth it specifically when the interaction pattern across multiple calls — not merely a single call’s isolated return value — genuinely matters to overall correctness.

Mock-based assertions — things like mock.assert_called_with(...) or assert_called_once() — inherently couple a test to specific implementation details: exactly how many times something was called, with precisely what arguments, sometimes in exactly what order. A purely internal refactor that changes how the code accomplishes something internally, without changing its externally observable behavior in any way, can still break a large pile of such tests — even though, from a correctness standpoint, genuinely nothing is actually wrong. Stubs, by contrast, only constrain what a dependency returns, which tends to remain far more stable across purely internal refactors, since a refactor rarely changes the fundamental shape of data flowing through the system even when it changes exactly how that data gets produced internally. This connects directly back to the general “test behavior, not implementation” principle.

Real, concrete safeguards, not just “be more careful”: periodically running contract tests against a real (sandboxed or staging) backend specifically to detect drift between the stub’s assumed shape and the API’s actual current shape; recording genuine real responses into fixtures/“cassettes” (VCR-style record-and-replay testing libraries, which capture real HTTP interactions once and replay them deterministically in future test runs) so stubs are effectively generated from reality rather than hand-typed once and left to silently go stale; or explicit schema validation applied to both the stub’s canned response and any genuinely-received real response, specifically to catch structural divergence between the two mechanically, rather than relying purely on someone noticing by inspection.

A .pyi file (a type stub file) contains only type signatures — function/method signatures, class attribute types — with genuinely no actual implementation code inside it at all. Real use cases: providing type information for a third-party library that ships with no inline type hints of its own (commonly distributed as a separately-installable *-stubs package); C-extension modules, which have no readable Python source code at all for a static type checker to introspect directly; or deliberately separating a library’s public, typed interface from its private internal implementation details. typeshed is worth knowing about specifically: the community-maintained repository of stub files covering the standard library and many popular third-party packages, which mypy/pyright both consult by default — a “why isn’t my type checker seeing this correctly” issue often traces back to exactly this, a missing stubs package.

The test pyramid describes a deliberately unequal distribution: many fast unit tests at the base, fewer integration tests in the middle, and very few end-to-end (e2e) tests at the top. Three concrete reasons this shape earns its keep: speed (unit tests run in milliseconds each, e2e tests in seconds or minutes — directly affecting how often people actually bother running them during development); isolation (a failing unit test points precisely at what broke, while a failing e2e test could indicate a problem genuinely anywhere across the whole stack, forcing a much longer investigation); and flakiness (e2e tests touch far more moving, real-world parts — actual network calls, real timing, external services — giving them far more opportunities to fail for reasons entirely unrelated to whether the code itself is actually correct). The pyramid isn’t dogma applied identically everywhere, though — some systems (thin orchestration/glue code wrapping complex external integrations) can genuinely benefit from a differently-shaped distribution, weighted more toward integration coverage.

Each parametrized case, using pytest.mark.parametrize, runs and is reported as a genuinely independent test by the test runner — meaning one failing case among many doesn’t hide or obscure the others: a hand-rolled loop inside a single test function, by contrast, typically stops entirely at the first failing assertion (since an unhandled exception halts the whole function), silently never even attempting the remaining cases in that run, and the failure report itself is far less precise about exactly which input combination failed.

A fixture’s scope controls how broadly one single instance of it is reused across multiple tests rather than being freshly recreated for each — session scope (the broadest) creates it exactly once for the entire test run. The concrete risk, specifically for a fixture producing mutable shared state: if any individual test mutates that shared object and doesn’t scrupulously clean up afterward, later tests can end up silently, unintentionally depending on leftover state from an earlier test — breaking the fundamental independence tests should have, and producing genuinely confusing, order-dependent failures (a test that passes perfectly fine in isolation but fails only when run as part of the full suite, or vice versa).

Ordinary example-based testing asserts specific, hand-picked input→output pairs you thought to write. Property-based testing instead asserts general properties that should hold true for any valid input (e.g., “sorting a list twice produces the same result as sorting it once” — an idempotence property) — and a library like hypothesis automatically generates many varied, often deliberately adversarial and edge-case-heavy inputs to try to violate that property, rather than relying on a human to have thought of the right examples by hand. A good illustrative example: a serialization round-trip function (deserialize(serialize(x)) == x) — property-based testing tends to find the one genuinely weird Unicode string, or unusually deeply nested structure, that a human writer simply wouldn’t have thought to construct by hand as an explicit test case.

Mutation testing tools (mutmut, cosmic-ray) automatically introduce small, deliberate bugs into your actual source code — flipping a comparison operator, changing a constant value — and then rerun your existing test suite against each individually mutated version. If your test suite still passes despite the deliberately-introduced bug (a “surviving mutant”), that’s a direct, concrete signal that your tests genuinely aren’t exercising that particular piece of logic meaningfully — revealing real gaps that a simple code-coverage percentage alone conceals, since coverage only proves a given line of code was executed during testing, not that its actual behavior was ever genuinely asserted on by anything.

Common root causes: unmocked reliance on real current time (datetime.now()), unseeded genuine randomness, real (rather than mocked) network calls, shared mutable state leaking between tests (the same fixture-scope risk described above), tests that implicitly depend on running in a specific order, or genuine race conditions inside async/threaded code under test. A sound triage process: attempt to reproduce it locally, ideally with the exact same failing random seed or execution order if that’s available; deliberately resist the tempting shortcut of just re-running the test until it happens to pass, or slapping an automatic @retry decorator onto the test itself (both approaches actively mask a genuinely real underlying bug rather than fixing anything); and treat the flakiness itself as a real bug requiring genuine root-causing, not an annoyance to be worked around.

Generally, this is a real smell — tests should be genuinely independent and safely runnable in any order, or in full isolation individually, because order-dependent tests reliably break the moment someone reorders them, filters to run only a subset (pytest -k some_pattern), or parallelizes test execution across multiple workers (pytest-xdist) — and the resulting failure mode is genuinely confusing to debug (“test B mysteriously fails, but only when test A happened to run first” is a real nightmare to trace back to its actual cause). A legitimate, rare exception: deliberately sequential integration/workflow tests explicitly modeling a genuine multi-step process — but even those are usually better expressed as one single test explicitly asserting the whole sequence in order, rather than relying implicitly on pytest’s default file-order execution behavior to accidentally achieve the same effect.