Testing and code review with AI

• Property-based testing — asserting invariants over many generated inputs rather than fixed examples.
• Mutation testing — deliberately injecting bugs to check whether the test suite catches them.
• Characterization test — a test that records current behavior of legacy code before changing it.
• SAST (Static Application Security Testing) — analysis of source code for vulnerabilities without running it.
• Flaky test — a test that passes and fails nondeterministically without code changes.

Tests derived only from the implementation tend to assert what the code does, bugs included. Generate tests from the spec or acceptance criteria, or ask for edge cases (empty, null, boundaries, concurrency, error paths) first and implementation-specific tests second.

A high coverage number from AI-written tests proves little. Mutation testing shows whether the tests fail when the code is broken; surviving mutants point at weak assertions the model should strengthen.

Write or generate failing tests, then let the agent implement until they pass. The failing tests are an executable spec and bound the agent’s freedom to drift. Watch for the agent “fixing” a failing test by weakening or deleting it instead of fixing the code.

For legacy code, have the model generate tests that capture present behavior, review them for sanity, then refactor with the safety net in place.

Give the agent the failure output and let it propose a cause and a minimal fix. For flaky tests it can spot common culprits (shared state, time, ordering, unawaited async work), but confirm by rerunning rather than trusting the explanation.

Automated reviewers summarise a PR, flag likely bugs, missing tests, security smells and style drift. Use them to catch the cheap things so human reviewers spend attention on design, naming, and domain correctness. They complement, not replace, linters and SAST, which are deterministic.

Generated changes are large and fluent, which invites skimming. Keep PRs small, require the author to explain the change, check for unrequested edits (touched unrelated files, removed tests, loosened assertions), and look specifically for deleted or weakened checks.

A fresh session with no memory of the implementation reviews with different blind spots than the one that wrote the code. Asking the author-agent to review its own work mostly confirms its own assumptions.