Automating Code Review and Documentation
Where automated review adds signal beyond linters and static analysis, and how to stop documentation from drifting away from the system it describes.
Code review and documentation are both bottlenecks that scale badly with team size, and both are dominated by work that is mechanical enough to automate but too context-dependent for a linter. That combination is where automated review assistance has become genuinely useful.
The review automation stack
Different tools catch different classes of problem, and they are complementary rather than competing.
flowchart TD
A[Pull request opened] --> B[Formatters and linters<br/><small>style, syntax — auto-fixed, not commented</small>]
B --> C[Static analysis<br/><small>provable defects: null paths, resource leaks</small>]
C --> D[Security and dependency scanning<br/><small>known vulnerability classes</small>]
D --> E[Policy checks<br/><small>organizational rules, IaC guardrails</small>]
E --> F[AI-assisted review<br/><small>intent mismatch, missing cases, unclear naming</small>]
F --> G[Human review<br/><small>design, trade-offs, business correctness</small>]
The ordering matters. Every layer should remove work from the layer above it, so that by the time a human reads the change they are spending attention on judgement rather than on things a machine could have caught.
Formatting should never be a review comment. If a formatter can fix it, the formatter fixes it on commit. A human commenting on indentation is a process failure, and it is a surprisingly common one.
What AI-assisted review adds
Static analysis proves properties; it is precise and limited. Language models reason about intent; they are broad and imprecise. The useful overlap is small but real:
| Finding type | Static analysis | AI-assisted review |
|---|---|---|
| Null dereference on a provable path | Reliable | Redundant |
| Unhandled error return | Reliable | Redundant |
| Code does not match its own comment or docstring | Cannot | Often catches it |
| Test asserts something different from what it claims | Cannot | Often catches it |
| Missing edge case implied by the surrounding code | Cannot | Sometimes catches it |
| Inconsistent with a pattern used elsewhere in the repo | Cannot | Sometimes catches it |
| Naming that misdescribes behaviour | Cannot | Often catches it |
| Design and architectural fit | Cannot | Unreliable — leave to humans |
The rules that keep it from becoming noise:
- Advisory only, never blocking. A non-deterministic reviewer must not be able to block a merge.
- Comment only on changed lines, for the same reason described in shift-left security.
- Cap the number of comments per pull request. Five useful comments get read; forty get collapsed and the tool gets muted.
- Measure the acceptance rate. If fewer than roughly a third of comments lead to a change, the tool is costing more attention than it returns and needs tuning or removal.
- Never auto-approve. Approval is an accountability statement by a person.
Automating the mechanical parts of review
Beyond the review itself, a good deal of review overhead automates cleanly:
- Reviewer assignment from code ownership rules, weighted by current load, so the same two people are not on everything.
- Change summarisation — a generated description of what a large pull request actually does, which helps most on the changes that are hardest to review.
- Risk labelling based on what was touched: database migrations, auth code, and payment paths get flagged for stricter review automatically.
- Blast radius annotation — which services depend on the changed code, drawn from the dependency graph.
- Stale review nudges, so a pull request does not sit for a week without anyone noticing.
Documentation drift
Documentation decays because nothing fails when it becomes wrong. The fix is to create failure conditions.
flowchart TD
A[Source of truth] --> B{Can the doc be<br/>generated from it?}
B -->|Yes| C["Generate: API references, config<br/>options, CLI help, schemas,<br/>dependency and architecture diagrams"]
B -->|No| D{Can it be<br/>tested?}
D -->|Yes| E["Executable docs: runnable examples,<br/>tutorials run in CI, link checking"]
D -->|No| F["Prose that needs review:<br/>concepts, decisions, rationale"]
F --> G["Attach an owner and a review date.<br/>Flag when the code it describes changes."]
Generate what can be generated. API references from schemas or annotations, configuration tables from option definitions, CLI documentation from the parser, architecture diagrams from the dependency graph. Generated documentation cannot drift, because it is derived.
Test what can be tested. Code examples in documentation should be extracted and compiled or run in CI. A tutorial that no longer works is a broken build, not a support ticket. Link checking belongs in the same category.
For prose, create a drift signal. Prose explaining a design decision cannot be generated, but you can detect when the code it describes has changed substantially since the document was last touched, and raise it for review. An AI-assisted check that reads both the document and the recent diff and asks “does this document still describe this code?” is a reasonable use of a model — it proposes, a human decides.
What AI does well in documentation
| Task | Value | Caveat |
|---|---|---|
| Draft a docstring from an implementation | High for boilerplate | Must be reviewed — it describes what the code does, including bugs |
| Draft release notes from merged changes | High | Needs editing for audience and emphasis |
| Flag docs contradicted by recent changes | High | Suggests; a human confirms |
| Improve clarity of existing prose | Medium | Can smooth away necessary precision |
| Generate a tutorial from an API reference | Medium | Must be executed to verify it works |
| Explain unfamiliar code to a reader | High | Explanation, not authority — verify before acting |
The recurring caveat is worth stating plainly: generated documentation describes the implementation, not the intention. If the code has a bug, the generated docstring documents the bug in confident prose. That is why generated prose gets reviewed and generated references — derived mechanically from schemas — do not need to be.
Measuring whether it helps
| Metric | What it tells you |
|---|---|
| Time to first review comment | Whether review is actually a bottleneck |
| Review cycles per pull request | Whether early automated feedback is reducing rework |
| Automated comment acceptance rate | Whether the assistant is signal or noise |
| Documentation link failure rate | Whether docs are decaying |
| “How do I…” support questions | The honest measure of documentation quality |
The last one is the outcome metric. Documentation exists so that people do not have to ask; a constant stream of the same question means the docs are not doing their job regardless of how complete they look.
Adoption checklist
- Formatting is auto-fixed, never reviewed by a human.
- Each automation layer removes work from the one above it.
- AI review is advisory, scoped to changed lines, and capped in comment count.
- Automated comment acceptance rate is measured; low-value tools are removed.
- Reviewer assignment, risk labelling, and blast-radius annotation are automated.
- API references, config docs, and architecture diagrams are generated, not written.
- Documentation examples are executed in CI; links are checked.
- Prose docs have owners, review dates, and a code-change drift signal.
Last updated 19 Aug 2026, 00:00 UTC.