Skip to content

security-guidance: never report a failed review as clean - #6375

Merged
mhegazy merged 1 commit into
mainfrom
mhegazy/sg-review-failures
Oct 13, 2026
Merged

mhegazy merged 1 commit into
mainfrom
mhegazy/sg-review-failures

Conversation

@mhegazy

@mhegazy mhegazy commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Human: @mhegazy

Summary

A review that got no answer was reported as clean (anthropics/claude-code#99840, also anthropics/claude-code#81057). When the model call failed, or its reply could not be read, the commit review logged "no security issues found" and marked the commit reviewed, so the push review skipped it.

What changes

  • A commit whose review got no verdict is not marked reviewed, so a later push review from Claude Code still checks it, unless the same command pushed it. No verdict means an API error, an unexpected error in the call, or a reply that cannot be read or lacks the verdict flag. A failed push review does not mark its range either.
  • Claude is told once per session for each kind of review (commit or push) and each kind of failure, for example "Commit security review did not complete: the API rejected the plugin's credentials (HTTP 401). Log: ". Later failures of the same kind are only logged. Nothing is blocked: the commit has already happened when the review runs.
  • In the agentic race, a single-shot fallback that got no verdict no longer wins while the agentic review can still answer. It waits up to 8 minutes from the start of the review. The fallback no longer starts after the agentic review has answered, and it never runs twice.
  • The Stop review uses the same signal, so an unreadable reply no longer records the diff as reviewed.

No version bump here. A separate PR bumps the plugin to 2.0.14 and should merge after this one.

Trade-offs

  • When a push review fails, or a commit review fails in a command that also pushed, a later push does not re-check those commits, because the pushed range has moved on. The notice says so.
  • A Stop review whose reply stays unreadable is retried on later turns with the growing diff, where it used to be recorded as reviewed.

Test plan

New tests/test_review_outcomes.py, 26 tests. Without the fix 25 of them fail. The plugin's suite passes, 214 tests, under pytest on Linux.

End to end. Run in tmux with public Claude Code 2.1.292, the plugin loaded by --plugin-dir, against a local fake API that scripts Claude and answers the plugin's review calls. The endpoint answers the plugin's review call with 401. Claude writes a file with a shell-injection bug and commits it.

Before (main) After
Commit recorded as reviewed Yes No
What Claude was told Nothing "The security review of this commit did not complete: the API rejected the plugin's credentials (HTTP 401)…"

Not verified: macOS and Windows.


Generated by Claude Code

@mhegazy mhegazy self-assigned this Oct 7, 2026
@mhegazy
mhegazy force-pushed the mhegazy/sg-review-failures branch 2 times, most recently from 5d02a8c to f1b1504 Compare October 8, 2026 01:23
@mhegazy mhegazy changed the title security-guidance: never report a failed review as clean, send gateway headers, deliver commit findings (2.0.12) security-guidance: never report a failed review as clean Oct 8, 2026
@mhegazy
mhegazy force-pushed the mhegazy/sg-review-failures branch from f1b1504 to 93f1a92 Compare October 9, 2026 16:03
When a commit review got no verdict (an API error, an unexpected error in
the call, or a reply that could not be read) it was logged as "no security
issues found" and the commit was marked reviewed, so the push review
skipped it (anthropics/claude-code#99840, also anthropics/claude-code#81057).

- A commit whose review got no verdict is not marked reviewed, so a later
  push review from Claude Code still checks it, unless the same command
  pushed it. A failed push review does not mark its range either.
- Claude is told once per session for each kind of review and failure.
  Later failures of the same kind are only logged.
- In the agentic race, a single-shot fallback with no verdict no longer
  wins while the agentic review can still answer, the fallback no longer
  starts after the agentic review has answered, and it never runs twice.
- The Stop and SubagentStop reviews use the same signal, so an unreadable
  reply no longer records a subagent's diff as reviewed.
@mhegazy
mhegazy merged commit d44ca8e into main Oct 13, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants