Multi-LLM PR review

A cross-family second opinion on every PR. Runs in CI, posts findings as inline diff comments anchored to the flagged lines, and never approves or requests changes — humans own merge.

Why a second opinion

Same-family review (Claude reviewing Claude) shares blind spots. A different model family catches different things — rule hallucinations, missed tenancy wrapping, dropped error paths, security regressions one family is structurally biased toward missing. The cost is one extra LLM call per PR push, bounded by a per-PR token budget; the benefit is a second pair of eyes that consistently disagrees with the first.

INF-63 shipped the workflow scaffold. INF-126 shipped the reviewer script that the workflow invokes. INF-155 added SSE streaming, inline diff comments, and neutral check-run signalling. The reviewer has been enabled in CI since 2026-06-02 on openai/gpt-5.5 and runs on every PR push. See How it was enabled for historical context.

What it does

On every PR push (opened, synchronize, reopened, ready_for_review):

.github/workflows/multi-llm-review.yml checks gating (feature flag, secret, escape-hatch markers, draft / fork / dependabot exclusions).
If gating passes, it invokes scripts/multi-llm-review.ts with the PR number, base SHA, head SHA, the configured model list, and the per-PR token budget.
The script:
- Computes git diff <base>...<head>.
- Reads .ai/constitution.md and any .ai/specs/SPEC-*.md paths it can extract from the PR body (regex + existence-check).
- For each configured model: builds a prompt (constitution + specs + diff + reviewer rules), streams the response from the Vercel AI Gateway (SSE), parses the model's findings from a fenced JSON block, and posts findings as inline diff comments via the GitHub Reviews API.
- Tracks token usage across models; aborts cleanly with a "budget exhausted" notice on the PR review if the budget would be exceeded.
- When findings exist, creates a neutral check-run in the PR checks panel so findings are visible at a glance without blocking merge.

Every finding is required by the prompt to cite the spec heading or AGENTS.md / constitution clause it flags. Uncited findings are dropped on the client side — reviewer noise is the failure mode we care most about.

How findings are posted

Findings are posted as inline diff comments anchored directly to the changed line where the issue was found. This means reviewers see the annotation right on the relevant code, not in a separate comment thread.

Anchorable findings — those whose path:line location falls within an actual changed hunk — appear as inline comments on the RIGHT (new) side of the diff.

Non-anchorable findings — those with no location, a hallucinated path, or a line number outside any changed hunk — fall back into the top-level review body summary. This ensures that off-target model guesses do not cause the entire review POST to fail (the GitHub API rejects inline comments on lines not in the diff).

A neutral check-run named Multi-LLM review findings is created in the PR checks panel whenever findings exist. Neutral never blocks merge — it provides an informational signal ("N findings posted — advisory") without turning the check red. Clean reviews (zero findings) and skipped/error runs do not create a check-run; the existing green status is preserved.

How the idle-timeout model works (streaming)

Gateway calls use SSE streaming with an idle/inactivity timer rather than a hard wall-clock timeout. The timer resets on every received chunk — a slow-but-progressing reasoning trace is never killed. Only a truly stalled upstream (no bytes arriving for GATEWAY_IDLE_TIMEOUT_MS = 60_000 ms) aborts the request.

This is strictly better than the previous wall-clock cap for reasoning models like openai/gpt-5 that may take well over 60 seconds to produce a full response on large diffs, but do so steadily without stalling. A stall (network partition, gateway unresponsive mid-generation) is still caught quickly via the idle timer.

The workflow's timeout-minutes: 15 provides an outer wall-clock guard for the job as a whole (raised from 8 by INF-249 — a sequential agentic panel can legally spend 5 × 180 s exploration calls plus two extraction attempts per model). Inside it, a soft run deadline (MULTI_LLM_REVIEW_RUN_DEADLINE_MS, workflow default 10 min) finalizes the run gracefully instead of letting the job timeout kill it mid-model: models that have not started are skipped with an explicit reason (a partial panel still synthesizes and posts), and a mid-review expiry stops tool exploration and jumps straight to the forced findings extraction (stoppedReason: 'deadline'). Per-call gateway timeouts under a deadline are clamped to the remaining run time plus a 30 s finalization grace, so a call admitted just before expiry cannot drag the run past the job timeout.

How it was enabled (one-time repo-admin step)

The reviewer was enabled on 2026-06-02. The one-time setup was:

Generated a Vercel AI Gateway token with access to openai/gpt-5.5 and added it as the VERCEL_AI_GATEWAY_TOKEN repo secret.
Set MULTI_LLM_REVIEW_ENABLED=true as a repo variable.
Set MULTI_LLM_REVIEW_MODELS=openai/gpt-5.5 as a repo variable.

The reviewer now runs on every PR push automatically. To tune the knobs:

MULTI_LLM_REVIEW_MODELS — comma-separated model ids. Current: openai/gpt-5.5.
MULTI_LLM_REVIEW_TOKEN_BUDGET — max total tokens per PR push. Default: 100000.

Agentic-mode knobs (INF-169 / INF-207 / INF-170 — gated behind `MULTI_LLM_REVIEW_AGENTIC`)

The agentic arm (read beyond the diff, ground in repo rules, defer to SonarCloud) is off by default. When MULTI_LLM_REVIEW_AGENTIC=true:

MULTI_LLM_REVIEW_MAX_TOOL_TURNS / MULTI_LLM_REVIEW_MAX_READ_FILE_BYTES / MULTI_LLM_REVIEW_PER_REVIEW_TOKEN_CEILING / MULTI_LLM_REVIEW_KEEP_RECENT_TOOL_RESULTS — the INF-207 cost levers. Unset ⇒ the tuned script defaults (5 turns, 8KB reads, 30k ceiling, keep-recent 2).
MULTI_LLM_REVIEW_GROUNDING_TOKEN_BUDGET (INF-170) — token budget for the repo rule-grounding block (AGENTS.md hard rules + linked-spec ACs + lessons + the SonarCloud summary). Unset ⇒ the script default.
SONAR_TOKEN (secret, INF-170) — SonarCloud user token for the division-of-labour read (PR issues + hotspots, summarised into grounding; LLM findings overlapping a Sonar issue on file+line are deduped). Unset ⇒ the Sonar read/dedupe is skipped cleanly — the review still runs.
SONAR_PROJECT_KEY (var, INF-170) — overrides the project key (env → sonar-project.properties → the documented default B2B-Online_constellation).

Panel-mode knobs (INF-171 — gated behind `MULTI_LLM_REVIEW_PANEL`)

Panel mode runs a panel of ≥2 distinct models independently over the same diff and reconciles their findings into one deduped, agreement-ranked review via a deterministic synthesizer. Each model runs through the same per-model path the single-model reviewer uses — diff-only by default, or the agentic/grounded path when MULTI_LLM_REVIEW_AGENTIC=true (panel composes with the agentic flag; it does not force it on). Panel is off by default; the single-model path is unchanged when off. When MULTI_LLM_REVIEW_PANEL=true and MULTI_LLM_REVIEW_MODELS lists ≥2 distinct ids:

MULTI_LLM_REVIEW_MODELS — the panel roster (e.g. openai/gpt-5.5,anthropic/claude-sonnet-4.6,google/gemini-2.5-pro). Duplicate ids collapse for the panel gate; fewer than 2 distinct ⇒ panel does not engage and the non-panel per-model loop runs unchanged (a pathological duplicate roster like m1,m1 still runs the per-model loop verbatim).
MULTI_LLM_REVIEW_PANEL_MIN_AGREEMENT — minimum cross-model agreement a synthesized finding must have to be posted. Default 1 (keep every finding); raise it to trade recall for precision.

Confirmation-mode knobs (INF-293 — gated behind `MULTI_LLM_REVIEW_DELTA`)

Every synchronize push has historically triggered a full fresh review of the whole BASE_SHA...HEAD_SHA diff, so a fix-push samples the same mostly-unchanged code again and surfaces a fresh batch of unrelated findings each round. Confirmation mode makes a follow-up push scoped to just the fix delta, revalidating prior findings instead of re-deriving them. Off by default:

MULTI_LLM_REVIEW_DELTA (repo var) — 'true' opts in. Unset/'false' ⇒ delta scoping is off: no state comment is read or written, no fingerprint is computed, and every run is a full review of the whole PR diff. Some behaviour is always on, independent of this flag: the INF-292 coverage-contract paragraph in the reviewer's system prompt (it sharpens the initial pass but touches no code path or API call); the INF-172 review_mode metrics field; and the INF-300 concurrency rules below — push runs and manual re-runs occupy separate concurrency groups, and a re-run never writes reviewer state.
The state comment. Each completed run (including a clean zero-findings one) upserts one hidden-marker issue comment on the PR — marker  followed by a fenced JSON block recording lastReviewedHeadSha, reviewedAt, mode, and this run's findings[] (title, severity, optional location, and a truncated detail). An issue comment is used deliberately — it creates no resolvable review thread and is trivially machine-findable. Only a marker comment authored by github-actions[bot] (the workflow's own token identity) is ever trusted as state; a marker comment from any other author is ignored and logged, so no PR commenter can forge a fake lastReviewedHeadSha to shrink review scope. An unparsable or missing state comment degrades to a full review, never a crash.
Delta scope. A push resolves to a confirmation review only when ALL of: the flag is 'true', a trusted state comment parsed successfully, its lastReviewedHeadSha differs from the new head, and git merge-base --is-ancestor confirms the old head is still an ancestor of the new one. The reviewed diff becomes lastReviewedHeadSha...HEAD_SHA (just the fix delta), and the prompt embeds the prior findings plus rules: revalidate each one (re-reporting an unfixed finding with its title prefixed STILL OPEN: ), report any new issue the delta introduces, and — the escape hatch — report a newly-noticed critical/high-severity issue anywhere in view even outside the delta. It must NOT raise a fresh non-critical finding in code the delta didn't touch.
Fallback-to-full conditions. Any of: the flag off, no state comment (first review, or the marker comment failed to parse / was untrusted), the head unchanged since the last review on an event other than edited, a broken ancestry chain (a force-push rewrites history), or a merge commit in the delta range (merging an updated base branch into the feature branch would flood the "fix delta" with unrelated upstream code) — all fall back to today's full-PR review, exactly as if the flag were off. A same-head edited event is the narrow INF-431 exception and still must pass the fingerprint and reviewer-roster trust checks described below.
State only advances after a completed review. If every configured model is skipped (gateway error, token budget, run deadline, or a diff-only response with no explicit findings block — prose/malformed output counts as incomplete in BOTH modes, mirroring the agentic rule, per INF-297), the state comment is left untouched — the next run re-reviews from the old SHA so a skipped diff can never escape review. An empty full-review diff clears the persisted findings and blanks the stored fingerprint, forcing the next run to a full review (the whole scope is clean); an empty confirmation delta carries them forward unchanged. Before every write the run confirms its head AND base ref are still the PR's live head/base (fail-closed), so a manually re-run older workflow execution — including one from before a retarget — cannot roll state backwards.
What busts the fingerprint (INF-300 / INF-431). Alongside the head SHA, a run fingerprints every input that determines what "a defect" means for this PR: the PR title, review-relevant body text, branch, base ref and merge-base, the effective reviewer configuration, the linked specs, and a digest of all resolved prompt grounding — .ai/constitution.md, the post-impl-review checklist, the repo hard-rule catalog, .ai/lessons.md, and (agentic mode) the live PT ticket and knowledge-base retrieval. Verdict-layer own-lines are deliberately removed before the body reaches linked-spec discovery, PT/KB grounding, or the hash: Review override: entries and strict own-line [skip ...] markers decide what a gate does with an existing review; they are not requirements for the model. Change any review-relevant input mid-PR and the next push falls back to one full review, so the already-reviewed code is re-checked against the new rules rather than being frozen behind confirmation mode's "no new non-critical findings in unchanged code" restriction. The SonarCloud summary is deliberately excluded from that digest: it is per-commit analysis output the head SHA already keys, and Sonar re-analyses on every push, so including it would bust the fingerprint every round and make confirmation mode unreachable. Excluding it from the hash is only half the job, though — while Sonar shared a token budget with the requirement chunks it could push a whole rule, ticket or KB chunk out of the prompt without the digest noticing. So the Sonar summary and the requirement grounding are budgeted in independent pools (a fixed Sonar reserve, and total − reserve for requirements). The requirement budget is a constant, so Sonar's size — or its absence — cannot change which requirements the model receives, and the digest is taken over that received set. Because the digest is resolved before the mode is, grounding retrieval is seeded by the full PR diff's changed paths in both modes — what the model sees is still delta-scoped. The fingerprint also covers the reviewer's own source, as the prompt contract it defines: the static reviewer rules, tool guidance and grounding header shape what counts as a defect just as much as the constitution does, and enumerating those constants individually would rot the moment someone added a new one. Four modules are hashed — scripts/multi-llm-review.ts, scripts/review-grounding.ts (which holds the grounding header and the hard-rule catalog), scripts/review-sonar.ts and scripts/review-synthesizer.ts — and a test asserts that list covers every sibling reviewer module the script imports, so a new one cannot escape the contract. Any edit to any of them forces one full review per open PR: the safe direction, and rare.
Verdict-only confirmation runs (INF-431). Adding or editing only verdict-layer own-lines leaves the normalized fingerprint unchanged. When the PR head still equals the trusted state's head, the edited run takes an empty confirmation delta, posts no LLM review, and records the distinct verdict_only terminal outcome. The workflow then runs the independent-review verdict against the existing state. An override quoting the state comment's current finding can therefore clear it without a fresh non-deterministic review re-titling the finding first. GitHub workflow re-runs replay their original event payload, so re-running that edited execution repeats the verdict refresh and stays read-only for reviewer state; manual re-runs of other same-head event types still run a full review. The verdict gate reads the live PR body, so a removed or edited override is not resurrected from that replayed payload. If the live read fails, a first attempt warns and falls back to its event body; a re-run ignores the known-stale payload's overrides and stays red until the body can be verified. A changed-head confirmation whose commits have zero net delta also emits verdict_only, carrying the trusted findings forward to the new head and refreshing the gate. Any substantive metadata edit still busts the fingerprint. The no-model path does not require gateway credentials, and the check summary says verdict refreshed, not reviewed, so state reuse is never presented as new review coverage.
Concurrency (INF-300). Push-triggered runs share one per-PR concurrency group (…-push) and still cancel in progress, so a rapid double-push cancels the older run exactly as before. A manual re-run gets its own run-scoped group, so it can neither cancel nor displace an in-flight or queued review of a newer head — which would otherwise leave the current head unreviewed until the next push, since the stale re-run then trips the fail-closed stale-head write guard. Gating cancel-in-progress on the attempt number is not enough for this: GitHub cancels an existing pending group member whenever another job enters the group, regardless of that flag, which only governs the running member. Because a re-run consequently is not serialised against push runs, a re-run is also made read-only for review state: an ordinary re-run may review and post its findings, while an INF-431 edited-event re-run only refreshes the verdict; neither writes the state comment. Otherwise two same-head runs could both pass the live-head guard and PATCH the same comment, the last writer silently dropping the other's findings from revalidation. State advancement belongs to push-triggered runs; the cost is one full review on the next push, which is the safe direction.
Prompt trust boundary. The fixed confirmation rules ride in the system prompt; the prior findings themselves (earlier model output influenced by the contributor's diff) are passed in the user message explicitly framed as untrusted data, so instruction-like text inside a recorded finding cannot steer the next review at system priority.
The initial pass also gained a coverage contract (INF-292) in SYSTEM_REVIEWER_RULES, unconditional on the flag: complete the review before reporting, report every independently actionable material finding (dedupe rather than truncate at an arbitrary count) — this front-loads the depth a later delta-scoped confirmation review depends on.
The INF-172 metrics record gains a review_mode field (full | confirmation), recorded on every run regardless of the flag, so convergence can be measured before the flag is enabled anywhere it matters.

Endpoint + deadline knobs (INF-249 — GPT-5.6 readiness)

OpenAI's GPT-5.6 guidance requires reasoning + function tools to go through the Responses API (Chat Completions function tools need effective reasoning none, while 5.6 defaults to medium), so the agentic reviewer routes endpoints per model:

MULTI_LLM_REVIEW_RESPONSES_MODELS — CSV of model-id prefixes whose agentic calls use the gateway's /v1/responses endpoint (e.g. openai/gpt-5.6 covers -sol, -terra, -luna). Unset ⇒ every model stays on Chat Completions — the GPT-5.5 path is byte-for-byte unchanged.
MULTI_LLM_REVIEW_REASONING_EFFORT — explicit reasoning effort sent on the Responses path (none | low | medium | high | xhigh | max, the GPT-5.6 set — there is no minimal on 5.6). Default medium; an invalid value fails loudly rather than silently running a different configuration.
MULTI_LLM_REVIEW_RUN_DEADLINE_MS — the soft in-run deadline described above. Workflow default 600000 (10 min); unset in local/eval runs ⇒ unbounded.
VERCEL_AI_GATEWAY_RESPONSES_URL — optional explicit Responses endpoint. Unset ⇒ derived from VERCEL_AI_GATEWAY_URL when that is a .../chat/completions URL; an underivable custom URL fails closed (never the public endpoint — that would route review content and the bearer token past a private gateway).

The per-run INF-172 metrics record embeds the effective runtime config (models, agentic/panel flags, responses routing, effort, budgets, deadline), so a metrics row stays interpretable without reconstructing repo-var state after the fact. Offline eval runs (scripts/review-eval/replay.ts) additionally persist a full reproducibility record — commit, per-model endpoint, requested/returned model ids, effort, grounding/KB modes, input hashes, budget knobs, duration — and keep the benchmark stationary by default (REVIEW_EVAL_KB=off; set on explicitly for the production-parity arm).

Cost scales ~linearly with the model count (per-PR spend ≈ the per-review ceiling × number of models). The posted review's footer breaks down per-model token cost and notes any model that was skipped (a partial panel is labelled as such, never reported as a clean pass). Measurement to date (INF-171): the 3-model union lifts recall but regresses precision — see the INF-163 measurement log before enabling in CI.

How to skip a PR

Add [skip ai-review] to the PR title — any substring match counts here, as with every other CI bypass marker in the repo (per bypass-marker matching) — or on its own line in the PR body. Inline backticked or prose mentions in the body do not trigger the skip; that strict-line rule only applies to the body, not the title. The title match is a plain case-insensitive substring — backticks do not prevent it — so to mention the concept in a PR title without triggering the skip, write it differently (e.g. skip-ai-review).

Adding or removing the marker re-triggers the check automatically (the workflow subscribes to pull_request.edited, and exemptions are always evaluated against the live PR metadata, not the triggering event's payload) — so a marker added after a red run turns the check green without a new push. With delta mode off, an ordinary title/body edit on a current-head reviewed PR uses the INF-297 cheap exit. With delta mode on, the script classifies the edit: verdict-only own-lines reuse state and refresh the verdict without an LLM call; review-relevant edits force a full review.

The workflow also deliberately skips — green, with the reason recorded in the check's step summary (INF-297):

Draft PRs (job-level; GitHub re-triggers on ready-for-review).
PRs from forks (the secret is not available to fork PRs, by design).
PRs opened by dependabot[bot].
PRs whose base branch is not develop, and release/* / hotfix/* head branches — the same exemption set as the other source-PR gates.
PRs with zero changed files (e.g. an already-merged back-merge).

How the budget works

The token budget is the cumulative ceiling per PR push across all configured models. The script:

Estimates the prompt size (≈4 chars/token) before each gateway call.
Skips remaining models cleanly if the estimate would overflow the budget — a "Skipped: token budget exhausted" review is posted so reviewers see the abort, rather than silent omission.
Records the gateway's reported usage.total_tokens from the terminal SSE usage chunk after each successful call. Falls back to estimateTokens when the gateway omits the usage chunk (some providers do not support stream_options).
Logs the running balance to the workflow's step output.

The default 100k/PR comfortably accommodates the constitution (~10k) + a typical spec (~5k) + a 30k diff + the response. Bigger refactor PRs may hit the budget; raise MULTI_LLM_REVIEW_TOKEN_BUDGET per-PR (gh variable set MULTI_LLM_REVIEW_TOKEN_BUDGET --body 250000) or [skip ai-review] them.

Failure modes (fail-loud since INF-297)

The review content is advisory — the bot never approves or requests changes; humans own merge. Its absence is not advisory: since INF-297 (motivated by the INF-172 shadow window, where 8 PRs — two of them security-relevant directory changes — shipped unreviewed behind green checks during two gateway-credit outages), every run ends with a determinate terminal status written to a status file and rendered into the check's step summary:

reviewed — at least one model's review actually posted (findings, or an explicit clean "no findings" review). This is the only outcome that represents fresh model coverage; "the workflow exited 0" is never, by itself, treated as reviewed.
verdict_only — an empty confirmation delta reused trusted review state and refreshed the independent-review verdict without a model call. Green and non-exempt, but explicitly not fresh review coverage.
skipped — a deliberate exemption (see the list above), green with the reason attributable in the step summary.
failed — everything else. The script exits non-zero and the required check goes RED.

Mode	Behaviour
`MULTI_LLM_REVIEW_ENABLED != true`	RED — a review was expected and the reviewer is disabled (accidental skip).
`VERCEL_AI_GATEWAY_TOKEN` missing on a model-review path	RED — the reviewer cannot call the gateway. A no-model `verdict_only` refresh does not require it.
Gateway error (5xx, 402 credit exhaustion, idle timeout)	A "Skipped: gateway error" review is posted for visibility, but no model reviewed → terminal status `failed`, RED.
Model returns malformed / unparseable findings JSON	Treated as an incomplete review (both agentic and diff-only paths), never as "zero findings — clean"; nothing else posted → RED.
Budget exhausted with nothing posted	"Skipped: budget exhausted" review posted for visibility; no model reviewed → RED.
Findings collected but the review POST fails	The review never reached the PR → RED.
Script crashes / job dies without writing a status file	The always-run terminal step fails closed: missing or unparseable status ⇒ RED.
Full PR diff is empty (zero changed files)	Deliberate skip → green, explicit.
Confirmation delta is empty	`verdict_only` → green; trusted findings carry forward and the verdict gate runs, but no fresh review coverage is claimed.
GitHub check-runs API returns 403/error (neutral check)	Warning logged; never throws — the neutral findings check-run is a bonus signal, not the terminal status.
Partial panel (some models skipped, ≥1 posted)	`reviewed`, with the partiality named in the posted review and the terminal detail.

The reason a review did or did not post is always visible on the PR: open the Multi-LLM review check → the step summary shows reviewed / verdict refreshed / deliberately skipped (reason) / failed (reason) without reading raw logs. To deliberately skip a red PR, add [skip ai-review] (title, or own body line) — the edit re-triggers the check and re-runs read live PR metadata.

Soak protocol (ongoing quality signal)

The reviewer has been live since 2026-06-02. The soak protocol (originally the activation gate) continues as an ongoing quality signal for deciding when to expand the panel:

The protocol as originally written (kept for context — see the note below on why it produced nothing):

Watch the bot's inline review comments on merged PRs.
React with 👍 on findings that genuinely helped.
React with 👎 on findings that were noise (uncited, wrong, or pedantic style preferences).
Decision gate for v2 (adding a second model):
- If 👍 outweighs 👎 across 5+ PRs → schedule v2 (add google/gemini-2.0-pro as a second reviewer — "panel of judges").
- If 👎 outweighs 👍 → tighten the system prompt's "what counts as a finding" rules and re-soak.

⚠ In practice this protocol produced no data. Measured 2026-07-16 (INF-254): zero 👍/👎 reactions exist across the whole shadow window — 89 findings on 5 sampled PRs, 0 reactions, and none anywhere in the 14-day window. The team resolves threads and replies fix-or-refute instead; it does not react with emoji. Treat the thumbs rate as unavailable, not as neutral, and use the two signals below.

The INF-163 offline eval harness (scripts/review-eval/) gives the objective signal: precision and recall against a labelled benchmark of real historic Constellation defects. The shadow collector (below) gives the live one, derived from review-thread state rather than reactions.

Shadow window + adjudication protocol (INF-172)

Phase 4 of INF-163 turns the soak signal into a MEASURED comparison against Copilot over a real shadow window (>= 20 PRs AND >= 2 weeks), feeding the INF-173 retire-Copilot decision. Both reviewers already run on every PR — this layer only measures.

Per-run metrics sink. When MULTI_LLM_REVIEW_METRICS_PATH is set (the workflow sets it), the reviewer appends one JSONL record per run — {run_id, pr, reviewer, models, finding_count, inline_finding_count, tokens_used, timestamp} — and the workflow uploads it as a review-metrics-<run_id>-<run_attempt> Actions artifact (90-day retention; the attempt suffix keeps re-runs of failed jobs from colliding with the immutable artifact of the previous attempt). finding_count is the total postable findings; inline_finding_count is the anchored subset actually posted as diff comments — the only ones the collector can harvest (non-anchorable findings go to the review summary body, which the collector never reads). Append-only JSONL by design — no DB table (INF-scope ownership decision on the INF-172 ticket).

Adjudication protocol (what humans do during the window). Nothing extra — just review PRs the way you already do (INF-254).

The window originally asked reviewers to react 👍/👎 on inline findings. Nobody ever did: adjudication sat at 0% for the window's whole first half (0 reactions across 89 findings on 5 sampled PRs). The signal the team actually emits is the one AGENTS.md § PR Review Loop already mandates — resolve-thread / fix-or-refute — so the collector now reads that instead.

Review PRs normally. Both reviewers' findings arrive as inline diff comments.
Fix the finding, or reply with a brief reason — then resolve the thread. That is the existing convention; there is nothing new to remember.
Reactions still work and still win when present: 👍 → true positive, 👎 → false positive. They are an override, not the mechanism.
Don't agonise: an unresolved thread, or a reply that asserts no clear verdict, is reported as unadjudicated and is never imputed either way. Contradictory signals are reported as conflicted and excluded from precision.
Don't change the reviewer config mid-window (models, panel, budgets, prompts). The pinned config and window start date are recorded on the wiki page inf-163-first-measurement-2026-06-26. If it changes anyway, the collector partitions per model rather than blending — see below.

How the verdict is derived. Resolution alone cannot tell TP from FP, because the convention resolves threads in both the fix and the refute case (measured: 241 of 243 window threads are resolved — a near-constant). So resolution is a gate, and the human reply is the discriminator: its leading clause is matched against a conservative fix/refute vocabulary (Fixed in <sha>: … → TP, Refuted (no change): … → FP). Anything that does not clearly assert a verdict stays unadjudicated rather than being guessed. The full rule, its vocabulary and its known failure modes live in .ai/specs/SPEC-inf-254-shadow-adjudication.md.

⚠ What adjudicated precision measures. It is the accept-and-fix rate, not correctness, and it is not symmetric between reviewers: a cheap, obviously-correct nit gets fixed unargued and scores TP, while a deep, contestable finding invites scrutiny and is often refuted — and a refuted deep finding may still have been worth raising. The metric rewards triviality. Read it alongside the offline benchmark (which scores against ground-truth defect labels), never as a standalone verdict.

Collector. npx tsx scripts/review-eval/shadow-collect.ts --prs <csv> (or --since <date> [--until <date>], optionally --model <id>) harvests both reviewers' inline findings for each PR in the window (ours via the Multi-LLM PR review review marker, Copilot via the Copilot login), derives per-reviewer adjudicated precision from review-thread state, and computes agreement (same file + overlapping lines between the two finding sets). It writes per-PR JSONL plus an aggregate markdown report to scripts/review-eval/results/shadow/ (gitignored).

Per-model partitioning. Every agent finding is attributed to the model named by its review's **Model:** line, so a roster change mid-window partitions the measurement instead of blending it. --model openai/gpt-5.5 narrows a run to the INF-172 pinned model (precision, agreement, per-PR rows and the JSONL all honour it). Pass no --model for the unfiltered superset; a blank --model is rejected.
Only real reviews count. A PR is measured only if the model under measurement actually reviewed it. A run that was budget-skipped or errored still posts a marked review (> Skipped: …) but no findings, so it does not count as a review — this is not hypothetical: on #1304 all 12 marked reviews were gateway-error skips, so gpt-5.5 never reviewed it at all, and ~8% of the window's apparent gpt-5.5 coverage was this kind of phantom. Anything the measured model did not review is reported as agent-absent and excluded, so the agent is never scored as "found nothing" on work it never saw.

What shadow data can NEVER claim: live recall. There are no ground-truth defect labels in a shadow window, so the collector's aggregate has no recall field and its report says so explicitly. Recall remains the offline benchmark's job (scripts/review-eval/replay.ts against scripts/review-eval/benchmark/).

⚠ Cross-reviewer agreement is low, and that is a real result, not a bug. Measured over the window: 5.0% of the agent's locatable findings overlap a Copilot one (24/482), and 6.5% the other way (20/308) — low, but not zero. The reviewers are complementary, not redundant: the agent finds RLS / outbox / cache-scoping defects; Copilot finds spec surfaces:-list gaps and docs drift (on #1345 both had 6 locatable findings and 0/6 overlap each way). A ~5% overlap is far too thin to corroborate findings, so agreement cannot serve as a precision filter.

v2 sketch — panel of judges (deferred)

The YAML scaffold and the script both already loop over --models. v2 adds a second model to the list and lets the workflow post two sets of inline reviews per PR. Open questions for when v2 is up:

Do we want a third "summariser" pass that reconciles findings across the two reviewers, or is two raw review sets fine?
Do we want to surface a combined neutral check verdict that aggregates all models' finding counts?

Defer until v1 has soaked successfully.

Where it lives

File	What
`.github/workflows/multi-llm-review.yml`	The workflow. Owns gating, environment, escape hatches.
`scripts/multi-llm-review.ts`	The reviewer script. Pure I/O at the edges, pure functions for unit-tested logic.
`scripts/multi-llm-review.test.ts`	Unit tests: arg parsing, spec extraction, budget tracker, prompt assembly, rendering, streaming, inline posting.
`.ai/specs/SPEC-inf-126-multi-llm-review-activation.md`	The activation spec.
`.ai/specs/SPEC-inf-155-multillm-stream-inline.md`	The streaming + inline comments spec.
`scripts/review-eval/`	Offline eval harness (INF-168). Scores any reviewer arm against a labelled benchmark; reports precision + recall.
`scripts/review-eval/benchmark/seed.json`	3 seed labelled cases from real historic Constellation defects.
`.ai/specs/SPEC-inf-168-review-eval-harness.md`	Spec for the offline eval harness (Phase 0 of INF-163).
`scripts/review-eval/shadow-collect.ts`	Shadow-window collector (INF-172): adjudicated precision + cross-reviewer agreement from live PRs.
`.ai/specs/SPEC-inf-172-shadow-mode.md`	Spec for shadow mode + the per-run metrics sink (Phase 4 of INF-163).
`.ai/specs/SPEC-inf-293-delta-scoped-multi-llm-rereview.md`	Spec for confirmation mode: the state comment, mode resolution, and delta-scoped re-review.

Why a second opinion​

What it does​

How findings are posted​

How the idle-timeout model works (streaming)​

How it was enabled (one-time repo-admin step)​

Agentic-mode knobs (INF-169 / INF-207 / INF-170 — gated behind MULTI_LLM_REVIEW_AGENTIC)​

Panel-mode knobs (INF-171 — gated behind MULTI_LLM_REVIEW_PANEL)​

Confirmation-mode knobs (INF-293 — gated behind MULTI_LLM_REVIEW_DELTA)​

Endpoint + deadline knobs (INF-249 — GPT-5.6 readiness)​

How to skip a PR​

How the budget works​

Failure modes (fail-loud since INF-297)​

Soak protocol (ongoing quality signal)​

Shadow window + adjudication protocol (INF-172)​

v2 sketch — panel of judges (deferred)​

Where it lives​