Add a Claude-engine model-routing smoke (smoke-claude-routed) #67795

Problem

gh-aw has three model-routing smokes: .github/workflows/smoke-copilot-routed.md, smoke-copilot-sdk-routed.md, and smoke-pi-routed.md. All three are checked by actions/setup/js/smoke_model_routing_assertions.cjs (#67523, #67595), and all three passed on main on 2026-10-11 (runs 38173405739, 38173407881, 38173410300). None of them uses the Claude engine.

The Claude engine has the most routing-specific behavior, and CI does not exercise it:

• AWF selects one endpoint, and Claude calls a different one. For Claude models, AWF's selection endpoint is /chat/completions. The Claude harness calls /v1/messages instead. Each main-agent request record in model-routing.jsonl then has routed: "as_selected", deviations: ["endpoint"], requested_endpoint and upstream_endpoint set to /v1/messages, and selected_endpoint: "/chat/completions". The runner writes model_routing to aw_info.json and agent_usage.json with status: selected, endpoint: /v1/messages, and selected_endpoint: /chat/completions.
• There was a regression here, and CI would not catch another. In Model routing: Claude-engine runs recorded as 'routing rejected (unsupported_endpoint)' although AWF routing succeeded #67012, successful routed Claude runs were recorded as model_routing.status: rejected, with failure_code: unsupported_endpoint and the footer "routing rejected". Correct model-routing attribution for endpoint overrides #67013 fixed it. No CI job runs a routed Claude workflow.
• The routing guide documents a log line that CI never checks. Per docs/src/content/docs/reference/model-routing.md, the Claude harness logs the selected model, effort, effective endpoint, and selected endpoint before inference. Real line: [claude-harness] inference routing: mode=awf-routed model=claude-sonnet-5 effort=medium endpoint=/v1/messages selected_endpoint=/chat/completions.

Evidence

These are private sandbox runs in githubnext/gh-aw-routing-sandbox, audited with a current-main build of gh aw audit. Both use engine: claude, model-provider: github, mode: auto, and candidates claude-haiku-4.5, claude-sonnet-5, and claude-opus-5. They ran AWF v0.28.50, gh-aw-router 0.1.3, and Claude Code 2.1.288.

• Run 38003537540 (3 main-agent requests) and run 37890995735 (5 requests)
  • Classification: mode: auto resolved to economy, with labels explain/local/trivial.
  • Classifier: claude-opus-5 at low effort on /chat/completions, purpose: routing_classification.
  • Selection: github-copilot/claude-sonnet-5, effort medium, endpoint /chat/completions.
  • Every main-agent token-usage record has model: claude-sonnet-5, path: /v1/messages?beta=true, and status 200.
• /reflect routing_models[].supported_endpoints (from sandbox/firewall/awf-reflect.json):
  • claude-haiku-4.5: /chat/completions and /v1/messages
  • claude-sonnet-5 and claude-opus-5: /v1/messages and /chat/completions
• What gh aw audit shows today for these runs:
  • model_routing: status=selected objective=cost mode=auto selected=github-copilot/claude-sonnet-5:medium endpoint=/chat/completions router=gh-aw-router@0.1.3
  • requests: routed=as_selected=3
  • In JSON output, engine_config.harness_outcome.effective_endpoint is /v1/messages. The top-level audit model_routing section has neither /v1/messages nor selected_endpoint. The runner-written aw_info.json / agent_usage.json in the agent artifact do have both.
• Redacted real Claude routed records already exist in pkg/cli/testdata/model_routing_golden/claude-awf-selected-messages/ (source run 37971713032).
• Non-routed Claude-on-GitHub setup: smoke-claude-on-copilot.md. This smoke has failed every scheduled run on main since at least 2026-10-01 (latest: 38098509801) with API Error: 400 output_config.effort "high" was provided, but model claude-haiku-4.5 does not support reasoning effort.

Replaying the current assertion helper on run 38003537540

I ran checkModelRoutingEvidence from origin/main against the run's real evidence, using engine: "claude", mainEndpoint: "/v1/messages", and the three-model allow list:

Proposed fix

Add .github/workflows/smoke-claude-routed.md, mirroring smoke-copilot-routed.md.

Keep these the same as the Copilot routed smoke:

• Triggers: schedule: every 2 days, workflow_dispatch, and the smoke-routing label command on pull_request.
• Permissions, including copilot-requests: write (the Claude engine supports it with model-provider: github).
• private: true, concurrency, strict: true, sandbox.agent.id: awf, and features.gh-aw-detection: false. Keep the detection setting the same as the other routed smokes. Routing smokes never run the default threat-detection engine, and nothing checks that detection stays unrouted #67789 tracks threat detection in routed smokes; don't address it here.
• Imports shared/smoke-test-brevity.md and shared/reporting.md, and tools.bash: ["echo"].
• The create-issue safe output and the 10-minute timeout.
• The same trivial echo task.

Change only:

• Engine: id: claude, model-provider: github, bare: true.
• model-routing with goal: cost, mode: economy, and a small Claude allowed-models pool. All three fields are required.

Choosing the model pool. gh-aw-router 0.1.3's cost-economy table covers claude-haiku-4.5 (no effort) and claude-sonnet-5 (low through max effort), and gh-aw workflows on main already use both models. Here is how they rank:

• For a pool of [claude-haiku-4.5, claude-sonnet-5], every economy ranking group puts a claude-sonnet-5 effort first and claude-haiku-4.5 last. Haiku is only selected if no sonnet-5 choice is eligible.
• If haiku is selected (null effort), it would probably fail with the same "effort high not supported" 400 that breaks smoke-claude-on-copilot on main.

So use claude-sonnet-5 as the routed pool. Include claude-haiku-4.5 only with a note explaining why.

Effort varies by classification. allowed-models can't pin effort; the schema pattern has no :effort form. The selected effort depends on how the task is classified:

• explain/local/trivial selects sonnet-5:medium. The sandbox runs cost about 18–23 AIC each.
• chore/cross_system/trivial (what the Copilot routed smoke got on main for the same echo task) ranks sonnet-5:max first.

The assertions don't depend on effort.

Post-step. Add the same "Assert model-routing evidence" post-step with engine: "claude", allowedModels matching the pool, and mainEndpoint: "/v1/messages", so M1 checks Claude's effective endpoint. Recompile with make recompile and commit the .lock.yml.

R3: no change needed for normal runs. R3 takes supported endpoints from /reflect and falls back to selection.endpoint only when /reflect has no metadata for the model.

• Real Claude runs list /v1/messages for claude-sonnet-5, so R3 passes on /v1/messages.
• The Claude harness refuses an endpoint mismatch before inference (claude_harness.test.cjs: "starts the harness with the routed model and refuses an endpoint mismatch").
• Leave R3 as an endpoint-support check. M1 (mainEndpoint: "/v1/messages") is the check that pins Claude's effective endpoint.
• Optional: when /reflect is missing, the fallback could also accept the runner-written workflow.info.modelRouting.effectiveEndpoint. Without that, R3 would fail a correct Claude run whenever /reflect is missing. If you change this, also update the docstring and the existing fallback test.

Routing status: R1 already covers it. R1 requires the runner-written model_routing.status to be selected. That is the field that read rejected in #67012, and the replay above shows R1 fails on that shape. Don't add a separate check; add Claude-shaped unit tests instead (acceptance case 3).

Acceptance cases

1. Smoke run on main.
   • Input: a scheduled run, or gh workflow run smoke-claude-routed.lock.yml on main.
   • Observable result: the run is green, and the "Assert model-routing evidence" step prints all of these PASS lines:
     • E2
     • E1
     • R1 routing selected claude-sonnet-5 (runner status=selected, endpoint=/chat/completions)
     • R2 (one routing_classification request)
     • R3 main-agent request on selected model: claude-sonnet-5 /v1/messages 200
     • R4
     • M1 all N main-agent request(s) used /v1/messages
2. Audit of the same run.
   • Input: gh aw audit <run-id> (current-main build), plus the run's agent artifact.
   • Observable result:
     • Audit shows model_routing: status=selected ... selected=github-copilot/claude-sonnet-5:<effort> endpoint=/chat/completions and routed=as_selected=N. It does not say "routing rejected".
     • Audit JSON has engine_config.harness_outcome.effective_endpoint: /v1/messages.
     • The runner-written aw_info.json / agent_usage.json model_routing has status: selected, endpoint: /v1/messages, and selected_endpoint: /chat/completions.
     • The agent log has the [claude-harness] inference routing: ... endpoint=/v1/messages selected_endpoint=/chat/completions line.
3. Negative unit cases in smoke_model_routing_assertions.test.cjs.
   • Input: fixtures built only from real Claude-engine record shapes, either redacted from sandbox run 38003537540 or taken from pkg/cli/testdata/model_routing_golden/claude-awf-selected-messages/. Use the selection with endpoint: /chat/completions, main-agent token usage with path: /v1/messages?beta=true, /reflect supported_endpoints, and runner aw_info.json model_routing. Don't invent fields.
   • Observable result: npx vitest run smoke_model_routing_assertions.test.cjs (from actions/setup/js) passes with these cases:
     • The real shape with engine: "claude" and mainEndpoint: "/v1/messages" passes every check.
     • Main-agent traffic on /chat/completions fails M1. R3 still passes because /reflect lists /chat/completions for Claude models.
     • Runner model_routing {status: "rejected", failure_code: "unsupported_endpoint", ...} from Model routing: Claude-engine runs recorded as 'routing rejected (unsupported_endpoint)' although AWF routing succeeded #67012, with the agent-written outcome still selected, fails R1.
4. Green run on the PR branch before review.
   • Input: a new workflow can't be dispatched with workflow_dispatch until it exists on the default branch or has run at least once, and the Copilot cloud agent can't trigger runs. So:
     1. A maintainer applies the smoke-routing label to the PR. This runs the PR's copy of the workflow.
     2. After that first run, a maintainer can run gh workflow run smoke-claude-routed.lock.yml --ref <branch>.
   • Observable result: the PR description links a green smoke-claude-routed run from the PR branch that shows the PASS lines from case 1.

Scope

• Keep the PR to this one fix. Do not change files or behavior unrelated to it.
• Before fixing a failing check, compare it with the same check on main. If it also fails on main, don't fix it in this PR. Merge main once main is fixed, and note in the PR which checks are known failures on main. For example, smoke-claude-on-copilot currently fails on main with the claude-haiku-4.5 effort 400.
• This applies even if a bot comment asks you to fix a failing check. Fix only failures caused by this PR's changes.