You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
docs: scripts/README.md evals table is missing 13 scripts added in the vally migration #2572
The scripts/README.md evals section lists only 7 scripts, but scripts/evals/ now contains 20 scripts after the beval-to-vally migration in #2552. The following 13 scripts are undocumented in the table:
Script
Purpose
Get-AgentDependencyMap.ps1
Build a JSON map of agent dependencies for the baseline-equivalence dispatcher
Get-ChangedAIArtifact.ps1
Emit a JSON manifest of AI customization artifacts changed between two git refs
Get-ChangedSpecStimulus.ps1
Emit a JSON manifest of synthetic artifacts derived from changed eval specs
Invoke-ArtifactModeration.ps1
Content-moderation pre-job for all eval specs plus changed AI artifacts
New-AgentMatrixDashboard.ps1
Render a self-contained HTML matrix dashboard for the per-agent agent-behavior eval suite
New-AgentSurfaceSignatures.ps1
Generate per-agent surface signature YAML for baseline equivalence runs
New-EquivalenceDashboard.ps1
Render a self-contained HTML dashboard for a local baseline-equivalence eval run
Test-CopilotToken.ps1
Pre-flight probe for the COPILOT_GITHUB_TOKEN secret used by vally evals
Test-EvalSpec.ps1
Validate vally eval spec files against the embedded schema and enforce per-agent behavioral eval coverage
Test-EvalSpecText.ps1
Run alex.js and retext-profanities against the AI-artifact markdown corpus
Test-StimulusPresence.ps1
Verify every changed AI artifact has at least one matching eval-spec stimulus backlink
Test-VallyTestSafety.ps1
Repo-wide safety lint that flags eval stimuli and corpora matching the skill-local refusal taxonomy
Update-AgentMatrixSummariesFromLogs.ps1
Rebuild per-agent matrix JSON summaries from existing vally logs without re-running npx vally
Expected Behavior
All scripts in scripts/evals/ should appear in the evals table in scripts/README.md, consistent with how other sections document every script.
Hi, I’d like to work on this documentation update. I noticed #2571
appears to describe the same missing eval-script entries. Could you
confirm which issue should be treated as canonical before I open the PR?
This looks resolved on main. #2988 (8147cb2) added the missing evals entries to scripts/README.md and closed the duplicate #2571, but not this one. 12 of the 13 scripts in the table above are now documented there. The 13th, New-AgentSurfaceSignatures.ps1, no longer exists: #2597 removed it. So this can be closed.
Thank you for reporting the evaluation-script documentation gap. PR #2988 completed the Evals table, and the current documentation covers the scripts requested here; its issue reference appears to have targeted the wrong issue number. We’re closing this as completed; please reopen it if a script remains undocumented.
Component
Documentation
Bug Description
The
scripts/README.mdevals section lists only 7 scripts, butscripts/evals/now contains 20 scripts after the beval-to-vally migration in #2552. The following 13 scripts are undocumented in the table:Get-AgentDependencyMap.ps1Get-ChangedAIArtifact.ps1Get-ChangedSpecStimulus.ps1Invoke-ArtifactModeration.ps1New-AgentMatrixDashboard.ps1New-AgentSurfaceSignatures.ps1New-EquivalenceDashboard.ps1Test-CopilotToken.ps1Test-EvalSpec.ps1Test-EvalSpecText.ps1Test-StimulusPresence.ps1Test-VallyTestSafety.ps1Update-AgentMatrixSummariesFromLogs.ps1npx vallyExpected Behavior
All scripts in
scripts/evals/should appear in the evals table inscripts/README.md, consistent with how other sections document every script.Steps to Reproduce
ls scripts/evals/*.ps1— 20 scripts found.scripts/README.md— only 7 scripts listed.Additional Context
scripts/evals/*.ps1(added/modified in chore(evals): migrate agent behavioral conformance from beval to Vally #2552)scripts/README.md.SYNOPSISin the script header.