[little Kitchen (littleKitchen)]

Pull Request

Description

A PR that only edits one of the RPI skills (rpi-research, rpi-plan, rpi-implement, rpi-review) skips the rpi-agent baseline-equivalence run, even though the equivalence harness copies those skill directories into the surface it measures.

Root cause, as described in the issue: Get-AgentDependencyMap.ps1 records only skills referenced by path. rpi-agent.agent.md names its skills in backticks, so the map has no skills for it, and Get-ChangedAIArtifact.ps1 emits affectedAgents: []. The harness resolves the backtick form through Get-AgentSkillReference, so the gate and the harness disagree.

Three places decide on equivalence from affectedAgents, and all of them saw the empty list:

• .github/workflows/eval-validation.yml:339 sets the equivalence-required output.
• scripts/evals/New-AgentEvalPlan.ps1:291 sets baseline.required and the baseline producers in the plan.
• scripts/evals/Invoke-VallyEvals.ps1:1081 onward promotes affected agents into the equivalence dispatch set.

Fix

The fix goes into the manifest itself, so all three consumers stay consistent:

• EquivalenceEnvironment.psm1 gets a new Get-AgentSurfaceSkill function, extracted from New-CustomizedEnvironment. It returns the skill directories the harness materializes for an agent: the skills the agent references plus, transitively, those of its declared subagents. New-CustomizedEnvironment now calls it, so its behavior is unchanged (same set, same sort order).
• Get-ChangedAIArtifact.ps1 checks every changed path, including the previous path of renames, against those directories for each equivalence subject. A subject with a changed path inside its surface is added to affectedAgents.
• The check covers the whole skill directory, not just SKILL.md, because the harness copies the whole directory. That is also why the check lives in Get-ChangedAIArtifact.ps1: the manifest's artifacts list records only SKILL.md for skills, while this script sees every changed path.
• Each nonblank subject must match exactly one <slug>.agent.md under .github/agents. A missing or duplicate subject throws, so manifest generation fails closed: the entry script exits 2 and writes no manifest.
• Paths that exist at HEAD are checked against the surface resolved from the checkout. Deleted paths, and the previous path of a rename or copy, are checked against the surface resolved at comparisonBase, because a deleted or renamed-away skill no longer exists at HEAD. The base surface is resolved from that commit's .github/agents and .github/skills, exported with git archive. It is only exported when such a path falls under .github/skills. Eval Validation checks out with fetch-depth: 0, so the comparison base is available there.

Scope

The issue notes that applying the harness resolver to the whole dependency map is broad (181 skill references across 51 of 64 agents). This change applies it only to the equivalence subjects:

• There is a new -EquivalenceSubject parameter that defaults to rpi-agent. That matches Invoke-VallyEvals.ps1 -EquivalenceSubject (line 138) and the ValidateSet('rpi-agent') on Invoke-BaselineEquivalence.ps1 -Agent (line 105).
• Get-AgentDependencyMap.ps1 and Get-AffectedAgentSlugs are unchanged.

Documentation

docs/contributing/evals-ci.md said that baseline equivalence runs when the plan marks baseline.required: true, but not when that happens. It now explains both routes into affectedAgents.

Related Issue(s)

Fixes #3080

Type of Change

Select all that apply:

Code & Documentation:

• Bug fix (non-breaking change fixing an issue)
• New feature (non-breaking change adding functionality)
• Breaking change (fix or feature causing existing functionality to change)
• Documentation update

Infrastructure & Configuration:

• GitHub Actions workflow
• Linting configuration (markdown, PowerShell, etc.)
• Security configuration
• DevContainer configuration
• Dependency update

Other:

• Script/automation (.ps1, .sh, .py)

Testing

Issue reproduction, before and after

I followed the issue's steps: a change set containing only .github/skills/rpi/rpi-research/SKILL.md, passed through Get-ChangedAIArtifact.ps1 and then New-AgentEvalPlan.ps1.

	affectedAgents	baseline in the plan
main (d085729)	[]	required: false, reason: agent-not-affected:rpi-agent
This branch	[rpi-agent]	required: true, reason: affected-agent:rpi-agent, producers baseline:gpt-6-luna and baseline:claude-sonnet-5

With single-file change sets against this branch:

• These mark rpi-agent affected:
  • the SKILL.md of each of rpi-research, rpi-plan, rpi-implement and rpi-review;
  • rpi-research/references/research.md.
• These do not:
  • data-science-engineering/evaluation-design/SKILL.md;
  • README.md.

Pester

There are 14 new tests.

EquivalenceEnvironment.Tests.ps1, Get-AgentSurfaceSkill:

• It includes the skills of the agent and of its declared subagents.
• Its output equals the skill directories New-CustomizedEnvironment actually materializes.
• For the real repository, the rpi-agent surface contains the four RPI skills.

Get-ChangedAIArtifact.Tests.ps1, Equivalence surface gate, using a fixture whose rpi-agent names its skill in backticks:

• A SKILL.md change marks rpi-agent affected.
• A nested reference-file change marks rpi-agent affected.
• An unrelated skill does not.
• An empty -EquivalenceSubject disables the check.
• A subject that matches no agent file, or two agent files, throws. Through the entry script, both cases exit 2 without writing a manifest.

Get-ChangedAIArtifact.Tests.ps1, Equivalence surface gate at the comparison base, using real two-commit git fixtures and New-EvalChangeSet:

• Deleting the backtick-named skill marks rpi-agent affected.
• Renaming it away marks rpi-agent affected.
• Deleting an unrelated skill does not.

Fixtures that contain no rpi-agent now pass -EquivalenceSubject @() explicitly. Those are two existing scan calls in Get-ChangedAIArtifact.Tests.ps1 and one in ChangedSpecStimulus.Tests.ps1. The entry-script fixture gains an rpi-agent in its base commit.

I also ran two mutation checks:

• With the line that merges the surface result into affectedAgents disabled, the two positive gate tests fail and the two negative ones still pass.
• With the base-path check disabled, the deletion and rename-away tests fail and the unrelated-deletion test still passes.

The full scripts/tests/evals/ suite on this branch: 1601 passed, 7 failed. The 7 failures are the Rust unit-test network-trace tests, which need docker, and they fail the same way on unmodified main (1587 passed, 7 failed) in the same environment.

Repository checks

I ran each npm run validate:local step individually, plus npm run spell-check, with the repository-pinned actionlint 1.7.10 installed and every Python project synced with uv sync --locked --only-dev, which provides the locked ruff. 30 of 31 pass, including lint:yaml, lint:py, lint:ps and spell-check.

The remaining failure is lint:md-links in full-repository mode. It reports 5 external links in 4 files this PR does not touch, and the same links fail on unmodified main:

• eur-lex.europa.eu: 202, three times.
• medium.com: 403.
• owasp.org/www-project-ai-security-and-privacy-guide/: 404, which looks genuinely dead.

Markdown-Link-Check.ps1 -ChangedFilesOnly, the mode markdown-link-check.yml uses in CI, completes successfully. That mode validates internal links repository-wide and external links in the changed files.

npm run validate:docs passes: docs:lint, docs:lint:labels, docs:typecheck and docs:test.

Checklist

Required Checks

• Documentation is updated (if applicable)
• Files follow existing naming conventions
• Changes are backwards compatible (if applicable)
• Tests added for new functionality (if applicable)

Required Local Checks

• Local validation aggregate: npm run validate:local. 30 of 31 steps pass; the remaining one is the full-repository external link check below.
• Documentation validation (if docs changed): npm run validate:docs
• Spell checking: npm run spell-check
• Link validation: npm run lint:md-links. The CI-equivalent -ChangedFilesOnly run passes. The full-repository run fails only on 5 external links in 4 untouched files, which also fail on main (see Testing); I'd ask that these be accepted for this PR.

Security Considerations

• This PR does not contain any sensitive or NDA information
• Any new dependencies have been reviewed for security issues
• Security-related scripts follow the principle of least privilege

Additional Notes

• Cost: PRs that touch only the four RPI skill directories will now run the two baseline model lanes. That is the intended outcome of fix(evals): RPI skill edits skip the rpi-agent baseline-equivalence run #3080.
• Deleted skills: a skill directory that is deleted entirely is no longer on disk at the head commit, so it cannot be resolved to a surface. In practice, removing an RPI skill would also edit rpi-agent.agent.md, which already marks the agent affected.

🤖 Generated with Claude Code

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.70%. Comparing base (d085729) to head (1c2b562).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3233      +/-   ##
==========================================
+ Coverage   86.61%   86.70%   +0.09%     
==========================================
  Files          87      103      +16     
  Lines       11554    11862     +308     
  Branches        0       63      +63     
==========================================
+ Hits        10007    10285     +278     
- Misses       1547     1566      +19     
- Partials        0       11      +11     

Flag	Coverage Δ
docusaurus	90.58% <ø> (?)
pester	86.32% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 17 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Bill Berry (WilliamBerryiii)]

Thank you for fixing the primary RPI skill-change gap and sharing one current-surface resolver between the gate and materializer. I left four inline comments covering destructive path states, subject resolution, PowerShell naming, and Pester isolation. Please also resolve or explicitly accept the two remaining Required Local Checks: the local validation aggregate and link validation, whose baseline-parity exceptions are documented in the PR.

[little Kitchen (littleKitchen)]

Thanks Bill Berry (@WilliamBerryiii)! All four are addressed in 56c447e, with details on each thread.

On the local checks: with actionlint and the uv envs set up, validate:local is now 30/31. The only remaining failure is the full-repo lint:md-links run, which trips on 5 external links in files this PR doesn't touch. They fail the same way on main, and the -ChangedFilesOnly run that CI uses passes. Is it OK to accept that one? I can also file an issue for the dead OWASP link.