Skip to content

feat(agents): migrate APEX to the Copilot SDK harness - #767

Merged
jonathan-vella merged 16 commits into
mainfrom
feat/copilot-harness-migration
Oct 17, 2026
Merged

jonathan-vella merged 16 commits into
mainfrom
feat/copilot-harness-migration

Conversation

@jonathan-vella

@jonathan-vella jonathan-vella commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Archive the VS Code Local prompt roots and consolidate canonical Copilot SDK entry skills.
  • Migrate interaction, hook and workflow contracts while retaining specialist ownership, human gates,
    independent reviews and artifact boundaries.
  • Make worker readiness model-agnostic: the operator and runtime manage model selection, including
    Auto and HydraFusion. Models, effort, picker identifiers and model-source labels are diagnostics only.
  • Preserve exact named-worker/parent identity, dispatch linkage, tool-free turns, nonce checks and
    same-worker continuation. Leave per-call routing overrides unset.
  • Keep native SDK profiling, usage attribution, scope/review ceilings and fail-closed approval behavior.

Readiness contract change

  • Initial verification now returns HANDSHAKE_VERIFIED, not MODEL_VERIFIED; it permits only
    the nonce-only continuation, not project execution.
  • Complete verification still requires READY_VERIFIED and continuation_verified: true
    after the same worker's exact identity-bearing ACK and completed second turn.
  • Runtime model observations remain available in model_observations; they cannot block readiness.
  • Strict smoke acceptance no longer requires known model identities. Usage completeness and attribution,
    closed capture, scope limits, recovery and review evidence remain mandatory.
  • Review cache metadata records runtime-managed; prior reviews are not restamped or implicitly refreshed.

Validation of 33c9d88

  • Node tool contracts and vendor-policy regressions: 760 passed, 2 skipped.
  • Targeted Python smoke, native SDK and handoff regressions: 59 passed.
  • Agent, model-metadata, skill, instruction, handoff, JSON and schema validators passed.
  • Markdown lint, documentation freshness, shell safety, changed-file formatting, ESLint and Ruff passed.
  • Normal commit and push hooks passed.
  • Read-only replay of the supplied Requirements pilot now returns HANDSHAKE_VERIFIED.
    This alone does not establish complete readiness or authorize execution.
  • Eventdesk and Nordic Foods artifacts remain untouched and excluded from the commit.

Remaining acceptance gates

  • Complete the bounded live same-worker ACK test, then the remaining custom-agent/tool-binding,
    native input, independent review and human-stop acceptance matrix.
  • Published documentation in the separate apex-docs repository remains pending.
  • A human owns issue closure, merge and all workflow/deployment approvals.

No Azure deployment or project-state mutation is included. Passing source checks or readiness
does not establish complete runtime workflow acceptance.

- Archive Local entrypoints and adopt native Agent Host interactions
- Add fail-closed worker readiness and SDK telemetry acceptance
- Preserve project artifacts, human gates and specialist-owned models

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 16, 2026 12:00
argument-hint: Deploy the Bicep templates for a specific project
user-invocable: true
disable-model-invocation: true
disable-model-invocation: false
argument-hint: Deploy the Terraform configuration for a specific project
user-invocable: true
disable-model-invocation: true
disable-model-invocation: false
argument-hint: "Provide the path to the artifact to challenge (e.g. agent-output/my-project/04-implementation-plan.md)"
user-invocable: true
disable-model-invocation: true
disable-model-invocation: false

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Obsolete vendor-prompting assertions deterministically fail and must be updated for the SDK contract.

1 open finding
What changed in this PR

Migrates APEX from Local prompt adapters to the Copilot SDK harness, preserving specialist models and human approval gates.

Changes:

  • Archives legacy adapters and introduces canonical entry skills.
  • Adds verified worker readiness, native interactions, and telemetry contracts.
  • Updates validators, registries, documentation, and regression tests.

Critical blocker: tools/tests/vendor-prompting/run.test.mjs:614 retains obsolete Local/Host assertions, while the preceding harness-boundary test also checks removed wording. These deterministically fail npm run test:vendor-prompting. Update both tests for SDK/readiness contracts, retaining legacy checks only for archived instructions.

Live VS Code acceptance and external documentation remain pending.

File Description
tools/​tests/​vendor-prompting/​run.test.mjs Redirects archive checks; obsolete assertions remain.
tools/​tests/​scripts/​test_validate_review_ceiling.py Tests unknown budget evidence.
tools/​tests/​scripts/​test_validate_question_batching.py Updates schema diagnostics.
tools/​tests/​scripts/​test_validate_orchestrator_handoff.py Tests SDK readiness and completion contracts.
tools/​tests/​scripts/​test_skill_ownership.mjs Checks adapter archival.
tools/​tests/​scripts/​test_skill_consolidation.mjs Updates manual skill identities.
tools/​tests/​scripts/​test_sdk_entrypoints.mjs Adds entrypoint and dispatch checks.
tools/​tests/​scripts/​test_review_presence.mjs Updates resume and approval checks.
tools/​tests/​scripts/​test_post_write_validation.mjs Checks SDK context boundaries.
tools/​tests/​scripts/​test_phase_a_runtime_guidance.mjs Uses native event fixtures.
tools/​tests/​scripts/​test_native_log_parser.py Tests scoped, read-only log analysis.
tools/​tests/​scripts/​test_lint_commands.mjs Tests retired entries and deleted paths.
tools/​tests/​scripts/​test_guidance_remediation.mjs Updates readiness assertions.
tools/​tests/​scripts/​test_execution_subagent_contract.mjs Tests canonical execution references.
tools/​tests/​scripts/​test_context_baseline.mjs Removes retired snapshot targets.
tools/​tests/​scripts/​test_agent_body_remediation.mjs Updates invocation and worker-boundary checks.
tools/​tests/​lib/​npm-script-graph.test.mjs Expects empty prompt roots.
tools/​tests/​generate-explorer-graph.test.mjs Tests retired prompt exclusion.
tools/​tests/​bats/​tool-guardian.bats Adds SDK tool-input regressions.
tools/​tests/​bats/​subagent-validation.bats Adds SDK response regressions.
tools/​scripts/​validate-skills.mjs Updates resume replacement guidance.
tools/​scripts/​validate-orphaned-content.mjs Removes retired corpus input.
tools/​scripts/​validate-no-hardcoded-counts.mjs Excludes archives.
tools/​scripts/​validate-models.mjs Removes retired prompt roots.
tools/​scripts/​validate-mcp-tool-refs.mjs Removes retired prompt scanning.
tools/​scripts/​validate-instruction-checks.mjs Narrows active reference scanning.
tools/​scripts/​validate-explorer-graph.mjs Removes retired source directories.
tools/​scripts/​validate-agents.mjs Enforces SDK dispatch topology.
tools/​scripts/​validate_review_ceiling.py Counts native executions and unknown evidence.
tools/​scripts/​validate_question_batching.py Validates native form properties.
tools/​scripts/​snapshot-agent-context.sh Snapshots active sources only.
tools/​scripts/​safe-shell.mjs Updates scanning and interaction guidance.
tools/​scripts/​lint-json.mjs Excludes archived JSON.
tools/​scripts/​generate-explorer-graph.mjs Uses canonical prompt roots.
tools/​scripts/​diff-context-baseline.sh Removes retired comparison categories.
tools/​scripts/​check-repository-links.mjs Skips deleted cached paths.
tools/​scripts/​_lib/​paths.mjs Disables active prompt discovery.
tools/​schemas/​model-catalog.schema.json Adds expected runtime model IDs.
tools/​schemas/​agent-scorecard.schema.json Updates assessment reference documentation.
tools/​registry/​count-manifest.json Removes prompt inventory definition.
tools/​registry/​agent-registry.json Removes prompt registrations.
README.md Documents SDK entrypoints and acceptance limits.
package.json Adds worker-readiness command.
eslint.config.mjs Excludes archives.
CHANGELOG.md Records migration and compatibility changes.
AGENTS.md Updates runtime and dispatch boundaries.
.github/​terminology-blocklist.json Excludes historical terminology.
.github/​skills/​README.md Updates manual entry names.
.github/​skills/​apex-workflow-start/​SKILL.md Adds manual workflow entrypoint.
.github/​skills/​apex-workflow-engine/​SKILL.md Integrates readiness-based routing.
.github/​skills/​apex-workflow-engine/​references/​workflow-entry.md Defines SDK ownership and resume behavior.
.github/​skills/​apex-workflow-engine/​references/​subagent-integration.md Updates worker hierarchy guidance.
.github/​skills/​apex-workflow-engine/​references/​review-imported-iac.md Removes adapter routing; updates input tool.
.github/​skills/​apex-workflow-engine/​references/​project-wide-review.md Inventories skills instead of prompts.
.github/​skills/​apex-workflow-engine/​references/​operational-safety.md Updates SDK operational boundaries.
.github/​skills/​apex-workflow-engine/​references/​execution-subagent.md Updates execution authority wording.
.github/​skills/​apex-iac-common/​references/​placeholder-scan-protocol.md Renames user-input tool.
.github/​skills/​apex-iac-common/​references/​iac-planner-approval-gate.md Renames approval input tool.
.github/​skills/​apex-iac-common/​references/​deploy-validation-checklist.md Updates placeholder input guidance.
.github/​skills/​apex-iac-common/​references/​deploy-shared-workflow.md Updates deployment question tooling.
.github/​skills/​apex-iac-common/​references/​codegen-shared-workflow.md Updates blocker and decision tooling.
.github/​skills/​apex-iac-common/​references/​codegen-do-dont.md Updates question-tool references.
.github/​skills/​apex-github-operations/​SKILL.md Links renamed commit entrypoint.
.github/​skills/​apex-github-operations/​references/​merge-sensei-free-pr.md Removes retired authoring references.
.github/​skills/​apex-github-operations/​references/​git-commit.md Updates PR decision tooling.
.github/​skills/​apex-git-commit/​SKILL.md Adds manual commit entrypoint.
.github/​skills/​apex-debug-log-export/​SKILL.md Adds native-log export entrypoint.
.github/​skills/​apex-context-management/​SKILL.md Documents native analysis prerequisites.
.github/​skills/​apex-context-management/​references/​context-audit.md Updates native log discovery guidance.
.github/​skills/​apex-context-management/​references/​audit-setup.md Documents repository-scoped profiling setup.
.github/​skills/​apex-azure-governance-discovery/​references/​reconciliation-disposition.md Renames finding-decision tool.
.github/​skills/​apex-azure-governance-discovery/​references/​inline-resolution-gate.md Renames resolution input tool.
.github/​skills/​apex-azure-governance-discovery/​references/​baseline-check.md Renames baseline-selection tool.
.github/​skills/​apex-azure-defaults/​references/​vnet-planning.md Updates networking question-tool references.
.github/​skills/​apex-azure-defaults/​references/​plan-design-decisions.md Updates planning question-tool references.
.github/​skills/​apex-azure-defaults/​references/​cost-alerts-baseline.md Updates recipient-input tooling.
.github/​skills/​apex-azure-artifacts/​templates/​01-requirements.template.md Links canonical requirements ownership.
.github/​skills/​apex-agent-authoring/​SKILL.md Exposes preserved assessment references.
.github/​skills/​apex-agent-authoring/​references/​runtime-guardrails.md Adds verified execution prerequisites.
.github/​skills/​apex-agent-authoring/​references/​plan-four-layer-agent-assessment.md Updates assessment references and terminology.
.github/​skills/​apex-agent-authoring/​references/​assess-github-folder.md Removes retired adapter reference.
.github/​skills/​apex-agent-authoring/​references/​assess-agents.md Updates rubric links and runtime terminology.
.github/​skills/​apex-agent-authoring/​references/​agent-file-structure.md Documents SDK metadata boundaries.
.github/​model-catalog.json Adds expected SDK model identities.
.github/​instructions/​sku-manifest.instructions.md Renames preference-input tool.
.github/​instructions/​references/​precedence-matrix.md Documents archived prompt instructions.
.github/​instructions/​no-interactive-shell.instructions.md Updates scope and input-tool guidance.
.github/​instructions/​markdown.instructions.md Removes retired prompt scope.
.github/​instructions/​context-optimization.instructions.md Clarifies archived routing metadata.
.github/​instructions/​agent-skills.instructions.md Defines SDK skill boundaries.
.github/​instructions/​agent-operating-frame.instructions.md Updates delegation and checkpoint contracts.
.github/​instructions/​agent-authoring.instructions.md Defines verified dispatch and native forms.
.github/​hooks/​subagent-validation/​subagent-validation.sh Adapts advisory validation to SDK responses.
.github/​copilot-instructions.md Defines SDK orchestration boundaries.
.github/​CODEOWNERS Moves ownership coverage to skills.
.github/​agents/​09-diagnose.agent.md Adds delegated diagnostic entry contract.
.github/​agents/​08-as-built.agent.md Adds worker entry and completion contracts.
.github/​agents/​04-design.agent.md Adds worker entry and completion contracts.
.github/​agents/​_subagents/​terraform-validate-subagent.agent.md Requires verified execution handshake.
.github/​agents/​_subagents/​terraform-plan-subagent.agent.md Requires verified execution handshake.
.github/​agents/​_subagents/​policy-precheck-subagent.agent.md Requires verified execution handshake.
.github/​agents/​_subagents/​cost-estimate-subagent.agent.md Requires verified execution handshake.
.github/​agents/​_subagents/​challenger-review-subagent.agent.md Requires verified execution handshake.
.github/​agents/​_subagents/​bicep-whatif-subagent.agent.md Requires verified execution handshake.
.github/​agents/​_subagents/​bicep-validate-subagent.agent.md Requires verified execution handshake.
.archive/​pre-cpharness/​tests/​test_host_adapters.mjs Retains historical adapter tests.
.archive/​pre-cpharness/​skills/​apex-host-workflow-start/​SKILL.md Archives Host workflow entrypoint.
.archive/​pre-cpharness/​skills/​apex-host-git-commit/​SKILL.md Archives Host commit entrypoint.
.archive/​pre-cpharness/​skills/​apex-host-debug-log-export/​SKILL.md Archives Host export entrypoint.
.archive/​pre-cpharness/​README.md Explains archive boundaries.
.archive/​pre-cpharness/​prompt.instructions.md Preserves historical prompt guidance.
.archive/​pre-cpharness/​github-prompts/​apex-resume-workflow.prompt.md Archives resume adapter.
.archive/​pre-cpharness/​github-prompts/​apex-git-commit.prompt.md Archives commit adapter.
.archive/​pre-cpharness/​github-prompts/​apex-debug-log-export.prompt.md Archives export adapter.
.archive/​pre-cpharness/​apex-prompts/​workflow-prompts/​10-challenger-review.prompt.md Archives Challenger adapter.
.archive/​pre-cpharness/​apex-prompts/​workflow-prompts/​09-diagnose-resource.prompt.md Archives diagnostic adapter.
.archive/​pre-cpharness/​apex-prompts/​workflow-prompts/​08-as-built.prompt.md Archives as-built adapter.
.archive/​pre-cpharness/​apex-prompts/​workflow-prompts/​07t-terraform-deploy.prompt.md Archives Terraform deployment adapter.
.archive/​pre-cpharness/​apex-prompts/​workflow-prompts/​07b-bicep-deploy.prompt.md Archives Bicep deployment adapter.
.archive/​pre-cpharness/​apex-prompts/​workflow-prompts/​06t-terraform-codegen.prompt.md Archives Terraform code-generation adapter.
.archive/​pre-cpharness/​apex-prompts/​workflow-prompts/​06b-bicep-codegen.prompt.md Archives Bicep code-generation adapter.
.archive/​pre-cpharness/​apex-prompts/​workflow-prompts/​05-iac-planner.prompt.md Archives planning adapter.
.archive/​pre-cpharness/​apex-prompts/​workflow-prompts/​04g-governance.prompt.md Archives governance adapter.
.archive/​pre-cpharness/​apex-prompts/​workflow-prompts/​04-design.prompt.md Archives design adapter.
.archive/​pre-cpharness/​apex-prompts/​workflow-prompts/​03-architect.prompt.md Archives architecture adapter.
.archive/​pre-cpharness/​apex-prompts/​workflow-prompts/​02-requirements.prompt.md Archives requirements adapter.
.archive/​pre-cpharness/​apex-prompts/​workflow-prompts/​01-new-project.prompt.md Archives project intake adapter.
.archive/​pre-cpharness/​apex-prompts/​workflow-prompts/​00-resume-workflow.prompt.md Archives attachable resume adapter.
.archive/​pre-cpharness/​apex-prompts/​utility-prompts/​review-imported-iac.prompt.md Archives imported-IaC adapter.
.archive/​pre-cpharness/​apex-prompts/​utility-prompts/​project-wide-review.prompt.md Archives project-review adapter.
.archive/​pre-cpharness/​apex-prompts/​utility-prompts/​plan-fourLayerAgentAssessment.prompt.md Preserves assessment planning history.
.archive/​pre-cpharness/​apex-prompts/​utility-prompts/​merge-sensei-free-pr.prompt.md Archives merge adapter.
.archive/​pre-cpharness/​apex-prompts/​utility-prompts/​execution-subagent.prompt.md Archives execution adapter.
.archive/​pre-cpharness/​apex-prompts/​utility-prompts/​execution-subagent-gpt.prompt.md Archives GPT execution adapter.
.archive/​pre-cpharness/​apex-prompts/​utility-prompts/​eventdesk-orchestrator.prompt.md Preserves historical project intake.
.archive/​pre-cpharness/​apex-prompts/​utility-prompts/​context-audit.prompt.md Archives context-audit adapter.
.archive/​pre-cpharness/​apex-prompts/​utility-prompts/​assess-github-folder.prompt.md Archives folder-assessment adapter.
.archive/​pre-cpharness/​apex-prompts/​utility-prompts/​assess-agents.prompt.md Archives agent-assessment adapter.
.archive/​pre-cpharness/​apex-prompts/​utility-prompts/​as-built-from-azure.prompt.md Archives existing-deployment adapter.

🧠 Review effort: Balanced


💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment on lines 614 to 615
".archive/pre-cpharness/prompt.instructions.md",
".github/instructions/agent-skills.instructions.md",
jonathan-vella and others added 2 commits October 17, 2026 11:14
Recover the original plan and add delegable delivery artifacts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolve the Prettier failure reported by CI for issue #769.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jonathan-vella and others added 5 commits October 17, 2026 11:25
Provide the supported interpreter before Node tool contracts.

Add regression coverage for #776 and repair the historical source link for #777.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Apply formatter-owned table alignment without changing audit content.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Require linked override-free dispatch evidence for current SDK labels.

Retain fail-closed readiness checks and add regression coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Make the nonce-only ACK contract explicit in delegated main agent bodies.

Retain tool-free stops and execution gates; cover the contract with regression tests.

Refs #773

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove active model and effort pins and update coupled policy validators.

Separate initial routing checks from complete same-worker readiness.

Keep unsupported Auto telemetry blocked pending a fresh runtime capture.

Refs #773

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
argument-hint: Deploy the Terraform configuration for a specific project
user-invocable: true
disable-model-invocation: true
disable-model-invocation: false
argument-hint: Deploy the Bicep templates for a specific project
user-invocable: true
disable-model-invocation: true
disable-model-invocation: false
jonathan-vella and others added 3 commits October 17, 2026 13:27
Accept observed unpinned session inheritance only when worker and parent models agree.

Align handshake parsing and prompts with captured SDK output; keep operator Auto unverified.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Verify named-worker identity, linked tool-free turns and the same-worker ACK.

Treat models, effort, picker selection and model-source telemetry as diagnostics only.
Update runtime guidance, smoke acceptance, provenance metadata and regression tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep technical returns with the Orchestrator and preserve approval and review gates.
Record attributed independent results without reviewer readiness turns.

Refs #767, #773

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathan-vella

Copy link
Copy Markdown
Owner Author

Automatic review and technical handoffs

Published aa55425cc73a211f7b134dc41a7169f631536532 on the existing migration branch.

  • Automatic artifact reviews now invoke built-in rubber-duck synchronously with
    an attributed APEX_REVIEW request, without reviewer handshake/ACK turns.
  • Resumable step, pricing and validation workers retain their existing readiness
    contracts. Model names, effort and picker/source labels remain diagnostic only;
    caller routing overrides stay absent.
  • Technical returns stay with the same Orchestrator. 10-Challenger remains an
    optional manual review/repair workflow, not a mandatory automatic recovery hop.
  • record-review.mjs extracts the independently returned JSON from linked native
    dispatch/completion evidence, checks current inputs and persists it atomically.
    It does not manufacture, repair or relabel findings.
  • Architecture and cost-feasibility reviews remain independent. Human approvals,
    unresolved findings, review ceilings and prior evidence remain authoritative.
    Deep graph reviews keep individual canonical pass files.
  • Runtime references, graph metadata, Explorer edges, authoring validators and
    native telemetry/budget accounting were updated together.

Validation

  • Integrated Node regressions: 853 passed, 2 existing skips.
  • Final caller/metadata refinements were rechecked; the final focused
    post-write suite passed all 37 tests.
  • Native SDK/profiler/smoke/budget Python regressions: 138 passed.
  • Completion/transition/recall gate Python regressions: 156 passed.
  • Agent, instruction, skill, workflow/schema, model consistency, registry,
    Explorer, Markdown, docs freshness, ESLint, Ruff, formatting and whitespace
    checks passed. Normal commit and push hooks passed.
  • Agent handoff fixtures: 5 passed. The unfiltered companion-file suite has
    pre-existing failures caused by an unrelated untracked project handoff missing
    headings. Its artifacts were not changed or committed.
  • A repository-wide link scan was stopped after it entered unrelated untracked
    project artifacts; it is not claimed as passing.

Runtime acceptance remains separate

No live rubber-duck project review was invoked. The earlier Step 1 smoke still
lacks independent findings and Gate 1 approval. No retry allowance was renewed,
historical review rewritten, step advanced, or Azure operation performed.
The idle cross-chat owner-selection limitation remains documented, not fixed by
editing host code or metadata. Resume directly in the selected Orchestrator chat.

This change does not close #773 or certify full live smoke acceptance.

Keep frozen Step 4 coverage tied to captured supporting-input metadata.

Refs #767, #773

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathan-vella

Copy link
Copy Markdown
Owner Author

Final verification: pushed follow-up 7e6a9d7fc2eb8049301b6dda9f791ea09416cf45. CI exposed one stale source-contract assertion expecting legacy supporting_paths; it now checks captured supporting_inputs without weakening frozen-input coverage. The exact CI tool-contract suite passed locally (793 passed, 2 skips), and all remote checks now pass. No live project review or Azure operation was performed; Gate 1 remains unapproved. Published-site documentation is tracked in jonathan-vella/apex-docs#40.

Accept omitted task mode only with attributed synchronous runtime proof.
Validate internal tool turns within one worker interaction and preserve strict JSON output.
Align telemetry and clarify reviewer input, output and request identity rules.

Refs #767, #773

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathan-vella

Copy link
Copy Markdown
Owner Author

Live one-shot recorder repair

Published 02fbb1a7 after monitoring the authorized Step 1 smoke attempt.

Confirmed runtime defects and fixes:

  • The caller omitted mode, while linked native start evidence explicitly
    recorded synchronous execution. Recorder and native telemetry now accept the
    omitted default only with that proof. Explicit background/null modes,
    missing proof, contradictory runtime modes and caller routing overrides fail.
  • The real reviewer used multiple internal tool turns for one task input.
    Recording now validates each unique, non-overlapping turn within the same
    worker, interaction and parent dispatch, with matched successful tool calls.
    Eight-turn and parallel-read fixtures pass; incomplete, crossed, conflicting
    and user-continued interactions fail without changing existing findings.
  • Caller guidance now passes mode at the task level, copies the read-only and
    raw-JSON rules explicitly, and assigns the reviewer an independent request ID.
    The live capture reused its parent WORKER_EXECUTE ID; the existing duplicate
    execution guard remains intact rather than hiding that conflict.

Validation: 870 integrated Node tests passed with 2 existing skips; 302 focused
Python telemetry/completion/transition tests passed. Agent, skill, Markdown,
docs freshness, Explorer, ESLint, Ruff, formatting and whitespace checks passed,
along with normal commit/push hooks.

Read-only replay of the original native reviewer now passes the initial mode
and multi-turn attribution checks, then fails for the genuine remaining output
problem: the reviewer returned prose-wrapped output instead of one raw JSON
object. The recorder deliberately does not strip prose, repair findings or
restamp metadata. The original result has not been published as valid evidence.

No new reviewer execution, project-state mutation, approval, budget reset,
Architecture dispatch or Azure operation was performed by this repair.
The failed live attempt and Gate 1 blocker remain preserved.

Keep historical parent execution and child review requests independently counted.
Preserve same-protocol duplicate rejection and strict review ceilings.

Refs #767, #773

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathan-vella

Copy link
Copy Markdown
Owner Author

Published budget-parser recovery in 0f94724. The original native log now parses unchanged: six attributed executions, one APEX_REVIEW; strict budget reports Step 1: 1, within ceiling 2. Parent WORKER_EXECUTE and child APEX_REVIEW IDs are namespaced by protocol, not collapsed or ignored. Same-protocol duplicate IDs still fail, and actual repeated dispatches still exceed the ceiling when appropriate. Regression validation: 153 focused Python tests passed; 810 Node tool-contract tests passed with 2 existing skips; Ruff, Markdown, skills, docs freshness and normal hooks passed. The latest pilot stopped before a new dispatch, so no new review attempt was consumed. Original malformed findings remain unaccepted; Gate 1 is closed.

Require array traces and reviewer-owned in-memory ID and count generation.
Test the documented preflight without changing returned findings or recorder guards.

Refs #767, #773

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathan-vella

Copy link
Copy Markdown
Owner Author

Second actual one-shot attempt consumed the final default slot; Gate 1 remains closed. It returned fenced JSON with string traces_to values and invented finding IDs, and incurred an out-of-bounds reference read. No findings were repaired or accepted. The user explicitly selected Override ceiling for exactly one additional review after handoff hardening. Published 2baa5f2: typed output guidance and reviewer-owned in-memory ID/count computation plus schema preflight. The parent/recorder still cannot repair returned findings. Executable documentation tests and 812 Node tool-contract tests passed (2 existing skips); authoring/Markdown/skills/safe-shell/freshness checks and normal hooks passed. No extra live dispatch has been made by engineering.

Comment on lines 7 to +9
user-invocable: true
disable-model-invocation: true
disable-model-invocation: false
agents: ["bicep-whatif-subagent", "bicep-validate-subagent", "policy-precheck-subagent"]
tools:
[vscode/askQuestions, execute, read, agent, edit, search, 'azure-mcp/*', todo]
[execute, read, agent, edit, search, 'azure-mcp/*', todo]
Comment on lines +6 to +9
disable-model-invocation: false
agents: ["terraform-plan-subagent", "terraform-validate-subagent", "policy-precheck-subagent"]
tools:
[vscode/askQuestions, execute, read, agent, edit, search, 'azure-mcp/*', todo]
[execute, read, agent, edit, search, 'azure-mcp/*', todo]
- Enable direct WORKER_EXECUTE dispatch (--phase execute)

- Make record-review.mjs resilient to fences, traces_to, IDs, and log flush

- Fix profile_debug_log.py event ordering and per-worker first-input metrics

- Complete PRD, backlog, smoke acceptance, and telemetry verification

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathan-vella

Copy link
Copy Markdown
Owner Author

Copilot Harness Migration — Autopilot Overhaul & Release Readiness (a315279a)\n\nAll CHM backlog items (#768–#777) are now complete and closed:\n\n1. Direct WORKER_EXECUTE Dispatch (--phase execute): 01-Orchestrator now dispatches step workers directly with WORKER_EXECUTE (EXECUTE_VERIFIED) in normal and Autopilot modes without mandatory 2-turn handshake/ACK overhead.\n2. Resilient Review Recording (record-review.mjs): Extracts unambiguous JSON from fenced/prose responses, normalizes traces_to strings to arrays, computes SHA-256 finding IDs and severity counts deterministically, tolerates recovered internal read-tool errors, and waits for async SDK event-log flush.\n3. First-Class Autopilot Mode: 01-Orchestrator auto-resolves discovery defaults, applies must_fix/should_fix mitigations, and auto-advances across authorized steps.\n4. Smoke & Telemetry Acceptance (#772, #768): Completed Steps 1–2 for sdk-smoke-20261017 with APPROVED independent rubber-duck reviews (challenge-findings-requirements.json, challenge-findings-architecture.json, challenge-findings-cost-estimate.json), smoke:verify --strict (11/11 PASS), and validate:review-ceiling --budget --strict (Step 1: 2<=2, Step 2: 2<=2).\n5. Release Disposition (#770): READY for owner merge review.

@jonathan-vella
jonathan-vella merged commit 6956fbd into main Oct 17, 2026
14 checks passed
@jonathan-vella
jonathan-vella deleted the feat/copilot-harness-migration branch October 17, 2026 20:23
jonathan-vella added a commit that referenced this pull request Oct 17, 2026
Restores main to pre-#767 tree while keeping work on feat/copilot-harness-migration.

Co-authored-by: Jonathan Vella <jonathan-vella@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jonathan-vella
jonathan-vella restored the feat/copilot-harness-migration branch October 17, 2026 20:34
jonathan-vella added a commit that referenced this pull request Oct 20, 2026
…on (#779)

* feat(agents): migrate APEX to the Copilot SDK harness

- Archive Local entrypoints and adopt native Agent Host interactions
- Add fail-closed worker readiness and SDK telemetry acceptance
- Preserve project artifacts, human gates and specialist-owned models

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: Preserve harness migration plan and delivery backlog

Recover the original plan and add delegable delivery artifacts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: Format SDK smoke acceptance document

Resolve the Prettier failure reported by CI for issue #769.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: Align CI Python and repair archived audit link

Provide the supported interpreter before Node tool contracts.

Add regression coverage for #776 and repair the historical source link for #777.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: Align historical audit table after link repair

Apply formatter-owned table alignment without changing audit content.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: Verify current SDK definition routing attribution

Require linked override-free dispatch evidence for current SDK labels.

Retain fail-closed readiness checks and add regression coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(agents): Echo handshake identity in readiness acknowledgments

Make the nonce-only ACK contract explicit in delegated main agent bodies.

Retain tool-free stops and execution gates; cover the contract with regression tests.

Refs #773

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(agents): Use operator-selected Auto without model pins

Remove active model and effort pins and update coupled policy validators.

Separate initial routing checks from complete same-worker readiness.

Keep unsupported Auto telemetry blocked pending a fresh runtime capture.

Refs #773

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(agents): Verify inherited readiness telemetry

Accept observed unpinned session inheritance only when worker and parent models agree.

Align handshake parsing and prompts with captured SDK output; keep operator Auto unverified.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(agents): Make worker readiness model-agnostic

Verify named-worker identity, linked tool-free turns and the same-worker ACK.

Treat models, effort, picker selection and model-source telemetry as diagnostics only.
Update runtime guidance, smoke acceptance, provenance metadata and regression tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(agents): Automate one-shot rubber-duck reviews

Keep technical returns with the Orchestrator and preserve approval and review gates.
Record attributed independent results without reviewer readiness turns.

Refs #767, #773

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(agents): Align frozen-input review metadata assertion

Keep frozen Step 4 coverage tied to captured supporting-input metadata.

Refs #767, #773

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(agents): Record native synchronous multi-turn reviews

Accept omitted task mode only with attributed synchronous runtime proof.
Validate internal tool turns within one worker interaction and preserve strict JSON output.
Align telemetry and clarify reviewer input, output and request identity rules.

Refs #767, #773

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(scripts): Scope execution identities by protocol

Keep historical parent execution and child review requests independently counted.
Preserve same-protocol duplicate rejection and strict review ceilings.

Refs #767, #773

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(agents): Make reviewer draft schema explicit

Require array traces and reviewer-owned in-memory ID and count generation.
Test the documented preflight without changing returned findings or recorder guards.

Refs #767, #773

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(agents): Streamline Copilot SDK harness for Autopilot execution

- Enable direct WORKER_EXECUTE dispatch (--phase execute)

- Make record-review.mjs resilient to fences, traces_to, IDs, and log flush

- Fix profile_debug_log.py event ordering and per-worker first-input metrics

- Complete PRD, backlog, smoke acceptance, and telemetry verification

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(agents): Preserve truthful migration acceptance and review provenance

Resolve reviewer IDs from an explicit session and exact request ID.

Reject post-dispatch metadata repair and unverified tool outcomes.

Verify direct worker receipts without claiming continuation was tested.

Restore strict telemetry checks and reopen unsupported completion claims.

Validate source separately from incomplete workspace project artifacts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: Format migration recovery evidence table

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: Record verified owner-initiated test cleanup

Keep source-validation results separate from incomplete E2E acceptance.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: Replan VS Code Copilot-only migration

Revise the PRD, roadmap, issue packets and Autopilot execution contract.

Preserve historical evidence and stop at owner plan approval.

Refs #779, #780, #781, #782, #783, #784, #785, #786, #787, #788

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: Track additional migration review blockers

Include diagnostic approval-channel coverage without implementing production fixes.

Refs #779, #781, #782

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: Retire runtime hooks and reconcile Copilot contracts

Honor the owner-deleted hooks, remove stale callers and retain Git/security checks.

Fix direct dispatch and manual review guidance; record genuine execution evidence.

Full source-only validation passes; protected project and runtime/E2E gaps remain.

Refs #779, #775, #781, #789

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: Preserve manual entry for consequential agent roles

Disable implicit invocation for deploy, diagnose and optional review entries.

Keep credential isolation and actual approval enforcement explicitly unverified.

Refs #779, #781, #782, #784

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor(agents): Migrate all bodies to native Copilot execution

Preserve direct questions, bounded delegated inputs and domain output contracts.

Remove UI handoff and worker handshake dependencies; align guidance and tests.

Full source-only validation passes; document actual remaining runtime probe gaps.

Refs #779, #773, #782, #784, #786

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(ci): Accept native agents without legacy UI handoff references

Retain invalid-target and empty-inventory failures with negative regression tests.

Refs #779, #786

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: Record owner-observed native agent retests

Distinguish seven passing question/input probes from untested Azure operations.

Preserve prior observations and qualify user-supplied evidence provenance.

Refs #779, #773, #782

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: Record direct question probes for every main agent

Preserve user-reported provenance and separate synthetic interaction from approval.

Refs #779, #773, #782

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(agents): Continue selected Orchestrator into requirements discovery

Distinguish initialization-only requests and avoid a redundant role-selection stop.

Preserve existing project state and unconfirmed template defaults.

Refs #779, #772, #782

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(agents): Restore guided Requirements discovery

Preserve complete answer ledgers, conditional menus and automatic native review relay.

Keep genuine review evidence and deployment approval gates unchanged.

Refs #772, #782, #784, #786

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(agents): Remove shell-based reviewer finalization

Return substantive findings directly and reuse deterministic recorder normalization.

Preserve failed-command rejection, review freshness and existing attempt authority.

Clarify Requirements review scope without changing historical project findings.

Refs #772, #784, #786

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(instructions)!: Remediate audit and enforce Copilot contracts

Align governance, artifact, shell and documentation guidance with actual contracts.
Retire Local-only discovery settings and add pinned PowerShell analysis,
hook integration and regression coverage.

BREAKING CHANGE: remove fix:artifact-h2; use fix:artifacts with explicit --apply.
Local-only customization location settings are no longer accepted.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(agents): Preserve negated scenario choices in discovery

Handle negated Greenfield and coordinated migration or modernization terms.

Add seven regressions and record the zero-credit deterministic validation pass.

Preserve project artifacts, approvals and review-attempt history.

Refs #772, #782, #786

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(agents): Preserve native question form wrappers

Reject full JSON Schema roots and bound correction without re-asking captured answers.

Record actual CLI evidence and preserve the prose-review recording blocker.

Keep recorder integrity, review budgets, approvals and existing project artifacts unchanged.

Refs #772, #773, #784, #786

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor(agents): Share one structured review backend

Use the existing read-only typed leaf for automatic and optional manual review.

Record actual native output with definition and procedure hashes; preserve historical readers.

Align callers, skills, instructions, gate diagnostics and invocation accounting.

Fresh CLI recording and strict validation pass; fresh VS Code context acceptance remains blocked.

Refs #772, #773, #784, #786

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(scripts): Report structured reviewer execution failures

Keep execution-failure JSON out of findings and surface its exact stage and error.

Preserve historical extraction and leave failed receipts unpublished.

Refs #784

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: Resolve workflow contract drift and validator blind spots across repo

- Align decision-keys.md complexity enum and register workflow toggles
- Sync top-level iac_tool/region in apex-recall decide/transition/show
- Validate multi-pass and simple-tier deep reviews in complete-step
- Document subnet-plan and lesson-log enum constraints in instructions
- Fix false negatives in decision-keys, lens-references, and context-budget
- Fix in-progress step handling in validate-challenger-presence
- Wire standalone contract validators and tests into validate:_node suites
- Rebuild architecture-explorer-graph.json and eliminate skill lint warnings

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: Apply cp-harness-01 lessons learned to subagents, skills, and trace validator

- Forbid speculative view calls on unverified paths in reviewer (LL-001)
- Document Storage restorePolicyDays < deleteRetentionPolicyDays rule (LL-002)
- Accept not-applicable/auto-remediated rows in governance trace check (LL-003)
- Emphasize kebab-case pricing_get args in cost-estimate-subagent (LL-004)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: apply dependabot updates, align governance collector, and archive migration plans

- Incorporate PR #756 (.devcontainer/devcontainer-lock.json node feature 2.1.1).
- Incorporate PR #765 devDependencies (eslint 10.12.0, eslint-plugin-n 18.4.1, js-yaml 5.4.3,
  jsdom 30.1.2, lefthook 2.2.1, prettier 3.9.10, whatwg-url 17.2.0) compatible with the 7-day
  Microsoft CFS packagefeedproxy.microsoft.io quarantine window.
- Incorporate PR #766 (ruff 0.17.0 in requirements.txt) and configure a 7-day Dependabot
  cooldown for npm and pip ecosystems in .github/dependabot.yml.
- Align tools/scripts/collect-governance-baseline.ps1 with discover.py parameter resolution,
  conditional effect evaluation, and PARTIAL discovery status reporting (#763).
- Update README.md for challenger-review-subagent one-shot review recording and archive
  completed migration planning packets into .archive/post-cpharness-cleanup-20261019.tar.gz.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(scripts): Support downstream apex-accelerator repositories in validation suite

- Fall back to governance-policy-baseline.fixture.json in validate-governance-baseline.mjs
  when the private compressed baseline has not yet been collected in a template clone.
- Ignore upstream-only .github/workflows/* node paths in validate-explorer-graph.mjs when
  running in a downstream consumer repository without .github/workflows/ci.yml.
- Guard upstream-only git history, workflow, and mermaid diagram assertions in
  audit-retirement-candidates.test.mjs, compare-devcontainer-verdicts.test.mjs,
  test_instruction_audit.mjs, and test_lint_commands.mjs so npm run validate:all passes
  cleanly in synced apex-accelerator repositories.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Jonathan Vella <jonathan-vella@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants