Skip to content
Open
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
c340e44
Azure SRE skills
Sep 29, 2026
fe1fd4d
Work in progress on RCA skills
Oct 1, 2026
ef8c70f
Work on the RCA skillset
Oct 2, 2026
14cdad5
Merge remote-tracking branch 'upstream/main' into eugenebo-rca-skill
Oct 5, 2026
6c23a13
RCA skills eval suites
Oct 5, 2026
e3701a8
Removed language locale from the url
Oct 5, 2026
a901541
Fixing formatting issues
Oct 5, 2026
e523a57
Spelling fixes
Oct 5, 2026
6d61375
Merge branch 'main' into eugenebo-rca-skill
eugeneboms Oct 5, 2026
f8c4258
Addressed "High" code review feedback
Oct 8, 2026
b1aa016
Merge remote-tracking branch 'upstream/main' into eugenebo-rca-skill
Oct 8, 2026
ec19563
Spell check update
Oct 8, 2026
da40bf5
Merge remote-tracking branch 'upstream/main' into eugenebo-rca-skill
Oct 9, 2026
5a89e7b
style(skills): format RCA markdown tables
Oct 9, 2026
b69b781
feat(skills): add root cause analysis skill package
Oct 12, 2026
d1adb8d
Merge branch 'main' into eugenebo-rca-skill
eugeneboms Oct 12, 2026
a355311
spell check issue fixed
Oct 12, 2026
ab68e5f
Merge branch 'eugenebo-rca-skill' of https://github.com/eugeneboms/hv…
Oct 12, 2026
e747869
removed representative LR(E)
Oct 13, 2026
96b79f7
Merge branch 'main' into eugenebo-rca-skill
eugeneboms Oct 14, 2026
7495f7b
Address CR-02,04,11,12,14,15 (all Medium)
Oct 14, 2026
9aae223
Merge branch 'eugenebo-rca-skill' of https://github.com/eugeneboms/hv…
Oct 14, 2026
6384504
Merge remote-tracking branch 'upstream/main' into eugenebo-rca-skill
Oct 15, 2026
6580849
spellcheck and doc links fixes
Oct 15, 2026
18b218c
Merge branch 'main' into eugenebo-rca-skill
eugeneboms Oct 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions .github/skills/root-cause-analysis/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
---
title: Generic Root Cause Analysis Skills
description: Configure the hybrid RCA skill package and its companion assessment skills
---

This package contains three portable skills:

* `root-cause-analysis/SKILL.md` owns the hybrid investigation and disposition workflow.
* `statistical-hypothesis-testing/SKILL.md` compares eligible binary rates or numeric averages.
* `causal-evidence-likelihood/SKILL.md` estimates how one evidence item changes hypothesis
plausibility through a reasoned likelihood ratio.

The package does not define a custom agent. Install all three skills so the RCA workflow can invoke
its companion assessments. An investigation can begin with empty or populated sets of evidence,
hypotheses, and observations.

## Host behavior

The RCA skill selects one of two host modes:

* In Azure SRE mode, it loads `root-cause-analysis/references/azure-sre-tools.md` and automatically
uses relevant, safe, read-only tools from that catalog. Azure terminology and source guidance
become authoritative for the connected Azure evidence.
* In generic mode, it inventories other available tools, proposes a bounded evidence-expansion
action, and obtains user approval before invoking those tools. It can still assess supplied
evidence when no additional tools are approved or available.

Tool availability does not grant data access or permission to alter a system. Enforce read-only

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CR-12: Package README: installation, scope, placement (Medium; findings #27, #44, #59, #75, #85, #94, #22, #77, #76, #42, #29)

Standards: MCP07:2025, Supply Chain Security § SLSA Build L2, GDPR planning reference § Art. 35, Markdown § Miscellaneous, COGA § 4.7, CM.AW-P, HVE Artifact Authoring § Package and Documentation Contract, Markdown § Recommended Fields, Writing Style § Language and Vocabulary, MP-1.1, MS-2.1

Requested changes:

  • Installation: least-privilege identity and approval mode; DPIA confirmation before attaching production telemetry; connector data scope, residency, and retention; version and digest with an update and withdrawal process; concrete verify steps.
  • Content: intended use and limits; move eval claims out of 'Deferred work' and stop calling advisory stimuli verification; fix the title, frontmatter, and product-name casing.
  • Placement: the README is not distributed; move install guidance into a skill reference or the generated docs page.

Already resolved at 6d61375: #21 (Resolved in e3701a8: /en-us removed from both Learn URLs.)

Original findings and justification:

#27 Azure SRE install guidance is conditional and omits least-privilege connector and approval-mode hardening (Medium)

Location: .github/skills/root-cause-analysis/README.md lines 28-29, 37-39. Raised by: Security#8, Security-Agentic#13.

Standards: MCP07:2025 Insufficient Authentication and Authorization (OWASP MCP Top 10 2025; .github/skills/security/owasp-mcp/references/00-vulnerability-index.md#Vulnerability catalog); PA Privileged Access (domain-level) (Microsoft Cloud Security Benchmark v2; .github/skills/security/mcsb/references/00-control-index.md#Control domain catalog); SBD-05 Usable Security Controls (Secure by Design; .github/skills/security/secure-by-design/references/00-principle-index.md#Principle catalog)

Standards basis: MCP07 requires tool access to be paired with concrete authorization configuration rather than assumed from connector availability. The README says attach the Azure SRE tools available to the agent and only conditionally says Keep write tools disabled, leaving least-privilege connector setup out of the installation path.

Severity: Low -> Medium (likelihood Medium, impact Medium). The skill delegates enforcement to host least privilege and approval controls, so missing install guidance removes the stated control, not just documentation.

The README correctly says enforcement belongs to 'host controls', but the install steps the operator follows do not configure any. Step 5 attaches 'the Azure SRE tools available to the agent' (all of them), step 6 is conditional ('Keep write tools disabled for a read-only investigation'), which implies write tools are acceptable for other RCA runs, and nothing tells the operator to scope the connector identity to read-only roles or to keep secret-returning tools away from an auto-invoking skill. Abuse path: an operator attaches every available tool under the agent's default managed identity, which commonly holds Contributor or app-settings list rights. The skill's Azure SRE mode then auto-invokes GetAppSetting and RunAzCliReadCommands (findings 1-2) with no backstop, because the 'host controls' the skill relies on were never set up. A deployed copy also persists independently of repo fixes (README 42-43), which makes initial hardening more important.

Corroborated by Security-Agentic: Nit: README presents disabling write tools as optional and gives no least-privilege guidance for the agent identity.

Suggested fix:

5. Attach only the read-only tools this investigation needs. Run connectors under a
   least-privilege identity (for example, Azure `Reader`, Log Analytics Reader, and a Kusto viewer
   role) with no Key Vault data-plane access and no `listKeys`, connection-string, or app-settings
   list permissions.
6. Do not attach write-capable tools to an agent that runs this skill. Attach secret-returning
   tools such as `GetAppSetting` only when required, and only with per-call approval enabled.
7. Configure the agent's run mode so actions outside the read-only catalog require human
   approval, and verify that a denied write is reported as a blocker.

#44 Manual Azure SRE install has no version, digest, provenance, update, or withdrawal process (Medium)

Location: .github/skills/root-cause-analysis/README.md lines 31-43. Raised by: SSSC#1.

Standards: SLSA Build L2 Hosted build platform, signed provenance (Supply Chain Security; .github/skills/security/supply-chain-security/references/slsa-levels.md#SLSA v1.0 Build Track Levels); Sigstore (cosign) Standards Build provenance + SBOM attestation and artifact-signing maturity (Supply Chain Security; .github/skills/security/supply-chain-security/references/sigstore-maturity.md#Sigstore (cosign) Standards); SBD-04 Supply Chain Security (Secure by Design; .github/skills/security/secure-by-design/references/00-principle-index.md#Principle catalog)

Standards basis: SLSA Build L2 expects a hosted build platform and provenance so consumers can identify what artifact was produced. The manual Azure SRE install path copies skills by hand and gives no version, digest, provenance, update, or withdrawal procedure.

Severity: Low -> Medium (likelihood Medium, impact Medium). Uploaded copies cannot be versioned or recalled, so every Azure SRE agent that loaded the vulnerable version keeps autonomous mode after these issues are fixed.

The plugin and VSIX path inherits the repository's controlled release: an immutable release tag, a per-VSIX SPDX SBOM, and build-provenance attestation in extension-provenance-signer.yml. The new README adds a second channel that bypasses all of it: people paste SKILL.md and reference files by hand into Azure SRE Agent Builder > Skills. The instructions name no source ref (tag or commit SHA), no verified artifact, no per-file digest, and no record of which version was deployed. They also imply copying from a local working tree, which may hold uncommitted or unreviewed edits ('Local repository changes do not update a deployed Azure SRE agent'). These files steer autonomous tool use against production telemetry, so a tampered or stale copy changes agent behavior with nothing to detect it. Deployed copies also drift silently. The repo cannot revoke or update them, and the README describes no way to learn about a fix or withdrawal (release watch, advisory, removal step).

Suggested fix:

## Azure SRE installation

Install only from a released, verified source:

1. Download `hve-core-<version>.vsix` from the `v<version>` GitHub release and verify its provenance:
   `gh attestation verify hve-core-<version>.vsix --repo microsoft/hve-core`.
2. Extract the three skill directories from the verified VSIX, or check out the exact release tag commit.
   Do not upload from a working tree with uncommitted or unreviewed changes.
3. Record the source tag, commit SHA, and SHA-256 of each uploaded file in the skill description,
   for example `Source: microsoft/hve-core v<version> @ <sha>; SKILL.md sha256:<digest>`.
4. Open the Azure SRE agent and go to **Builder > Skills**.
5. Create skills named `root-cause-analysis`, `statistical-hypothesis-testing`, and
   `causal-evidence-likelihood`, and add each corresponding `SKILL.md`.
6. Include `root-cause-analysis/references/azure-sre-tools.md` with the RCA skill.
7. Attach the Azure SRE tools, verify connector credentials, and keep write tools disabled.
8. Start a new thread and verify that the host is detected as Azure SRE mode.

### Updates and withdrawal

Deployed copies do not update automatically. Watch microsoft/hve-core releases and security
advisories. When a release changes or withdraws these skills, re-upload from the newly verified
release or delete the skills from **Builder > Skills**. Periodically compare deployed file digests
with the recorded release digests to detect drift.

#59 Azure SRE installation does not require confirming a privacy impact assessment before production telemetry is attached (Medium)

Location: .github/skills/root-cause-analysis/README.md lines 31-40. Raised by: Privacy-2#3.

Standards: Art. 35 Data protection impact assessment (GDPR planning reference; .github/skills/project-planning/privacy-standards/references/dpia-thresholds.md#DPIA threshold heuristics); GV.PO-P Policies, processes, and governance structures (NIST Privacy Framework; .github/skills/project-planning/privacy-standards/references/nist-privacy-framework.md#Core functions and categories); MS-2.10 Privacy risk assessment (NIST AI RMF 1.0 Measure; .github/skills/rai/rai-standards/references/nist-ai-rmf-measure.md#Measure subcategories)

Standards basis: GDPR Article 35 DPIA screening is expected before high-risk processing such as large-scale telemetry, employee monitoring, or new AI technology. The Azure SRE install steps attach connectors for autonomous production reads without asking operators to confirm DPIA status.

Severity: Low -> Medium (likelihood Medium, impact Medium). Privacy screening found a DPIA likely required for production telemetry, but installation does not require one.

The installation steps attach every available tool and connector for autonomous LLM processing of production telemetry. The screening here comes out positive (end-user identifiers at scale, new technology, employee monitoring), but no step points to an assessment or data-owner approval. #80 is about missing documentation of data scope, residency and retention; this finding is about the missing approval step before enablement.

Extends pass-1 finding #80 (now #94): [Privacy] Azure SRE installation guidance omits connector data scope, residency, and thread or memory retention.

Suggested fix:

Insert before step 5: 'Confirm with the data owner or privacy team that a privacy impact assessment (a DPIA where GDPR applies) covers LLM-agent processing of each connected source, including end-user identifiers, employee activity data, thread and memory retention, and inference location. Attach only the connectors it covers.'

#75 Product name capitalization: Azure SRE agent vs Azure SRE Agent (Low)

Location: .github/skills/root-cause-analysis/README.md lines 33, 42. Raised by: Standards#16.

Standards: Miscellaneous Use proper capitalization for product names and project terms (Markdown Instructions; .github/instructions/hve-core/markdown.instructions.md#Miscellaneous); Language and Vocabulary Choose precise terms and consistent vocabulary (Writing Style Instructions; .github/instructions/hve-core/writing-style.instructions.md#Language and Vocabulary)

Standards basis: Markdown miscellaneous guidance requires correct capitalization for product and project terms. The README writes Azure SRE agent, while the skill and reference consistently use Azure SRE Agent.

The README writes 'the Azure SRE agent', but SKILL.md:34 and :322 and azure-sre-tools.md:13 use the product name 'Azure SRE Agent'.

Suggested fix:

1. Open Azure SRE Agent and go to **Builder > Skills**. (On line 42, write 'a deployed Azure SRE Agent instance'.)

#85 Azure SRE installation steps say 'verify' without stating how or what success looks like (Low)

Location: .github/skills/root-cause-analysis/README.md lines 38-40. Raised by: Accessibility#3.

Standards: 4.7 Use Clear Step-by-step Instructions (COGA; .github/skills/accessibility/accessibility/references/frameworks/coga.md#Control 4.7 Use Clear Step-by-step Instructions); 7.1 Clearly State the Results and Disadvantages of Actions, Options, and Selections (COGA; .github/skills/accessibility/accessibility/references/frameworks/coga.md#Control 7.1 Clearly State the Results and Disadvantages of Actions, Options, and Selections); Clarity Principles Write clear, direct instructions with enough context to act (Writing Style Instructions; .github/instructions/hve-core/writing-style.instructions.md#Clarity Principles)

Standards basis: COGA 4.7 calls for clear step-by-step instructions with enough detail for a user to know what to do next. The Azure SRE install steps say verify credentials and mode detection but do not state the check or success condition.

The installation list is numbered and has one action per step, which is good. However, steps 5 and 7 ask the reader to verify an outcome without saying how to check it or what a successful result looks like. Step 7 depends on a detection rule that only appears in SKILL.md and azure-sre-tools.md: explicit host identity, or a distinctive tool such as GetAnalysis, SearchIncidentKnowledge, or ExecuteClusterKustoQuery. Readers who rely on explicit instructions and low memory load cannot confirm success or recover from a wrong result without opening other files. Nit.

Suggested fix:

5. Attach the Azure SRE tools available to the agent and verify its connector credentials (see
   [Connectors and data access](https://learn.microsoft.com/azure/sre-agent/connectors)).
6. Keep write tools disabled for a read-only investigation.
7. Start a new thread and ask the agent which RCA host mode it detected. It should report Azure SRE
   mode. If it reports generic mode, confirm that a distinctive tool such as `GetAnalysis` or
   `ExecuteClusterKustoQuery` is attached (step 5).

#94 Azure SRE installation guidance omits connector data scope, residency, and thread or memory retention (Low)

Location: .github/skills/root-cause-analysis/README.md lines 38-40. Raised by: Privacy#7.

Standards: CM.AW-P Notice and transparency (NIST Privacy Framework; .github/skills/project-planning/privacy-standards/references/nist-privacy-framework.md#Core functions and categories); CT.DP-P Data processing and sharing controls (NIST Privacy Framework; .github/skills/project-planning/privacy-standards/references/nist-privacy-framework.md#Core functions and categories); predictability Predictability objective (NISTIR 8062 Privacy Engineering; .github/skills/project-planning/privacy-standards/references/nistir-8062.md#Objectives)

Standards basis: CM.AW-P requires notice and transparency about data processing and user interaction. The Azure SRE README tells operators to attach tools but does not disclose connector data scope, residency, thread storage, or memory retention.

Step 5 tells operators to attach tools and verify credentials. It does not mention that the attached connectors decide which personal data (logs, App Insights telemetry, GitHub/ADO content) the agent will read without per-step approval. It also does not say that investigation evidence and checkpoints will persist in Azure SRE threads and agent memory, under that service's region and retention settings. The README links the connectors doc (line 87) but gives no guidance on scoping connectors to the data an RCA needs, or on confirming that the deployment region and retention meet the organization's requirements for incident data.

Suggested fix:

5. Attach only the Azure SRE tools and connectors whose data the investigation needs, and verify
   their credentials. Connectors decide which logs, telemetry, and repository or work-item
   content, including personal data, the skill can read without per-step approval.
6. Keep write tools disabled for a read-only investigation.
7. Confirm that the agent's region and its thread and memory retention meet your organization's
   requirements for incident evidence. Checkpoints and evidence excerpts persist in agent threads.
8. Start a new thread and verify that the host is detected as Azure SRE mode.

#22 Package-level README has no precedent and is not distributed (Medium)

Location: .github/skills/root-cause-analysis/README.md lines 1-91. Raised by: Standards#3, Readiness#18.

Standards: Package and Documentation Contract Distributable artifacts live under package subdirectories and require docs/plugin synchronization (HVE Artifact Authoring; .github/skills/coding-standards/hve-artifact-authoring/SKILL.md#Package and Documentation Contract); Keep Context Focused Keep activation context focused and avoid duplicated canonical guidance (HVE Builder Instructions; .github/instructions/hve-core/hve-builder.instructions.md#Keep Context Focused); Readiness review Validation steps, test evidence, checklist completion, PR description, and changed documentation checks (Code Review Skill; .github/skills/coding-standards/code-review/references/lens-checklists.md#Readiness review)

Standards basis: The package and documentation contract keeps distributable artifacts under package-scoped component paths and uses docs/plugin synchronization for published surfaces. The new .github/skills/root-cause-analysis/README.md is package-level prose that is neither a component README precedent nor a distributed reference page.

No other package under .github/skills has a package-level README (checked accessibility, coding-standards, data-science-engineering, design-thinking, engagement-reporting, experimental, installer, rpi, plus orientation-checked rai, security, shared, project-planning). plugin.json ships only the three skill directories. As a result, the Azure SRE install steps, the 'keep write tools disabled' guidance, and the limits text never reach plugin or VSIX consumers, and the docs site does not link to them.

Corroborated by Readiness: Package README (Azure SRE install guidance) does not reach plugin/extension users and is not linked from generated docs.

Suggested fix:

Move the host-behavior, installation, and limits content into a skill-root reference such as root-cause-analysis/references/azure-sre-setup.md, or into a 'How to use it' tail section on docs/reference/skills/root-cause-analysis/root-cause-analysis.md (a generator-preserved region). Then delete the package-level README.

#77 README frontmatter lacks recommended ms.date and author (Low)

Location: .github/skills/root-cause-analysis/README.md lines 1-4. Raised by: Standards#18.

Standards: Recommended Fields Recommended frontmatter fields include ms.date and author where applicable (Markdown Instructions; .github/instructions/hve-core/markdown.instructions.md#Recommended Fields); YAML Frontmatter Use supported frontmatter fields and conventions (Markdown Instructions; .github/instructions/hve-core/markdown.instructions.md#YAML Frontmatter)

Standards basis: Markdown recommended-fields guidance calls out frontmatter such as ms.date and author where applicable. The new README frontmatter has only title and description, unlike sibling reference README files that carry those provenance fields.

The new Markdown file has only title and description. Sibling READMEs, such as evals/behavior-conformance/README.md, carry author and ms.date.

Suggested fix:

Add 'author: Microsoft' and 'ms.date: 2026-10-05'. If finding 3 is taken, these move with the content.

#76 README title says Generic but covers both host modes (Low)

Location: .github/skills/root-cause-analysis/README.md lines 2-3. Raised by: Standards#17.

Standards: Language and Vocabulary Choose precise terms and consistent vocabulary (Writing Style Instructions; .github/instructions/hve-core/writing-style.instructions.md#Language and Vocabulary); Readiness review Validation steps, test evidence, checklist completion, PR description, and changed documentation checks (Code Review Skill; .github/skills/coding-standards/code-review/references/lens-checklists.md#Readiness review)

Standards basis: The writing-style vocabulary rule requires precise terms that do not collide with project concepts. The README title says Generic RCA skill package, but Generic is only one host mode in a package the skill describes as hybrid.

'Generic' is one of the package's two host modes, so it is a misleading title for a package the skill calls 'hybrid'.

Suggested fix:

title: Root Cause Analysis Skills

#42 README has no intended-use or out-of-scope statement and says little about model limits (Medium)

Location: .github/skills/root-cause-analysis/README.md lines 58-72. Raised by: RAI#9.

Standards: MP-1.1 Intended purposes, laws, and settings documented (NIST AI RMF 1.0 Map; .github/skills/rai/rai-standards/references/nist-ai-rmf-map.md#Map subcategories); MP-2.2 Knowledge limits and human oversight requirements (NIST AI RMF 1.0 Map; .github/skills/rai/rai-standards/references/nist-ai-rmf-map.md#Map subcategories); GV-1.4 Transparent risk management policies established (NIST AI RMF 1.0 Govern; .github/skills/rai/rai-standards/references/nist-ai-rmf-govern.md#Govern subcategories)

Standards basis: MP-1.1 expects intended purpose, legal context, and operational setting to be documented. The README's Behavior and limits section covers scheduling and discoverability limits but omits appropriate-use boundaries, model limits, and the missing statistical runtime.

'Behavior and limits' covers workflow limits: no guarantee of a discoverable cause, no scheduling, no continuing after the host stops. It does not cover model limits or appropriate use. It never says that the package ships no statistical runtime, so p-values exist only when the host supplies one. It does not say that causal-likelihood bands are uncalibrated LLM judgments that can vary between runs. It does not warn that an LLM can produce a plausible but wrong causal account from telemetry it misreads. It does not say that results need human validation before adoption. It names no out-of-scope uses, such as sole basis for regulatory incident reports, safety determinations, legal conclusions, or evaluating an individual's performance. Readers deploying into Azure SRE, where copies persist on their own (lines 42-43), get no deployment-time guidance on these limits.

Suggested fix:

`Complete` means the causal completion gate passed as assessed by the agent. `Provisional`,
`Inconclusive`, `Blocked`, and `Paused` are not completed RCAs. The skill cannot guarantee that
available evidence contains a discoverable root cause, override host budgets, schedule itself, or
continue after the host stops.

## Intended use and limits

The package assists engineers in investigating operational, software, data, and process failures.
Its findings, including a `Complete` status, are AI-assessed and need validation by a qualified
human reviewer before they are adopted in a postmortem, incident record, or corrective-action plan.

* The package ships no statistical runtime. p-values exist only when the host executes the test;
  otherwise the test is recorded as `Blocked`.
* Causal-likelihood bands are uncalibrated, model-elicited judgments of evidence relevance. They
  can differ between runs or models and are not probabilities that a cause is real.
* The agent can misread telemetry, code, or lineage. Verify the cited evidence IDs and gate
  criteria rather than the narrative alone.

Do not use the output as the sole basis for regulatory incident reports, safety determinations,
legal conclusions, or any evaluation of an individual's performance or conduct. For those
contexts, require accountable domain review of the technical findings.

#29 Package README overstates eval coverage and mixes it into 'Deferred work'; the deferred item is untracked (Medium)

Location: .github/skills/root-cause-analysis/README.md lines 74-82. Raised by: Readiness#15, RAI#11, Functional#19, Standards#19.

Standards: MS-2.1 TEVV tools and documentation (NIST AI RMF 1.0 Measure; .github/skills/rai/rai-standards/references/nist-ai-rmf-measure.md#Measure subcategories); GV-4.2 Risk documentation and communication practices (NIST AI RMF 1.0 Govern; .github/skills/rai/rai-standards/references/nist-ai-rmf-govern.md#Govern subcategories); Tracking Contract Keep tracking references out of production documentation and artifacts (HVE Artifact Authoring; .github/skills/coding-standards/hve-artifact-authoring/SKILL.md#Tracking Contract)

Standards basis: MS-2.1 also requires evaluation evidence to be documented separately from future work or claims. The README's Deferred work section says behavior coverage now verifies key contracts while still admitting model-backed RCA evaluation remains pending.

Under '## Deferred work', after the single deferred item, the README says 'Behavior conformance coverage now verifies companion-skill disposition authority, the Complete gate, production-action approval, and resumable checkpoints...'. That paragraph has three problems. (a) It is placed under the wrong heading. (b) It uses a temporal marker ('now'). (c) It overstates the evidence: the stimuli are advisory, single-regex, and unexecuted (findings 9-10), so they do not verify behavior. The deferred item ('The following approved backlog remains outside this change') cites no issue, and the PR body and linked issue #3134 do not mention it, so the follow-up is unrecorded.

Corroborated by RAI: README says advisory regex stimuli 'verify' behavior; Functional: README overstates eval coverage as verification; Standards: PR-scoped narrative in durable README.

Suggested fix:

## Deferred work

* Expand statistical methods beyond binary rates and independent-sample means (tracked in #<issue>).

## Evaluation coverage

Advisory behavior-conformance stimuli exercise companion-skill disposition authority, the `Complete` gate, production-action approval, and checkpoints for non-complete stop states. A synthetic regression corpus covers incident, data, software, and process-failure cases. These stimuli are advisory and do not block builds.

access, approvals, and production boundaries through host controls.

## Azure SRE installation

1. Open the Azure SRE agent and go to **Builder > Skills**.
2. Create skills named `root-cause-analysis`, `statistical-hypothesis-testing`, and
`causal-evidence-likelihood`.
3. Add each corresponding `SKILL.md` as a skill procedure.
4. Include the `root-cause-analysis/references/azure-sre-tools.md` reference with the RCA skill.
5. Attach the Azure SRE tools available to the agent and verify its connector credentials.
6. Keep write tools disabled for a read-only investigation.
7. Start a new thread and verify that the host is detected as Azure SRE mode.

Local repository changes do not update a deployed Azure SRE agent. Upload each changed skill and
its references to refresh the deployment.

## Generic installation

Install the three skill directories in a host that supports on-demand skills. Expose only tools
appropriate for the investigation. The RCA skill will request approval before using non-Azure-SRE
tools to expand the evidence set.

Example invocation:

> Use the root-cause-analysis skill to investigate why the reported outcome occurred. Start with
> the hypotheses, evidence, and observations in this thread. Identify gaps, propose additional
> evidence collection when useful, and continue until the completion gate passes or a concrete
> evidence, access, approval, or tool blocker remains.

## Behavior and limits

The skill tests predictions before declaring a cause, actively seeks disconfirming evidence,
tracks source lineage, and preserves exact test parameters and observations. It invokes
statistical testing for eligible grouped observations and causal evidence likelihood assessment
for selected statistically significant pairs whose observed direction supports the hypothesis.
Statistical significance and elicited likelihood ratios inform evidence weighting but cannot
directly set a hypothesis disposition.

`Complete` means the causal completion gate passed. `Provisional`, `Inconclusive`, `Blocked`, and
`Paused` are not completed RCAs. The skill cannot guarantee that available evidence contains a
discoverable root cause, override host budgets, schedule itself, or continue after the host stops.

The procedure checkpoints after completed test cycles. Use approved durable storage when
available; a checkpoint emitted only in chat does not guarantee cross-thread recovery.

## Deferred work

The following approved backlog remains outside this change:

* Expand statistical methods beyond binary rates and independent-sample means.

Behavior conformance coverage now verifies companion-skill disposition authority, the `Complete`
gate, production-action approval, and resumable checkpoints for every non-complete stop state. A
synthetic regression corpus exercises incident, data, software, and process-failure cases.

## Microsoft documentation

* [Skills: creation, tool attachment, and lifecycle](https://learn.microsoft.com/azure/sre-agent/skills)
* [Connectors and data access](https://learn.microsoft.com/azure/sre-agent/connectors)

---

🤖 *Crafted with precision by ✨Copilot following brilliant human instruction, then carefully refined by our team of discerning human reviewers.*
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
---
name: causal-evidence-likelihood
description: 'Assess how one evidence item changes the plausibility of an RCA hypothesis using a reasoned likelihood ratio.'
compatibility: "Requires an Agent Skills host and the root-cause-analysis companion workflow."
---

# Causal Evidence Likelihood

## Goal

Assess whether one observed evidence item makes an RCA hypothesis more or less plausible through
logical and causal reasoning. Return a reasoned likelihood ratio for use by
`root-cause-analysis`, without presenting the estimate as measured probability or causal proof.

## Inputs

* One explicit, falsifiable hypothesis `H`, including its proposed mechanism
* One evidence item `E`, separated into direct observation and interpretation
* Evidence provenance, reliability, coverage, timing, and known transformations
* Plausible alternative explanations and relevant system context

Assess one material evidence-to-hypothesis link at a time. Do not combine multiple observations,
summaries, or duplicated telemetry into one item unless their dependence is explicit.
Do not provide or use p-values, confidence intervals, effect sizes, statistical-test conclusions,
correlation strengths, anomaly scores, sample counts, or likelihood estimates produced by another
skill. If supplied, set them aside before assessment. Use only the meaning of the direct observation
and general knowledge of how the system or class of systems behaves.

## Assessment

Ask:

> Suppose a reasonable observer initially assigns equal probability to `H` and `not H`. After
> learning only `E`, while using established logical and system knowledge, how should the
> observer's assessment of `H` change?

Evaluate this by comparing:

```text
LR(E) = P(E given H) / P(E given not H)
```

With prior odds of 1, the elicited posterior corresponding to the assessment is:

```text
q(E) = LR(E) / (1 + LR(E))
```

Reason in both directions before selecting a value:

1. State why `E` would be expected if `H` were true.
2. State why `E` could occur if `H` were false, including common causes, reverse causation,
measurement effects, and competing mechanisms.
3. Check whether the proposed cause precedes the effect and whether the mechanism could operate
in the observed scope.
4. Distinguish direct runtime, control, reproduction, or intervention evidence from source-code
intent, documentation, temporal proximity, and unexplained correlation.
5. Test logical compatibility explicitly. If `E` cannot be true when `H` is true under the stated
mechanism and scope, classify it as contradicting even when statistical observations favor
`H`.
6. Select the narrowest defensible likelihood-ratio band. Prefer `Neutral` when the connection is
unknown or equally expected under `H` and `not H`.

## Likelihood-Ratio Scale

| Assessment | `LR(E)` | Implied `q(E)` from a 50% prior |
|----------------------|-------------------------:|--------------------------------:|
| Strongly contradicts | at most 1/19 | at most 5% |
| Weakly contradicts | above 1/19 and below 0.8 | above 5% and below 44% |
| Neutral or unclear | 0.8 through 1.25 | 44% through 56% |
| Weakly supports | above 1.25 and below 19 | above 56% and below 95% |
| Strongly supports | at least 19 | at least 95% |

Return a band and, when the reasoning supports it, one representative `LR(E)` value. Use the Strong
bands only when general system reasoning makes `E` nearly entailed by `H` or nearly incompatible
with it after considering the strongest plausible alternative. If provenance or meaning is too
uncertain to reason from `E`, return `Not Assessed` rather than `Neutral`.

## Interpretation Constraints

* Label `LR(E)` and `q(E)` as elicited reasoning estimates, not empirical frequencies.
* Do not use `q(E)` as the RCA hypothesis's actual posterior probability. The 50% prior is a
standard reference point for evidence relevance.
* Do not infer support from evidence reliability alone. Reliability controls whether the
observation can be trusted; `LR(E)` controls whether it distinguishes `H` from `not H`.
* Do not inspect, cite, summarize, transform, or reproduce statistical inference from this or any
other skill. Statistical support has already been assessed elsewhere; using it here would count
the same support twice.
* Derive the direction and ratio only from logical compatibility, mechanism behavior, temporal
possibility, system constraints, and plausible alternatives. Association strength and causal
relevance are separate.
* Preserve logical contradiction. Reliable evidence that is incompatible with a necessary part of
`H` contradicts `H` even if numerous correlations or statistical tests favor it.
* Do not multiply likelihood ratios unless the RCA workflow has justified conditional
independence or modeled the dependence. Duplicates and downstream consequences of the same
event do not provide independent updates.
* A code comment or design document can support mechanism plausibility, but it does not prove that
the path executed during the incident.
* Missing expected evidence contradicts `H` only when source coverage is adequate and `H` predicts
that the evidence would be emitted, retained, and observed.

## RCA Record

Return:

* Hypothesis ID and exact statement
* Evidence ID, direct observation, provenance, and reliability
* `P(E given H)` rationale
* `P(E given not H)` rationale and strongest plausible alternative
* Direction: supports, contradicts, neutral, or `Not Assessed`
* Likelihood-ratio band and representative `LR(E)` value when defensible
* Implied `q(E)` from the standardized 50% prior, labeled as an elicited relevance estimate
* Reasoning confidence: Low, Medium, or High, with the decisive limitation
* Dependence notes identifying evidence that must not be multiplied with this result
* Next observation that would most change or calibrate the assessment

Return the assessment to `root-cause-analysis`. It informs evidence weighting but cannot directly
set a hypothesis disposition or satisfy the root-cause completion gate.
Loading
Loading