Repository navigation
feat(skills): add root cause analysis skill package #3136
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
eugeneboms
wants to merge
25
commits into
microsoft:main
Choose a base branch
from
eugeneboms:eugenebo-rca-skill
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 9 commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
c340e44
Azure SRE skills
fe1fd4d
Work in progress on RCA skills
ef8c70f
Work on the RCA skillset
14cdad5
Merge remote-tracking branch 'upstream/main' into eugenebo-rca-skill
6c23a13
RCA skills eval suites
e3701a8
Removed language locale from the url
a901541
Fixing formatting issues
e523a57
Spelling fixes
6d61375
Merge branch 'main' into eugenebo-rca-skill
eugeneboms f8c4258
Addressed "High" code review feedback
b1aa016
Merge remote-tracking branch 'upstream/main' into eugenebo-rca-skill
ec19563
Spell check update
da40bf5
Merge remote-tracking branch 'upstream/main' into eugenebo-rca-skill
5a89e7b
style(skills): format RCA markdown tables
b69b781
feat(skills): add root cause analysis skill package
d1adb8d
Merge branch 'main' into eugenebo-rca-skill
eugeneboms a355311
spell check issue fixed
ab68e5f
Merge branch 'eugenebo-rca-skill' of https://github.com/eugeneboms/hv…
e747869
removed representative LR(E)
96b79f7
Merge branch 'main' into eugenebo-rca-skill
eugeneboms 7495f7b
Address CR-02,04,11,12,14,15 (all Medium)
9aae223
Merge branch 'eugenebo-rca-skill' of https://github.com/eugeneboms/hv…
6384504
Merge remote-tracking branch 'upstream/main' into eugenebo-rca-skill
6580849
spellcheck and doc links fixes
18b218c
Merge branch 'main' into eugenebo-rca-skill
eugeneboms File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| --- | ||
| title: Generic Root Cause Analysis Skills | ||
| description: Configure the hybrid RCA skill package and its companion assessment skills | ||
| --- | ||
|
|
||
| This package contains three portable skills: | ||
|
|
||
| * `root-cause-analysis/SKILL.md` owns the hybrid investigation and disposition workflow. | ||
| * `statistical-hypothesis-testing/SKILL.md` compares eligible binary rates or numeric averages. | ||
| * `causal-evidence-likelihood/SKILL.md` estimates how one evidence item changes hypothesis | ||
| plausibility through a reasoned likelihood ratio. | ||
|
|
||
| The package does not define a custom agent. Install all three skills so the RCA workflow can invoke | ||
| its companion assessments. An investigation can begin with empty or populated sets of evidence, | ||
| hypotheses, and observations. | ||
|
|
||
| ## Host behavior | ||
|
|
||
| The RCA skill selects one of two host modes: | ||
|
|
||
| * In Azure SRE mode, it loads `root-cause-analysis/references/azure-sre-tools.md` and automatically | ||
| uses relevant, safe, read-only tools from that catalog. Azure terminology and source guidance | ||
| become authoritative for the connected Azure evidence. | ||
| * In generic mode, it inventories other available tools, proposes a bounded evidence-expansion | ||
| action, and obtains user approval before invoking those tools. It can still assess supplied | ||
| evidence when no additional tools are approved or available. | ||
|
|
||
| Tool availability does not grant data access or permission to alter a system. Enforce read-only | ||
| access, approvals, and production boundaries through host controls. | ||
|
|
||
| ## Azure SRE installation | ||
|
|
||
| 1. Open the Azure SRE agent and go to **Builder > Skills**. | ||
| 2. Create skills named `root-cause-analysis`, `statistical-hypothesis-testing`, and | ||
| `causal-evidence-likelihood`. | ||
| 3. Add each corresponding `SKILL.md` as a skill procedure. | ||
| 4. Include the `root-cause-analysis/references/azure-sre-tools.md` reference with the RCA skill. | ||
| 5. Attach the Azure SRE tools available to the agent and verify its connector credentials. | ||
| 6. Keep write tools disabled for a read-only investigation. | ||
| 7. Start a new thread and verify that the host is detected as Azure SRE mode. | ||
|
|
||
| Local repository changes do not update a deployed Azure SRE agent. Upload each changed skill and | ||
| its references to refresh the deployment. | ||
|
|
||
| ## Generic installation | ||
|
|
||
| Install the three skill directories in a host that supports on-demand skills. Expose only tools | ||
| appropriate for the investigation. The RCA skill will request approval before using non-Azure-SRE | ||
| tools to expand the evidence set. | ||
|
|
||
| Example invocation: | ||
|
|
||
| > Use the root-cause-analysis skill to investigate why the reported outcome occurred. Start with | ||
| > the hypotheses, evidence, and observations in this thread. Identify gaps, propose additional | ||
| > evidence collection when useful, and continue until the completion gate passes or a concrete | ||
| > evidence, access, approval, or tool blocker remains. | ||
|
|
||
| ## Behavior and limits | ||
|
|
||
| The skill tests predictions before declaring a cause, actively seeks disconfirming evidence, | ||
| tracks source lineage, and preserves exact test parameters and observations. It invokes | ||
| statistical testing for eligible grouped observations and causal evidence likelihood assessment | ||
| for selected statistically significant pairs whose observed direction supports the hypothesis. | ||
| Statistical significance and elicited likelihood ratios inform evidence weighting but cannot | ||
| directly set a hypothesis disposition. | ||
|
|
||
| `Complete` means the causal completion gate passed. `Provisional`, `Inconclusive`, `Blocked`, and | ||
| `Paused` are not completed RCAs. The skill cannot guarantee that available evidence contains a | ||
| discoverable root cause, override host budgets, schedule itself, or continue after the host stops. | ||
|
|
||
| The procedure checkpoints after completed test cycles. Use approved durable storage when | ||
| available; a checkpoint emitted only in chat does not guarantee cross-thread recovery. | ||
|
|
||
| ## Deferred work | ||
|
|
||
| The following approved backlog remains outside this change: | ||
|
|
||
| * Expand statistical methods beyond binary rates and independent-sample means. | ||
|
|
||
| Behavior conformance coverage now verifies companion-skill disposition authority, the `Complete` | ||
| gate, production-action approval, and resumable checkpoints for every non-complete stop state. A | ||
| synthetic regression corpus exercises incident, data, software, and process-failure cases. | ||
|
|
||
| ## Microsoft documentation | ||
|
|
||
| * [Skills: creation, tool attachment, and lifecycle](https://learn.microsoft.com/azure/sre-agent/skills) | ||
| * [Connectors and data access](https://learn.microsoft.com/azure/sre-agent/connectors) | ||
|
|
||
| --- | ||
|
|
||
| 🤖 *Crafted with precision by ✨Copilot following brilliant human instruction, then carefully refined by our team of discerning human reviewers.* | ||
118 changes: 118 additions & 0 deletions
118
.github/skills/root-cause-analysis/causal-evidence-likelihood/SKILL.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,118 @@ | ||
| --- | ||
| name: causal-evidence-likelihood | ||
| description: 'Assess how one evidence item changes the plausibility of an RCA hypothesis using a reasoned likelihood ratio.' | ||
| compatibility: "Requires an Agent Skills host and the root-cause-analysis companion workflow." | ||
| --- | ||
|
|
||
| # Causal Evidence Likelihood | ||
|
|
||
| ## Goal | ||
|
|
||
| Assess whether one observed evidence item makes an RCA hypothesis more or less plausible through | ||
| logical and causal reasoning. Return a reasoned likelihood ratio for use by | ||
| `root-cause-analysis`, without presenting the estimate as measured probability or causal proof. | ||
|
|
||
| ## Inputs | ||
|
|
||
| * One explicit, falsifiable hypothesis `H`, including its proposed mechanism | ||
| * One evidence item `E`, separated into direct observation and interpretation | ||
| * Evidence provenance, reliability, coverage, timing, and known transformations | ||
| * Plausible alternative explanations and relevant system context | ||
|
|
||
| Assess one material evidence-to-hypothesis link at a time. Do not combine multiple observations, | ||
| summaries, or duplicated telemetry into one item unless their dependence is explicit. | ||
| Do not provide or use p-values, confidence intervals, effect sizes, statistical-test conclusions, | ||
| correlation strengths, anomaly scores, sample counts, or likelihood estimates produced by another | ||
| skill. If supplied, set them aside before assessment. Use only the meaning of the direct observation | ||
| and general knowledge of how the system or class of systems behaves. | ||
|
|
||
| ## Assessment | ||
|
|
||
| Ask: | ||
|
|
||
| > Suppose a reasonable observer initially assigns equal probability to `H` and `not H`. After | ||
| > learning only `E`, while using established logical and system knowledge, how should the | ||
| > observer's assessment of `H` change? | ||
|
|
||
| Evaluate this by comparing: | ||
|
|
||
| ```text | ||
| LR(E) = P(E given H) / P(E given not H) | ||
| ``` | ||
|
|
||
| With prior odds of 1, the elicited posterior corresponding to the assessment is: | ||
|
|
||
| ```text | ||
| q(E) = LR(E) / (1 + LR(E)) | ||
| ``` | ||
|
|
||
| Reason in both directions before selecting a value: | ||
|
|
||
| 1. State why `E` would be expected if `H` were true. | ||
| 2. State why `E` could occur if `H` were false, including common causes, reverse causation, | ||
| measurement effects, and competing mechanisms. | ||
| 3. Check whether the proposed cause precedes the effect and whether the mechanism could operate | ||
| in the observed scope. | ||
| 4. Distinguish direct runtime, control, reproduction, or intervention evidence from source-code | ||
| intent, documentation, temporal proximity, and unexplained correlation. | ||
| 5. Test logical compatibility explicitly. If `E` cannot be true when `H` is true under the stated | ||
| mechanism and scope, classify it as contradicting even when statistical observations favor | ||
| `H`. | ||
| 6. Select the narrowest defensible likelihood-ratio band. Prefer `Neutral` when the connection is | ||
| unknown or equally expected under `H` and `not H`. | ||
|
|
||
| ## Likelihood-Ratio Scale | ||
|
|
||
| | Assessment | `LR(E)` | Implied `q(E)` from a 50% prior | | ||
| |----------------------|-------------------------:|--------------------------------:| | ||
| | Strongly contradicts | at most 1/19 | at most 5% | | ||
| | Weakly contradicts | above 1/19 and below 0.8 | above 5% and below 44% | | ||
| | Neutral or unclear | 0.8 through 1.25 | 44% through 56% | | ||
| | Weakly supports | above 1.25 and below 19 | above 56% and below 95% | | ||
| | Strongly supports | at least 19 | at least 95% | | ||
|
|
||
| Return a band and, when the reasoning supports it, one representative `LR(E)` value. Use the Strong | ||
| bands only when general system reasoning makes `E` nearly entailed by `H` or nearly incompatible | ||
| with it after considering the strongest plausible alternative. If provenance or meaning is too | ||
| uncertain to reason from `E`, return `Not Assessed` rather than `Neutral`. | ||
|
|
||
| ## Interpretation Constraints | ||
|
|
||
| * Label `LR(E)` and `q(E)` as elicited reasoning estimates, not empirical frequencies. | ||
| * Do not use `q(E)` as the RCA hypothesis's actual posterior probability. The 50% prior is a | ||
| standard reference point for evidence relevance. | ||
| * Do not infer support from evidence reliability alone. Reliability controls whether the | ||
| observation can be trusted; `LR(E)` controls whether it distinguishes `H` from `not H`. | ||
| * Do not inspect, cite, summarize, transform, or reproduce statistical inference from this or any | ||
| other skill. Statistical support has already been assessed elsewhere; using it here would count | ||
| the same support twice. | ||
| * Derive the direction and ratio only from logical compatibility, mechanism behavior, temporal | ||
| possibility, system constraints, and plausible alternatives. Association strength and causal | ||
| relevance are separate. | ||
| * Preserve logical contradiction. Reliable evidence that is incompatible with a necessary part of | ||
| `H` contradicts `H` even if numerous correlations or statistical tests favor it. | ||
| * Do not multiply likelihood ratios unless the RCA workflow has justified conditional | ||
| independence or modeled the dependence. Duplicates and downstream consequences of the same | ||
| event do not provide independent updates. | ||
| * A code comment or design document can support mechanism plausibility, but it does not prove that | ||
| the path executed during the incident. | ||
| * Missing expected evidence contradicts `H` only when source coverage is adequate and `H` predicts | ||
| that the evidence would be emitted, retained, and observed. | ||
|
|
||
| ## RCA Record | ||
|
|
||
| Return: | ||
|
|
||
| * Hypothesis ID and exact statement | ||
| * Evidence ID, direct observation, provenance, and reliability | ||
| * `P(E given H)` rationale | ||
| * `P(E given not H)` rationale and strongest plausible alternative | ||
| * Direction: supports, contradicts, neutral, or `Not Assessed` | ||
| * Likelihood-ratio band and representative `LR(E)` value when defensible | ||
| * Implied `q(E)` from the standardized 50% prior, labeled as an elicited relevance estimate | ||
| * Reasoning confidence: Low, Medium, or High, with the decisive limitation | ||
| * Dependence notes identifying evidence that must not be multiplied with this result | ||
| * Next observation that would most change or calibrate the assessment | ||
|
|
||
| Return the assessment to `root-cause-analysis`. It informs evidence weighting but cannot directly | ||
| set a hypothesis disposition or satisfy the root-cause completion gate. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CR-12: Package README: installation, scope, placement (Medium; findings #27, #44, #59, #75, #85, #94, #22, #77, #76, #42, #29)
Standards: MCP07:2025, Supply Chain Security § SLSA Build L2, GDPR planning reference § Art. 35, Markdown § Miscellaneous, COGA § 4.7, CM.AW-P, HVE Artifact Authoring § Package and Documentation Contract, Markdown § Recommended Fields, Writing Style § Language and Vocabulary, MP-1.1, MS-2.1
Requested changes:
Already resolved at 6d61375: #21 (Resolved in e3701a8:
/en-usremoved from both Learn URLs.)Original findings and justification:
#27 Azure SRE install guidance is conditional and omits least-privilege connector and approval-mode hardening (Medium)
Location:
.github/skills/root-cause-analysis/README.mdlines 28-29, 37-39. Raised by: Security#8, Security-Agentic#13.Standards: MCP07:2025 Insufficient Authentication and Authorization (OWASP MCP Top 10 2025;
.github/skills/security/owasp-mcp/references/00-vulnerability-index.md#Vulnerability catalog); PA Privileged Access (domain-level) (Microsoft Cloud Security Benchmark v2;.github/skills/security/mcsb/references/00-control-index.md#Control domain catalog); SBD-05 Usable Security Controls (Secure by Design;.github/skills/security/secure-by-design/references/00-principle-index.md#Principle catalog)Standards basis: MCP07 requires tool access to be paired with concrete authorization configuration rather than assumed from connector availability. The README says attach
the Azure SRE tools available to the agentand only conditionally saysKeep write tools disabled, leaving least-privilege connector setup out of the installation path.Severity: Low -> Medium (likelihood Medium, impact Medium). The skill delegates enforcement to host least privilege and approval controls, so missing install guidance removes the stated control, not just documentation.
The README correctly says enforcement belongs to 'host controls', but the install steps the operator follows do not configure any. Step 5 attaches 'the Azure SRE tools available to the agent' (all of them), step 6 is conditional ('Keep write tools disabled for a read-only investigation'), which implies write tools are acceptable for other RCA runs, and nothing tells the operator to scope the connector identity to read-only roles or to keep secret-returning tools away from an auto-invoking skill. Abuse path: an operator attaches every available tool under the agent's default managed identity, which commonly holds Contributor or app-settings list rights. The skill's Azure SRE mode then auto-invokes
GetAppSettingandRunAzCliReadCommands(findings 1-2) with no backstop, because the 'host controls' the skill relies on were never set up. A deployed copy also persists independently of repo fixes (README 42-43), which makes initial hardening more important.Corroborated by Security-Agentic: Nit: README presents disabling write tools as optional and gives no least-privilege guidance for the agent identity.
Suggested fix:
#44 Manual Azure SRE install has no version, digest, provenance, update, or withdrawal process (Medium)
Location:
.github/skills/root-cause-analysis/README.mdlines 31-43. Raised by: SSSC#1.Standards: SLSA Build L2 Hosted build platform, signed provenance (Supply Chain Security;
.github/skills/security/supply-chain-security/references/slsa-levels.md#SLSA v1.0 Build Track Levels); Sigstore (cosign) Standards Build provenance + SBOM attestation and artifact-signing maturity (Supply Chain Security;.github/skills/security/supply-chain-security/references/sigstore-maturity.md#Sigstore (cosign) Standards); SBD-04 Supply Chain Security (Secure by Design;.github/skills/security/secure-by-design/references/00-principle-index.md#Principle catalog)Standards basis: SLSA Build L2 expects a hosted build platform and provenance so consumers can identify what artifact was produced. The manual Azure SRE install path copies skills by hand and gives no version, digest, provenance, update, or withdrawal procedure.
Severity: Low -> Medium (likelihood Medium, impact Medium). Uploaded copies cannot be versioned or recalled, so every Azure SRE agent that loaded the vulnerable version keeps autonomous mode after these issues are fixed.
The plugin and VSIX path inherits the repository's controlled release: an immutable release tag, a per-VSIX SPDX SBOM, and build-provenance attestation in extension-provenance-signer.yml. The new README adds a second channel that bypasses all of it: people paste SKILL.md and reference files by hand into Azure SRE Agent Builder > Skills. The instructions name no source ref (tag or commit SHA), no verified artifact, no per-file digest, and no record of which version was deployed. They also imply copying from a local working tree, which may hold uncommitted or unreviewed edits ('Local repository changes do not update a deployed Azure SRE agent'). These files steer autonomous tool use against production telemetry, so a tampered or stale copy changes agent behavior with nothing to detect it. Deployed copies also drift silently. The repo cannot revoke or update them, and the README describes no way to learn about a fix or withdrawal (release watch, advisory, removal step).
Suggested fix:
#59 Azure SRE installation does not require confirming a privacy impact assessment before production telemetry is attached (Medium)
Location:
.github/skills/root-cause-analysis/README.mdlines 31-40. Raised by: Privacy-2#3.Standards: Art. 35 Data protection impact assessment (GDPR planning reference;
.github/skills/project-planning/privacy-standards/references/dpia-thresholds.md#DPIA threshold heuristics); GV.PO-P Policies, processes, and governance structures (NIST Privacy Framework;.github/skills/project-planning/privacy-standards/references/nist-privacy-framework.md#Core functions and categories); MS-2.10 Privacy risk assessment (NIST AI RMF 1.0 Measure;.github/skills/rai/rai-standards/references/nist-ai-rmf-measure.md#Measure subcategories)Standards basis: GDPR Article 35 DPIA screening is expected before high-risk processing such as large-scale telemetry, employee monitoring, or new AI technology. The Azure SRE install steps attach connectors for autonomous production reads without asking operators to confirm DPIA status.
Severity: Low -> Medium (likelihood Medium, impact Medium). Privacy screening found a DPIA likely required for production telemetry, but installation does not require one.
The installation steps attach every available tool and connector for autonomous LLM processing of production telemetry. The screening here comes out positive (end-user identifiers at scale, new technology, employee monitoring), but no step points to an assessment or data-owner approval. #80 is about missing documentation of data scope, residency and retention; this finding is about the missing approval step before enablement.
Extends pass-1 finding #80 (now #94): [Privacy] Azure SRE installation guidance omits connector data scope, residency, and thread or memory retention.
Suggested fix:
#75 Product name capitalization: Azure SRE agent vs Azure SRE Agent (Low)
Location:
.github/skills/root-cause-analysis/README.mdlines 33, 42. Raised by: Standards#16.Standards: Miscellaneous Use proper capitalization for product names and project terms (Markdown Instructions;
.github/instructions/hve-core/markdown.instructions.md#Miscellaneous); Language and Vocabulary Choose precise terms and consistent vocabulary (Writing Style Instructions;.github/instructions/hve-core/writing-style.instructions.md#Language and Vocabulary)Standards basis: Markdown miscellaneous guidance requires correct capitalization for product and project terms. The README writes
Azure SRE agent, while the skill and reference consistently useAzure SRE Agent.The README writes 'the Azure SRE agent', but SKILL.md:34 and :322 and azure-sre-tools.md:13 use the product name 'Azure SRE Agent'.
Suggested fix:
#85 Azure SRE installation steps say 'verify' without stating how or what success looks like (Low)
Location:
.github/skills/root-cause-analysis/README.mdlines 38-40. Raised by: Accessibility#3.Standards: 4.7 Use Clear Step-by-step Instructions (COGA;
.github/skills/accessibility/accessibility/references/frameworks/coga.md#Control 4.7 Use Clear Step-by-step Instructions); 7.1 Clearly State the Results and Disadvantages of Actions, Options, and Selections (COGA;.github/skills/accessibility/accessibility/references/frameworks/coga.md#Control 7.1 Clearly State the Results and Disadvantages of Actions, Options, and Selections); Clarity Principles Write clear, direct instructions with enough context to act (Writing Style Instructions;.github/instructions/hve-core/writing-style.instructions.md#Clarity Principles)Standards basis: COGA 4.7 calls for clear step-by-step instructions with enough detail for a user to know what to do next. The Azure SRE install steps say
verifycredentials and mode detection but do not state the check or success condition.The installation list is numbered and has one action per step, which is good. However, steps 5 and 7 ask the reader to verify an outcome without saying how to check it or what a successful result looks like. Step 7 depends on a detection rule that only appears in SKILL.md and azure-sre-tools.md: explicit host identity, or a distinctive tool such as
GetAnalysis,SearchIncidentKnowledge, orExecuteClusterKustoQuery. Readers who rely on explicit instructions and low memory load cannot confirm success or recover from a wrong result without opening other files. Nit.Suggested fix:
#94 Azure SRE installation guidance omits connector data scope, residency, and thread or memory retention (Low)
Location:
.github/skills/root-cause-analysis/README.mdlines 38-40. Raised by: Privacy#7.Standards: CM.AW-P Notice and transparency (NIST Privacy Framework;
.github/skills/project-planning/privacy-standards/references/nist-privacy-framework.md#Core functions and categories); CT.DP-P Data processing and sharing controls (NIST Privacy Framework;.github/skills/project-planning/privacy-standards/references/nist-privacy-framework.md#Core functions and categories); predictability Predictability objective (NISTIR 8062 Privacy Engineering;.github/skills/project-planning/privacy-standards/references/nistir-8062.md#Objectives)Standards basis: CM.AW-P requires notice and transparency about data processing and user interaction. The Azure SRE README tells operators to attach tools but does not disclose connector data scope, residency, thread storage, or memory retention.
Step 5 tells operators to attach tools and verify credentials. It does not mention that the attached connectors decide which personal data (logs, App Insights telemetry, GitHub/ADO content) the agent will read without per-step approval. It also does not say that investigation evidence and checkpoints will persist in Azure SRE threads and agent memory, under that service's region and retention settings. The README links the connectors doc (line 87) but gives no guidance on scoping connectors to the data an RCA needs, or on confirming that the deployment region and retention meet the organization's requirements for incident data.
Suggested fix:
#22 Package-level README has no precedent and is not distributed (Medium)
Location:
.github/skills/root-cause-analysis/README.mdlines 1-91. Raised by: Standards#3, Readiness#18.Standards: Package and Documentation Contract Distributable artifacts live under package subdirectories and require docs/plugin synchronization (HVE Artifact Authoring;
.github/skills/coding-standards/hve-artifact-authoring/SKILL.md#Package and Documentation Contract); Keep Context Focused Keep activation context focused and avoid duplicated canonical guidance (HVE Builder Instructions;.github/instructions/hve-core/hve-builder.instructions.md#Keep Context Focused); Readiness review Validation steps, test evidence, checklist completion, PR description, and changed documentation checks (Code Review Skill;.github/skills/coding-standards/code-review/references/lens-checklists.md#Readiness review)Standards basis: The package and documentation contract keeps distributable artifacts under package-scoped component paths and uses docs/plugin synchronization for published surfaces. The new
.github/skills/root-cause-analysis/README.mdis package-level prose that is neither a component README precedent nor a distributed reference page.No other package under .github/skills has a package-level README (checked accessibility, coding-standards, data-science-engineering, design-thinking, engagement-reporting, experimental, installer, rpi, plus orientation-checked rai, security, shared, project-planning). plugin.json ships only the three skill directories. As a result, the Azure SRE install steps, the 'keep write tools disabled' guidance, and the limits text never reach plugin or VSIX consumers, and the docs site does not link to them.
Corroborated by Readiness: Package README (Azure SRE install guidance) does not reach plugin/extension users and is not linked from generated docs.
Suggested fix:
#77 README frontmatter lacks recommended ms.date and author (Low)
Location:
.github/skills/root-cause-analysis/README.mdlines 1-4. Raised by: Standards#18.Standards: Recommended Fields Recommended frontmatter fields include ms.date and author where applicable (Markdown Instructions;
.github/instructions/hve-core/markdown.instructions.md#Recommended Fields); YAML Frontmatter Use supported frontmatter fields and conventions (Markdown Instructions;.github/instructions/hve-core/markdown.instructions.md#YAML Frontmatter)Standards basis: Markdown recommended-fields guidance calls out frontmatter such as ms.date and author where applicable. The new README frontmatter has only
titleanddescription, unlike sibling reference README files that carry those provenance fields.The new Markdown file has only title and description. Sibling READMEs, such as evals/behavior-conformance/README.md, carry author and ms.date.
Suggested fix:
#76 README title says Generic but covers both host modes (Low)
Location:
.github/skills/root-cause-analysis/README.mdlines 2-3. Raised by: Standards#17.Standards: Language and Vocabulary Choose precise terms and consistent vocabulary (Writing Style Instructions;
.github/instructions/hve-core/writing-style.instructions.md#Language and Vocabulary); Readiness review Validation steps, test evidence, checklist completion, PR description, and changed documentation checks (Code Review Skill;.github/skills/coding-standards/code-review/references/lens-checklists.md#Readiness review)Standards basis: The writing-style vocabulary rule requires precise terms that do not collide with project concepts. The README title says
Generic RCA skill package, but Generic is only one host mode in a package the skill describes as hybrid.'Generic' is one of the package's two host modes, so it is a misleading title for a package the skill calls 'hybrid'.
Suggested fix:
#42 README has no intended-use or out-of-scope statement and says little about model limits (Medium)
Location:
.github/skills/root-cause-analysis/README.mdlines 58-72. Raised by: RAI#9.Standards: MP-1.1 Intended purposes, laws, and settings documented (NIST AI RMF 1.0 Map;
.github/skills/rai/rai-standards/references/nist-ai-rmf-map.md#Map subcategories); MP-2.2 Knowledge limits and human oversight requirements (NIST AI RMF 1.0 Map;.github/skills/rai/rai-standards/references/nist-ai-rmf-map.md#Map subcategories); GV-1.4 Transparent risk management policies established (NIST AI RMF 1.0 Govern;.github/skills/rai/rai-standards/references/nist-ai-rmf-govern.md#Govern subcategories)Standards basis: MP-1.1 expects intended purpose, legal context, and operational setting to be documented. The README's
Behavior and limitssection covers scheduling and discoverability limits but omits appropriate-use boundaries, model limits, and the missing statistical runtime.'Behavior and limits' covers workflow limits: no guarantee of a discoverable cause, no scheduling, no continuing after the host stops. It does not cover model limits or appropriate use. It never says that the package ships no statistical runtime, so p-values exist only when the host supplies one. It does not say that causal-likelihood bands are uncalibrated LLM judgments that can vary between runs. It does not warn that an LLM can produce a plausible but wrong causal account from telemetry it misreads. It does not say that results need human validation before adoption. It names no out-of-scope uses, such as sole basis for regulatory incident reports, safety determinations, legal conclusions, or evaluating an individual's performance. Readers deploying into Azure SRE, where copies persist on their own (lines 42-43), get no deployment-time guidance on these limits.
Suggested fix:
#29 Package README overstates eval coverage and mixes it into 'Deferred work'; the deferred item is untracked (Medium)
Location:
.github/skills/root-cause-analysis/README.mdlines 74-82. Raised by: Readiness#15, RAI#11, Functional#19, Standards#19.Standards: MS-2.1 TEVV tools and documentation (NIST AI RMF 1.0 Measure;
.github/skills/rai/rai-standards/references/nist-ai-rmf-measure.md#Measure subcategories); GV-4.2 Risk documentation and communication practices (NIST AI RMF 1.0 Govern;.github/skills/rai/rai-standards/references/nist-ai-rmf-govern.md#Govern subcategories); Tracking Contract Keep tracking references out of production documentation and artifacts (HVE Artifact Authoring;.github/skills/coding-standards/hve-artifact-authoring/SKILL.md#Tracking Contract)Standards basis: MS-2.1 also requires evaluation evidence to be documented separately from future work or claims. The README's
Deferred worksection says behavior coveragenow verifieskey contracts while still admitting model-backed RCA evaluation remains pending.Under '## Deferred work', after the single deferred item, the README says 'Behavior conformance coverage now verifies companion-skill disposition authority, the
Completegate, production-action approval, and resumable checkpoints...'. That paragraph has three problems. (a) It is placed under the wrong heading. (b) It uses a temporal marker ('now'). (c) It overstates the evidence: the stimuli are advisory, single-regex, and unexecuted (findings 9-10), so they do not verify behavior. The deferred item ('The following approved backlog remains outside this change') cites no issue, and the PR body and linked issue #3134 do not mention it, so the follow-up is unrecorded.Corroborated by RAI: README says advisory regex stimuli 'verify' behavior; Functional: README overstates eval coverage as verification; Standards: PR-scoped narrative in durable README.
Suggested fix: