How should trace evidence survive into reviewable agent change proposals?

[vittorfp]

I am looking for feedback from people working with OpenTelemetry-style LLM/agent traces.

I am testing a small open-source artifact called change_proposal: a plain JSON review artifact downstream of traces/evals. It records the observed behavior, evidence references, proposed behavior change, validation criteria, risk, and rollback.

Repo: https://github.com/vittorfp/agent-change-proposals
Example: https://github.com/vittorfp/agent-change-proposals/blob/main/examples/declared-intent/change_proposal.example.json
Discussion: vittorfp/agent-change-proposals#6

This is not intended to be a tracing standard or observability backend. The question is what information from traces/evals needs to survive into the review artifact so a human can approve or reject a proposed agent behavior change.

I would especially value feedback on:

1. Which trace identifiers or evidence references should be immutable in a proposal?
2. Would baseline/candidate artifact IDs, evaluator and dataset versions, metric thresholds, sample counts, and rollback triggers be enough for CI gating?
3. Should ACP stay plain JSON first, or align early with OpenTelemetry/OpenInference concepts?
4. Where would this artifact fit badly in real observability workflows?

[IlyaKhalafi]

I'd keep the artifact as JSON, with OpenTelemetry identifiers as evidence references. That lets a review system consume it without becoming another tracing backend.

For immutable evidence, keep the trace ID and span ID together, the source project/environment, and an immutable export version or digest of the evidence itself. A proposal digest does not freeze externally referenced traces. A dashboard URL helps navigation, but retention, later updates and redaction can change what a reviewer sees. Prefer references to copying sensitive trace contents. Record the instrumentation and semantic-convention versions too. The OpenTelemetry SpanContext specification provides the identity contract.

The first gap I'd test is in your retrieval example. It correctly calls the diagnosis a hypothesis. I'd also require evidence about trace coverage before interpreting an absent retrieval span. Export loss, sampling, an uninstrumented retriever or retrieval in another process can all produce that observation. With incomplete coverage, report that the diagnosis is unknown rather than treating absence as evidence that retrieval did not happen.

For CI, pin the dataset revision, case IDs, evaluator/model and prompt versions, scorer configuration, and baseline/candidate code/config artifacts. For RAG, include the corpus/index snapshot and retrieval settings. Record evaluated, failed and missing cases separately, and compare the same cases in both arms. For stochastic evaluations, include repeated paired measurements and uncertainty around the difference, with a minimum sample requirement. Your structured checks and CI-gate RFC are the right place to make missing required evidence block a gate. Validating the proposal JSON alone should not imply that its acceptance criteria passed.

This fits PR or evaluation review before a prompt, retriever or tool policy changes. It fits poorly as a mandatory human review artifact for every high-volume production incident. Replay also needs recorded or stubbed side-effectful tools, otherwise validation could repeat real writes.

A concrete first validation would use a deterministic fixture or recorded model/tool outputs to hold agent behavior constant, first with retrieval uninstrumented, then with instrumentation restored. The first trace should leave the retrieval diagnosis unknown. The second should provide evidence that can confirm or refute the routing hypothesis. Trace structure alone should not authorize a behavior change.