Repository navigation
chore(bench): publish proxy and memory HNSW and ranking-quality and write-pipeline and extraction-quality benchmark data - #1118
ibex-harness-benchmark[bot] wants to merge 142 commits into
Conversation
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
|
Important Review skippedBot user detected. Ask an organization administrator to verify that this bot has an active assigned CodeRabbit review seat. See the seat-assignment guide. After the seat is confirmed, request a review again.
|
✅ All benchmarks passedTL;DR: All collected suites passed. Suite matrix
Deep Dive: ProxyLoad test (k6)
Synthetic overhead (Go)
k6 P99 is the SLA. Stage / ns/op values are synthetic Go microbenchmarks, not live traces. Deep Dive: Extraction quality
Environment & meta
Generated by IBEX Benchmark Bot
|
Up to standards ✅🟢 Issues
|
|
Tick the box to add this pull request to the merge queue (same as
|
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
| "run_url": "https://github.com/Rick1330/ibex-harness/actions/runs/44848498709", | ||
| "methodology": { | ||
| "index": "idx_memories_embedding_hnsw (m=16, ef_construction=64, cosine)", | ||
| "ef_search": 64, |
There was a problem hiding this comment.
🔍 HNSW methodology note conflicts with results
The PR methodology note lists ef_search=40, but both published cells use 64, matching the publish contract.
Was this helpful? React with 👍 or 👎 to provide feedback.
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
| "regression_vs_baseline_pct": null, | ||
| "baseline_sha": "48ea560b8e757aa7114a84c103dacdbef8ee8d34", | ||
| "profile": "fast", | ||
| "run_number": 2248, |
There was a problem hiding this comment.
🟡 Missing proxy benchmark history entry
Publishing run 2248 drops run 2241 from runs. Proxy latency history compares adjacent entries, so it shows p99 falling 2.8% instead of rising 3.8%.
Learn more
The benchmark workflow loads history from the repository, so a main-branch artifact does not include entries still pending in the shared data PR. When run 2248 was added, its history omitted the already-staged run 2241. The runs have distinct commit SHAs, and the history merge keeps distinct main-branch runs within its 365-entry limit. Proxy latency history uses adjacent entries for its deltas.
Example: Run 2248 has a p99 of 1.704 ms. Compared with run 1860 at 1.753 ms, that is a 2.8% decrease. Compared with run 2241 at 1.642 ms, it is a 3.8% increase.
Recommended fix: Restore run 2241 directly below run 2248. When updating the shared data PR, merge the incoming proxy history with existing pending main-branch runs, deduplicate by SHA, and keep the newest entries within the retention limit.
Was this helpful? React with 👍 or 👎 to provide feedback.
| }, | ||
| { | ||
| "t_s": 10.0, | ||
| "req_per_s": 1311.11 |
There was a problem hiding this comment.
🔍 Synthetic throughput shown as peak
The throughput generator derives throughput_series from the run-wide mean, not k6 interval measurements. The benchmark page reports the largest generated point as peak throughput without labeling it as synthetic.
Was this helpful? React with 👍 or 👎 to provide feedback.
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
There was a problem hiding this comment.
No application code in the PR — skipped Code Health checks.
See analysis details in CodeScene
Quality Gate Profile: Pay Down Tech Debt
Install CodeScene MCP: safeguard and uplift AI-generated code. Catch issues early with our IDE extension and CLI tool.
|



What and Why
Automated publish of public benchmark history for ibex-harness. Suites in this PR: Proxy, Memory HNSW, Ranking quality, Write pipeline, Extraction quality.
Keeps
/benchmarksand/benchmarks/memoryhistory current without contributor-authored commits.Tracking issue
N/A (GitHub App data publish)
How
Proxy suite
Status: ✅ PASS
1.57 ms176.47 µs/opsite/public/benchmarks/benchmark-data.json,site/public/benchmarks/badge.svgk6 p99 is the authoritative SLA metric. Stage breakdown and ns/op values are synthetic Go microbenchmarks.
Memory HNSW suite
1.0000site/public/benchmarks/hnsw-benchmark-data.jsonCorpus cells
Production-like knobs expected (
ef_search=40,min_similarity≈0.70,iterative_scan=off, bulk index build).Ranking-quality suite
1.00001.0000site/public/benchmarks/ranking-quality-benchmark-data.jsonWrite-pipeline suite
67.06 mssite/public/benchmarks/write-pipeline-benchmark-data.jsonSLA: p95 <= 200 ms.
Extraction-quality suite
1.00001.0000site/public/benchmarks/extraction-quality-benchmark-data.jsonTesting
/benchmarksand/or/benchmarks/memorypreview updatedPerformance
No runtime change — history JSON / badge only. Review suite tables above for SLA deltas.
Security
No secrets; App-signed commits; artifacts re-verified via Actions API before publish.
Migrations / Ops
None.
Docs
Public benchmark history only (
site/public/benchmarks/*). Suite JSON files stay separate.Checklist (Definition of Done)
benchmark-data,hnsw-benchmark-data,ranking-quality-benchmark-data,write-pipeline-benchmark-data,extraction-quality-benchmark-data)chore/bench-data-publishThe PR still needs the previously reported schema mismatch fixed before merging.
Fix with agent prompt
Summary
Publishes benchmark results for the proxy, HNSW, ranking quality, write pipeline, and extraction quality.
Reviews (29) · Last reviewed commit: "chore(bench): proxy benchmark data updat..." · Reviewed by Greptile