Skip to content

chore(bench): publish proxy and memory HNSW and ranking-quality and write-pipeline and extraction-quality benchmark data - #1118

Open
ibex-harness-benchmark[bot] wants to merge 142 commits into
mainfrom
chore/bench-data-publish
Open

ibex-harness-benchmark[bot] wants to merge 142 commits into
mainfrom
chore/bench-data-publish

Conversation

@ibex-harness-benchmark

@ibex-harness-benchmark ibex-harness-benchmark Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

IBEX Benchmark Bot IBEX Benchmark Bot

What and Why

Automated publish of public benchmark history for ibex-harness. Suites in this PR: Proxy, Memory HNSW, Ranking quality, Write pipeline, Extraction quality.

Keeps /benchmarks and /benchmarks/memory history current without contributor-authored commits.

Tracking issue

N/A (GitHub App data publish)

How

Proxy suite

Status: ✅ PASS

Field Value
Run number 2250
Head SHA f2a0376
Proxy p99 (k6) 1.57 ms
Proxy overhead (Go) 176.47 µs/op
Paths site/public/benchmarks/benchmark-data.json, site/public/benchmarks/badge.svg

k6 p99 is the authoritative SLA metric. Stage breakdown and ns/op values are synthetic Go microbenchmarks.

Memory HNSW suite

Field Value
Run number 912
Head SHA 0a8ae77
Mean recall@10 1.0000
Path site/public/benchmarks/hnsw-benchmark-data.json

Corpus cells

Size Recall@10 p95 ef min_sim iterative build
10000 1.0000 6.5ms 64 0.70 off bulk
100000 1.0000 10.9ms 64 0.70 off bulk

Production-like knobs expected (ef_search=40, min_similarity≈0.70, iterative_scan=off, bulk index build).

Ranking-quality suite

Field Value
Run number 912
Head SHA 0a8ae77
Precision@5 1.0000
Recall@10 1.0000
Path site/public/benchmarks/ranking-quality-benchmark-data.json

Write-pipeline suite

Field Value
Run number 912
Head SHA 0a8ae77
p95 latency 67.06 ms
Path site/public/benchmarks/write-pipeline-benchmark-data.json

SLA: p95 <= 200 ms.

Extraction-quality suite

Field Value
Run number 423
Head SHA 0a8ae77
Precision macro 1.0000
Recall macro 1.0000
Path site/public/benchmarks/extraction-quality-benchmark-data.json

Testing

  • Bot workflow validation passed (suite(s) above)
  • Harness CI green on this PR
  • /benchmarks and/or /benchmarks/memory preview updated

Performance

No runtime change — history JSON / badge only. Review suite tables above for SLA deltas.

Security

No secrets; App-signed commits; artifacts re-verified via Actions API before publish.

Migrations / Ops

None.

Docs

Public benchmark history only (site/public/benchmarks/*). Suite JSON files stay separate.

Checklist (Definition of Done)

  • Correct files only (proxy JSON+badge and/or suite JSON files)
  • Labels match files (benchmark-data, hnsw-benchmark-data, ranking-quality-benchmark-data, write-pipeline-benchmark-data, extraction-quality-benchmark-data)
  • Single shared data PR branch chore/bench-data-publish

RetriggerView in GreptileConfidence Score: 4/5

The PR still needs the previously reported schema mismatch fixed before merging.

Fix All in CodexFindings

  1. P1 New fields fail benchmark checks. ▶
Fix with agent prompt
### Issue 1
site/public/benchmarks/hnsw-benchmark-data.json:86-88
Both the HNSW and ranking records now include `provenance`, but their schemas forbid extra run fields and do not declare it. The memory benchmark workflow checks the full published files against these schemas, so it rejects these records. Later runs keep older records, so publishing another run will not clear the failure. Update both schemas to accept the new fields, or remove the fields from both published files.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Publishes benchmark results for the proxy, HNSW, ranking quality, write pipeline, and extraction quality.

  • The proxy history adds a new run and the badge shows its p99.
  • The HNSW history adds a run with recall and search timings at two corpus sizes.
  • The ranking history adds a run with quality scores and gate results.
  • The write-pipeline history adds a run with latency percentiles and gate results.
  • The extraction history adds a run with gold-set scores and gate results.

Reviews (29) · Last reviewed commit: "chore(bench): proxy benchmark data updat..." · Reviewed by Greptile

Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

Ask an organization administrator to verify that this bot has an active assigned CodeRabbit review seat. See the seat-assignment guide. After the seat is confirmed, request a review again.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 0e9c9cd0-ad72-41dc-8efd-b10ce7bcb67a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions github-actions Bot added area/web Public site and published docs content area/benchmarks Benchmark collection, dashboard, published data labels Nov 3, 2026
codescene-delta-analysis[bot]

This comment was marked as outdated.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

View IBEX dashboard →

✅ All benchmarks passed

TL;DR: All collected suites passed.

Suite matrix

Suite Primary SLA vs Baseline Status
Proxy 0.86 ms (P99) n/a ✅ PASS
Extraction quality 1.000 (Precision macro) — ✅ PASS
Deep Dive: Proxy

Load test (k6)

  • P99: 0.86 ms (SLA < 20 ms)
  • Throughput: 1.4k req/s
  • Error rate: 0.00%

Synthetic overhead (Go)

  • Total: 109.90 µs/op
  • Stages: Auth LRU 0.35 µs · gRPC 0.15 µs · Rate limit 0.08 ms · Directive 0.02 ms · Prompt 34 ns

k6 P99 is the SLA. Stage / ns/op values are synthetic Go microbenchmarks, not live traces.

Deep Dive: Extraction quality
  • Precision macro: 1.000
  • Recall macro: 1.000
  • Category accuracy: 1.000
  • Temporal accuracy: 1.000
  • Conversations: 125
  • Provider: openai
Environment & meta
Generated by IBEX Benchmark Bot

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@mergify

mergify Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
@ibex-harness-benchmark ibex-harness-benchmark Bot changed the title chore(bench): publish extraction-quality benchmark data chore(bench): publish memory HNSW and extraction-quality benchmark data Nov 3, 2026
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
@ibex-harness-benchmark ibex-harness-benchmark Bot changed the title chore(bench): publish memory HNSW and extraction-quality benchmark data chore(bench): publish memory HNSW and ranking-quality and extraction-quality benchmark data Nov 3, 2026
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
@ibex-harness-benchmark ibex-harness-benchmark Bot changed the title chore(bench): publish memory HNSW and ranking-quality and extraction-quality benchmark data chore(bench): publish memory HNSW and ranking-quality and write-pipeline and extraction-quality benchmark data Nov 3, 2026
codescene-delta-analysis[bot]

This comment was marked as outdated.

codescene-delta-analysis[bot]

This comment was marked as outdated.

codescene-delta-analysis[bot]

This comment was marked as outdated.

Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
@ibex-harness-benchmark ibex-harness-benchmark Bot changed the title chore(bench): publish memory HNSW and ranking-quality and write-pipeline and extraction-quality benchmark data chore(bench): publish proxy and memory HNSW and ranking-quality and write-pipeline and extraction-quality benchmark data Nov 3, 2026
@ibex-harness-benchmark ibex-harness-benchmark Bot added the benchmark-data Automated benchmark data commits label Nov 3, 2026
codescene-delta-analysis[bot]

This comment was marked as outdated.

Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
codescene-delta-analysis[bot]

This comment was marked as outdated.

Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
codescene-delta-analysis[bot]

This comment was marked as outdated.

Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
codescene-delta-analysis[bot]

This comment was marked as outdated.

ibex-harness-benchmark[bot] added 4 commits November 20, 2026 18:38
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
codescene-delta-analysis[bot]

This comment was marked as outdated.

codescene-delta-analysis[bot]

This comment was marked as outdated.

codescene-delta-analysis[bot]

This comment was marked as outdated.

codescene-delta-analysis[bot]

This comment was marked as outdated.

Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
codescene-delta-analysis[bot]

This comment was marked as outdated.

Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
codescene-delta-analysis[bot]

This comment was marked as outdated.

ibex-harness-benchmark[bot] added 4 commits November 21, 2026 01:15
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
codescene-delta-analysis[bot]

This comment was marked as outdated.

codescene-delta-analysis[bot]

This comment was marked as outdated.

codescene-delta-analysis[bot]

This comment was marked as outdated.

codescene-delta-analysis[bot]

This comment was marked as outdated.

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 1 new potential issue.

Devin Review

"run_url": "https://github.com/Rick1330/ibex-harness/actions/runs/44848498709",
"methodology": {
"index": "idx_memories_embedding_hnsw (m=16, ef_construction=64, cosine)",
"ef_search": 64,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 HNSW methodology note conflicts with results

The PR methodology note lists ef_search=40, but both published cells use 64, matching the publish contract.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>
codescene-delta-analysis[bot]

This comment was marked as outdated.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 new potential issues.

Devin Review

"regression_vs_baseline_pct": null,
"baseline_sha": "48ea560b8e757aa7114a84c103dacdbef8ee8d34",
"profile": "fast",
"run_number": 2248,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Missing proxy benchmark history entry

Publishing run 2248 drops run 2241 from runs. Proxy latency history compares adjacent entries, so it shows p99 falling 2.8% instead of rising 3.8%.

Learn more

The benchmark workflow loads history from the repository, so a main-branch artifact does not include entries still pending in the shared data PR. When run 2248 was added, its history omitted the already-staged run 2241. The runs have distinct commit SHAs, and the history merge keeps distinct main-branch runs within its 365-entry limit. Proxy latency history uses adjacent entries for its deltas.

Example: Run 2248 has a p99 of 1.704 ms. Compared with run 1860 at 1.753 ms, that is a 2.8% decrease. Compared with run 2241 at 1.642 ms, it is a 3.8% increase.

Recommended fix: Restore run 2241 directly below run 2248. When updating the shared data PR, merge the incoming proxy history with existing pending main-branch runs, deduplicate by SHA, and keep the newest entries within the retention limit.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

},
{
"t_s": 10.0,
"req_per_s": 1311.11

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Synthetic throughput shown as peak

The throughput generator derives throughput_series from the run-wide mean, not k6 interval measurements. The benchmark page reports the largest generated point as peak throughput without labeling it as synthetic.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Signed-off-by: ibex-harness-benchmark[bot] <4215784+ibex-harness-benchmark[bot]@users.noreply.github.com>

@codescene-delta-analysis codescene-delta-analysis Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No application code in the PR — skipped Code Health checks.

See analysis details in CodeScene

Quality Gate Profile: Pay Down Tech Debt
Install CodeScene MCP: safeguard and uplift AI-generated code. Catch issues early with our IDE extension and CLI tool.

@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/benchmarks Benchmark collection, dashboard, published data area/web Public site and published docs content automated Bot-generated commits benchmark-data Automated benchmark data commits extraction-quality-benchmark-data hnsw-benchmark-data ranking-quality-benchmark-data write-pipeline-benchmark-data

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants