Skip to content

perf: reduce glob traversal and Buffer write overheads - #29

Merged
CoderSerio merged 22 commits into
feat/node-api-alignmentfrom
perf/native-batch-overheads
Oct 1, 2026
Merged

CoderSerio merged 22 commits into
feat/node-api-alignmentfrom
perf/native-batch-overheads

Conversation

@CoderSerio

@CoderSerio CoderSerio commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Problem and native value

Four compatible glob patterns previously scanned the same tree four times and locked shared result vectors per match. This change shares walkers for identical roots/hidden policies/root inclusion, collects results per worker, and skips cross-walk deduplication when only one walk remains. Separate roots retain their previous traversal semantics.

Buffer writes previously copied synchronous input once and async input twice. Sync calls now borrow bytes; async calls keep their entry snapshot and write it directly, preserving mutation isolation and slice boundaries. Snapshots are released on the worker after I/O, without waiting for JS Promise resolution. String encoding and public routing/signatures are unchanged.

Based on #28 (feat/node-api-alignment). The integrated change also includes the reproducible peer comparisons and official documentation from #30/#31, with glob traversal/exclusion and Windows scan metadata corrections.

Compatibility and tests

  • 452 tests passed on each of Node 22.22.0 and 24.21.0 locally, macOS arm64 release build.
  • Public grouped glob tests compare sync/Promise names and Dirents against Node, covering duplicate/overlapping patterns, exclusions, hidden roots, gitignore and symlink roots.
  • Public Buffer tests cover subarray offsets, empty inputs, concurrent snapshots, mutation after invocation, unchanged source bytes and sequential append ordering.
  • Snapshot semantics are Vooya's existing ownership guarantee, not a claim about mutable-input behavior in Node.
  • Node 22/24 Linux, Windows, macOS x64/arm64 CI all passed for d0ef0d5: CI run. All four native builds, eight runtime jobs, Lint and Documentation succeeded. CI now accepts all PR base branches so stacked PRs receive the same checks. Automated CodeRabbit review was skipped by its base-branch configuration; CI success does not imply a human review.

Performance evidence

Detailed report, Node 22 raw samples, Node 24 raw samples, reproduction harness.

Release/public-entry comparison against db835af on Apple M4 Pro, macOS arm64. Two warmups, ten measured samples, rotating Node/before/after order, median latency; warm local filesystem; exact results validated every sample. Glob concurrency 4, write concurrency 1, no explicit flush. Source/binary hashes and per-sample RSS deltas are recorded.

Workload / Node 22 Node Before After Limits
2,000 files, 4 patterns 13.59 ms 14.31 ms 3.79 ms Compatible roots/policies only
20,000 files, 4 patterns 118.27 ms 49.86 ms 14.07 ms Fully materialized results
8 files, 4 patterns 0.38 ms 7.47 ms 1.78 ms Still slower than Node
8 MiB async writeFile 0.93 ms 0.90 ms 0.78 ms One snapshot remains
8 MiB sync writeFile 0.75 ms 0.81 ms 0.75 ms Not consistently faster than Node

Both runtimes show improvement over the previous 8 MiB Buffer paths in final runs. Small writes have mixed results/regressions; exploratory disk-writeback noise also affected large writes. Single-pattern glob is broadly comparable; no independent speedup is claimed for the collector alone. RSS deltas are not peak memory; worker-local vectors may retain spare capacity and overlap the merged allocation. One payload-sized redundant allocation is removed from each Buffer operation, not all copies or filesystem work.

Documentation and validation

  • Relevant public tests, API SDDs, API pages and CHANGELOG updated.
  • Snapshot and grouping boundaries documented; integrated glob routing corrections have public regression coverage and explicit Node fallback documentation.
  • Before/after and Node release evidence includes small inputs and limitations.
  • Release build, full tests, typecheck, lint, cargo fmt and Clippy pass locally.
  • Node 22/24 platform CI complete: Linux, Windows, macOS x64/arm64 all passed.
  • Documentation site build passed.

No package publication is included. The integrated stack has 468 passing local tests per Node 22/24 runtime plus 12 publication/adapter contract checks; final platform CI passed and source-tree equality is verified during integration. Current per-API evidence retains all wins, losses and Node fallback routes.

Final verified code tree: 27d9beaac4576bdb74ddafb4a7bfceb58ff7bb06. Passing cross-platform CI. Integration preserves this exact tree.

@coderabbitai

coderabbitai Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: b52614a9-2571-43d4-9368-9d629950bfd2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@CoderSerio
CoderSerio merged commit ad26c3b into feat/node-api-alignment Oct 1, 2026
4 checks passed
@CoderSerio
CoderSerio deleted the perf/native-batch-overheads branch October 1, 2026 06:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant