Repository navigation
Conversation
The C kernels in kernel/x86_64/sbgemv_n.c and sbgemv_t.c are the default SBGEMV kernels (kernel/Makefile.L2) of every kernel table that has no SBGEMV kernel of its own: all x86_64 cores except COOPERLAKE/SAPPHIRERAPIDS, arm64 except NEOVERSEV1/N2/V2, POWER6 and older, riscv64 except ZVL256B, and the other architectures when built with BUILD_BFLOAT16=1. On every call they allocated a bf16 copy and an fp32 copy of A (6*m*n bytes) plus x, never checked the allocations and dereferenced NULL when they failed; the copies (and the stride-m inner loop over the fp32 copy in the N kernel) also made them very slow. Compute from A, x and y in place instead, with the same bf16 -> fp32 conversion as SBF16TOS_K (denormals to signed zero, quiet NaNs), the same summation order for every output, and no allocation at all (the x/y copies for non-unit increments are not needed either). Results are unchanged apart from code generation effects at rounding level: they are bit-identical except for NaN payloads on x86_64, arm64, POWER6, z13, riscv64, loongarch64, mips64 and arm32 in testing; with x87 arithmetic (32-bit x86) the N kernel now accumulates in single instead of extended precision, and the -O3 z14 build fuses alpha*acc + beta*y differently. The Cooperlake path still packs strided x/y into small buffers; it now checks those allocations and falls back to the C code if they fail.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
kernel/x86_64/sbgemv_n.c and sbgemv_t.c are the default SBGEMV kernels: all x86_64 targets except Cooperlake/Sapphire Rapids, and other architectures without bf16 kernels of their own (e.g. most arm64 targets).
On every call they malloc a bf16 and an fp32 copy of A (6mn bytes) and convert all of A before computing.
These allocations are not checked, so a large SBGEMV segfaults when malloc fails.
This PR computes directly from A, x and y; the generic path no longer allocates anything.
The conversion and the summation order are unchanged, so results are bit-identical to develop (apart from NaN payloads).
The Cooperlake/Sapphire Rapids path keeps its kernels, but now checks the allocations it makes for strided x/y.
On a Zen 2 (one thread, median of 9 interleaved runs), N gets 16-27x and T 7-12x faster for m = n = 256-6000, and small sizes (n = 4-64) 1.5-8x.
The Haswell, Sandybridge and Nehalem kernels gain 11-27x (N) and 7-12x (T).
Tested with test_sbgemv and a bit-for-bit comparison against develop on all x86_64 cores (Cooperlake/Sapphire Rapids under Intel SDE), on arm64, POWER, s390x, riscv64, loongarch64, mips64 and armv6/v7 under qemu, and on 32-bit x86.
With x87 math (32-bit x86) and GCC -O3 on z14, results differ at rounding level only.