Float64 miscompile with FP64 emulation on Arc A750 at sub-group size 32 (NEO 26.35 / IGC 2.41.5); KernelAbstractions kernels hit it by default #750

On an Arc A750 (no native FP64, so Float64 is emulated), this kernel gives a wrong Float64 result when compiled with sub_group_size=32. Sub-group sizes 8 and 16, the default @oneapi launch, and Float32 are all correct.

In oneAPI.jl 2.10 the KernelAbstractions backend compiles every kernel for sub-group size 32 (KI.kernel_function → device_limits().sub_group_size). So with oneAPI.jl 2.10, any KA kernel with this pattern returns wrong results on this device. We found it through wrong Hessians in ExaModels.jl (madsuite-org/ExaModels.jl#347).

Reproducer

using oneAPI

# `a * b`, except that a zero `a` gives zero (branchless)
@inline amul(a, b) = (p = a * b; ifelse(iszero(a), zero(p), p))

@noinline function body(y, x, adj, adj2)
    @inbounds a = x[1]; @inbounds b = x[2]
    s = a + 1; c = b + 1
    v = amul(adj2, b * b)
    @inbounds y[1] += amul(adj * b, -s) + v * c * c
    @inbounds y[2] += (adj + adj2 * s * b)
    return
end

function kernel(y, x, adj, adj2)
    body(y, x, adj, adj2)
    return
end

for T in (Float32, Float64)
    x = T[-1.2, 1.0]; adj = T(2); adj2 = T(0)
    ref = zeros(T, 2); body(ref, x, adj, adj2)
    println(T, "  cpu: ", ref)
    for sg in (nothing, 8, 16, 32)
        y = oneAPI.zeros(T, 2)
        if sg === nothing
            @oneapi kernel(y, oneArray(x), adj, adj2)
        else
            @oneapi sub_group_size=sg kernel(y, oneArray(x), adj, adj2)
        end
        gpu = Array(y)
        println("  sub_group_size=", rpad(something(sg, "default"), 8), gpu, gpu ≈ ref ? "" : "   <-- WRONG")
    end
end

Output (oneAPI.jl 2.10.0, fresh environment):

Float32  cpu: Float32[0.4000001, 2.0]
  sub_group_size=default Float32[0.4000001, 2.0]
  sub_group_size=8       Float32[0.4000001, 2.0]
  sub_group_size=16      Float32[0.4000001, 2.0]
  sub_group_size=32      Float32[0.4000001, 2.0]
Float64  cpu: [0.3999999999999999, 2.0]
  sub_group_size=default [0.3999999999999999, 2.0]
  sub_group_size=8       [0.3999999999999999, 2.0]
  sub_group_size=16      [0.3999999999999999, 2.0]
  sub_group_size=32      [2.0, 2.0]   <-- WRONG

With adj2 = 0, y[1] should be -adj*b*(a+1) = 0.4. Instead it gets 2.0, the value computed for y[2] (adj + 0). Changing adj changes the wrong y[1] to match: it is always the y[2] value.

The same KA kernel (@kernel function k(y, x, adj, adj2); body(y, x, adj, adj2); end) gives [2.0, 2.0] for any workgroupsize, and also with oneAPIBackend(; always_inline = true).

It is very sensitive to the code shape

Small changes make it go away:

• inlining body into the kernel
• writing amul out by hand
• dropping either store, or using = instead of +=
• dropping * c * c

The trigger is not specific to any math function: the version above has only FP64 add/mul/neg, compare and select, and the original case used sin/cos.

Regression in the driver, not the SPIR-V

• SPIR-V is the same on both stacks. Dumped with @device_code, the SPIR-V for this kernel is the same with oneAPI.jl 2.9.2 and 2.10.0 except for one type declaration: the kernel state struct is {ptr, i64, i64} instead of [1 x ptr], and the code never uses it. The SPIR-V body is below.
• The old driver is correct at SIMD32. oneAPI.jl 2.9.2 (NEO 26.18.38308 / IGC 2.34.4) gives the correct result for this kernel. That driver already compiles it at sub-group size 32 by default (get_sub_group_size() returns 32), and IGC_ForceOCLSIMDWidth=32 doesn't change the result.
• The new driver is wrong at SIMD32 however it's selected. On oneAPI.jl 2.10.0 (NEO 26.35.39758 / IGC 2.41.5), the default launch picks sub-group size 8 and is correct. IGC_ForceOCLSIMDWidth=32 makes the default launch wrong too, so it's the SIMD32 code generation, not the sub_group_size plumbing.
• The SPIR-V back-end doesn't matter. The same wrong result happens with the Khronos translator (oneAPI.oneL0.LTS[] = true).

So the same SPIR-V is correct with IGC 2.34.4 and wrong with IGC 2.41.5 at SIMD32 with FP64 emulation. This looks like an IGC regression to forward to intel/intel-graphics-compiler. On the oneAPI.jl side, it might be worth not forcing sub-group size 32 for KA kernels on devices without native FP64, or at least documenting it. KA kernels can't currently opt out, since KI.kernel_function throws for a different sub_group_size.

%julia_body = OpFunction %void DontInline %10
%y = OpFunctionParameter %_ptr_Function__ptr_CrossWorkgroup_uchar
%x = OpFunctionParameter %_ptr_Function__ptr_CrossWorkgroup_uchar
%adj = OpFunctionParameter %double
%adj2 = OpFunctionParameter %double
%top = OpLabel
%34 = OpBitcast %_ptr_Function__ptr_CrossWorkgroup_double %x
%35 = OpLoad %_ptr_CrossWorkgroup_double %34 Aligned 8
%36 = OpBitcast %_ptr_CrossWorkgroup_uchar %35
%37 = OpLoad %double %35 Aligned 8
%38 = OpInBoundsPtrAccessChain %_ptr_CrossWorkgroup_uchar %36 %ulong_8
%39 = OpBitcast %_ptr_CrossWorkgroup_double %38
%40 = OpLoad %double %39 Aligned 8
%41 = OpFAdd %double %37 %double_1
%42 = OpFAdd %double %40 %double_1
%43 = OpFMul %double %40 %40
%44 = OpFMul %double %adj2 %43
%45 = OpFUnordNotEqual %bool %adj2 %14
%46 = OpSelect %double %45 %44 %14
%47 = OpBitcast %_ptr_Function__ptr_CrossWorkgroup_double %y
%48 = OpLoad %_ptr_CrossWorkgroup_double %47 Aligned 8
%49 = OpBitcast %_ptr_CrossWorkgroup_uchar %48
%50 = OpLoad %double %48 Aligned 8
%51 = OpFMul %double %adj %40
%52 = OpFNegate %double %41
%53 = OpFMul %double %51 %52
%54 = OpFUnordNotEqual %bool %51 %14
%55 = OpSelect %double %54 %53 %14
%56 = OpFMul %double %42 %46
%57 = OpFMul %double %42 %56
%58 = OpFAdd %double %55 %57
%59 = OpFAdd %double %50 %58
OpStore %48 %59 Aligned 8
%60 = OpInBoundsPtrAccessChain %_ptr_CrossWorkgroup_uchar %49 %ulong_8
%61 = OpBitcast %_ptr_CrossWorkgroup_double %60
%62 = OpLoad %double %61 Aligned 8
%63 = OpFMul %double %adj2 %41
%64 = OpFMul %double %40 %63
%65 = OpFAdd %double %adj %64
%66 = OpFAdd %double %65 %62
OpStore %61 %66 Aligned 8
OpReturn
OpFunctionEnd

Versions

Bad (oneAPI.jl 2.10.0):

Binary dependencies:
- NEO: 26.35.39758+2
- libigc: 2.41.5+2
- gmmlib: 22.10.0+0
- SPIRV_LLVM_Backend: 23.1.1+8
- SPIRV_LLVM_Translator: 23.1.1+3
- SPIRV_Tools: 2025.4.0+0
- oneAPI_Support: 0.11.0+0 (oneMKL v2026.1.0)

Toolchain:
- Julia: 1.13.1
- LLVM: 20.1.8

Julia packages:
- oneAPI.jl: 2.10.0
- GPUArrays: 12.1.1
- GPUCompiler: 2.18.0
- KernelAbstractions: 0.10.3
- KernelInterface: 0.4.3
- LLVM: 10.4.2
- SPIRVIntrinsics: 1.5.1

1 driver:
- 00000000-0000-0000-18e2-6b5701039b4e (v1.3.39758, API v1.17.0)

1 device:
- Intel(R) Arc(TM) A750 Graphics

Good (oneAPI.jl 2.9.2, same machine):

Binary dependencies:
- NEO: 26.18.38308+0
- libigc: 2.34.4+5
- gmmlib: 22.10.0+0
- SPIRV_LLVM_Backend: 23.1.1+8
- SPIRV_LLVM_Translator: 23.1.1+3
- SPIRV_Tools: 2025.4.0+0
- oneAPI_Support: 0.10.0+0 (oneMKL v2025.3.0)

Julia packages:
- oneAPI.jl: 2.9.2
- GPUArrays: 11.5.16
- GPUCompiler: 2.10.0
- KernelAbstractions: 0.9.45
- LLVM: 9.13.2
- SPIRVIntrinsics: 1.1.4

1 driver:
- 00000000-0000-0000-18e2-6bb1010395a4 (v1.3.38308, API v1.15.0)

Linux 7.0.0-38-generic. The results are the same with or without OverrideDefaultFP64Settings=1 IGC_EnableDPEmulation=1, and all runs used NEO_CACHE_PERSISTENT=0.