[ambergorzynski]

Issue: #4299

@afd and I have been working on generating CTS tests through mutation testing, which we discussed some time ago in #4299. This draft proposes a first batch of five tests, so we can agree on their form before generating more.

Recap

The idea: we deliberately mutate (i.e. introduce a small bug into) part of a WebGPU implementation or a downstream driver, and run the CTS against the mutated version. If no test fails, the CTS has a gap: it doesn't detect that change. We then use a WGSL fuzzer to find a shader that does fail on the mutated code, and propose adding it as a test, so that the CTS will catch future bugs in that part of the code.

Each test is a small fuzzer-generated program that the unmodified compiler runs correctly and the mutated one does not. For these five tests:

1. We mutated Mesa's NIR shader compiler (through llvmpipe/lavapipe, run with Dawn) and ran the CTS against each mutant.
2. For mutants that no CTS test detected, we used the WGSL fuzzer WGSLsmith to find programs whose output changes when the mutant is present.
3. We reduced each program automatically, to what we hope is a reasonable size.

In the earlier discussion, @dneto0 made some helpful suggestions that we have incorporated here: the tests live in a separate bucket, organised by the implementation that was mutated, and each one has a machine-readable label saying what was mutated and where.

What the tests look like

The tests are in src/webgpu/mutation/mesa/, in two files: no_input.spec.ts (shaders without an input buffer) and uniform_input.spec.ts (shaders that read a uniform buffer). Each test runs its shader once and compares the whole output buffer with the expected bytes. For example:

kMutations['shader_10c2ef9c'] = [
  { package: 'dawn', version: '9de0fd67086127228e8aa28e69616da37220cfc3' },
  {
    package: 'mesa',
    version: 'ba8eaab4f07e33c0b74fa92c60852cba2518bf2e',
    mutated: {
      file: 'src/compiler/nir/nir_constant_expressions.c',
      line: 50895,
      generatedFrom: 'src/compiler/nir/nir_constant_expressions.py',
    },
  },
];
g.test('shader_10c2ef9c')
  .desc(
    `A fuzzer-generated compute shader. Found by mutation testing a constant folding compiler pass.`
  )
  .fn(t => {
    runGeneratedTest(
      t,
      {
        // s_output: c = 4
        output: new Uint8Array([4, 0, 0, 0]),
      },
      `
struct StorageBuffer {
    c: i32,
}
...
`
    );
  });

As suggested:

• They are in a separate bucket, organised by the implementation that was mutated (webgpu:mutation,mesa,*).
• Each test has a machine-readable label, kMutations: git commits and the mutated source location. (generatedFrom marks a file that Mesa generates during the build.) The Mutation type is shared, in src/webgpu/mutation/mutation_info.ts.

Test	Mutated Mesa source
shader_10c2ef9c	nir_constant_expressions.c, line 50895
shader_101299f4	nir_constant_expressions.c, line 48527
shader_7ea91b04	nir_range_analysis.c, line 1178
shader_429a8325	nir_constant_expressions.c, line 51437
shader_7be24930	nir_opt_if.c, line 839

Questions

• Where should the bucket live? We used a subdirectory, webgpu:mutation,mesa,*, as suggested by @dneto0. The alternative is a separate top-level tree, webgpu-mutation:*; the earlier discussion left this open.
• Is the kMutations format useful? Each entry is written just above its test and collected in a table each file exports. Should it contain more or less detail, e.g. the mutation itself rather than just its line?
• How much should each program be simplified? Further simplification is possible, but gets harder to scale hundreds of tests.
• Bytes or typed values? Buffer contents are byte arrays, as the fuzzer reports them, with a comment giving the values. We could instead write them as values of their WGSL types, member by member (e.g. new Float32Array([1, 1, 0, 0, 10, 10]) for a mat3x2<f32>), which is easier to read. Would you prefer the typed form?
• Padding: The output buffer starts zero-filled, and padding bytes must still be 0 afterwards. Should we pre-fill with a sentinel instead, as padding.spec.ts does? And if we move to typed values, should padding still be checked, or only the struct members?
• Test names: a hash of the shader text (shader_<8 hex digits>), with a description naming the area of the mutation. Is there a naming scheme you'd prefer?

+@afd