Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -26,5 +26,9 @@ a.out.wasm
/compat/wasm/*.wasm
/src/wasm/*.wasm

# emscripten JS glue, embedded inside wllama.wasm by scripts/pack_wasm.js
/compat/wasm/*.js
/src/wasm/*.js

# vitest browser screenshots of failed tests
__screenshots__/
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Common mistakes that AI agents usually make:
The `llama.cpp` submodule is bumped weekly by `.github/workflows/sync-upstream.yml`. To do it by hand, or when the workflow asks you to fix a broken sync:

1. Bump the submodule: `git -C llama.cpp fetch origin master && git -C llama.cpp checkout --detach FETCH_HEAD`
2. Rebuild: `./scripts/sync_upstream.sh`. It runs the wasm build (default + compat), regenerates the glue message types and the worker code, then formats. Everything it touches is tracked in git and must land in the same commit as the submodule bump. Set `SKIP_COMPAT=1` to skip the compat build, it doubles the build time - the sync workflow does this, so `compat/wasm/wllama.js` is normally not refreshed by an autosync PR
2. Rebuild: `./scripts/sync_upstream.sh`. It runs the wasm build (default + compat), regenerates the glue message types and the worker code, then formats. Everything it touches is tracked in git and must land in the same commit as the submodule bump. Set `SKIP_COMPAT=1` to skip the compat build, it doubles the build time - the sync workflow does this, so `compat/wasm/wllama.wasm` is normally not refreshed by an autosync PR
3. If the build fails, the break is almost always in `cpp/` - our glue calls a llama.cpp API that changed. Find the upstream change with `git -C llama.cpp log -p --since=2.weeks -- <path>`, then adapt our side to match
4. Rerun `./scripts/sync_upstream.sh` until it passes
5. Once it builds, run `npm run build` (some tests import from `esm/`), then `npm run test`. It runs the suite on Chrome, which is enough here - do NOT run `npm run test:firefox` or `npm run test:safari`
Expand Down
17 changes: 12 additions & 5 deletions README-dev.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,19 +109,26 @@ These steps are performed:

The main downside of this approach is that on WebGPU, even though some tensors can be offloaded to the GPU, we still need to allocate the full model in main memory. For example, a 4GB model will still occupy 4GB of main memory, even if half of the layers (~2GB) are offloaded to the GPU.

## Function names in stack traces
## Packing the wasm

The release wasm has no debug info, so `scripts/add_wasm_names.js` adds a standard wasm `name` section with cleaned names after each build. Browsers read this section themselves, so a stack trace shows `server_response::send` without any decoding in JS.
After each build, `scripts/pack_wasm.js` packs the emscripten output into a single `wllama.wasm`. It adds 2 custom sections:

Emscripten's `--emit-symbol-map` flag produces a `.js.symbols` file mapping each wasm function index to its demangled C++ name. The script reads this file from the build directory and writes the result to `src/wasm/` or `compat/wasm/`. It always reads from the build directory, so running it twice does not add a second section.
- `wllama_js`: the emscripten JS glue (`build/wllama.js`). At runtime, `src/worker.ts` compiles the wasm, reads the JS from this section and runs it in the worker. So the JS and the wasm of one build always ship together, and `wllama.js` is not tracked in git.
- `name`: function names for stack traces, see below.

### Function names in stack traces

The release wasm has no debug info, so `scripts/pack_wasm.js` adds a standard wasm `name` section with cleaned names after each build. Browsers read this section themselves, so a stack trace shows `server_response::send` without any decoding in JS.

Emscripten's `--emit-symbol-map` flag produces a `.js.symbols` file mapping each wasm function index to its demangled C++ name. The script reads this file from the build directory and writes the result to `src/wasm/` or `compat/wasm/`. It skips a section that is already there, so running it twice does not add a second one.

The script runs as part of the docker build (see `scripts/docker-compose.yml`). It can also be run by hand:

```sh
node scripts/add_wasm_names.js build/wllama.wasm src/wasm/wllama.wasm
node scripts/pack_wasm.js build/wllama.wasm src/wasm/wllama.wasm
```

### Name cleaning rules
#### Name cleaning rules

Raw demangled names can be hundreds of characters. The following rules are applied in order:

Expand Down
17 changes: 5 additions & 12 deletions compat/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ wllama.setCompat(null);
npm install @wllama/wllama-compat
```

Then copy the assets from `node_modules/@wllama/wllama-compat/wasm/` to your public directory and call `setCompat()` with the URLs pointing to those files:
Then copy `node_modules/@wllama/wllama-compat/wasm/wllama.wasm` to your public directory and call `setCompat()` with the URL pointing to that file:

```ts
import { Wllama } from '@wllama/wllama';
Expand All @@ -66,23 +66,16 @@ const wllama = new Wllama({ default: '/wasm/wllama.wasm' });

wllama.setCompat({
wasm: '/wllama-compat/wasm/wllama.wasm',
worker: '/wllama-compat/wasm/wllama.js',
});
```

**IMPORTANT**: for Vite, you will need to import the JS as `?raw`
The JS code of the build is embedded inside `wllama.wasm`, so there is only one file to host. For Vite:

```ts
import compatWasm from '@wllama/wllama-compat/wasm/wllama.wasm?url';
import compatWorker from '@wllama/wllama-compat/wasm/wllama.js?raw'; // IMPORTANT: ?raw, NOT ?url

export const WLLAMA_COMPAT_CONFIG = {
wasm: compatWasm,
worker: {
code: compatWorker,
},
};

const instance = new Wllama(WLLAMA_CONFIG_PATHS, { logger: DebugLogger });
instance.setCompat(WLLAMA_COMPAT_CONFIG);
instance.setCompat({ wasm: compatWasm });
```

> **Note:** in older versions, `setCompat()` also needed a `worker` (the `wllama.js` file). It is no longer needed and can be removed. It can still be passed to override the embedded JS code, for example for custom builds.
1 change: 0 additions & 1 deletion compat/wasm/wllama.js

This file was deleted.

1 change: 0 additions & 1 deletion examples/basic/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,6 @@ <h2>Embeddings</h2>
if (window.location.href.match(/localhost|127\.0\.0\.1/)) {
wllama.setCompat({
wasm: '../../compat/wasm/wllama.wasm',
worker: '../../compat/wasm/wllama.js',
});
} else {
wllama.setCompat('default');
Expand Down
1 change: 0 additions & 1 deletion examples/decision/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -305,7 +305,6 @@ <h2>Decision models</h2>
if (window.location.href.match(/localhost|127\.0\.0\.1/)) {
w.setCompat({
wasm: '../../compat/wasm/wllama.wasm',
worker: '../../compat/wasm/wllama.js',
});
} else {
w.setCompat('default');
Expand Down
1 change: 0 additions & 1 deletion examples/embeddings/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@
if (window.location.href.match(/localhost|127\.0\.0\.1/)) {
wllama.setCompat({
wasm: '../../compat/wasm/wllama.wasm',
worker: '../../compat/wasm/wllama.js',
});
} else {
wllama.setCompat('default');
Expand Down
2 changes: 1 addition & 1 deletion examples/main/src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ export const WLLAMA_CONFIG_PATHS = {
default: wllamaWasm,
};

// Either a local { wasm, worker } object (when compat package is built) or 'default' (CDN fallback)
// Either a local { wasm } object (when compat package is built) or 'default' (CDN fallback)
export const WLLAMA_COMPAT_CONFIG = compatConfig;

export const MAX_GGUF_SIZE = 2 * 1024 * 1024 * 1024; // 2GB
Expand Down
9 changes: 2 additions & 7 deletions examples/main/vite.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,7 @@ const COMPAT_WASM = resolve(
__dirname,
'node_modules/@wllama/wllama-compat/wasm/wllama.wasm'
);
const COMPAT_JS = resolve(
__dirname,
'node_modules/@wllama/wllama-compat/wasm/wllama.js'
);
const compatAvailable = existsSync(COMPAT_WASM) && existsSync(COMPAT_JS);
const compatAvailable = existsSync(COMPAT_WASM);

// https://vitejs.dev/config/
export default defineConfig({
Expand All @@ -32,8 +28,7 @@ export default defineConfig({
if (compatAvailable) {
return `
import wasm from '${COMPAT_WASM}?url';
import worker from '${COMPAT_JS}?raw';
export default { wasm, worker: { code: worker } };
export default { wasm };
`;
} else {
console.warn(
Expand Down
2 changes: 0 additions & 2 deletions examples/multimodal/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,6 @@ <h2>Multimodal (Vision) Completion</h2>
if (window.location.href.match(/localhost|127\.0\.0\.1/)) {
wllama.setCompat({
wasm: '../../compat/wasm/wllama.wasm',
worker: '../../compat/wasm/wllama.js',
});
} else {
wllama.setCompat('default');
Expand Down Expand Up @@ -260,7 +259,6 @@ <h2>Multimodal (Vision) Completion</h2>
if (window.location.href.match(/localhost|127\.0\.0\.1/)) {
wllama.setCompat({
wasm: '../../compat/wasm/wllama.wasm',
worker: '../../compat/wasm/wllama.js',
});
} else {
wllama.setCompat('default');
Expand Down
1 change: 0 additions & 1 deletion examples/omni/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -263,7 +263,6 @@ <h3>Transcribe an audio file</h3>
if (window.location.href.match(/localhost|127\.0\.0\.1/)) {
wllama.setCompat({
wasm: '../../compat/wasm/wllama.wasm',
worker: '../../compat/wasm/wllama.js',
});
} else {
wllama.setCompat('default');
Expand Down
1 change: 0 additions & 1 deletion examples/test-backend-ops/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,6 @@ <h2>test-backend-ops</h2>

const CONFIG_COMPAT_PATHS = {
wasm: '../../compat/wasm/wllama.wasm',
worker: '../../compat/wasm/wllama.js',
};

const elemOutput = document.getElementById('output');
Expand Down
2 changes: 0 additions & 2 deletions examples/tools/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -615,7 +615,6 @@ <h2>Tool Calling Demo (OAI-compat API)</h2>
if (window.location.href.match(/localhost|127\.0\.0\.1/)) {
wllama.setCompat({
wasm: '../../compat/wasm/wllama.wasm',
worker: '../../compat/wasm/wllama.js',
});
} else {
wllama.setCompat('default');
Expand Down Expand Up @@ -647,7 +646,6 @@ <h2>Tool Calling Demo (OAI-compat API)</h2>
if (window.location.href.match(/localhost|127\.0\.0\.1/)) {
wllama.setCompat({
wasm: '../../compat/wasm/wllama.wasm',
worker: '../../compat/wasm/wllama.js',
});
} else {
wllama.setCompat('default');
Expand Down
3 changes: 0 additions & 3 deletions scripts/build_worker.sh
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,5 @@ process_file() {
process_file ./src/workers-code/llama-cpp.js LLAMA_CPP_WORKER_CODE
process_file ./src/workers-code/opfs-utils.js OPFS_UTILS_WORKER_CODE

# emscripten
process_file ./src/wasm/wllama.js WLLAMA_EMSCRIPTEN_CODE

# build CDN paths
node ./scripts/generate_wasm_from_cdn.js
10 changes: 4 additions & 6 deletions scripts/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,9 @@ services:
# go back to root
cd ..

# copy output files
# pack output files (wasm + JS glue in one file)
mkdir -p src/wasm
cp build/wllama.js src/wasm
node scripts/add_wasm_names.js build/wllama.wasm src/wasm/wllama.wasm
node scripts/pack_wasm.js build/wllama.wasm src/wasm/wllama.wasm

# build compat package (skip if SKIP_COMPAT=1)
if [ -z "${SKIP_COMPAT}" ]; then
Expand All @@ -60,10 +59,9 @@ services:
# go back to root
cd ..

# copy output files (compat build)
# pack output files (compat build)
mkdir -p compat/wasm
cp build-compat/wllama.js compat/wasm
node scripts/add_wasm_names.js build-compat/wllama.wasm compat/wasm/wllama.wasm
node scripts/pack_wasm.js build-compat/wllama.wasm compat/wasm/wllama.wasm
fi

# list files (optional, for verification)
Expand Down
1 change: 0 additions & 1 deletion scripts/generate_wasm_from_cdn.js
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@ const WasmFromCDN = {
};

export const WasmCompatFromCDN = {
worker: 'https://cdn.jsdelivr.net/npm/@wllama/wllama-compat@${version}/wasm/wllama.js',
wasm: 'https://cdn.jsdelivr.net/npm/@wllama/wllama-compat@${version}/wasm/wllama.wasm',
};

Expand Down
96 changes: 79 additions & 17 deletions scripts/add_wasm_names.js → scripts/pack_wasm.js
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
#!/usr/bin/env node
// Usage: node scripts/add_wasm_names.js <input.wasm> <output.wasm>
// Copies a wasm file and adds a "name" section, so browsers show function names in stack traces.
// Names come from the .js.symbols file next to the input, made by --emit-symbol-map.
// Usage: node scripts/pack_wasm.js <input.wasm> <output.wasm>
// Pack the emscripten output into one wasm file: add the JS glue and the function names as custom sections. See README-dev.md

import { readFileSync, writeFileSync } from 'fs';
import { existsSync, readFileSync, writeFileSync } from 'fs';

// keep in sync with src/worker.ts
const JS_SECTION_NAME = 'wllama_js';

const [inputFile, outputFile] = process.argv.slice(2);
if (!inputFile || !outputFile) {
console.error('Usage: node scripts/add_wasm_names.js <input.wasm> <output.wasm>');
console.error('Usage: node scripts/pack_wasm.js <input.wasm> <output.wasm>');
process.exit(1);
}

Expand Down Expand Up @@ -127,23 +129,83 @@ function encodeNameSection(symbols) {
encodeUleb(names.length),
names,
]);
return encodeCustomSection(body);
}

function encodeCustomSection(body) {
const CUSTOM_SECTION = 0;
return Buffer.concat([Buffer.from([CUSTOM_SECTION]), encodeUleb(body.length), body]);
}

function encodeDataSection(name, data) {
return encodeCustomSection(Buffer.concat([encodeString(name), data]));
}

// -- wasm decoder -------------------------------------------------------------

function decodeUleb(buf, pos) {
let value = 0, shift = 0, b;
do {
b = buf[pos++];
value += (b & 0x7F) * 2 ** shift;
shift += 7;
} while (b & 0x80);
return [value, pos];
}

// list the names of all custom sections
function listCustomSections(wasm) {
const names = [];
let pos = 8; // skip magic + version
while (pos < wasm.length) {
const id = wasm[pos++];
let size;
[size, pos] = decodeUleb(wasm, pos);
if (id === 0) {
const [len, p] = decodeUleb(wasm, pos);
names.push(wasm.subarray(p, p + len).toString());
}
pos += size;
}
return names;
}

// -- main ---------------------------------------------------------------------

const wasm = readFileSync(inputFile);
const existingSections = listCustomSections(wasm);
if (existingSections.includes(JS_SECTION_NAME)) {
writeFileSync(outputFile, wasm);
process.stderr.write(`${outputFile}: already packed, copied as-is\n`);
process.exit(0);
}

const sections = [];

// the name section needs ascending function indices
const symbols = readFileSync(inputFile.replace(/\.wasm$/, '.js.symbols'), 'utf8')
.split('\n')
.map((line) => {
const colon = line.indexOf(':');
return [parseInt(line.slice(0, colon)), line.slice(colon + 1).trim()];
})
.filter(([id]) => !isNaN(id))
.sort(([a], [b]) => a - b);
const symbolsFile = inputFile.replace(/\.wasm$/, '.js.symbols');
if (existingSections.includes('name')) {
process.stderr.write(`${outputFile}: already has a name section, skipped\n`);
} else if (!existsSync(symbolsFile)) {
process.stderr.write(`${outputFile}: ${symbolsFile} not found, name section skipped\n`);
} else {
const symbols = readFileSync(symbolsFile, 'utf8')
.split('\n')
.map((line) => {
const colon = line.indexOf(':');
return [parseInt(line.slice(0, colon)), line.slice(colon + 1).trim()];
})
.filter(([id]) => !isNaN(id))
.sort(([a], [b]) => a - b);
const section = encodeNameSection(symbols);
sections.push(section);
process.stderr.write(`${outputFile}: ${symbols.length} symbols, name section is ${section.length.toLocaleString()} bytes\n`);
}

const wasm = readFileSync(inputFile);
const section = encodeNameSection(symbols);
writeFileSync(outputFile, Buffer.concat([wasm, section]));
process.stderr.write(`${outputFile}: ${symbols.length} symbols, name section is ${section.length.toLocaleString()} bytes\n`);
const jsFile = inputFile.replace(/\.wasm$/, '.js');
const js = readFileSync(jsFile);
const jsSection = encodeDataSection(JS_SECTION_NAME, js);
sections.push(jsSection);
process.stderr.write(`${outputFile}: embedded ${jsFile}, ${JS_SECTION_NAME} section is ${jsSection.length.toLocaleString()} bytes\n`);

writeFileSync(outputFile, Buffer.concat([wasm, ...sections]));
1 change: 0 additions & 1 deletion src/wasm-from-cdn.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ const WasmFromCDN = {
};

export const WasmCompatFromCDN = {
worker: 'https://cdn.jsdelivr.net/npm/@wllama/wllama-compat@3.13.0/wasm/wllama.js',
wasm: 'https://cdn.jsdelivr.net/npm/@wllama/wllama-compat@3.13.0/wasm/wllama.wasm',
};

Expand Down
1 change: 0 additions & 1 deletion src/wasm/wllama.js

This file was deleted.

7 changes: 5 additions & 2 deletions src/wllama.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -619,9 +619,12 @@ test.sequential('allowOffline', async () => {
allowOffline: true,
});

// Mock fetch to simulate offline
// Mock fetch to simulate offline, but let wasm load (like from HTTP cache)
const origFetch = window.fetch;
window.fetch = () => Promise.reject(new Error('offline'));
window.fetch = (input, init) =>
String(input instanceof Request ? input.url : input).endsWith('.wasm')
? origFetch(input, init)
: Promise.reject(new Error('offline'));

try {
await wllama.loadModelFromUrl(TINY_MODEL);
Expand Down
7 changes: 3 additions & 4 deletions src/wllama.ts
Original file line number Diff line number Diff line change
Expand Up @@ -185,8 +185,9 @@ export class WllamaRuntimeError extends Error {
* By default, these are set to URL of the latest builds on CDN, which requires internet to download. If you want to use local assets or have your own CDN, follow the instruction from @wllama/wllama-compat package.
*/
export interface WllamaCompat {
worker: string | { code: string };
wasm: string;
/** Optional, override the JS code embedded inside the wasm file */
worker?: string | { code: string } | undefined;
}

/**
Expand Down Expand Up @@ -1609,9 +1610,7 @@ export class Wllama {
: '')
);
} else {
const isUsingDefault =
this.compat.worker === WasmCompatFromCDN.worker &&
this.compat.wasm === WasmCompatFromCDN.wasm;
const isUsingDefault = this.compat.wasm === WasmCompatFromCDN.wasm;
if (isUsingDefault) {
this.logger().warn(
'Compatibility mode is activated, using resources from CDN. To use local resources, please refer to @wllama/wllama-compat package.'
Expand Down
Loading
Loading