WAI Extension: Neural Dispatch

Mirrored from the canonical text at commit c5e7133e (2026-11-11).

Status: Draft. Says which unit of a sink’s device — its neural processor, its GPU or its CPU — runs a neural capability’s networks, under the determinism tier the decode needs (determinism-tiers), and what the sink reports about the choice. A path that computes in floating point, a neural processor’s included, never runs a decode that needs decode-equivalence. Defines the report wai-neural-dispatch/1 and eight refusal codes. No manifest member and no wire format: the envelope is unchanged, and the choice is the sink’s. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

1. Scope

A neural capability’s decode has an entropy stage, which turns the payload into symbols and a latent, and networks, which turn the latent into the medium. A sink can run the networks on more than one unit of its device:

• a neural processor (npu), driven through the platform’s model runtime or through a browser’s WebNN;
• a GPU (gpu);
• the CPU (cpu).

A path is a unit, the backend that drives it, and the arithmetic the networks are computed in there:

The arithmetic is integer (the integer arithmetic the capability’s registration fixes, end to end) or float (floating point, at whatever precision, order and fusion the backend chooses: a neural processor computes in reduced precision, a GPU and a CPU reorder and fuse).

2. The tier a decode needs

A decode of capability C needs decode-equivalence when any of these holds:

1. C is registered at decode-equivalence (its registry class is NeuralIntegerExact, determinism-tiers §3);
2. the envelope’s intent is replicate (SPEC §7);
3. the pin over C’s parameter set states tier: "decode-equivalence" (SPEC §3.1).

Otherwise it needs C’s registered tier: entropy-consistency for a NeuralEntropyExact capability, none for every other. A pin’s other tokens, and a token the vocabulary does not register, add nothing. A pin applies to the capability it pins; a decode through the fallback reads no pin of the primary’s.

3. Which paths may run it

• A float path MUST NOT run a decode that needs decode-equivalence (dispatch_tier_refused), whichever unit it names, the CPU included. Determinism-tiers §1 says why: the result of a float network depends on what the backend chooses per device.
• A float path MAY run any other decode. The decode holds C’s registered tier, never more: a neural processor’s output is the reconstruction of that device, and no tier over the reconstruction is claimed for it.
• An integer path runs only a capability registered at decode-equivalence, whose networks are integer; offered for another, it is refused (dispatch_arithmetic_mismatch). It holds decode-equivalence where determinism-tiers §5 says the device keeps the integer arithmetic: on the CPU always, on a GPU under Claiming the tier on a GPU.

A neural processor that runs integer graphs exactly, as wai/wai-ops-npu does for wai-ops/1-npu (integer-operators §9.2), would be an integer path; no decoder runs its graphs there in this revision (determinism-tiers §5), so no integer path names npu here.

The entropy stage is not a path’s: every path runs it on the CPU, in the integer arithmetic the capability’s registration fixes (§6.3). So at entropy-consistency a decode’s symbols are the same whichever path runs its networks.

4. The choice

A sink offers its paths in an order of its choosing; the reference sink offers the neural processor first, then the CPU, then the integer interpreter (§6). For each offered path in order, the sink:

1. applies §3; a refused path is not tried;
2. asks whether the device can run it: whether the unit is there (dispatch_unit_absent), whether it passed the exactness check an integer backend makes before the unit runs anything (dispatch_unit_inexact; determinism-tiers §5, Claiming the tier on a GPU, item 5), whether the parameter set holds a program for the backend (dispatch_program_absent), whether that program takes this payload’s shape (dispatch_shape_mismatch), and, where the backend states a plan, whether the plan places any operation on the unit (dispatch_not_placed, §5);
3. runs the decode’s networks there; a backend error refuses the path (dispatch_backend_failed).

The first path that runs is the decode’s. When every offered path is refused, the decode fails with the report (§5), and SPEC §4 treats the capability as one this sink cannot decode for this envelope.

5. The report

A sink that dispatches a neural decode SHOULD give a report, a JSON object:

A path is {"unit", "backend", "arithmetic"} with the tokens of §1. The path in ran adds tier, the tier the decode holds (§3), and placement: where the backend states a plan, {"npu", "gpu", "cpu", "unknown"}, how many of the program’s operations the plan puts on each unit, constants left out; where it states none, null. A refused path adds reason, a code of §7.

Placement is the backend’s statement. A model runtime that compiles a program for a device chooses, per operation, among the devices the program may use, and states its choice in a compute plan. A plan that puts no operation on the unit a path names is not a run there: the path is refused (dispatch_not_placed) and the next is tried. A plan that puts some operations elsewhere is reported as it is. WebNN states no placement: a page reports the unit it created its context for, with placement null, and the report says no more than that.

A report states which unit ran and at which tier. It is not an energy figure, and is never read as one (energy-measurement).

A sink of a fleet carries its reports in its run report, which holds each decode’s set and tier to what the operator’s model policy lets the sink’s class run, and refuses a report whose float path claims decode-equivalence (model-governance §6.2).

6. The paths of the reference implementation

wai-rs/src/neural_dispatch.rs holds §2–§5 as pure functions: demand, admit, choose and Report::to_json. A page runs the same functions (wai-web’s neural_npu bundle). wai.neural.bmshj2018 has a path on each unit the reference reaches; the integer capabilities have the GPU’s and the CPU’s in a native sink (§6.5); every other neural capability has the one path it always had, which the report states, with the neural processor refused (dispatch_tier_refused where the decode needs decode-equivalence, dispatch_program_absent otherwise).

6.1 The platform’s model runtime (native, macOS)

Feature neural_npu of wai-rs (src/neural/npu.rs, src/neural/npu_runtime.rs). A parameter set holds a program for the runtime as two files: decoder.mlmodel, an ML Program’s specification, and decoder.weight.bin, its weight file. The sink places the bytes it read from the set — the bytes a pin verified, for a pinned decode — in a package of its own, compiles the package with the runtime, loads it for the CPU and the neural processor, and reads the runtime’s compute plan. The program’s one input fixes the latent’s shape; a payload of another shape is dispatch_shape_mismatch. neural::npu::decode_envelope_dispatched and decode_payload_dispatched decode with an offer (Offer::native(): the neural processor, then ONNX Runtime on the CPU, then the integer interpreter) and return the report beside the output; neural::npu::npu_plan gives the plan without decoding. On another platform, or without the feature, the path is dispatch_unit_absent. Every other entry point decodes on the CPU, as before.

tools/wai_bmshj2018_npu_set.py writes such a set for wai.neural.bmshj2018 from CompressAI’s pretrained bmshj2018-factorized model: decoder.onnx and cdfs_q3.json as the CPU path reads them, the program computed in float16 with its input fixed at the latent of a 256 × 256 picture, the WebNN form of §6.2, and the set’s usage statement (wai-usage.json, prior-carriage §18.1), which records the licence the weights are published under, BSD-3-Clause-Clear. It also writes one payload, a picture it draws from a seed and codes with the model’s own encoder. GDN’s reparameterised parameters are taken as values, so the program holds plain convolutions, multiplications and square roots; the tool refuses to write a set whose plain synthesis differs from CompressAI’s by any amount.

6.2 WebNN (a page)

wai-web/demo/npu.html, with the neural_npu bundle. The page decodes the entropy stage through the bundle (bmshj2018Latent), builds the synthesis as a WebNN graph from decoder.webnn.json (the layers, each a transposed convolution or an inverse GDN, as offsets into decoder.f32, the float32 weights), and tries the contexts of the neural processor, the GPU and the CPU in that order (createContext with the deviceType the browser of §8 takes). A transposed convolution is built with no padding and followed by the slice its padding and output padding select, the same values: one browser’s WebNN backend refuses explicit padding on a transposed convolution. npuDispatch applies §2–§4 to what the page found and gives the report. A browser with no navigator.ml gives every WebNN path dispatch_unit_absent; the page has no other path for the networks. A decode that needs decode-equivalence is refused before any context is created.

6.3 The entropy stage

neural_dispatch::bmshj2018::latent: the payload’s header, the rANS stream decoded against the set’s tables (cdfs_q<N>.json) to the symbols, and each symbol plus its channel’s median, a value the set carries, as the latent. Integer and table reads only, so the symbols and the latent are the same on every machine and in every page. Their digest (symbols_digest) is the BLAKE3 of the symbols, each as four bytes little-endian.

6.4 Agreement with the reference

A float path’s decode agrees with the reference at its tier when:

1. the symbols are the reference decode’s (§6.3: the same digest); and
2. at none, which promises nothing of the reconstruction, the implementation holds its own bound: the RGB8 reconstruction within 40 dB PSNR of the reference sink’s decode on the CPU (ONNX Runtime), on the set’s payload. The bound is the implementation’s check that the path decodes the capability, not a property of the capability.

At entropy-consistency the first condition is the tier’s criterion (determinism-tiers §2). Nothing here compares reconstructions across devices for a tier: no tier above the registered one is claimed for a float path’s output.

6.5 The integer capabilities in a native sink

wai-rs/src/sink_decode.rs (feature sink_decode), the decode a native host runs in its own process (native-sink §13). It offers the GPU’s path, the integer operator set’s WGSL kernels (gpu, wgsl, integer), then the CPU’s (cpu, wai-ops, integer). The GPU’s path runs only on a GPU that passed the decoder’s start-up exactness check when it opened (dispatch_unit_inexact otherwise), and none after a device error (dispatch_backend_failed). Its placement counts the decode’s graph operators run on the GPU and those run on the CPU (a graph with an operator outside wai-ops/1-i32); a decode none of whose operators ran on the GPU reports that path dispatch_not_placed, and the CPU’s as the one that ran. On the GPU’s path each graph runs on the unit measured cheaper for its shape on that device, in energy where the platform gives a reading and in time otherwise (native-sink §13.2); the placement counts where each ran. The entropy stage and the stage operators run on the CPU on both paths, in the capability’s integer arithmetic, so the two paths give the same bytes.

7. Codes

8. Evidence