WAI Extension: Integer Payloads

Mirrored from the canonical text at commit 75a74f31 (2026-10-04).

Status: Draft. Pins the payload formats of the WAI-native integer codecs that SPEC.md §5 registers without one (Payloads of the WAI-native integer codecs): WIV1 for wai.video.int_motion and wai.video.int_hyper, WIH1 for wai.neural.int_hyper, WIA1 for wai.audio.int_codec and WIS1 for wai.splat.int_codec. Defines each layout, its decode, the checks a sink makes and the closed list of refusal codes; names each capability’s parameter-set members; and makes the JSON the encode and export tools write a converter input, never a payload. Reference impl: wai-rs — int_payload::{wire, decode_int, resolve}, int_payload::{wiv1_from_json, wih1_from_json, wia1_from_json, wis1_from_json}, codecs::PINNED_INT_PAYLOADS, container::check_int_payload_framing. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

1. Scope

Why this pin is permitted. Until this revision, these capabilities’ SPEC §5 rows named the decoder that read their payload, and no byte layout. The rows of wai.video.int_motion, wai.audio.int_codec and wai.splat.int_codec also named the encode or export tool that wrote a bitstream, a JSON document; those of wai.neural.int_hyper and wai.video.int_hyper named their decoders only. SPEC §5 called every row’s payload the bytes its codec normally emits, which for these rows was the tools’ JSON. So this revision changes what these strings carry. SPEC §8 permits it under Completing a registration: a revision may complete the registration of a WAI-native string whose row named a decoder and fixed no byte layout by pinning one, and it lists the envelopes that become non-conforming (§11; SPEC §5). SPEC §8’s rule that a registered string’s payload format is fixed forever applies to these strings from this revision on. The encode and export tools still write JSON that carries the same fields; it is a converter input (§10).

wai.neural.int_synth and wai.neural.int_mlicpp keep the payload formats SPEC §5 gives them. A sink decodes them and the five strings above (wai.video.int_motion, wai.video.int_hyper, wai.neural.int_hyper, wai.audio.int_codec, wai.splat.int_codec) through one path (int_payload::decode_int), and §8 names the codes it reports for those two.

2. Common rules

Encoding. Every multi-byte field is little-endian. Offsets are in bytes.

Framing. A payload starts with its four-byte magic and a version byte. This revision defines version 1 for every format. Every reserved field is zero. Nothing follows the last section.

rANS streams. A stream is a u32 length L followed by L bytes. L is a multiple of 4 and at least 8. It is decoded as follows (precision 16, with a bypass escape of 4-bit groups):

• The state x starts as (w1 << 32) | w0, from the stream’s first two u32 words. Renormalising: if x < 2^31, then x = (x << 32) | w, where w is the next unread word; if no word is left, the stream has run out and x is left as it is.
• bits(n): v = x mod 2^n, x = x >> n, renormalise, return v.
• Decoding a symbol against a table cdf of length n with offset o: let cum = x mod 2^16 and s be the largest index with cdf[s] ≤ cum (0 ≤ s ≤ n − 2); then x = (cdf[s+1] − cdf[s]) · (x >> 16) + cum − cdf[s], and renormalise. If s < n − 2, the value is s. If s = n − 2 (the escape): let v = bits(4) and count = v; then, at most 8 times, while v = 15 and the stream has not run out, v = bits(4) and count = count + v; then count = min(count, 8) and raw = Σ_{j < count} bits(4) << 4j. The value is −(raw >> 1) − 1 if raw is odd and (raw >> 1) + (n − 2) if it is even, saturated to i32. The symbol is value + o, saturated to i32.
• A decode stops at the first symbol during whose decode its stream runs out, and refuses the payload (stream_exhausted): the value that symbol would give is not used, and nothing after it is decoded. A stream that never runs out decodes every symbol, so a conforming payload is unaffected.
• After the last symbol a decode reads from a stream, the stream ends canonically: every word of it has been read, and the state is 2^31. Otherwise the payload is refused (stream_noncanonical). An encoder starts its state at 2^31 and writes the words it flushes and no others, so its streams end this way. A word after the last symbol’s, or a state the encoder did not start from, codes the same symbols in other bytes; the rule closes both, so neither can give one decode several content hashes. A stream that codes no symbol (the residual stream of a one-frame WIV1 with keyframe 1) is the state 2^31 alone: the words 2^31 and 0.

One clip, several payloads. These rules fix how a payload is read, not the only payload of a decoded output: tables that differ in entries the stream never selects (a symbol the clip does not use, say), or a different escape layout, code the same output in other bytes. A payload’s hash therefore identifies the payload, not what it decodes to. Where one output must have one identity, it is identified by the digest of its decoded output: for a clip, its frame-state-equivalence digests (SPEC §7, a video receipt’s clip_hash); for the other formats, their capabilities’ equivalence digests.

CDF tables. A table is n entries of u32 with 3 ≤ n ≤ 4097, cdf[0] = 0, cdf[n − 1] = 65536 and every entry greater than the one before. These rules are what keep the symbol search and the state update above in range.

Order of the checks. A parser checks, in this order, and refuses with the first that fails:

1. the magic: the capability’s (§1), or else format_capability_mismatch if it is another of the four magics, or bad_magic;
2. the version byte: present (stream_length), and 1 (version);
3. the fixed header: present (stream_length);
4. the reserved fields (reserved_nonzero);
5. the header’s fields, in the order the format’s section lists their rules (field_range), then, for WIV1, the keyframe kind (keyframe_capability_mismatch);
6. the caps of §7 on the sizes the header declares (too_large), in the order the format’s section lists them;
7. each later section in order: present (stream_length), then its rules (field_range, cdf_malformed). A section of one table per channel (WIA1, WIS1) is read table by table: each table present, then checked, before the next;
8. nothing after the last section (trailing, or nested_length for WIV1’s nested keyframe).

A decode then makes the checks of §7, and refuses a stream that runs out or does not end canonically (§2). Two conforming sinks refuse the same bytes with the same code.

3. WIV1 — integer inter-frame video

The header’s field rules are checked in the order mv_precision, keyframe, n_frames, h and w, block, q, cdf_len, then, for a one-frame clip, mv_precision 1 and block 1: such a clip codes no motion vector, so neither field would change its decode, and they are fixed so that it has one header. keyframe 0 is wai.video.int_motion’s and 1 is wai.video.int_hyper’s; the other is keyframe_capability_mismatch. The frame h·w·3, the clip n_frames·h·w·3, the frame count n_frames and the motion-vector field 2·nb·(n_frames − 1) are capped (§7), in that order.

Motion vectors. nb = ⌈h/block⌉·⌈w/block⌉ and nbx = ⌈w/block⌉. The vectors are frames 1 to n_frames − 1 in order, each its block grid row-major: the vector of pixel (y, x) in frame f is entry (f − 1)·nb + (y div block)·nbx + (x div block), in units of 1/mv_precision pel.

Decode. One decoder reads the residual stream for the whole clip, against the residual table at offset. For each frame in order, for each pixel in raster order (y, x), for each channel c of R, G, B: decode the symbol r, and the sample is clamp(pred + r·q, 0, 255). Since |r| < 2^31 and q < 2^16, every step fits i64.

• Frame 0 with keyframe 0: pred = 128.
• Frame 0 with keyframe 1: the frame is the nested WIH1’s decode (§4), which reads no symbol of the residual stream. It must be h × w (geometry_mismatch).
• Frame f ≥ 1: pred reads the previous decoded frame R with the pixel’s vector (dy, dx), where p(Y, X) is R at row clamp(Y, 0, h − 1), column clamp(X, 0, w − 1), channel c:
  • whole-pel: pred = p(y + dy, x + dx);
  • half-pel: iy = y + (dy >> 1), fy = dy & 1, ix = x + (dx >> 1), fx = dx & 1 (arithmetic shift); then p(iy, ix) when both are 0, (p(iy, ix) + p(iy, ix + 1) + 1) >> 1 when only fx is 1, (p(iy, ix) + p(iy + 1, ix) + 1) >> 1 when only fy is 1, and (p(iy, ix) + p(iy, ix + 1) + p(iy + 1, ix) + p(iy + 1, ix + 1) + 2) >> 2 when both are.

The output is n_frames RGB8 frames of h × w (rows top to bottom, pixels left to right, R, G, B bytes): the buffers of the capability’s frame-state-equivalence digests (SPEC §7 Digests).

4. WIH1 — the integer conditional image codec

The caps (§7) apply to n_z·zh·zw and to 2·m·sh·sw.

Decode, against the set of §9 (g_s.whm and h_s.whm, two integer networks at one fixed-point scale 2^afrac; entropy.json, the tables):

1. For each hyper-latent channel k and each of its zh·zw elements, decode a symbol s from the hyper-latent stream against channel k’s table and offset; the element is s·2^afrac + median_k.
2. h_s on the hyper-latent [n_z, zh, zw] gives [2·m, sh, sw]: its first m channels are the scales σ, the rest the means μ.
3. For each latent element p in order (channel, then row, then column), its table is b, where, with the scale edges e_0 … e_{N−1} and σ' = max(σ_p, e_0), b = N − 1 − #{ i < N − 1 : σ' ≤ e_i }. Decode a symbol r from the latent stream against table b and its offset; the element is r·2^afrac + μ_p.
4. g_s on the latent [m, sh, sw] gives three channels; each value v is clamped to [0, 2^afrac] and becomes (v·255 + 2^(afrac−1)) >> afrac, as RGB8.

Every value is formed in i64 and must fit it (§7).

5. WIA1 — the integer learned audio codec

The header’s field rules are checked in the order afrac, then sr, n_ch, lat_len and t_out, then cdf_len. The caps (§7) apply to n_ch·lat_len and to t_out.

Decode, against the set of §9 (model.wam, the integer synthesis):

1. For each channel k and each of its lat_len elements, decode a symbol s against channel k’s table at offset; the element is s·2^afrac + median_k.
2. The synthesis (one-dimensional integer transposed convolutions and integer inverse GDN) on [n_ch, lat_len] gives one channel of t ≥ t_out samples.
3. The first t_out samples v become (v·32767 + 2^(afrac−1)) >> afrac, clamped to [−32768, 32767].

The output is t_out samples of i16 PCM, one channel, at sr hertz: the buffer of the capability’s sample-equivalence digest (signed 16-bit little-endian samples).

6. WIS1 — the integer learned Gaussian-splat attribute codec

The header’s field rules are checked in the order afrac, the dimensions, n_gaussians. The caps (§7) apply to c·sh·sw and to attr·s·s.

Decode, against the set of §9 (g_s.whm, the integer synthesis):

1. For each channel k and each of its sh·sw elements, decode a symbol s against channel k’s table and offset; the element is s·2^afrac + median_k.
2. g_s on [c, sh, sw] gives [attr, s, s] (geometry_mismatch otherwise).
3. Each value v is clamped to [0, 2^afrac] and becomes (v·255 + 2^(afrac−1)) >> afrac, a byte.

The output, plane after plane, is the buffer of the capability’s splat-equivalence digest. The first n_gaussians cells of the planes, in raster order, hold one Gaussian each. The ranges are presentation data: a sink presents a plane’s byte u at about lo + (hi − lo)·u/255. They are outside the digest, and this extension does not fix presentation.

7. Caps and the checks a decode makes

Caps, the same on every target. No tensor a payload sizes may exceed 2^26 elements: a frame h·w·3, a WIV1 motion-vector field 2·nb·(n_frames − 1), a latent, a hyper-latent, a network’s output, an attribute tensor, a clip’s t_out. A WIV1 clip may not exceed 2^30 bytes, nor 2^20 frames.

Planning the networks. Before a decode allocates a latent, it plans every network on the payload’s shapes from the layers’ geometry and parameter lengths alone. The network’s input must be within the cap (too_large); then, layer by layer in order:

1. the layer fits the running channel count (model_mismatch): a convolution’s output channels, kernel and stride are non-zero, it has c·out·k·k weights (c·out·k for the audio synthesis) and out multipliers and biases, and its requantisation shift is under 128; an inverse GDN has c·c and c parameters and shifts under 128; a leaky ReLU’s shift is under 64;
2. every size the layer forms is within 2^26 (too_large): a convolution’s padded input side n + 2·pad, a transposed convolution’s uncropped side (n − 1)·stride + k + opad and uncropped plane, and the output;
3. the layer leaves an output (model_mismatch): a kernel no larger than the padded input, a crop 2·pad smaller than the uncropped side.

Every size is formed in 64-bit arithmetic with overflow checked, and a size past 64 bits is past the cap, so a 32-bit sink refuses exactly what a 64-bit one does, with the same code; once a network is planned, every size it forms fits a 32-bit index.

Memory and work. A payload’s header declares its decode’s output and the work that output takes, and a sink can read both before it decodes anything. A stream does not bound them: a table can make one symbol cost a small fraction of a bit, so a few kilobytes of stream can code a clip at the 2^30-byte cap, as any codec can describe a large output in a small file. The bound is the header’s, within the caps:

• the output: a WIV1 clip’s n_frames·h·w·3 bytes (one per sample), a WIH1 image’s 3·h·w bytes at its synthesis network’s size, a WIA1 clip’s 2·t_out bytes (i16 samples), a WIS1 attribute tensor’s attr·s·s bytes;
• the entropy decode: one step per symbol, n_frames·h·w·3 for a keyframe-0 WIV1 (h·w·3 fewer, plus its keyframe’s, for keyframe 1), n_z·zh·zw + m·sh·sw for a WIH1, n_ch·lat_len for a WIA1, c·sh·sw for a WIS1;
• the buffers: latents and network activations of 8 bytes per element (i64); while a network layer runs, its input, its output and, for a transposed convolution, the uncropped plane it accumulates into, each at most 2^26 elements (Planning the networks); a WIV1 decode’s frames (one byte per sample) and its motion vectors (4 bytes each);
• each network’s work, its multiply-accumulates layer by layer: a convolution’s every output times c·k·k (every tap of its padded input), a transposed convolution’s every input times out·k·k (out·k for the audio synthesis), an inverse GDN’s every element times c; a WIV1 with keyframe 0 runs no network. A kernel may skip a zero term, so it performs at most this many.

A WIA1’s latent is the one its t_out needs and no longer: the synthesis of lat_len samples gives at least t_out samples and that of lat_len − 1 fewer (geometry_mismatch), so a payload cannot declare a large latent to present a few samples.

A decode refuses a stream that runs out at its first missing symbol (§2), so a short stream stops early, but a stream that carries its symbols is decoded in full: a sink that will not spend what a header declares refuses before decoding, by its own size policy, applied to the declared cost (int_payload::declared_cost and int_payload::check_cost, with the set; neural::declared_cost over the set a registry resolved; the C ABI’s wai_int_payload_declared, given the set’s path; wai-web’s intPayloadCost; the header alone through container::declared_size), cost_over_limit (§8). The reference neural sink applies one when it is given one (neural::ModelRegistry::with_cost_limit); its C ABI does not decode integer video, PCM or splat attributes, which it has no output kind for. This extension fixes the caps; a sink’s policy may be stricter.

The parameter set must fit the payload (model_mismatch):

• every network parses (the WHM1 and WAM1 forms, §9), and the networks of one set share one fixed-point scale afrac in 1..=30;
• a WIA1 or WIS1 payload’s afrac is the model’s;
• the entropy tables of entropy.json are consistent (a table, length and offset per scale edge, at least one edge; a table, length, offset and median per hyper-latent channel) and every table is a CDF of §2 over its length; a payload has no more hyper-latent channels than the tables;
• each layer’s parameters have the lengths the running channel count needs, and its geometry leaves an output (Planning the networks);
• the audio synthesis gives one channel; the conditional codec’s g_s gives three.

The shapes the payload declares must be the ones the decode gives (geometry_mismatch): h_s’s output and [2·m, sh, sw]; a WIV1 keyframe and h × w; the audio synthesis’s length and t_out (it may be longer, by less than one latent sample’s worth: the synthesis of lat_len − 1 samples gives fewer than t_out); g_s’s output and [attr, s, s].

The order of a decode’s checks. After the parse (§2), a decode checks in this order and refuses with the first that fails:

1. the set’s members, in the order of §9’s list, each present (set_incomplete) and parsed (model_mismatch);
2. the parameter set against the payload (model_mismatch): one fixed-point scale, the payload’s afrac, the entropy tables, the hyper-latent channels;
3. every network planned, in decode order (h_s then g_s; the audio synthesis; the splat g_s), each layer by the three rules above (model_mismatch, too_large): the caps come before any shape is compared;
4. what the networks give against the payload: for WIH1, h_s’s shape (geometry_mismatch), then g_s’s three channels (model_mismatch); for WIV1 with keyframe 1, its keyframe’s as for WIH1, then the keyframe’s size against h × w (geometry_mismatch); for WIA1, one channel (model_mismatch), then at least t_out samples, then a latent no longer than that needs (both geometry_mismatch); for WIS1, g_s’s shape (geometry_mismatch);
5. each stream in decode order (WIH1: the hyper-latent’s, then, after h_s runs, the latent’s; WIV1 with keyframe 1: the keyframe’s streams, then the residual stream), symbol by symbol: a symbol that runs its stream out (stream_exhausted) before its value’s range (field_range); after its last symbol, its canonical end (stream_noncanonical);
6. each network as it runs, layer by layer (field_range, below).

So a run-out is reported before the value it would have given, and a size past the caps before a shape that disagrees. A sink that applies a cost limit (Memory and work) compares the declared cost with it after step 4, once everything the cost reads has been checked, and before any symbol is decoded (cost_over_limit, §8).

Exactness. A latent rebuilt as s·2^afrac + base must fit i64; before each network layer runs, the decode checks on the values present that no product, partial sum, requantised value or inverse-GDN term can leave the type its arithmetic is formed in. A stream whose values fail either check is refused (field_range). So a decode with overflow checks and one without compute the same integers, and neither panics.

8. Refusal codes

The list is closed: a sink reports exactly one of these, the first that applies in the order of §2 and §7.

A refusal of the payload or of the parameter set against it, bad_magic through model_mismatch in the table below (and payload_malformed), is final: the sink does not decode the payload by another path, and SPEC §4’s fallback is not tried, since it would decode the same bytes. The other codes say the sink cannot decode through the capability at all, and route as SPEC §4 routes such a capability:

• set_incomplete is a pin that did not resolve (SPEC §3.1): a WAI1 continues at SPEC §4 step 4 (or reports it at step 6), and a WAI2 moves on to its next entry (SPEC §7.1);
• not_dispatchable and unsupported_capability name a capability the sink does not decode: SPEC §4 does not take step 3 (or 5) with it, so a WAI1 continues at step 4 and a WAI2 at its next entry, as for a capability the sink does not advertise;
• cost_over_limit is the sink’s own policy, not a fault of the payload: a payload whose declared cost exceeds the limit the sink applies (§7, Memory and work) is refused before any symbol is decoded, and routes as unsupported_capability does, so a WAI1 continues at step 4 and a WAI2 at its next entry.

9. Parameter sets

A capability that decodes against a parameter set reads these members, by the names a set’s listing gives them (prior-carriage §7):

A set of several files is pinned through its listing (wai.prior.set); a one-file set MAY be pinned form-less, by its file’s SHA-256 (SPEC §3.1). wai.audio.int_codec and wai.splat.int_codec were registered with sets of several files, because no one-file format had been named for them; this revision registers each as one file, the file its decoder reads, which holds every parameter its output depends on. That widens their registrations, as SPEC §8 allows a MINOR revision to; a set pinned through its listing still verifies.

WHM1 (int_hyper_synth::IntModel::from_bytes): magic, then afrac, gf, shift and the layer count (u32 each), then per layer a tag byte: 0 a transposed convolution and 1 a convolution (input and output channels, kernel, stride, padding and, for 0, output padding as u32; weights as i16; per-output-channel multipliers and biases as i64), 2 an inverse GDN (channels as u32; gamma as i32, beta as i64), 3 a ReLU, 4 a leaky ReLU (slope i64, its shift u32). WAM1 (int_audio::AudioModel::from_bytes): the same header, then layers of tag 0, a one-dimensional transposed convolution, and tag 2, an inverse GDN. In both forms nothing follows the last layer: a file with bytes after it is refused (model_mismatch), so one network has one file and one digest. The layers’ integer arithmetic is the reference decoders’ (int_hyper_synth.rs, int_audio.rs, int_transform.rs), which the conformance vectors pin.

10. The tools’ JSON forms

The encode and export tools under tools/ write a JSON bitstream. It is a converter input and a debugging aid, never a payload. An encoder MUST NOT carry it as the payload of any capability of §1, and a sink refuses it (bad_magic). The converters (int_payload::{wiv1,wih1,wia1,wis1}_from_json, and the independent int-payload-conformance/int_payload_verify.py) make explicit what the JSON leaves implicit:

• WIV1: an absent mv_precision is 1, and only 1 and 2 are converted; a one-frame clip is written with mv_precision 1 and block 1, whatever the JSON states; mvs has one list per frame, and the first is empty; a frame’s list shorter than its block grid is padded with (0, 0), the vector the JSON decoder read for a missing block; a list longer than the grid, or a vector outside i16, is refused; rmax is not carried. The keyframe of wai.video.int_hyper is the WIH1 converted from its own JSON.
• WIH1: the fields of cond_bitstream.json, the streams as they are.
• WIA1: sr is the export’s sample_rate (in audio.json, which the bitstream does not carry); lo is the symbol offset; every channel’s table must have the first table’s length; afrac is carried and must be the model’s.
• WIS1: the ranges lo and hi must each be a binary32 value, carried exactly (a JSON integer included: 2^60 + 1 is refused, not rounded to 2^60); afrac is carried and must be the model’s.

A payload and the JSON it is converted from decode to the same bytes; the conformance vectors check that for every committed JSON bitstream.

11. Envelopes this revision makes non-conforming

This revision completes these strings’ registrations with a payload format they did not have (SPEC §8, Completing a registration). It makes non-conforming, and they still parse:

• a WAI1 envelope whose capability or fallback is a string of §1 and whose payload is not that string’s format: a tool’s JSON form among them;
• a WAI2 envelope with a component of a string of §1 whose bytes are not that string’s format.

A sink refuses such a payload when it decodes it (§8), with no fallback attempt. An encoder MUST NOT write one; the reference implementation’s emit paths (wai wrap, the C packers and the browser writers) parse the payload as a sink does and refuse to. An encrypted envelope’s payload (SPEC §6) is ciphertext until a sink decrypts it, so the emit paths do not check it; the sink checks the plaintext when it decodes it (SPEC §6.4). No committed envelope or conformance corpus carried such a payload when this revision was made.

12. Reference implementation and conformance

• int_payload::wire parses and writes the four formats, with no dependency on the rest of the crate; int_payload::decode_int decodes every integer capability from parameter files held in memory, with no ONNX runtime and no filesystem, for native and wasm32 sinks; int_payload::resolve verifies a pin over sets held in memory (pin::verify_bytes_with) and runs SPEC §3.1’s check 5, for a pinned decode and over the files an unpinned decode reads (a listing the set carries must match them, and the digests include the listing of the files read); it is the only way to build the parameters a decode reads (IntParams). The neural sink (neural::decode_payload) delegates every integer capability to decode_int, and its registry reads an unpinned integer set once and runs check 5 over those bytes.
• codecs::PINNED_INT_PAYLOADS is the one table of strings and magics; container::check_int_payload_framing parses a payload with int_payload::wire and the capability’s keyframe kind, which needs no integer decoder and no feature, so every emit path runs it: Wai::to_emit_bytes and WaiMulti::to_emit_bytes (and so the C packers), wai-web’s pack and packMulti, and wai wrap; wai inspect reports it.
• video_receipt::VideoReceipt::seal_decode_wiv1 seals a wai.video.int_motion receipt whose identity is BLAKE3 over the WIV1 payload.
• wai-web decodes the four formats in the browser (decodeWiv1, decodeWih1, decodeWia1, decodeWis1).
• Conformance: the payloads beside their JSON sources in wai-rs/tests/vectors/, and the refusal fixtures and index in int-payload-conformance/, written by wai_int_payload_vectors gen and checked by wai_int_payload_vectors verify; the same corpus checked by int_payload_verify.py, which shares no code with wai-rs and decodes the WIV1 keyframe-0 payloads in full; and the cross-architecture goldens of byte-exact-conformance, which parse and decode the WIV1 and WIH1 payloads on every platform of its matrix.