Skip to main content

WAI Extension: Browser Sink

Mirrored from the canonical text at commit b1681667 ().

Status: Draft. How a sink that runs in a web page answers SPEC §4’s “the sink advertises cap”, and what the frames it decodes can be checked against. For each object it is about to decode, the page routes the object’s capability to the user agent’s decoder, a native host’s or WAI’s own WebAssembly decoder (§4): the user agent is asked only about codec strings the WebCodecs Codec Registry lists, and its decoder is preferred where it is exact for the tier the object needs. The page evaluates the constraint sets of capability-constraints itself (§5), and reports the joules a decode was measured to cost beside the user agent’s powerEfficient boolean, which stays the user agent’s word (§6). Every frame the page decodes carries a content binding (§7): the content hash of the receipted object it was decoded from, the link hash of the group receipt that object verified under, and the tier its samples hold. The binding’s life through clone(), transfer, copyTo() and re-encode is in §8, its carriage in a page in §9, and the state of the frame on screen, which joins the binding to the receipts and to live-provenance’s verdict, in §10. The member a frame carries, contentBinding, is proposed for the WebCodecs VideoFrame Metadata Registry in proposals/webcodecs-content-binding.md, a draft that has not been filed; until a registry lists the member, a page carries it in a side table (§9.2). No transport, no record, no change to the wire format or to SPEC §4’s algorithm. The reference implementation is the browser_sink module of wai-rs (feature browser_sink), which wai/wai-live-web builds for the page; the page’s own code is joule-code/joule-code-rs/crates/joule-moq-web/web/room/binding.js; the corpus is live-conformance/browser/, and its second implementation is live-conformance/browser_sink_verify.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

Texts read for this revision, each pinned to the version named: WebCodecs, W3C Working Draft 19 October 2026: the VideoDecoder’s Output VideoFrames steps, Create a VideoFrame, the VideoFrameMetadata dictionary and metadata(), Copy VideoFrame metadata, Clone VideoFrame, the constructors’ metadata steps, the serialization and transfer steps, copyTo() and EncodedVideoChunk; the WebCodecs Codec Registry, W3C Registry Draft 24 September 2026, its video and audio tables; the WebCodecs VideoFrame Metadata Registry, W3C Registry Draft 14 April 2026, its entries and its registration requirements; and Media Capabilities, W3C Working Draft 18 October 2026, MediaCapabilitiesInfo (supported, smooth, powerEfficient) and its privacy considerations. A section number in this text is this extension’s unless it names another document.

1. Scope

In scope: the capability report a page builds (§3); routing an object to a decoder (§4); constraint sets evaluated in the page (§5); the energy entry beside powerEfficient (§6); the content binding, its form and when a page writes one (§7); what happens to it through a frame’s life (§8); how a page carries it (§9); the state of the frame on screen (§10); and the reasons (§11).

Out of scope: a transport; the user agent’s decoders and whether they conform to their standards, which a page cannot see beyond the answers they give and a probe (§4.4); the decoders of a native host; how a meter measures (energy-measurement §3); and what a party does with a state.

Relationship to the core spec and the other extensions:

2. Terms

Hexadecimal is lowercase. JSON in this text is RFC 8259.

3. The capability report

A page states what it knows about its decoders, and what it is about to decode, in a capability report: one JSON object.

{ "browser_sink": 1,
  "ua": { "<codec string>": { "supported": true, "smooth": true, "power_efficient": false,
                              "probe": "match" | "mismatch" | "not_run" } },
  "host": { "<capability>": "<tier>" },
  "wasm": [ "<capability>" ],
  "constraints": { "<capability>": [ { "<parameter>": { "enum": [ ], "minimum": 0, "maximum": 0 } } ] },
  "energy": { "<capability> <route>": { "joules_micro": 0, "decodes": 0, "window_us": 0,
                                        "class": "<class>", "relative_ppm": 0,
                                        "filtering": "On" | "Off" | "Unknown",
                                        "calibration_ref": "<text>",
                                        "coverage": "Whole" | "Partial", "fabrics": [ "cpu" ] } },
  "objects": [ { "name": "<text>", "capability": "<capability>", "codec": "<codec string>" | null,
                 "need": "<tier>", "params": { "<parameter>": 0 } } ] }

A report that does not read in this shape, or whose browser_sink is not 1, is refused whole (report_malformed): every member above is required, objects may be empty, and a member this text does not define is ignored. A codec string with an answer in ua that §4.2’s gate refuses is never read (§4.2).

4. Routing

4.1 The user agent’s decoders

A capability is put to a UA decoder only through a row of this table:

capabilityUA decodercodec stringUA tier
wai.video.vp9VideoDecodervp09. and moredecode-equivalence
wai.video.avcVideoDecoderavc1. or avc3. and moredecode-equivalence
wai.video.hevcVideoDecoderhev1. or hvc1. and moredecode-equivalence
wai.video.av1, wai.video.av1.losslessVideoDecoderav01. and morenone
wai.video.vp8VideoDecodervp8none
wai.audio.opus, wai.audio.opus.packetAudioDecoderopusnone
wai.audio.flacAudioDecoderflacnone
wai.audio.aacAudioDecodermp4a. and morenone

The UA tier is what a UA decoder’s output holds when the decoder conforms to its standard:

A capability with no row has no UA decoder: wai.video.av2, wai.video.vvc and wai.audio.iamf among the registered ones, since the registry of §4.2 lists no codec string for them, and every WAI-native capability, whose decoder is WAI’s own. ImageDecoder has no row in this revision: an image decode in a user agent may apply colour conversion, and the user agent does not say whether it did.

4.2 The registry gate

The WebCodecs Codec Registry (W3C Registry Draft, 24 September 2026) lists these codec strings:

mediarows
videoav01.*, avc1.*, avc3.*, hev1.*, hvc1.*, vp8, vp09.*
audioflac, mp3, mp4a.*, opus, vorbis, ulaw, alaw, pcm-*

A codec string matches a row equal to it; a row ending .* or -* is matched by a string that begins with the row up to and including its . or - and has at least one character after it. A codec string that matches no row:

The registry is how WebCodecs names a codec. Its abstract says no implementer is required to support any entry; a string outside it names nothing the registry defines, so two user agents’ answers about it need not be about the same codec, and a page has no basis to read one. Such a capability resolves through a host decoder or a wasm decoder, or not at all. A row the registry adds later enters this table by a revision of this extension.

4.3 The steps

For each object o of a report, in order, the page computes its route:

  1. Kind. If o.capability is not dispatchable (SPEC §4: a record, a derivation, a candidate, a companion-only or a refinement-only capability), the route is none (not_dispatchable).
  2. Need. If o.need is above the capability’s registered tier, the route is none (tier_exceeds_class, determinism-tiers §3).
  3. Constraints. §5 over the report’s sets for o.capability and o.params: if they cannot be evaluated the route is none (constraints_unevaluable), and if no set is satisfied it is none (constraints_unmet) — the capability is unsupported for this object, which is not a decode error (capability-constraints §2).
  4. The UA leg. It is taken when every check holds; otherwise its reason is the first that fails:
    1. o.codec is a string (no_ua_mapping: the object names nothing to ask a UA decoder);
    2. o.codec passes §4.2’s gate (codec_unregistered), whatever the capability;
    3. o.capability has a row in §4.1 (no_ua_mapping);
    4. o.codec is a string of that row’s form (codec_mismatch);
    5. the report holds an answer for o.codec (ua_not_asked), and its supported is true (ua_unsupported);
    6. the need is at most the row’s UA tier (ua_inexact);
    7. where the need is decode-equivalence, the answer’s probe is match (ua_unprobed for not_run, ua_probe_mismatch for mismatch).
  5. The host leg. It is taken when o.capability is in host (no_host) with a tier at least the need (host_inexact).
  6. The wasm leg. It is taken when o.capability is in wasm (no_wasm). A wasm decoder is WAI’s own, so its output holds the registered tier, which step 2 has already held to be at least the need.
  7. Route. The route is the first leg taken, in the order UA, host, wasm. Where none is taken the route is none (no_decoder), and each leg’s reason is reported.

The UA leg comes first because a decoder the device ships is the one a page is expected to use, and checks 4.6 and 4.7 take it only where its output is exact for what the object needs; the wasm decoder is the fallback that depends on nothing the device must provide. A page MUST decode an object only through its route, and MUST NOT decode one whose route is none.

Each object’s entry in the routing table states: its name, its route, its reason (null unless the route is none), each leg’s outcome ("taken", "not_reached" for a leg after the one taken and for every leg when steps 1 to 3 decide, or the leg’s reason), asked (whether the page may put o.codec to the user agent: true exactly when steps 1 to 3 pass and checks 4.1 to 4.4 hold), the route’s tier, and the energy entry of §6. The entry’s reason is the code of the step that decided, or no_decoder from step 7. The route’s tier is the row’s UA tier for ua (lowered to none unless the probe matched), the host’s declared tier for host, the registered tier for wasm, and none for none; in each case no higher than the registered tier.

4.4 The probe

An answer of supported says a UA decoder accepts the configuration; it does not say the decoder’s output is the decoding process’s. Where an object needs decode-equivalence, the page decodes a probe: an object of the same capability and configuration whose decoded frames’ SHA-256 digests are published (http-delivery §12.4’s probe object), each over the frame’s samples as copyTo() writes them with no format, rect or layout option (planes in order, rows tightly packed). A frame output in NV12 is compared after its interleaved chroma plane is split into its two planes, which reorders samples and changes none. The probe is match when every frame’s digest is the published one, mismatch when any is not, and not_run when the page decoded no probe. This revision publishes no probe for a UA codec, so a page reports not_run and a need of decode-equivalence is never routed to a UA decoder.

4.5 The user agent’s answer is a constraint check too

isConfigSupported answers for a whole configuration, before any decoder is configured, and returns the configuration it would accept. It already does, for the parameters a configuration carries, what capability-constraints asks of a sink: an answer before dispatch, for one object, that is “unsupported” rather than a decode error. A page SHOULD build the configuration from the object’s own declared parameters, and SHOULD configure the decoder with the configuration the answer returns. The two checks are separate and both apply: an object routes ua only when its constraint sets are satisfied (step 3) and the user agent answered supported (check 4.5). Neither covers the neural, quantum, world or haptic families, which have no UA decoder; their constraints are evaluated by §5 alone.

4.6 What a page asks, and what it advertises

A page builds a report in two passes: from the objects alone it computes asked for each, and puts to the user agent exactly the codec strings an entry marks asked; then it adds the answers and computes the routes. Media Capabilities is asked only about a codec string the user agent answered supported (http-delivery §12.2). A capability routed ua, wasm or host is, for http-delivery §12.1’s descriptor, via "ua" (with the codec strings asked and their answers, provisional true), "wasm" or "host"; one routed none is absent from it.

5. Constraint sets in the page

A page evaluates capability-constraints §2 in this form:

evaluate(sets, params) is:

  1. No sets for the capability: unconstrained.
  2. The sets break the form above — not an array, an empty array, a set that is not an object, a constraint that is not an object, a constraint with no member or with a member other than the three, an empty enum or one holding something other than a value, a minimum or maximum that is not an integer, or a minimum above a maximum: unevaluable (constraint_malformed).
  3. A set names a parameter outside the register: unevaluable (parameter_unregistered). A page MUST NOT advertise a capability whose sets it cannot evaluate (capability-constraints §2).
  4. A set is satisfied when, for each parameter it names, the object declares a value for it and the value meets each member of the constraint: for enum, it equals one of the values, a string only a string and an integer only an integer; for minimum and maximum, it is an integer at least, or at most, the bound. An undeclared parameter, or a declared value that is neither a string nor an integer, meets no constraint. An empty set is satisfied by every object.
  5. satisfied when one set is; otherwise unsatisfied (constraints_unmet).

The grammar is the one capability-constraints adopts, enumerations, minima and maxima ORed over sets, with integers in place of rationals, so every evaluation is exact.

6. Energy beside powerEfficient

Each entry of the routing table carries an energy entry:

{ "power_efficient": true | false | null, "figure": null | { … }, "reason": null | "<reason>" }

A measurement is read as energy-measurement §2 has it, and refused, with figure null and the entry’s reason, when:

A figure that reads is reported with every member it was given and per_decode_micro = ⌊joules_micro / decodes⌋; joules_micro and decodes stay beside it, so the floor loses nothing. An object with no measurement is unmetered: figure and reason are both null, and a page presents it as unmetered, never as 0 J. In this revision a figure never changes a route; it is reported, not steered on. How a figure is measured, bracketed and attributed is energy-measurement §3 and §4’s, and a Partial coverage says the meter saw only the fabrics it names.

7. The content binding

7.1 The value

A content binding is a JSON object, and in a page a dictionary of the same members, the contentBinding member of a frame’s metadata: proposed for the WebCodecs VideoFrame Metadata Registry, and not registered (§9):

membervalue
algthe algorithm digest is computed with. This revision defines jwp-object-1: the content hash of jwp-receipts §2.1, BLAKE3("moq-jwp:object\x01" ‖ object).
digest64 hexadecimal digits: the content hash of the receipted object the frame was decoded from.
receiptoptional; 64 hexadecimal digits: the link hash (jwp-receipts §3 step 3) of the group receipt that object verified under.
tierthe tier the frame’s samples hold (§7.2).

Its canonical form is the JSON text of those members in that order, receipt left out when absent, with no whitespace and no other member. A reader:

7.2 When a page binds a frame

A page binds a frame it decoded, and only then, when:

  1. the frame was output for a chunk whose coded data came from one receipted object, and the page computed digest over that object’s bytes as received, or holds it as the content hash of the object receipt that object verified against;
  2. the content hash of every object the decoder was given since its last key frame — its last key chunk, or its configuration — was established: the object verified against its JWP object receipt (jwp-receipts §3 steps 1 to 5), or it was fetched whole under that hash as its content identifier, or through a verified head proof (http-delivery §6, §8.3). A frame predicted from data whose hash nobody established is not bound (binding_unverified_reference). A page that verifies every object before a decoder sees it meets this by construction.

receipt is written exactly when the object verified against a group receipt (jwp-receipts §3 steps 1 to 5), and is that receipt’s link hash; a page MUST NOT write a receipt it did not verify the object against. For an object carried over MoQ, digest and receipt are then the content_hash and group_receipt of its WAI_RECEIPT property (moq-streaming-format §5). tier is the route’s tier (§4.3), lowered to none where:

Where the frame’s tier is below the object’s need, the page MUST NOT present the frame as meeting it.

On the wasm route the wasm decoder writes the binding itself: it holds the object’s bytes, hashes them, decodes them, and returns every frame with the binding.

8. Through the frame’s life

operationthe binding
a decoder’s outputnone from the decoder: WebCodecs’ Create a VideoFrame takes no metadata. The page binds it (§7.2, §9).
clone()carried: the clone references the same samples, and Clone VideoFrame assigns it every slot but the resource reference, [[metadata]] among them.
serialization (postMessage without transfer)carried: the serialization steps copy [[metadata]].
transfercarried to the receiving realm; the source is closed and keeps none.
close()gone with the frame.
copyTo()the bytes written carry none. With no format option they are the frame’s samples, and a page MAY check them against a published digest (§4.4); with one they are a conversion, which no tier covers.
new VideoFrame(frame, init)carried only when the page puts it in init.metadata: the constructor copies init’s metadata, not the source’s. The new frame references the source’s samples; a page MAY copy the binding into it, and a narrower visibleRect presents part of the same samples, from the same object.
new VideoFrame from a canvas, an image or a buffera frame of other samples: none, except on the wasm route, where the samples are the ones the wasm decoder output from the bound object (§7.2).
drawing the frame (drawImage, a texture)the drawn pixels are a rendering and carry none; §10 says which frame a region of the screen shows.
VideoEncoder.encode(), and a decode of its outputdoes not survive. A re-encoded frame is a new object: its chunks carry no binding, and a frame decoded from them is bound, if at all, to that new object and its own receipt.

9. Carriage in a page

9.1 As metadata

Where the user agent’s VideoFrameMetadata has a contentBinding member, a page binds a decoded frame f by constructing new VideoFrame(f, { metadata: { ...f.metadata(), contentBinding } }) and closing f. clone(), serialization and transfer then carry it by WebCodecs’ own steps (§8).

9.2 In a side table

VideoFrameMetadata is a dictionary whose members are the registry’s, and a dictionary keeps only the members it declares, so a member no registry lists is dropped from init.metadata. Until contentBinding is listed, a page carries the binding in a side table:

  1. Pending. Before it gives a decoder a chunk, the page enters the chunk’s binding under (that decoder, the chunk’s timestamp). A decoder outputs a frame with the timestamp of the chunk it came from (WebCodecs Output VideoFrames), and nothing else in a frame says which chunk that was. Where a second chunk is given the same decoder under a timestamp still pending, both entries are struck and neither frame is bound (binding_ambiguous). A page bounds the entries pending per decoder, and strikes the oldest past the bound.
  2. Output. On a decoder’s output, the page takes the entry under (that decoder, the frame’s timestamp), removes it, and attaches it to the frame object, held weakly so that it lives as long as the frame does. An output with no entry is unbound.
  3. Reset. reset(), close() and a decoder error strike that decoder’s pending entries, and so does flush() once it resolves, for the entries whose frames it did not output; a frame output afterwards for a struck entry is unbound.
  4. Clone. clone() attaches the source’s binding to the clone.
  5. Transfer. A frame posted to another realm is posted with its binding beside it, and the receiver attaches it on arrival. A frame that arrives without one is unbound.
  6. Close. close() detaches it.

A page MAY test whether the user agent keeps the member (construct a frame with it in init.metadata and read metadata()), and use §9.1 where it does.

10. The frame on screen

For each region of the screen a page draws frames into (a tile), the page keeps the binding of the frame it last drew there. The state of that frame is computed from:

The state is, in this order:

  1. unbound, with no binding or with one the reader refuses (§7.1);
  2. refused, when object, chain or provenance is refused;
  3. pending, when object is unverified, chain is held or provenance is missing;
  4. verified otherwise.

anchor and history are admissions: a chain a joiner anchored, or filled below its anchor. A page that shows verified for them MUST also say from which group it holds the chain, as jwp-receipts §3 step 3 requires of a verifier that re-anchors, and MUST NOT state the groups before it verified.

A page that shows the state, behind a control the reader turns on, shows for each tile the state, the tier, the first digits of the digest and of the receipt, the provenance, the route and the energy entry. The state is about the frame the tile last drew, not about the pixels on the display: what the display emits is a rendering of the frame, scaled and converted to the display’s colour space, which no tier covers.

11. Reasons

reasonraised at
report_malformed§3
not_dispatchable§4.3 step 1
tier_exceeds_class§4.3 step 2
constraints_unevaluable§4.3 step 3
constraints_unmet§4.3 step 3, §5 step 5
no_ua_mapping§4.3 steps 4.1, 4.3
codec_mismatch§4.3 step 4.4
codec_unregistered§4.2, §4.3 step 4.2
ua_not_asked§4.3 step 4.5
ua_unsupported§4.3 step 4.5
ua_inexact§4.3 step 4.6
ua_unprobed§4.3 step 4.7
ua_probe_mismatch§4.3 step 4.7
no_host§4.3 step 5
host_inexact§4.3 step 5
no_wasm§4.3 step 6
no_decoder§4.3 step 7
constraint_malformed§5 step 2
parameter_unregistered§5 step 3
energy_class_invalid§6
unmetered_as_zero§6
energy_count_invalid§6
energy_evidence_missing§6
binding_malformed§7.1
binding_alg_unknown§7.1
binding_unverified_reference§7.2
binding_ambiguous§9.2 step 1

An object routed none is inert at the sink for that object, as SPEC §4 step 6 has it; a refused energy measurement leaves the route as it is; a refused binding leaves the frame unbound.

12. Conformance

live-conformance/browser/ holds the corpus, vectors.json, written by wai_browser_sink_vectors gen. It has:

wai_browser_sink_vectors verify and the test tests/browser_sink.rs check that vectors.json regenerates byte for byte, that wai-rs gives every case its stated outcome, that every reason of §11 appears, and that §11 lists exactly the reference’s codes. live-conformance/browser_sink_verify.py is the second implementation, written from this text: it shares no code with wai-rs, reads each capability’s kind, status and replicate class from the registry export (registry/capabilities.json), checks signatures with WAI’s strict rule and hashes with BLAKE3 from claim-conformance/, and mutates copies of the corpus and requires every copy to fail. The page runs the route, constraint, energy, binding and frame cases through its wasm build, and its side table through the sequences, in a browser (joule-moq-web’s end-to-end scenario binding).

13. Security and privacy considerations