WAI Extension: Machine Descriptors

Mirrored from the canonical text at commit c5e7133e (2026-11-11).

Status: Draft. A machine descriptor is what a model produced about one media object: labels, regions of the picture, and embeddings named by digest. It is carried as the payload of a signed meta event of live-events, anchored to the object it describes, and it states the model that produced it by digest. No new event kind, no new record and no new transport: the event is a WEV1 as live-events §3 lays it out, signed by the machine that ran the model, and it rides an MSF event timeline, an EXT-X-DATERANGE tag, an emsg box, a data channel and a recording as every event does (live-events §8). The reference implementation is the machine_coding module of wai-rs (feature machine_coding); the corpus is live-conformance/machine-coding/, and its second implementation is live-conformance/machine_coding_verify.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

1. Scope

In scope: the descriptor’s payload (§3), how a reader reads it from an event (§4), what its carriage is (§5), and its refusals (§6).

Out of scope: how a model is run, what its labels mean, and how good they are. A descriptor is a statement by its signer that a named model gave these outputs for a named object. It is not a claim about a task’s accuracy (task-fidelity §2 is the shape of one), and a reader that takes a score for a measurement has misread it. A descriptor carries no embedding vector: an embedding travels as an object of its own and is named here by its digest. The descriptors that video coding for machines codes inside its own bitstream (ISO/IEC 23888-2) are that format’s; wai.video.vcm is a candidate with no pinned payload (format-probes §19), and nothing here reads them.

2. Terms

• Machine: the principal that signs the event (live-events §2): the key of the device or service that ran the model.
• Model: named by a model digest, 32 bytes: the SHA-256 a pin of the model’s parameter set carries (SPEC §3.1), as a caption’s model is (live-events §4.10). A digest of 32 zero bytes names nothing.
• Object: the media object the descriptor is about, named by the event’s anchor (live-events §3.1): a track, a group and an object.
• Picture extent: a region’s coordinates are in ten-thousandths of the picture’s width and of its height, from its top left corner, so they hold at any size the object is presented at.

3. The payload

The payload of the meta event, integers little-endian:

An item’s form:

A text is one length byte and that many bytes of UTF-8 (RFC 3629), with no code point below U+0020 and no U+007F. A score is in thousandths, 0 to 1000, or 0xFFFF when the model stated none; it is the model’s own output and has no unit this extension defines. A region has w and h of at least 1, x + w at most 10 000 and y + h at most 10 000.

A descriptor of three items is under 200 bytes; live-events’ default max_payload is 1024 (live-events §5). A model that says more about one object signs more events.

4. Reading

A reader is given one event record, the room it reads for, and, where it has one, the list of models it admits. In this order; the first rule that fails names the refusal:

1. The record reads as live-events §3.3 reads one, its room is the reader’s, and its signature verifies (live-events §3.2): otherwise descriptor_event.
2. Its kind is meta (live-events §4.2): otherwise descriptor_not_meta.
3. It has an anchor: otherwise descriptor_unanchored. A descriptor is about one object.
4. Its payload begins with WMD1: otherwise descriptor_no_magic. A meta event’s payload is opaque to live-events, and one that is not a descriptor is some other application’s.
5. version is 1: otherwise descriptor_version. A reader reads no further field of a later version.
6. The fields are read in §3’s order. A payload that ends inside a field is descriptor_truncated. As each is read: an origin other than 2 or 3 is descriptor_origin; a model of 32 zero bytes is descriptor_no_model; a count of 0 or above 16 is descriptor_count; a form this section does not define, a score above 1000 that is not 0xFFFF, a region outside the picture extent or of no width or height, or an embedding with dims of 0 or a zero space or digest, is descriptor_item; a text that is not UTF-8, holds a code point this section excludes, or is empty where a label must not be, is descriptor_text.
7. No byte follows the last item: otherwise descriptor_trailing.
8. Where the reader was given the models it admits, model is one of them: otherwise descriptor_model_unknown.

A reader that accepts a descriptor has the machine that signed it, the object it is about, the model it names and its items. It has not checked the model’s outputs: nothing in a descriptor lets it. A sink that shows a descriptor’s labels SHOULD say that a machine produced them, as live-events §4.10 has it label a machine’s caption, and takes the label from origin.

A room’s verifier (live-events §6) carries the event as any meta event: it holds the machine to its policy, its chain and its rates, and reads the payload for its length alone. The rules of this section are a descriptor reader’s, applied to events that verifier accepted or to one record alone.

5. Carriage

A descriptor event is an event, and live-events §8 carries it unchanged. On MoQ it is a record of the machine’s MSF event timeline (live-events §8.1): the event is anchored, so its record is {"l":[<group>,<object>],"data":{"wev1":"<base64url>"}}, indexed by the MOQT Location of the object it describes, and a subscriber that joins mid-stream is not owed earlier descriptors (live-events §8.2 lists what a join set must hold; a publisher MAY repeat a descriptor whose duration_ms spans the group’s start). The machine publishes on a track of its own, as every principal does, so a sink that wants no descriptors subscribes to none.

An embedding’s bytes are an object the publisher names elsewhere; a reader that fetches them checks their SHA-256 against the item’s digest and their length against dims under the element type the space fixes. This extension defines no space.

6. Reasons

7. Conformance

live-conformance/machine-coding/ holds, in vectors.json, descriptor events with the outcome each was written to have: the descriptor read, with the MSF record that carries its event, or its refusal. Every reason of §6 is given. wai_machine_coding_vectors verify and wai-rs/tests/machine_coding.rs replay it; machine_coding_verify.py, written from this text and live-events and sharing no code with wai-rs, reads each record with its own readers, writes each MSF record again, requires every mutated copy of the corpus to fail, and drops each of its own checks in turn.

8. Security and privacy considerations

A descriptor is as trustworthy as its signer. Its signature shows which machine made the statement and that the statement was not changed; it does not show that the named model was the one run, nor that the model is right. A reader that acts on labels admits models by digest (§4 step 8) and signers by the room’s policy (live-events §5).

Labels and regions can describe people. A descriptor says where in a picture something is and what a model called it; a publisher that sends descriptors of people sends what a viewer could not have computed without the model, and the basis on which that may be processed is not this extension’s to grant (jwp-receipts §2.4.6 states one). An embedding is carried by digest so that a sink which does not fetch it does not hold it.