WAI Extension: Task Fidelity
Mirrored from the canonical text at commit 498e6b88 ().
Status: Draft. Names the contract for media whose correctness is a downstream model’s behaviour, not a byte comparison — the class
determinism-tierscannot express. Sibling to that file, not a tier inside it: the tiers there are reproducibility guarantees a sink can check from the bytes alone, and a task claim is not one. No new wire format; a claim shape + an evidence grading over §7, modelled onenergy-measurement’s acquisition class. §5 registers one receipt class,split-computation(jwp-receipts §2.4.4): what each side of a network cut in two ran, and the energy it spent. Its reference is thesplitmodule ofwai-rs’sclaim_receiptand themachine_codingmodule (featuremachine_coding); its corpus islive-conformance/machine-coding/, and its second implementationlive-conformance/machine_coding_verify.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.
1. Why a second axis
determinism-tiers.md answers one question: does the sink reproduce the same
bytes? A growing class of coded media is not answering that question at all.
Three independent arrivals, none of them WAI’s:
- Video coding for machines. ISO/IEC 23888-2 (MPEG-AI Part 2; not published: its project page gives its status as ongoing) codes video and descriptors extracted from it, jointly optimised for bitrate and for machine-task performance after decoding. Its fidelity target is a task’s accuracy, not a pixel comparison.
- Feature coding for machines. ISO/IEC 23888-4 (MPEG-AI Part 4; not published, status ongoing) codes intermediate feature tensors from inside a machine-vision network — the origin runs the first N layers, ships the tensor, the sink runs the remainder. No pixels are reconstructed at any point, so there is no medium to hash.
- Vendor framing. At least one vendor markets compression as safe for machine consumption when the change it induces in a downstream model’s behaviour is statistically indistinguishable from the variance that model already tolerates. No standards body, no designation — and the vendor’s own material states the industry lacks a shared framework for the claim, which is what makes the seam worth naming rather than ceding.
WAI registers the two parts as candidates, wai.video.vcm and wai.tensor.fcm (SPEC
§5; neither is emitted or dispatched, and neither has a probe:
format-probes §19), but a registration cannot state what conformance
means for them. ReplicateClass::Behavioral is a disclaimer, not a criterion.
Task-equivalent but not bit-equal is unstateable, so an implementation either
overclaims a reproducibility tier it does not meet or says nothing at all.
The distinction that makes this a separate file. A reproducibility tier is
checkable by the sink from the bytes it received. A task-fidelity claim is not:
checking it requires a model, a corpus, and a test run. Listing the two together
would blur the one property determinism-tiers.md currently keeps clean — that
its tiers are verifiable without leaving the object.
2. The claim
A task-fidelity claim is a tuple, and it is meaningless with any element missing:
| element | meaning |
|---|---|
model | content address of the network the claim is about — the identity under which δ was observed |
test | content address of the task, corpus and metric the claim was measured under |
tolerance | τ — the behavioural change the claimant asserts is acceptable, in the metric test names |
observed | δ — the behavioural change actually measured under test |
evidence | the class in §3 |
A claim asserts δ ≤ τ under test, for model. It asserts nothing about any
other model, any other corpus, or any metric test does not name.
- A claim MUST carry all five elements. A tolerance without an observation is a target, not a claim, and MUST NOT be sealed as one.
modelandtestMUST be content addresses, not names. “Tested on a standard detector” is unfalsifiable; a hash is not.- A claim MUST NOT be restated for a different model or corpus without a new observation. Behavioural tolerance does not transfer by assertion.
3. Evidence class (REQUIRED)
What separates the rungs is provenance and scope, not accuracy — the same
principle as energy-measurement.md §2, and for the same reason: a confidently
formatted δ says nothing about whether anyone ran the test.
| class | meaning | requires |
|---|---|---|
Measured | δ was observed by running test against model on the stated corpus. | retained artifacts sufficient to re-run |
Transferred | δ was observed under a different model or corpus and is argued to carry over. | the source claim, and the argument, both named |
Asserted | δ is claimed without an observation. | — |
- An implementation MUST NOT promote a claim’s evidence class in transit, as
§2 of
energy-measurement.mdforbids for acquisition class. - A verifier that cannot resolve
modelandtestMUST treat the claim asAssertedregardless of the class stated, and SHOULD say so rather than discard the claim silently. TransferredMUST NOT be used to cross a modality or a task family. A detector’s tolerance is not a segmenter’s.
4. What the claim is, and is not
- It is not a reproducibility tier, and MUST NOT be presented as one. A
capability carrying a task-fidelity claim MUST NOT thereby claim
entropy-consistencyordecode-equivalence; those are separate assertions about separate properties, and a capability may hold both, one, or neither. - It is not sink-checkable from the bytes. This is the defining limitation, and stating it is the point: a receipt bearing a task-fidelity claim carries a graded assertion, not a verification, and a relying party that treats it as the latter has misread it.
- It is not a quality metric. δ is behavioural change in a named downstream task, not perceptual fidelity, and the two MUST NOT be reported in the same field.
- It does not license degradation elsewhere. A task-fidelity claim says
nothing about energy, and a payload an encoder altered to save energy, such as
one produced in answer to a decoding-operation-reduction request, is still bound
by
determinism-tiers.md§2.
5. The split-computation receipt
Feature coding for machines makes the work partition between an origin and a sink a
parameter of the wire: a network is cut at a layer, one party runs the layers before the
cut, a tensor crosses, and another party runs the rest. That breaks an assumption of
jwp-receipts, where joules_micro is one figure for producing an
object by one producer. A receipt of a split computation needs three things: the energy
spent before the cut, the energy spent after it, and an identity of the network under
which the two halves provably belong together.
This section gives them as a pair of claims of one class, split-computation
(a claim of jwp-receipts §2.4.3, registered here as its §2.4.4 provides). Each party signs its own half: a claim carries the authority of
its signer and no more (jwp-receipts §2.4.4), and neither party read the other’s meter.
The side that runs the layers before the cut is the head; the side that runs the rest
is the tail. Both claims are about the tensor object, the object that carried the
feature tensor across the cut, by its content hash (jwp-receipts §2.1).
A task-fidelity claim over a split computation MUST state the cut point, and MUST NOT report a single joule figure as if one producer had done all the work.
5.1 The body
A split-computation claim’s subject_type is object and its subject the tensor
object’s content hash. Its body has exactly these members, read at jwp-receipts §3.1 step
2 in this order:
| member | type | meaning |
|---|---|---|
side | string | head or tail. |
network | hash | The network’s identity (below). |
head_set | hash | The digest of the head’s parameter set: the SHA-256 a pin of that set carries (SPEC §3.1). |
tail_set | hash | The digest of the tail’s parameter set, likewise. |
cut | object | The cut: { "layer", "dtype", "shape" }. layer is the number of layers the head runs, an integer of at least 1; dtype the element type of the tensor that crosses, one of u8, i8, u16, i16, i32, f16, f32, f64; shape its shape, an array of 1 to 8 integers, each at least 1. |
consumed | hash | The content hash of what this half took in: for a head, the source object; for a tail, the tensor object. |
produced | hash | The content hash of what this half gave: for a head, the tensor object; for a tail, its result’s bytes. |
ran_on | string | The fabric this half ran on: cpu, gpu, media or npu (energy-measurement §6.3). |
energy | object or null | The energy this half spent, written as jwp-receipts §2.4.12 writes a session’s energy: { "joules_micro", "energy_provenance", "energy_uncertainty", "energy_counter_filtering", "energy_calibration_ref" }. null when this half was not metered, never a figure of 0 (energy-measurement §2). |
attests | string | MUST be "computed". Any string passes step 2; another value is refused at step 4. |
side, dtype and ran_on are read against their enumerations at step 2 (field_type),
and so are layer and shape against their bounds. A hash is 32 bytes in lowercase
hexadecimal. The body has no member for a total, and one that adds any is refused
(unknown_member).
The network identity is
network = SHA-256("wai:split-network\x01" ‖ JCS({"cut": cut, "head_set": head_set, "tail_set": tail_set}))
over the RFC 8785 canonical form of the object of exactly those three members, cut as the
body writes it and each set in lowercase hexadecimal. It names the two parameter sets in
their order and the interface between them, so a head split for one tail does not name the
network of another, and exchanging the sets names a different network.
5.2 Class rules
At jwp-receipts §3.1 step 4, in this order:
attestsiscomputed(split_not_computed).networkis §5.1’s digest of the body’s owncut,head_setandtail_set(split_network_mismatch).- The
subjectis the tensor object: a head’sproduced, a tail’sconsumed(split_subject_mismatch). consumedandproduceddiffer: a half that gives back what it took ran nothing (split_no_work).- Where
energyis notnull, itsjoules_microis not 0 (energy_zero_figure), and its class is admitted as jwp-receipts §2.4.5 rule 2 admits one (energy_unknown_acquisition,energy_legacy_acquisition,energy_acquisition_evidence).
The energy figure of a half that is metered is an item of its own for signing authority
(jwp-receipts §7.5): its signer needs split-computation and the energy authority of the
figure’s class.
5.3 The pair
At step 6 a verifier checks one half against the other, each already through steps 1 to 5, and against the parameter sets it holds. In order:
- The other claim verifies through steps 1 to 4 and is of this class (
context_invalid). - One is a head and the other a tail (
split_side_mismatch). - Their subjects are equal: the tail consumed the tensor the head produced
(
split_tensor_mismatch). - Their
cuts are equal, member for member (split_cut_mismatch). - Their
network,head_setandtail_setare equal (split_halves_do_not_compose). - Where the verifier holds the head’s parameter set or the tail’s, the digest it computes
over the set it holds is the one the claims name (
split_set_mismatch).
A pair that passes composes: both parties stated one network, cut at one place, and one tensor passed between them. A verifier that holds one half alone has no pair, and reports the half as verified through step 5 and nothing about the other side. Nothing here shows that either half computed what its network computes: that needs the parameter sets, the tensor and a run, and a claim about the task’s result is §2’s.
5.4 The account
A pair that composes gives an account, and the account is all a verifier reports of the pair’s energy:
- for each side, its signer, the fabric it ran on, and its
energyas its claim wrote it; - by class: for each acquisition class under which a side was metered, strongest first (energy-measurement §2), the sum of the figures of that class and the sides it covers. Two figures of one class are added; figures of two classes are never added, and the account has no total;
- the sides that were not metered, named. An unmetered side is listed here and has no row: it is never a row of 0 J;
- its coverage:
wholewhen both sides were metered,partialwhen one was,nonewhen neither.
A reader MUST NOT present a partial account’s figures as the energy of the whole
computation, and MUST NOT add rows of different classes into one figure. A sum of a
class’s figures past 2^53 − 1 has no account (context_invalid).
5.5 Reasons
| reason | raised at |
|---|---|
split_not_computed | §5.2 rule 1 |
split_network_mismatch | §5.2 rule 2 |
split_subject_mismatch | §5.2 rule 3 |
split_no_work | §5.2 rule 4 |
split_side_mismatch | §5.3 step 2 |
split_tensor_mismatch | §5.3 step 3 |
split_cut_mismatch | §5.3 step 4 |
split_halves_do_not_compose | §5.3 step 5 |
split_set_mismatch | §5.3 step 6 |
5.6 What it does not say, and conformance
The class is independent of how the tensor is coded. wai.tensor.fcm, registered for
ISO/IEC 23888-4, is a candidate with no pinned payload and no probe
(format-probes §19), so no object of it is emitted; the claims name the
tensor object by its content hash and say nothing of its bytes. A pair states what each
party says it ran and spent. It is not a measurement of the network’s accuracy, and it does
not make either half’s figure the other’s.
The corpus live-conformance/machine-coding/ holds
network digests, single claims verified or refused at their step, and pairs with the
account or the refusal each was written to have; wai_machine_coding_vectors verify and
wai-rs/tests/machine_coding.rs replay it, and machine_coding_verify.py, written from
this text and sharing no code with wai-rs, replays it with claim_verify.py for steps 1
to 4, requires every mutated copy to fail, and drops each of its own checks in turn.
6. Framing
Byte-comparability is becoming one fidelity contract among several rather than the only one. WAI’s position is unchanged by that — it names contracts and grades the evidence behind them, and does not adjudicate which contract a domain should want. What would be wrong is silence: a ratified class of media arriving with a conformance notion the container cannot state, described instead in the vocabulary of a guarantee it does not make.