WAI Extension: Conformance Results

Mirrored from the canonical text at commit 260cbd09 (2026-11-08).

Status: Draft. Defines the record conform-result/1: the signed outcome of running one of WAI’s conformance corpora against an implementation, whoever wrote it. The record names the corpus by digest, the implementation by what ran, and each case by its expected and its actual answer; its summary and its verdict follow from its cases; and it is signed with Ed25519 over its RFC 8785 form. A verifier checks the record as written, offline: that it is well formed, that the signature holds, and that the summary and the verdict are what the cases say. No new transport. The reference implementation is the conform_result module of wai-rs (feature conform_result), which wai conform writes records with and wai verify checks them with; the corpus is conform-result-conformance/, and its second implementation is conform-result-conformance/conform_result_verify.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

A section number in this text is this extension’s unless it names another document.

1. Scope

In scope: the record (§2), its signing input and signature (§3), what a verifier checks and in what order (§4), what a relying party that admits an implementation on a record checks besides (§4.1), and the codes a verifier refuses a record with (§5).

Out of scope: how a run reaches an implementation (a command line, a URL), which corpora and suites exist, and what makes an implementation conformant beyond the run a record states. A record states what one run observed; it is not a certificate.

2. The record

A conformance result is one JSON text (RFC 8259) in UTF-8: an object with exactly the members below, each of the type given, and no member named twice at any level. Every number in it is an integer from 0 to 2^53 − 1 written in decimal digits, without a sign, a fraction or an exponent. Every other number, and every member not listed, makes the record malformed.

A writer MUST set summary.cases to the number of cases, summary.passed to the number whose pass is true and summary.failed to the number whose pass is false; and MUST set conformant to true exactly when there is at least one case and every case passed.

2.1 The corpus digest

The corpus digest is sha256: followed by 64 lowercase hexadecimal digits: the SHA-256 of the concatenation of one line for every corpus file the run read, each line the file’s SHA-256 in lowercase hexadecimal, two spaces, the file’s path relative to wai/ with / between its segments, and a line feed; the lines sorted in ascending byte order, and a line that occurs twice taken once. A party that holds the corpus can compute the digest and compare it; a record made against another corpus has another digest.

2.2 The implementation

For the command adapter, files holds the program the command line runs, as found on the runner’s PATH, and every argument of the command line that is a file, each with the SHA-256 of its bytes when the run started. For the http adapter it is empty, and impl is the URL. These name what ran on the runner’s machine; they are statements of the runner, covered by the signature like every other member.

3. The signature

The signing input is the 18 bytes wai:conform-result followed by the byte 0x01, followed by the record without its sig member, serialised as RFC 8785 specifies (its members sorted, no insignificant whitespace). signer_pubkey is part of the signed record.

sig is a pure Ed25519 signature (RFC 8032 §5.1) over the signing input by the key signer_pubkey. It is verified by WAI’s strict rule (SPEC §7 Signatures): the key canonical and not of small order, S < L, R not of small order, and the cofactorless equation compared byte for byte.

4. Verification

A verifier checks a record in this order and refuses it at the first step that fails, with the code that step names (§5):

1. It reads. The bytes are one JSON text in UTF-8 whose value is an object, and no object in it names a member twice. Otherwise malformed.
2. It is this format. The member wai is the string conform-result/1. Otherwise unknown_format.
3. Its members. It has exactly the members of §2, each of the type and form §2 gives. Otherwise malformed.
4. Its signature. sig verifies over the signing input of §3 under signer_pubkey, by the strict rule. Otherwise signature_invalid.
5. Its summary. summary.cases, summary.passed and summary.failed are what its cases count (§2). Otherwise summary_mismatch.
6. Its verdict. conformant is true exactly when there is at least one case and every case passed. Otherwise conformant_mismatch.

A record that passes every step is accepted, and its verdict is its conformant. A verifier writes its outcome as accept conformant=true, accept conformant=false, or reject step=<n> reason=<code>. A record that says its run was not conformant is a genuine record of that run, and is accepted as one.

4.1 Admitting an implementation

Acceptance shows that a record is intact and consistent; it admits nothing (§8). A relying party that admits an implementation on a record, such as a certification of a sink or a gate in a build, holds the corpus the record names, and applies these rules in this order. It refuses the implementation at the first rule that fails, with the code that rule names:

1. The record is accepted (§4). Otherwise the code of the step of §4 that refused it.
2. Its conformant is true (not_conformant). A record of a run that did not pass is genuine, and admits nothing.
3. Its kind and suite name a suite whose corpus the relying party holds and reads (suite_unknown).
4. Its corpus.digest is the corpus digest (§2.1) of that suite’s files as the relying party holds them, and its cases are that suite’s, each once, by name (corpus_mismatch). A run against another corpus, an earlier revision’s or one changed to suit an implementation, admits nothing, and neither does a whole-corpus run of a program over those files, whose one case states its exit status and no case’s answer.

These rules read the record and the corpus the relying party holds, and nothing that depends on the runner, so two relying parties that hold the same corpus admit the same records. They do not say who ran the run: a relying party that requires a known runner also requires key to be supplied and confirms signer_pubkey (§6). wai verify --conformant applies them, against the corpus of a checkout of the standard.

5. Reasons

The codes a verifier refuses a record with, each with the step of §4 that raises it (§4.1 names its own):

6. What a signature shows

A record signed with a key made for its run ("key": "ephemeral") shows that the record is the one the run wrote: no case, count, digest or name was changed after it was signed. It does not say who ran it. A record signed with a key the runner holds ("key": "supplied") says, in addition, that the holder of that key ran it; which party that is, a relying party learns as it learns any key, for instance from a key log (jwp-receipts §7). Neither makes the implementation’s other behaviour conform, and neither makes the run’s statements true: a runner can run a corpus other than the one it names, and the digest of §2.1 is how a party holding the corpus checks which one ran.

7. Conformance

conform-result-conformance/ holds records signed with a published test key: accepted ones (conformant, not conformant, with no cases, from either adapter and either kind of key) and refused ones (a case’s outcome changed after signing, the corpus digest changed, the summary alone changed, a signature changed in one bit, an S past L, a key of small order, a summary that disagrees with the cases though signed (each of its three counts wrong alone, and two at once), a verdict that disagrees with them, an unknown version, a member missing, a member added, a member named twice, a count with a fraction, a digest out of form, bytes that are not JSON), each with the outcome §4 gives it. wai_conform_result_vectors verify (in wai-rs, feature conform_result) and the independent conform_result_verify.py, which shares no code with wai-rs, reach every vector’s outcome.

The rules of §4.1 are held by wai-rs’s tests/cli_conform_sinks.rs, over records of runs of the native sink, through its module and its C interface, and of the score decoder: each is admitted; a sink that presents an object whose digest does not match is refused not_conformant; the same sink run against a copy of the corpus changed to suit it is admitted by a party holding that copy and refused corpus_mismatch against WAI’s; and a record of a suite the build does not run is refused suite_unknown. Rule 4’s cases are held by tests/cli_conform.rs: a whole-corpus run of a program that exits 0, whose one case is the corpus, is refused corpus_mismatch, and a run of the same suite’s cases is admitted. §4.1 has that reference implementation only: this corpus holds no admission vectors, and conform_result_verify.py does not apply its rules.

8. Security and privacy considerations

• What a record reveals. The command line and the paths of the files it names are the runner’s, and can name a user or a machine; the URL names where the implementation ran. A runner SHOULD run from paths and addresses it is willing to publish with the record.
• Accepted is not trusted. Acceptance shows the record is intact and consistent. Whether the corpus it names is WAI’s, and whether its signer is a party to rely on, a relying party decides from the digest (§2.1) and from what it knows of the key (§6).
• Bounded reading. A verifier reads one JSON object and its members once. It MAY refuse a record larger than it is prepared to read, as malformed.