Conformance runs: `wai conform`

wai conform <kind> --suite <suite> --impl <command or URL> runs one of WAI’s conformance corpora against an implementation, yours or anyone’s, and writes the outcome as a signed record. Every case’s expected answer is the corpus’s own: a verdict, a digest or a reason the corpus states, set from the specification’s text.

wai conform --list                                                    # every kind and suite
wai conform verifier --suite claim --impl "my-verifier verdicts {corpus}"
wai conform decoder --suite score --impl "my-renderer {in} {out}"
wai conform origin --suite http --impl http://127.0.0.1:8380/
wai verify claim.wai-conform.json                                     # the signed result, checked

It runs in a checkout of the standard, which holds the corpora; --root <checkout>/wai names one elsewhere.

Kinds and suites

Kind	Suite	Corpus	A case is	It passes when
verifier	`claim`	`claim-conformance/`	One vector	The verdict is the one declared: `accept link_hash=<hex>`, `accept ancestor=<hex>` or `reject step=<n> reason=<label>` (jwp-receipts §3.1)
verifier	`jwp`	`jwp-conformance/`	One vector	The verdict is the one declared (jwp-receipts §2–§6)
verifier	`signer`	`signer-conformance/`	One vector	The verdict is the one declared (jwp-receipts §7)
verifier	`conform-result`	`conform-result-conformance/`	One record	The outcome is the one declared: `accept conformant=<true or false>` or `reject step=<n> reason=<code>` (conformance-results §4)
decoder	`score`	`score-conformance/`	A `WSCR` score	The rendered `i16le` PCM has the declared SHA-256
packager	`policy`	`pack-conformance/`	A packaging policy	It is admitted with the declared digest (`ok <digest>`), or refused with the declared reason (packaging §3, §12)
origin	`http`	`http-conformance/objects/`	An object	It is served as http-delivery §5.3 and §6.2 require: its bytes, `ETag`, cache and cross-origin headers, a byte range, `HEAD`, its head proof and the preflight

Each verifier corpus’s README states its verdict lines.

Adapters

A command. --impl is a command line, run by the system shell, with these placeholders replaced by paths:

Placeholder	Replaced by
`{corpus}`	The corpus file whole: the command answers every case at once, one `name<TAB>verdict` line each (the verifier suites)
`{case}`	A file holding one case, as the corpus writes it
`{in}`	The case’s input file
`{out}`	A file to write the output to

Without {corpus}, {case} or {in}, the case or input is given on standard input; without {out}, a decoder’s output is read from standard output. The first line a command prints is its verdict; a decoder that refuses exits non-zero. A command is stopped after --timeout seconds (60 unless given).

HTTP. --impl is an http:// URL. Each case is a POST to it: a verifier’s case as JSON, a decoder’s or packager’s input as bytes. The answer is the response’s first line, or the member verdict of a JSON response; a decoder answers 200 with the output, or another status with its refusal. For an origin, the URL is where it serves the corpus’s objects: <url>/<cid>.wai and <url>/<cid>.wai.hp.

The result record

Each run writes a record, <suite>.wai-conform.json; --out names another file, and a file that exists is replaced only with --yes. The record, conform-result/1, is specified in conformance-results: its members and corpus digest (§2), its signature (§3), what a verifier checks and in what order (§4), and the codes it refuses with (§5). Its corpus, conform-result-conformance/, holds accepted and refused records with their outcomes, and wai conform verifier --suite conform-result runs a verifier of these records against it.

{ "wai": "conform-result/1", "kind": "verifier", "suite": "claim",
  "standard": "v0.4.0", "toolchain": { "name": "wai", "version": "0.3.0" },
  "corpus": { "dir": "claim-conformance", "files": 1, "digest": "sha256:…" },
  "implementation": { "adapter": "command", "impl": "my-verifier verdicts {corpus}",
                      "name": null, "version": null, "files": [{ "path": "…", "sha256": "…" }] },
  "started_at": "2026-10-27T12:00:00Z",
  "cases": [{ "name": "…", "pass": true, "expected": "…", "got": "…" }],
  "summary": { "cases": 1, "passed": 1, "failed": 0 }, "conformant": true,
  "key": "ephemeral", "signer_pubkey": "…", "sig": "…" }

corpus.digest: the SHA-256 of the lines <SHA-256 of the file> <path under wai/>, one for every corpus file the run read, sorted.
implementation: the command and the SHA-256 of every file it names (its program found on PATH, and each argument that is a file), or the URL; --name and --version add what the implementer calls it.
sig: Ed25519 (RFC 8032) by signer_pubkey, over the bytes wai:conform-result and 0x01 followed by the record without sig in RFC 8785 form. A key made for the run ("key": "ephemeral") shows the record is as the run wrote it; a key given with --key <file> (a 32-byte seed in hexadecimal) also says who ran it.

wai verify <path> checks every record under a path, in the order of conformance-results §4: that it reads, its version, its members, its signature, and that its summary and conformant are what its cases say. A record changed after it was signed is refused.

Exit status and JSON

Exit	Meaning
0	Every case passed
1	A case failed
2	A usage error
4	The record would replace a file, and `--yes` was not given
5	The kind, suite, corpus or program is not found
6	The implementation’s URL does not answer; safe to retry
7	An internal error

--dry-run lists the cases and what would run, and writes nothing. With --json it prints one JSON envelope ("op": "conformance.run"), its data the counts, the failures and the record’s path; without --json the output is text, on a terminal or not.

WAI’s continuous integration runs the corpora’s independent verifiers, second implementations written from the specification’s text and sharing no code with the reference, through wai conform, as it would run any other implementation.

Conformance runs: wai conform

Kinds and suites

Adapters

The result record

Exit status and JSON

Conformance runs: `wai conform`