IF Commons GitHub

Formats

0001Draft

0001-formats · owner if · domain commons

Text formats

FR-001checkpassing

Formats MUST read JSON5 1.0 (comments, trailing commas, unquoted ECMAScript keys, single-quoted and continued strings, hexadecimal, Infinity, NaN, a leading +, a leading or trailing decimal point) into a typed Json value, or say the line and column where the text stops being JSON5; MUST read every JSON text as JSON.parse does; MUST define a key named __proto__ as a member, never as the prototype; and MUST write a value as a person edits it (a // comment at its head, bare keys where they can be, two spaces of indent, a comma after every member, a short list of plain values on one line) that it reads back as the same value, -0, NaN and the infinities included.

Enforced (check): .github: cd commons/formats && bun test suite/ — formats.test.ts: round trips with JSON and with the writer, the specification's examples, __proto__, refusals with line and column

Proved by: formats/suite/formats.test.ts

Written in: formats/src/json5.ts

Tests: passed: suite/formats.test.ts a JSON value written as JSON5 reads back the same; JSON read as JSON5 is what JSON.parse g; passed: suite/formats.test.ts the examples of the JSON5 specification are read; passed: suite/formats.test.ts the writer: a comment at the head, keys bare where they can be, short lists of plain value; passed: suite/formats.test.ts what is not JSON5 is refused with its line and column

Cited by: Formats, json5

FR-002checkpassing

Formats MUST read YAML 1.2 as data: block mappings and sequences (a sequence at its key's indentation, a mapping begun on a dash), flow collections over lines, plain scalars folded over lines, single- and double-quoted scalars with every YAML 1.2 escape, literal and folded block scalars with their indicators, comments and one document; resolve plain scalars by the core schema or, asked, the failsafe one; refuse, with the line, column and reason, anchors and aliases, tags, complex keys, directives, a second document, a key given twice and a tab for indentation; and write a value as block YAML it reads back as the same value.

Enforced (check): .github: cd commons/formats && bun test suite/ — data.test.ts: random values written and read back, a front matter with every construct, failsafe, every refusal with its place and reason

Proved by: formats/suite/data.test.ts

Written in: formats/src/yaml.ts

Tests: passed: suite/data.test.ts a value written as YAML reads back as the same value; passed: suite/data.test.ts front matter, configuration and the scalar styles are read as YAML 1.2 reads them; passed: suite/data.test.ts the writer: block style, text plain only where it reads back as the same text; passed: suite/data.test.ts what is ambiguous or unsafe to read is refused with its line and column

Cited by: Formats, yaml

FR-003checkpassing

Formats MUST read TOML 1.0.0 whole (bare, quoted and dotted keys; tables, arrays of tables and inline tables; the four kinds of string; decimal, hexadecimal, octal and binary integers; floats with inf and nan; the four kinds of date and time, kept as their text with their kind) and hold to its rules (no key or table defined twice, no inline table or static array extended, no table made by dotted keys reopened by a header), refusing an integer JavaScript cannot hold exactly rather than rounding it and saying the line and column of what is wrong; and MUST write a table as TOML it reads back as the same table.

Enforced (check): .github: cd commons/formats && bun test suite/ — data.test.ts: random tables written and read back, TOML 1.0's examples, every rule broken and named with its place

Proved by: formats/suite/data.test.ts

Written in: formats/src/toml.ts

Tests: passed: suite/data.test.ts a table written as TOML reads back as the same table; passed: suite/data.test.ts the examples of TOML 1.0 are read as the language defines them; passed: suite/data.test.ts what breaks a rule of TOML is refused with its line and column

Cited by: Formats, toml

FR-004checkpassing

Formats MUST write XML from a typed tree with every value escaped and the characters XML 1.0 forbids removed, and MUST read XML (elements, attributes in either quote, the five named entities and character references, CDATA, comments, processing instructions, the declaration) into the same tree, checking that each element is closed by its own name and each attribute is given once, saying the line and column of what is wrong, and refusing a document type with an internal subset so that no entity is ever expanded.

Enforced (check): .github: cd commons/formats && bun test suite/ — formats.test.ts: random trees written and read back, entities, CDATA, prefixes, the finders, refusals with line and column, no entity expansion

Proved by: formats/suite/formats.test.ts

Written in: formats/src/xml.ts

Tests: passed: suite/formats.test.ts a tree written and read back is the same tree; passed: suite/formats.test.ts the reader: entities, references, CDATA, comments, prefixes, white space; the finders; passed: suite/formats.test.ts the writer escapes every value and drops what XML forbids; passed: suite/formats.test.ts what is not well formed, or would expand an entity, is refused with its line and column

Cited by: Formats, xml

FR-005checkpassing

Formats MUST read and write CSV by RFC 4180 (any one-character separator, quoted fields with doubled quotes and line breaks, CRLF or LF), refuse a quote in an unquoted field and a quoted field left open with the line and column, give records by a header's names (refusing a record of another length), and read back what it writes.

Enforced (check): .github: cd commons/formats && bun test suite/ — data.test.ts: random records written and read back with each separator and line ending, RFC 4180's cases, refusals

Proved by: formats/suite/data.test.ts

Written in: formats/src/csv.ts

Tests: passed: suite/data.test.ts records written as CSV read back as the same records; passed: suite/data.test.ts RFC 4180: quotes, doubled quotes, line breaks in fields, a header; what breaks it is named

Cited by: csv, Formats

FR-006checkpassing

Formats MUST write data as TypeScript object literals and read back exactly that subset without running it, as a Result.

Enforced (check): .github: cd commons/formats && bun test suite/ — ts-literal.test.ts: literals written and read back, keys, comments, refusals

Proved by: formats/suite/ts-literal.test.ts

Written in: formats/src/ts-literal.ts

Tests: passed: suite/ts-literal.test.ts a literal written and read back is the same value (round trip); passed: suite/ts-literal.test.ts keys bare where they can be; comments and trailing commas read; anything else refused

Cited by: ts-literal

FR-007checkpassing

tsLiteral MUST take any value whose type is shaped as JSON, checked by the compiler at every depth: an interface or a type with no index signature, optional properties (left out where undefined), readonly arrays and tuples; and MUST NOT compile for a value holding a function, a Date, a Map, a bigint, a symbol, unknown, or undefined anywhere but as an optional property's value.

Enforced (check): .github: cd commons/formats && bun test suite/ — ts-literal.test.ts: tsLiteral takes an interface with no index signature, refuses a Date, a function, undefined in an array (type-level, with negative controls)

Proved by: formats/suite/ts-literal.test.ts

Tests: passed: suite/ts-literal.test.ts tsLiteral takes any type shaped as JSON and refuses one that is not

Cited by: ts-literal

Binary formats

FR-008checkpassing

Formats MUST encode and decode base64 and base64url by RFC 4648 (base64url written without padding and read with or without it), refuse rather than skip what is not of the alphabet asked for, and read back what it writes.

Enforced (check): .github: cd commons/formats && bun test suite/ — der.test.ts: RFC 4648's vectors in both alphabets, random bytes read back, malformed text refused

Proved by: formats/suite/der.test.ts

Written in: formats/src/base64.ts

Tests: passed: suite/der.test.ts bytes read back as written, in both alphabets, large ones too; passed: suite/der.test.ts RFC 4648's test vectors, both alphabets

Cited by: base64, Formats

FR-009checkpassing

Formats MUST encode ASN.1 DER (X.690) in its one form (minimal lengths and two's complement integers, negative ones too; OIDs; SEQUENCE; a SET OF sorted; strings, booleans, NULL, explicit and implicit tags) and read it as a Result that says the byte where the input stops being DER (an indefinite or non-minimal length, a high tag number, content past the end, a non-minimal integer or OID arc), reading back integers, OIDs and nested structures as written.

Enforced (check): .github: cd commons/formats && bun test suite/ — der.test.ts: X.690's encodings, random integers, OIDs and structures read back, every malformed input named at its byte

Proved by: formats/suite/der.test.ts

Written in: formats/src/der.ts

Tests: passed: suite/der.test.ts integers, OIDs and nested sequences read back as written; passed: suite/der.test.ts the encodings X.690 and RFC 5652 give; passed: suite/der.test.ts what is not DER is refused with the byte where it goes wrong

Cited by: der, Formats

FR-010checkpassing

Formats MUST write archives byte for byte from the files alone (a ustar tar, a gzip of stored blocks, a zip of stored entries, in the byte order of their paths, with fixed times and modes); where asked, a zip of deflated entries (method 8, through the web standard CompressionStream("deflate-raw"), an entry stored where deflate would not shrink it), the same bytes for the same files on one runtime and the same files unzipped on every runtime (the deflaters of Node and Bun may choose different bytes, and do); and read them (tar with pax and GNU long names, gzip and zip entries stored or deflated through DecompressionStream), checking each header checksum, CRC-32 and size, and refusing every path that is absolute, has a .., . or empty segment, a backslash, a drive letter or a NUL.

Enforced (check): .github: cd commons/formats && bun test suite/ — archive.test.ts: round trips of tar, zip and gzip, fixed bytes, unsafe and damaged archives, deflated zip read and written, pax and GNU names

Proved by: formats/suite/archive.test.ts

Written in: formats/src/archive.ts

Tests: passed: suite/archive.test.ts a zip of deflated entries: the files back, the same bytes twice, stored where deflate woul; passed: suite/archive.test.ts tar then untar gives the files back, the same bytes every time; passed: suite/archive.test.ts zip then unzip, and gzip then gunzip, give the bytes back; passed: suite/archive.test.ts a damaged archive is refused: a tar checksum, a zip CRC, a truncated stream; passed: suite/archive.test.ts a long path is split into the ustar prefix and name; one that cannot be is refused and 6 more

Cited by: archive, Formats

FR-011checkpassing

Formats MUST read a tar archive as a stream (untarStream), as @std/tar's UntarStream does: a TransformStream from the archive's bytes, gzipped or not, to its entries, each with its header and its bytes as a stream, with the checks of FR-010 on every header and path, so an archive larger than memory can be read.

Enforced (check): .github: cd commons/formats && bun test suite/ — archive.test.ts: untarStream equals untar, damaged headers refused

Proved by: formats/suite/archive.test.ts

Tests: passed: suite/archive.test.ts untarStream gives the entries and bytes untar gives, gzipped or not, however the bytes are; passed: suite/archive.test.ts untarStream refuses a damaged header, a truncated archive, an unsafe path and a damaged gz; passed: suite/archive.test.ts a large gzipped archive fed in small chunks is read an entry at a time, a body cancelled s; passed: suite/archive.test.ts cancelling the entries cancels the bytes they are read from

Cited by: archive

FR-012checkpassing

Formats MUST read PNG of every colour type and bit depth PNG defines (palette with tRNS, greyscale and RGB with tRNS, 16-bit samples), with every filter type, into 8-bit RGBA; check every chunk's CRC; refuse, by a typed error, a wrong signature, truncation, malformed headers, an out-of-range palette index, an unknown critical chunk and interlacing; refuse a picture of more than 2^24 pixels before allocating or inflating anything, and stop inflating at the size the header declares; write 8-bit RGB and RGBA PNG that it reads back as the same pixels; and give ICO files of PNG frames, resizing (Lanczos, on premultiplied alpha), compositing, thumbnail sizes, colour parsing and WCAG luminance.

Enforced (check): .github: cd commons/formats && bun test suite/ — png.test.ts: random pictures written and read back, hand-built PNGs of every filter, colour type and depth with tRNS, every corrupt and hostile file refused by its kind, resize and composite laws, ICO round trips

Proved by: formats/suite/png.test.ts

Written in: formats/src/png.ts

Tests: passed: suite/png.test.ts a fully transparent overlay, wherever it is laid, changes nothing; passed: suite/png.test.ts a picture written and read back is the same picture, as RGBA; passed: suite/png.test.ts a resize gives the size asked, and a picture of one colour stays that colour; passed: suite/png.test.ts an icon file written and read back holds one frame per size that fits, the full-size one t; passed: suite/png.test.ts an opaque overlay replaces the pixels it covers and 12 more

Cited by: Formats, png

FR-013checkpassing

Formats MUST write bytes as lower-case hexadecimal, two digits each (toHex), and read hexadecimal text as bytes (fromHex), refusing an odd length or a character that is not a digit, so that the digests of crypto are written through it.

Enforced (check): .github: cd commons/formats && bun test suite/ — hex.test.ts: toHex and fromHex round trip as a property, an odd length and a non-digit refused

Proved by: formats/suite/hex.test.ts

Written in: formats/src/hex.ts

Tests: passed: suite/hex.test.ts fromHex refuses an odd length and a character that is not a hexadecimal digit; passed: suite/hex.test.ts toHex writes two lower-case digits for each byte and fromHex reads them back

FR-014checkno test run

Formats MUST read CBOR (RFC 8949: the definite-length forms of every major type, tags, the simple values and half, single and double floats) into a Result whose error says what and at which byte (an indefinite length, a reserved value, text that is not UTF-8, a map key that is neither an integer nor a text string, a key given twice, nesting deeper than 64 levels, bytes left over), and write it in the core deterministic encoding (RFC 8949 §4.2.1: the shortest heads, definite lengths, map keys sorted by their encoded bytes), so that the same value is the same bytes every time (cbor, taken with COSE and WebAuthn from 0014-webauthn FR-001).

Enforced (check): .github: cd commons/formats && bun test suite/ — cbor.test.ts: RFC 8949 Appendix A's examples, the deterministic encoding, every malformed input at its byte

Written in: formats/src/cbor.ts

Cited by: semver

Names and patterns

FR-015checkpassing

Formats MUST parse, print, order and bump semantic versions by semver.org 2.0.0 (§11 precedence, the build ignored, a leading v accepted) and compare the dotted versions programs print, part by part (compareDotted, atLeast, versionIn).

Enforced (check): .github: cd commons/formats && bun test suite/ — semver.test.ts: semver.org's example order, round trip, order laws, bumps, dotted versions

Proved by: formats/suite/semver.test.ts, system/suite/fs-more.test.ts

Written in: formats/src/semver.ts

Tests: passed: suite/semver.test.ts a bump is above what it bumps and suffices for its change; passed: suite/semver.test.ts a version printed and parsed is the same version (round trip); passed: suite/semver.test.ts the order is reflexive, antisymmetric, transitive and total; passed: suite/semver.test.ts dotted versions compare part by part, a missing part as 0; passed: suite/semver.test.ts the order is semver.org's (§11): its own example list is sorted, and the build is ignored and 2 more

Cited by: semver

FR-016checkpassing

Formats MUST read npm's version ranges (node-semver's grammar): comparators (<, <=, >, >=, =), sets of them joined by spaces, sets joined by ||, hyphen ranges, x-ranges (*, x, X, a missing part), tilde (~, ~>) and caret (^) ranges, each made into comparators exactly as node-semver documents them (^1.2.3 as >=1.2.3 <2.0.0-0, ~0.2 as >=0.2.0 <0.3.0-0, 1.2.3 - 2.3 as >=1.2.3 <2.4.0-0); a range that is not of the grammar MUST be an error as a Result. satisfies(version, range) MUST hold where the version satisfies every comparator of one set, and a pre-release version MUST satisfy a set only where one of its comparators names a pre-release of the same major, minor and patch, unless pre-releases are asked for.

Enforced (check): .github: cd commons/formats && bun test suite/ — semver.test.ts: ranges against npm's documented desugaring and laws (caret, tilde, x-range, hyphen, union), pre-releases

Proved by: formats/suite/semver.test.ts

Tests: passed: suite/semver.test.ts a range printed and read back is the same range (round trip); passed: suite/semver.test.ts laws: caret, tilde and x-ranges are their comparators; a hyphen is >= and <=; || is or; a ; passed: suite/semver.test.ts node-semver's own cases: the versions each range includes and excludes, pre-releases among; passed: suite/semver.test.ts ranges desugar into comparators exactly as node-semver documents them; passed: suite/semver.test.ts what is not a range is an error naming it, never a range that holds every version

FR-017checkpassing

Formats MUST turn a path glob into a regular expression and match paths with it as @std/path's globToRegExp does (*, **, ?, character classes, {a,b} alternatives, a leading ! refused where it cannot mean anything), on / paths (as a Result, a leading ! refused; ? and [!…] never matching /, where @std/path's can), and fs.expandGlob MUST give the files of a folder that a glob matches, in byte order.

Enforced (check): .github: cd commons/formats && bun test suite/ — glob.test.ts: globs against @std/path's cases; expandGlob

Proved by: formats/suite/glob.test.ts

Written in: formats/src/glob.ts

Tests: passed: suite/glob.test.ts * never crosses a /, and **/ crosses any number of them; passed: suite/glob.test.ts a glob of plain characters matches itself and nothing else; passed: suite/glob.test.ts a leading ! that is not an extglob group is refused; passed: suite/glob.test.ts a table of globs matches the paths @std/path's semantics give it, and no others; passed: suite/glob.test.ts isGlob tells a glob from a path; globRoot gives the folder a walk starts from and 1 more

Cited by: glob, Formats

FR-018checkpassing

Formats MUST give the media type of a path or an extension from one table of the types the harnesses serve and publish, charset=UTF-8 with every textual type, and application/octet-stream where the extension is not known.

Enforced (check): .github: cd commons/formats && bun test suite/ — data.test.ts: the types and charsets of paths and extensions, unknown extensions

Proved by: formats/suite/data.test.ts

Written in: formats/src/media-types.ts

Tests: passed: suite/data.test.ts media types by extension, with a charset for text

Cited by: Formats, media-types

FR-019checkpassing

Formats MUST read the Link header (RFC 8288) into its links and find a rel, read Retry-After as seconds or an HTTP date, and say which methods are safe (RFC 9110 §9.2.1).

Enforced (check): .github: cd commons/formats && bun test suite/ — http.test.ts: Link headers, Retry-After in both forms, safe methods

Proved by: formats/suite/data.test.ts, formats/suite/http.test.ts

Written in: formats/src/http.ts

Tests: passed: suite/data.test.ts If-Match compares strongly and If-None-Match weakly, as RFC 9110 says; passed: suite/http.test.ts a Link header written from links reads back to the same targets and rels; passed: suite/http.test.ts Link headers are read into their links, and rel=next is found among them; passed: suite/http.test.ts Retry-After as seconds and as an HTTP date, never negative; passed: suite/http.test.ts only GET, HEAD, OPTIONS and TRACE are safe

Cited by: http, Formats

The package

FR-020checkno test run

Formats MUST depend on core alone, MUST import nothing outside its own files, MUST name no host global, and MUST be published as compiled modules with their declarations as @intellectual-frontiers/formats, with an entry for the package (.) and one for each of its modules (./yaml, ./der, ...), each with its types (commons 0032-libraries-by-concern FR-005, FR-013), so that it runs on Bun, on Node 26 and in a browser with no build step.

Enforced (check): .github: cd commons/formats && bun run verify — the package is built, inspected and packed by tools/pack.ts; tests/commons-packages.test.ts installs the tarball and runs it on Node 26 and Bun and type-checks a consumer of every entry

Written in: formats/about.ts, formats/library.ts, formats/src/mod.ts, formats/tools/check.ts, formats/tools/pack.ts

FR-021checkpassing

Formats MUST read an xz file in pure code (unxz, from @intellectual-frontiers/formats/xz): the container (one stream or several, with the zero padding between them), the LZMA2 filter, and the checks the file carries (the CRC-32 of each header and of the index, the CRC-32, CRC-64 or SHA-256 of each block's data, the sizes the headers and the index state, and the list of blocks the index holds), and MUST refuse the whole file, giving no bytes, for a check that fails, a truncation, a filter other than LZMA2 or a check it does not know.

Enforced (check): .github: cd commons/formats && bun test suite/ — xz.test.ts: files made by the system xz (every check, several blocks, several streams, uncompressed chunks), every flipped byte and every truncation refused

Proved by: formats/suite/xz.test.ts

Written in: formats/src/xz.ts

Tests: passed: suite/xz.test.ts where xz is installed, random and repetitive data it compresses reads back, and a tar it c; passed: suite/xz.test.ts every flipped byte and every truncation of small files is refused with a typed error, and ; passed: suite/xz.test.ts files made by the system xz are read back: each check, no data, stored chunks, several blo; passed: suite/xz.test.ts flipped bytes and truncations of a large file, with several blocks, are refused and none t

Cited by: Formats, xz

FR-022checkpassing

unxz MUST NOT make more bytes than its limit (a gigabyte unless the caller gives another) however few bytes it is given, MUST grow its output only as data is made and not by what a header declares, and MUST reach no file, network or program; a tar that was xz-compressed is read by untar of what unxz gives, as a gzipped one is read of what gunzip gives.

Enforced (check): .github: cd commons/formats && bun test suite/ — xz.test.ts: a file that expands to sixteen mebibytes stops at a limit of one, the output is a copy, a tar.xz untars

Proved by: formats/suite/xz.test.ts

Written in: formats/src/xz.ts

Tests: passed: suite/xz.test.ts where xz is installed, random and repetitive data it compresses reads back, and a tar it c; passed: suite/xz.test.ts a file that expands to sixteen mebibytes stops at a smaller limit, and is read whole at a

FR-023checkpassing

untar MUST give each symbolic link and each hard link of a tar as an entry of its own, of the kind symlink or hardlink, with the path it names in linkName, and linksOf MUST give the links of a list of entries as typed values (the path, whether it is a symbolic or a hard link, and its target) in the order of the archive, so that a consumer decides what to do with a link and never finds one among its files.

Enforced (check): .github: cd commons/formats && bun test suite/ — archive.test.ts: a tar with symbolic and hard links gives link entries and linksOf lists them

Proved by: formats/suite/archive.test.ts

Written in: formats/src/archive.ts

Tests: passed: suite/archive.test.ts a tar with symbolic and hard links gives each link as an entry of its own and linksOf list

Cited by: Formats, xz

Acceptance scenarios

  1. Given a YAML front matter, When it is read and written as JSON5, Then the same value comes back, per FR-001 and FR-002.
  2. Given a DER structure of integers, OIDs and sets, When it is encoded and decoded, Then the same structure comes back and a malformed input is named at its byte, per FR-009.
  3. Given the packed package installed under node_modules, When Node 26 and Bun import @intellectual-frontiers/formats and @intellectual-frontiers/formats/yaml, Then both give the same parse, per FR-020.
  4. Given a consumer that imports every entry of the package, When it is type-checked against the installed tarball, Then it compiles, per FR-020.

Out of scope

Edge cases

  • A YAML or TOML document with a refused construct: refused with its line and column, per FR-002 and FR-003.
  • Hexadecimal of odd length: refused, per FR-018.

Assumptions

  • ECMAScript 2024 and the web standards are all that the code needs (CompressionStream and DecompressionStream for gzip and deflate).

Open questions

  • None.

Key entities

  • Format - a text or binary form, read into a typed value or an error with its place, and written back deterministically.

Success criteria

SC-001success criterion

A harness reads and writes its files with these and with no outside package.

Review & acceptance checklist

  • [x] Every requirement is testable (MUST / MUST NOT), not aspirational
  • [x] No company fact is asserted here
  • [x] Every open item is marked, not silently decided

Cited by

archive, base64, csv, der, glob, http, Formats, json5, media-types, png, semver, toml, ts-literal, xml, xz, yaml

Titles, headings, exports, tables, columns and requirements. Esc closes.