IF Commons GitHub

Test evidence

0008Draft

0008-test-evidence · owner if · domain commons

Capture (runtime)

FR-001reviewpassing

The runtime MUST take the record of every test it registers from the library test (@intellectual-frontiers/test, test 0001-test FR-004), where it is specified and proved (0032-libraries-by-concern FR-010), and MUST NOT specify it here.

Enforced (review): 0032-libraries-by-concern FR-010 — kept by the library test: 0001-test FR-004

Tests: passed: suite/host-test-run.test.ts a fixture suite run into one record, its reports, a ledger and signatures, and verified; passed: suite/test-report.test.ts a test's id is the SHA-256 of its file, a NUL and its name; a text is cut to 4096 characte

Cited by: ifqs/ifqs, Quality, pure/ladder, pure/names, Test, record, test, Quality evidence

FR-002reviewpassing

The runtime MUST take the span and log record of a test with the test conventions from the library test (@intellectual-frontiers/test, test 0001-test FR-005), where it is specified and proved (0032-libraries-by-concern FR-010), and MUST NOT specify it here.

Enforced (review): 0032-libraries-by-concern FR-010 — kept by the library test: 0001-test FR-005

Tests: passed: suite/host-test-run.test.ts a fixture suite run into one record, its reports, a ledger and signatures, and verified; passed: suite/test-report.test.ts a test is a test span and a log record with the test.* conventions, the ifc.test.* attribu

Cited by: Quality evidence

FR-003reviewpassing

The runtime MUST take the installation record as the resource of a test's record from the library test (@intellectual-frontiers/test, test 0001-test FR-006), where it is specified and proved (0032-libraries-by-concern FR-010), and MUST NOT specify it here.

Enforced (review): 0032-libraries-by-concern FR-010 — kept by the library test: 0001-test FR-006

Tests: passed: suite/host-test-run.test.ts a fixture suite run into one record, its reports, a ledger and signatures, and verified; passed: suite/test-report.test.ts the installation record: runtime, system, revision, digests and role; a person's address i

Cited by: pure/record, Quality evidence

FR-004reviewpassing

The test run requirement MUST be taken from the library quality (@intellectual-frontiers/quality, quality 0001-quality-evidence FR-001), where it is specified and proved (0032-libraries-by-concern FR-010), and MUST NOT be specified here.

Enforced (review): 0032-libraries-by-concern FR-010 — kept by the library quality: 0001-quality-evidence FR-001

Proved by: telemetry/suite/otel-host.test.ts, test/suite/test.test.ts

Tests: passed: suite/host-test-run.test.ts a fixture suite run into one record, its reports, a ledger and signatures, and verified; passed: suite/test-report.test.ts IFC_TEST_RECORD lines decode back; a cut-short last line is left out; a test that started ; passed: suite/otel-host.test.ts a pipeline hands a span or a log record made elsewhere to every processor, with its source; passed: suite/test.test.ts each test is recorded in memory: id, file from the caller, requirements, risk, times, outc; passed: suite/test.test.ts without IFC_TEST_RECORD, each test goes to the installed pipeline

Cited by: host/otel, Test, record, test, Quality evidence, Test

The run (runtime and tools)

FR-005reviewpassing

The test run requirement MUST be taken from the library quality (@intellectual-frontiers/quality, quality 0001-quality-evidence FR-002), where it is specified and proved (0032-libraries-by-concern FR-010), and MUST NOT be specified here.

Enforced (review): 0032-libraries-by-concern FR-010 — kept by the library quality: 0001-quality-evidence FR-002

Tests: passed: suite/host-test-run.test.ts a fixture suite run into one record, its reports, a ledger and signatures, and verified; passed: suite/host-test-run.test.ts a runtime that is not installed is reported and fails the run; passed: suite/test-report.test.ts a run's record is one OTLP JSON document read back through the decoders: its run span, ins

Cited by: Test

FR-006reviewpassing

The test run requirement MUST be taken from the library quality (@intellectual-frontiers/quality, quality 0001-quality-evidence FR-003), where it is specified and proved (0032-libraries-by-concern FR-010), and MUST NOT be specified here.

Enforced (review): 0032-libraries-by-concern FR-010 — kept by the library quality: 0001-quality-evidence FR-003

Tests: passed: suite/host-test-run.test.ts a fixture suite run into one record, its reports, a ledger and signatures, and verified; passed: suite/host-test-run.test.ts a runtime that is not installed is reported and fails the run; passed: suite/test-report.test.ts the exit rule: a failed or cancelled test, a runner that failed or is missing, or an unpro

Cited by: test

FR-007reviewpassing

The test run requirement MUST be taken from the library quality (@intellectual-frontiers/quality, quality 0001-quality-evidence FR-004), where it is specified and proved (0032-libraries-by-concern FR-010), and MUST NOT be specified here.

Enforced (review): 0032-libraries-by-concern FR-010 — kept by the library quality: 0001-quality-evidence FR-004

Tests: passed: suite/host-test-run.test.ts a fixture suite run into one record, its reports, a ledger and signatures, and verified; passed: suite/test-report.test.ts a run's record is one OTLP JSON document read back through the decoders: its run span, ins

Cited by: Names that carry a schema

Reports (kernel)

FR-008reviewpassing

The test run requirement MUST be taken from the library quality (@intellectual-frontiers/quality, quality 0001-quality-evidence FR-005), where it is specified and proved (0032-libraries-by-concern FR-010), and MUST NOT be specified here.

Enforced (review): 0032-libraries-by-concern FR-010 — kept by the library quality: 0001-quality-evidence FR-005

Tests: passed: suite/test-report.test.ts CTRF: results.tool, summary and tests, with requirements and risk under extra; passed: suite/test-report.test.ts JUnit XML: a testsuite per file and runtime, testcases with classname and seconds, failure; passed: suite/test-report.test.ts OSCAL 1.2.3 assessment results: one observation per test and one finding per requirement; passed: suite/test-report.test.ts SARIF 2.1.0: each failed or cancelled test a result at its file, its requirement the rule; passed: suite/test-report.test.ts TAP 14: a plan, a subtest per file, ok and not ok numbered, SKIP and TODO with their reaso

FR-009reviewpassing

The test run requirement MUST be taken from the library quality (@intellectual-frontiers/quality, quality 0001-quality-evidence FR-006), where it is specified and proved (0032-libraries-by-concern FR-010), and MUST NOT be specified here.

Enforced (review): 0032-libraries-by-concern FR-010 — kept by the library quality: 0001-quality-evidence FR-006

Tests: passed: suite/test-report.test.ts the completion report: summary, installation, trace matrix, incidents, high-risk review ne

FR-010reviewpassing

The test run requirement MUST be taken from the library quality (@intellectual-frontiers/quality, quality 0001-quality-evidence FR-007), where it is specified and proved (0032-libraries-by-concern FR-010), and MUST NOT be specified here.

Enforced (review): 0032-libraries-by-concern FR-010 — kept by the library quality: 0001-quality-evidence FR-007

Tests: passed: suite/test-report.test.ts every rendering's counts are the model's, and rendering a record twice gives the same byte

Evidence (kernel and runtime)

FR-011reviewpassing

The test run requirement MUST be taken from the library quality (@intellectual-frontiers/quality, quality 0001-quality-evidence FR-008), where it is specified and proved (0032-libraries-by-concern FR-010), and MUST NOT be specified here.

Enforced (review): 0032-libraries-by-concern FR-010 — kept by the library quality: 0001-quality-evidence FR-008

Tests: passed: suite/host-test-run.test.ts a fixture suite run into one record, its reports, a ledger and signatures, and verified; passed: suite/test-report.test.ts the in-toto Test Result statement of a run, and the ledger entries of a run and of a signa

Cited by: Trust

FR-012reviewpassing

The test run requirement MUST be taken from the library quality (@intellectual-frontiers/quality, quality 0001-quality-evidence FR-009), where it is specified and proved (0032-libraries-by-concern FR-010), and MUST NOT be specified here.

Enforced (review): 0032-libraries-by-concern FR-010 — kept by the library quality: 0001-quality-evidence FR-009

Tests: passed: suite/host-test-run.test.ts a fixture suite run into one record, its reports, a ledger and signatures, and verified; passed: suite/test-report.test.ts verification: the digest in the ledger, the chain whole, each envelope verified and naming

FR-013reviewpassing

The test run requirement MUST be taken from the library quality (@intellectual-frontiers/quality, quality 0001-quality-evidence FR-010), where it is specified and proved (0032-libraries-by-concern FR-010), and MUST NOT be specified here.

Enforced (review): 0032-libraries-by-concern FR-010 — kept by the library quality: 0001-quality-evidence FR-010

Tests: passed: suite/host-test-run.test.ts a fixture suite run into one record, its reports, a ledger and signatures, and verified; passed: suite/test-report.test.ts the completion report: summary, installation, trace matrix, incidents, high-risk review ne; passed: suite/test-report.test.ts verification: the digest in the ledger, the chain whole, each envelope verified and naming

Cited by: pure/record, Quality evidence

FR-014checkno test run

CI MUST run the suite of a library through the test command of the library quality (src/test-cli.ts run) on each runtime of 0005-typescript-practice FR-018 and keep the record and its reports as the job's artifacts.

Enforced (check): .github: .github/workflows/ifc-typescript.auto.yml test job — CI runs the suite through ifc test on each runtime and system and uploads the record and reports

Cited by: Test

Out of scope

  • Electronic signatures as 21 CFR Part 11 defines them for a regulated record (signer identity, signature manifestations with a name): a harness that holds identities adds them through its own signer; this repository is public and names roles alone.
  • Validation of the tools themselves (TypeScript, the runtimes, the runners) beyond recording their versions.
  • Performance qualification: what a harness measures in use, not what a test suite records.

Edge cases

  • A test that throws an AssertionError with expected and actual values: fail, with both values kept, per FR-001.
  • A test whose actual value holds Bearer abc: written Bearer REDACTED, per FR-001.
  • A test skipped with the reason windows has no file modes: skip with that reason, in TAP as # SKIP windows has no file modes, per FR-001 and FR-008.
  • A test process killed in the middle of a test: the test is cancelled, the tests before it kept, and the run fails, per FR-004 and FR-006.
  • ifc.test.role=alice@example.com: refused, as a person's address, per FR-003.
  • A requirement in the list that no test proves and no review declares: the run fails naming it, per FR-006.
  • The same record rendered twice to JUnit: the same bytes, per FR-010.
  • A record changed by one byte after the run: test-cli verify of the library quality fails, as its digest is not in the ledger, per FR-012.
  • A high-risk requirement with no signature of meaning reviewed: verify --require-review fails, per FR-013.

Assumptions

  • OpenTelemetry's test.* attributes are in development; their names are used as they stand, with the rest under ifc.test.*.
  • TAP 14 is stable though no longer maintained, and CTRF's version 1 schema may still change; both are renderings, so a change touches one function.
  • The TRACEPARENT carrier reaches the test processes of all three runners, which pass their environment on.

Open questions

OQ-1open question

Whether to add the Linux kernel's KTAP, or JUnit's later forms, when a consumer asks.

OQ-2open question

Whether performance measurements belong in the record as metrics (0006-signals FR-013) for performance qualification.

Key entities

  • Test record - the OTLP file of one run: its run span, its installation record and a span and log record per test.
  • Installation record - the resource of a record: runtime, system, revision, lockfile and pins, and the role.
  • Rendering - TAP, JUnit, CTRF, SARIF, OSCAL or a completion report, a pure function of a record.
  • Signature - a DSSE envelope over a record, entered in the ledger with its meaning and role.

Success criteria

SC-001success criterion

One record per run, on any of the three runtimes, from which every report is rendered.

SC-002success criterion

A reviewer can trace every requirement of a specification to the tests that prove it, their outcomes and the installation they ran on, from the completion report alone.

SC-003success criterion

A change to a kept record is detected by test-cli verify of the library quality.

Review & acceptance checklist

  • [x] Every requirement is testable (MUST / MUST NOT), not aspirational
  • [x] No company fact is asserted here
  • [x] Every open item is marked, not silently decided

Cited by

Libraries, oscal-register, ifqs/ifqs, Quality, pure/ladder, pure/names, pure/record, pure/render, test-report, host/otel, Test, record, test, Quality evidence, Test, Names that carry a schema, Intellectual Frontiers Quality System, Trust

Titles, headings, exports, tables, columns and requirements. Esc closes.