IF Commons GitHub

host/test-run

Written to: 1 requirement: 0 proved, 0 partly, 0 failing, 1 with no test recorded. 0001-quality-evidence FR-002

A test run as evidence (commons 0001-quality-evidence FR-002 to FR-004, FR-008, FR-009): runTests starts one trace with a test run span, runs bun test or node --test (one or both in turn) with TRACEPARENT and IFC_TEST_RECORD set, passes each runner's own output on, gathers every process's records into one OTLP JSON file (<out>/<trace id>.otlp.json) with the run's span, applies the exit rule, renders the reports asked for from the record as written, appends the record's digest to an evidence ledger and, where a key is given, signs an in-toto Test Result statement of it in a DSSE envelope whose signature is entered in the ledger with its meaning and role. signRun signs a record later (a review, an approval), and verifyRun checks a record against the ledger and the envelopes. A runtime that is not installed is reported and fails the run. test-cli.ts is the command line.

import { runTests } from "./test-run.ts";
const r = await runTests({ runtimes: ["node"], paths: ["tests"], out: "test-results", reports: ["junit", "md"] });
if (r.kind === "ok") console.log(r.value.recordPath, r.value.code);

Source: commons/quality/src/host/test-run.ts

6 of 34 exports are imported by a test of the run. A test that imports an export may not call it or check what it returns.

testRunnamespace
namespace testRun
testRun.Runtimetypeno test imports it
type Runtime = "bun" | "node";

A runtime whose runner a run uses.

testRun.RUNTIMESconstno test imports it
const RUNTIMES: readonly Runtime[]

Every runtime.

testRun.ReportFormattypeno test imports it
type ReportFormat = "tap" | "junit" | "ctrf" | "sarif" | "md" | "html" | "oscal";

A report a run renders from its record (FR-005, FR-006).

testRun.REPORT_FORMATSconstno test imports it
const REPORT_FORMATS: readonly ReportFormat[]

Every report format.

testRun.reportNamefunctionno test imports it
function reportName(traceId: string, format: ReportFormat): string

The file name of a report of a run.

testRun.SigningKeytypeno test imports it
type SigningKey = CryptoKey | string;

A key to sign with: a Web Crypto private key, or the path of a PKCS#8 PEM file of an Ed25519 or ECDSA P-256 key.

testRun.SignOptionstypeno test imports it
type SignOptions = { readonly key: SigningKey; readonly meaning: Meaning; readonly role: string; readonly keyid?: string };

A signature to make: the key, its meaning, the role that signs, and the key's id in the envelope.

testRun.RunOptionstypeno test imports it
type RunOptions = { /** The runners, in turn (the runtime this code runs on by default). */ readonly runtimes?: read… (1431 characters)
type RunOptions = {
  /** The runners, in turn (the runtime this code runs on by default). */
  readonly runtimes?: readonly Runtime[];
  /** Test files and folders (each `*.test.ts`, `*.test.mts`, `*.test.js` or `*.test.mjs` under a folder); `tests` by default. */
  readonly paths?: readonly string[];
  /** Where the record and the reports go (`test-results` by default). */
  readonly out?: string;
  /** The working folder of the runners (this process's by default). */
  readonly cwd?: string;
  /** A file of requirement ids, or a spec-kit enforcement register (FR-003). */
  readonly requirements?: string;
  /** Keep only the requirements of this specification (`0005-typescript-practice`). */
  readonly spec?: string;
  /** A file of reviewed requirements: the requirement, a tab and the reason, one per line. */
  readonly reviewed?: string;
  /** The evidence ledger (JSON lines) the record's digest is appended to (FR-008). */
  readonly ledger?: string;
  readonly sign?: SignOptions;
  readonly reports?: readonly ReportFormat[];
  /** Whether the runners' output is passed on (`inherit`, the default) or dropped (`null`). */
  readonly output?: "inherit" | "null";
  /** Variables added to the runners' environment. */
  readonly env?: Readonly<Record<string, string>>;
  /** The executable of a runtime, where it is not the one on `PATH`. */
  readonly executables?: Readonly<Partial<Record<Runtime, string>>>;
};

How to run a suite.

testRun.RunResulttypeno test imports it
type RunResult = {
  /** 0 where the run passed, 1 where it failed (FR-003). */
  readonly code: 0 | 1;
  readonly recordPath: string;
  /** The SHA-256 of the record's bytes (FR-004). */
  readonly digest: Sha256;
  readonly run: TestRun;
  readonly reports: readonly string[];
  readonly ledger?: readonly LedgerEntry[];
  readonly envelopePath?: string;
};

What a run did.

testRun.testFilesfunctionno test imports it
function testFiles(paths: readonly string[], cwd: string): Promise<Result<readonly string[], string>>

The test files of some paths: a file as given, a folder as the test files under it, each absolute and once, sorted.

testRun.readLedgerfunctionno test imports it
function readLedger(path: string): Promise<Result<readonly LedgerEntry[], string>>

A ledger read from its file, empty where there is none.

testRun.loadPrivateKeyfunctionno test imports it
function loadPrivateKey(pathOrPem: string, extractable?: boolean): Promise<Result<CryptoKey, string>>

A private key from a PKCS#8 PEM file (or its text) of an Ed25519 or ECDSA P-256 key.

testRun.loadPublicKeyfunctionno test imports it
function loadPublicKey(pathOrPem: string): Promise<Result<CryptoKey, string>>

A public key from an SPKI PEM file (or its text), or the public half of a PKCS#8 private key, of Ed25519 or ECDSA P-256.

testRun.writeReportsfunctionno test imports it
function writeReports(run: TestRun, out: string, formats: readonly ReportFormat[], extra?: { readonly digest?: Sha256 | undefined; readonly ledger?: readonly LedgerEntry[] | undefined; }): Promise<readonly string[]>

The reports asked for, rendered from the record as read back, written to out; their paths.

testRun.runTestsfunctionno test imports it
function runTests(options?: RunOptions): Promise<Result<RunResult, string>>

A suite run as evidence (FR-002 to FR-004, FR-008): every runner in turn, one record, the exit rule, the reports, the ledger and the signature asked for. An error is something that kept the run from being recorded (a path that does not exist, a list that cannot be read); a run that fails is ok with code 1.

testRun.loadRecordfunctionno test imports it
function loadRecord(path: string): Promise<Result<{ readonly run: TestRun; readonly digest: Sha256; }, string>>

A record read from its file, with its digest.

testRun.signRunfunctionno test imports it
function signRun(options: { readonly record: string; readonly ledger: string; readonly sign: SignOptions; }): Promise<Result<{ readonly envelopePath: string; readonly ledger: readonly LedgerEntry[]; }, string>>

A record signed after its run (a review, an approval): the envelope beside it, and its signature in the ledger (FR-008, FR-010).

testRun.verifyRunfunctionno test imports it
function verifyRun(options: { readonly record: string; readonly ledger: string; readonly envelopes?: readonly string[] | undefined; readonly publicKey?: string | CryptoKey | undefined; readonly keyid?: string | undefined; readonly requireReview?: boolean | undefined; }): Promise<Result<Verification & { readonly digest: Sha256; }, readonly string[]>>

A record checked (FR-009, FR-010): its digest in the ledger, the ledger's chain whole, each envelope verified with the key and naming the digest, and, with requireReview, a signature of meaning reviewed in the ledger. Each problem as a sentence.

Runtimetype
type Runtime = "bun" | "node";

A runtime whose runner a run uses.

Imported by 2 tests of the run (2 passed, 0 failed): the library's results

RUNTIMESconstno test imports it
const RUNTIMES: readonly Runtime[]

Every runtime.

ReportFormattypeno test imports it
type ReportFormat = "tap" | "junit" | "ctrf" | "sarif" | "md" | "html" | "oscal";

A report a run renders from its record (FR-005, FR-006).

REPORT_FORMATSconst
const REPORT_FORMATS: readonly ReportFormat[]

Every report format.

Imported by 2 tests of the run (2 passed, 0 failed): the library's results

reportNamefunctionno test imports it
function reportName(traceId: string, format: ReportFormat): string

The file name of a report of a run.

SigningKeytypeno test imports it
type SigningKey = CryptoKey | string;

A key to sign with: a Web Crypto private key, or the path of a PKCS#8 PEM file of an Ed25519 or ECDSA P-256 key.

SignOptionstypeno test imports it
type SignOptions = { readonly key: SigningKey; readonly meaning: Meaning; readonly role: string; readonly keyid?: string };

A signature to make: the key, its meaning, the role that signs, and the key's id in the envelope.

RunOptionstypeno test imports it
type RunOptions = { /** The runners, in turn (the runtime this code runs on by default). */ readonly runtimes?: read… (1431 characters)
type RunOptions = {
  /** The runners, in turn (the runtime this code runs on by default). */
  readonly runtimes?: readonly Runtime[];
  /** Test files and folders (each `*.test.ts`, `*.test.mts`, `*.test.js` or `*.test.mjs` under a folder); `tests` by default. */
  readonly paths?: readonly string[];
  /** Where the record and the reports go (`test-results` by default). */
  readonly out?: string;
  /** The working folder of the runners (this process's by default). */
  readonly cwd?: string;
  /** A file of requirement ids, or a spec-kit enforcement register (FR-003). */
  readonly requirements?: string;
  /** Keep only the requirements of this specification (`0005-typescript-practice`). */
  readonly spec?: string;
  /** A file of reviewed requirements: the requirement, a tab and the reason, one per line. */
  readonly reviewed?: string;
  /** The evidence ledger (JSON lines) the record's digest is appended to (FR-008). */
  readonly ledger?: string;
  readonly sign?: SignOptions;
  readonly reports?: readonly ReportFormat[];
  /** Whether the runners' output is passed on (`inherit`, the default) or dropped (`null`). */
  readonly output?: "inherit" | "null";
  /** Variables added to the runners' environment. */
  readonly env?: Readonly<Record<string, string>>;
  /** The executable of a runtime, where it is not the one on `PATH`. */
  readonly executables?: Readonly<Partial<Record<Runtime, string>>>;
};

How to run a suite.

testFilesfunctionno test imports it
function testFiles(paths: readonly string[], cwd: string): Promise<Result<readonly string[], string>>

The test files of some paths: a file as given, a folder as the test files under it, each absolute and once, sorted.

loadPrivateKeyfunctionno test imports it
function loadPrivateKey(pathOrPem: string, extractable?: boolean): Promise<Result<CryptoKey, string>>

A private key from a PKCS#8 PEM file (or its text) of an Ed25519 or ECDSA P-256 key.

loadPublicKeyfunctionno test imports it
function loadPublicKey(pathOrPem: string): Promise<Result<CryptoKey, string>>

A public key from an SPKI PEM file (or its text), or the public half of a PKCS#8 private key, of Ed25519 or ECDSA P-256.

writeReportsfunction
function writeReports(run: TestRun, out: string, formats: readonly ReportFormat[], extra?: { readonly digest?: Sha256 | undefined; readonly ledger?: readonly LedgerEntry[] | undefined; }): Promise<readonly string[]>

The reports asked for, rendered from the record as read back, written to out; their paths.

Imported by 2 tests of the run (2 passed, 0 failed): the library's results

runTestsfunction
function runTests(options?: RunOptions): Promise<Result<RunResult, string>>

A suite run as evidence (FR-002 to FR-004, FR-008): every runner in turn, one record, the exit rule, the reports, the ledger and the signature asked for. An error is something that kept the run from being recorded (a path that does not exist, a list that cannot be read); a run that fails is ok with code 1.

Imported by 4 tests of the run (4 passed, 0 failed): the library's results

loadRecordfunctionno test imports it
function loadRecord(path: string): Promise<Result<{ readonly run: TestRun; readonly digest: Sha256; }, string>>

A record read from its file, with its digest.

signRunfunction
function signRun(options: { readonly record: string; readonly ledger: string; readonly sign: SignOptions; }): Promise<Result<{ readonly envelopePath: string; readonly ledger: readonly LedgerEntry[]; }, string>>

A record signed after its run (a review, an approval): the envelope beside it, and its signature in the ledger (FR-008, FR-010).

Imported by 2 tests of the run (2 passed, 0 failed): the library's results

verifyRunfunction
function verifyRun(options: { readonly record: string; readonly ledger: string; readonly envelopes?: readonly string[] | undefined; readonly publicKey?: string | CryptoKey | undefined; readonly keyid?: string | undefined; readonly requireReview?: boolean | undefined; }): Promise<Result<Verification & { readonly digest: Sha256; }, readonly string[]>>

A record checked (FR-009, FR-010): its digest in the ledger, the ledger's chain whole, each envelope verified with the key and naming the digest, and, with requireReview, a signature of meaning reviewed in the ledger. Each problem as a sentence.

Imported by 2 tests of the run (2 passed, 0 failed): the library's results

Titles, headings, exports, tables, columns and requirements. Esc closes.