IF Commons GitHub

recovery-proof-identity

0064Active

0064-recovery-proof-identity · owner if · domain data

Requirements

FR-001checkpassing

The model MUST be a module of governance named proof that requires nothing, with a drill as a record of its kind (a closed code list: replay, rollback, restore), what it was done to, when it began, how many seconds it took, what it found (a closed code list: passed, failed, abandoned), the hash of what was there before and after and the evidence, annotated append-only.

Enforced (check): .github: cd commons/data && bun run verify — suite/proof.test.ts: the proof model has drills (kept, only added to), calculation runs with the hash of their inputs and outputs, reference cases, lineage edges and entity links, with closed code lists and every concept defined

Proved by: data/suite/proof.test.ts

Written in: data/models/proof/mod.ts

Tests: passed: suite/proof.test.ts the proof model has drills (kept, only added to), calculation runs with the hash of their

FR-002checkpassing

A calculation run MUST be a record of the calculation and its version, the parameters as JSON, the SHA-256 of its inputs and of its outputs, how many rows it gave, when and the load it was for, annotated append-only.

Enforced (check): .github: cd commons/data && bun run verify — suite/proof.test.ts: the proof model has drills (kept, only added to), calculation runs with the hash of their inputs and outputs, reference cases, lineage edges and entity links, with closed code lists and every concept defined

Proved by: data/suite/proof.test.ts

Written in: data/models/proof/mod.ts

Tests: passed: suite/proof.test.ts the proof model has drills (kept, only added to), calculation runs with the hash of their

FR-003checkpassing

A reference case MUST be a record of the calculation, what it shows, the inputs as JSON, the value expected, the tolerance and where the value comes from.

Enforced (check): .github: cd commons/data && bun run verify — suite/proof.test.ts: the proof model has drills (kept, only added to), calculation runs with the hash of their inputs and outputs, reference cases, lineage edges and entity links, with closed code lists and every concept defined

Proved by: data/suite/proof.test.ts

Written in: data/models/proof/mod.ts

Tests: passed: suite/proof.test.ts the proof model has drills (kept, only added to), calculation runs with the hash of their

FR-004checkpassing

A lineage edge MUST be a record of the dataset something is made from, the dataset that is made and the transform that makes it, unique together.

Enforced (check): .github: cd commons/data && bun run verify — suite/proof.test.ts: the proof model has drills (kept, only added to), calculation runs with the hash of their inputs and outputs, reference cases, lineage edges and entity links, with closed code lists and every concept defined

Proved by: data/suite/proof.test.ts

Written in: data/models/proof/mod.ts

Tests: passed: suite/proof.test.ts the proof model has drills (kept, only added to), calculation runs with the hash of their

FR-005checkpassing

An entity link MUST be a record of the entity, the source and the key of the record in it (unique together), the rule that found it (a closed code list: exact, normalized, similar, confirmed) and the score.

Enforced (check): .github: cd commons/data && bun run verify — suite/proof.test.ts: the proof model has drills (kept, only added to), calculation runs with the hash of their inputs and outputs, reference cases, lineage edges and entity links, with closed code lists and every concept defined

Proved by: data/suite/proof.test.ts

Written in: data/models/proof/mod.ts

Tests: passed: suite/proof.test.ts the proof model has drills (kept, only added to), calculation runs with the hash of their

FR-006checkpassing

Every concept of the model MUST be defined.

Enforced (check): .github: cd commons/data && bun run verify — suite/proof.test.ts: the proof model has drills (kept, only added to), calculation runs with the hash of their inputs and outputs, reference cases, lineage edges and entity links, with closed code lists and every concept defined

Proved by: data/suite/proof.test.ts

Written in: data/models/proof/mod.ts

Tests: passed: suite/proof.test.ts the proof model has drills (kept, only added to), calculation runs with the hash of their

FR-007checkpassing

rollbackDrill MUST count what a load wrote in each table, undo it by its id, and show that the rows removed are the rows written, that none of the load's rows remain and that the rows of the other loads are what they were, and keep the drill as passed or failed with the problems found.

Enforced (check): .github: cd commons/data && bun run verify — suite/proof.test.ts: a rollback drill counts what a load wrote, undoes it by its id, shows that what was removed is what was written, that none of its rows remain and that the rows of the four other loads are as they were, and keeps the drill

Proved by: data/suite/proof.test.ts

Written in: data/sql/proof/mod.ts

Tests: passed: suite/proof.test.ts a rollback drill counts what a load wrote, undoes it by its id, shows that what was remove

FR-008checkpassing

stateDrill MUST read a table, do the steps (a replay, or a backup, a loss and a restore), read it again and say whether it is as it was (or, where asked, that it is not), keep the drill with how long it took, and say what did not hold where it failed; fingerprint MUST be the count and the SHA-256 of the columns asked for in the order asked for.

Enforced (check): .github: cd commons/data && bun run verify — suite/proof.test.ts: a replay that is idempotent and a restore leave a table as it was and pass, one that changes it fails and says so, and each drill is kept with how long it took

Proved by: data/suite/proof.test.ts

Written in: data/sql/proof/mod.ts

Tests: passed: suite/proof.test.ts a replay that is idempotent and a restore leave a table as it was and pass, one that chang

FR-009checkpassing

recordCalculation MUST run a calculation and record the hashes of its inputs and outputs with its version and parameters, and reproduce MUST say reproduced where the same inputs give the recorded outputs, different-outputs with both hashes where they do not, different-inputs where the inputs are not the recorded ones and unknown-run for a run that was not recorded.

Enforced (check): .github: cd commons/data && bun run verify — suite/proof.test.ts: a calculation is recorded with the hash of its inputs and outputs and reproduced from them: reproduced with the same inputs and version, and different-outputs after a change of rounding, different-inputs with other inputs, unknown for a run not recorded

Proved by: data/suite/proof.test.ts

Written in: data/sql/proof/mod.ts

Tests: passed: suite/proof.test.ts a calculation is recorded with the hash of its inputs and outputs and reproduced from them

FR-010checkpassing

runReferenceCases MUST run each case of a calculation and say for each the value expected, the value given, the difference and whether it is within the tolerance.

Enforced (check): .github: cd commons/data && bun run verify — suite/proof.test.ts: reference cases hold a calculation to the value expected within the tolerance: a case within passes, one outside fails with the difference

Proved by: data/suite/proof.test.ts

Written in: data/sql/proof/mod.ts

Tests: passed: suite/proof.test.ts reference cases hold a calculation to the value expected within the tolerance: a case with

FR-011checkpassing

lineage MUST walk upstream or downstream from a dataset, nearest first, each dataset once, without looping on a cycle, with its depth and the transform; edgesOf MUST give the edges as the label check of 0062 reads them.

Enforced (check): .github: cd commons/data && bun run verify — suite/proof.test.ts: lineage is walked upstream and downstream, nearest first, each dataset once and without looping on a cycle, and its edges feed the label check of the contract model

Proved by: data/suite/proof.test.ts

Written in: data/sql/proof/mod.ts

Tests: passed: suite/proof.test.ts lineage is walked upstream and downstream, nearest first, each dataset once and without lo

FR-012checkpassing

linkRecords MUST link records by rules tried in order (exact on fields as they are, normalized on fields written one way, similar above a threshold across sources), give each record the entity it is in with the rule and score that joined it, leave a record nothing links as its own entity, and give the same entities whatever order the records come in.

Enforced (check): .github: cd commons/data && bun run verify — suite/proof.test.ts: records of three sources are linked by rules tried in order (a key as it is, a key written one way, a similarity), each with the rule and score that joined it and a stable entity id, a record nothing links stays alone, and the links are kept

Proved by: data/suite/proof.test.ts

Written in: data/sql/proof/mod.ts

Tests: passed: suite/proof.test.ts records of three sources are linked by rules tried in order (a key as it is, a key written

FR-013checkpassing

saveLinks MUST keep the links in the table of entity links, a record linked to nothing as an exact link to itself.

Enforced (check): .github: cd commons/data && bun run verify — suite/proof.test.ts: records of three sources are linked by rules tried in order (a key as it is, a key written one way, a similarity), each with the rule and score that joined it and a stable entity id, a record nothing links stays alone, and the links are kept

Proved by: data/suite/proof.test.ts

Written in: data/sql/proof/mod.ts

Tests: passed: suite/proof.test.ts records of three sources are linked by rules tried in order (a key as it is, a key written

Out of scope

  • Probabilistic record linkage with learned weights and a review queue: similar is a threshold a person confirms, and confirmed is how a link they confirmed is kept.
  • Backups of a database: the engine's; the drill proves a restore, and does not make one.
  • A lineage extracted from SQL text: the edges are given, and a parser of lineage is a later extension.

Edge cases

  • A drill that throws leaves no row: it was not done, per FR-007.
  • A rollback drill runs the rollback for real, so it is done on a copy or in a test, per FR-007.
  • similar never links two records of the same source, per FR-012.

Assumptions

  • The tables are made by bind from the model on any of SQLite, DuckDB and PostgreSQL; the fixtures of the tests are synthetic.

Open questions

OQ-1open question

Whether a drill's schedule (how often each kind must be done) should be a control of 0062, so that an overdue drill shows in the control matrix.

Key entities

  • Drill - a recovery put to the test, kept with what it found.
  • Calculation run - a calculation with the hashes of what went in and came out.
  • Reference case - a value a calculation must give.
  • Lineage edge - a dataset made from another.
  • Entity link - a record of a source linked to an entity, with the rule.

Success criteria

SC-001success criterion

A load can be undone and the others shown intact, a replay or restore shown to return what was there, a figure reproduced or its change shown, and who is the same person across sources is a fact with a rule.

Review & acceptance checklist

  • [x] Every requirement is testable (MUST / MUST NOT), not aspirational
  • [x] No company fact is asserted here
  • [x] Every open item is marked, not silently decided

Cited by

Data, models/proof/mod, sql/proof/mod

Titles, headings, exports, tables, columns and requirements. Esc closes.