hashRowsconstconst hashRows: (rows: readonly unknown[]) => stringThe SHA-256 of rows, in the order given: the same rows in the same order always give the same hash.
Recovery, proof and identity at work (commons data 0064-recovery-proof-identity): the code that works on the tables of the proof model (models/proof). A drill puts a recovery to the test and keeps what it found: a rollback of a load by its id is first counted and then done, and what other loads wrote must be what it was; a replay that is idempotent leaves a table as it was; a restore brings back what was destroyed. A calculation is run with the hash of its inputs and outputs so that it can be reproduced; a reference case holds it to a value; lineage says what a dataset is made from and what a change reaches; and records of different sources are linked to one entity by rules that are written down.
hashRowsconstconst hashRows: (rows: readonly unknown[]) => stringThe SHA-256 of rows, in the order given: the same rows in the same order always give the same hash.
Fingerprintinterfaceno test imports itinterface FingerprintWhat a table held, as a count and a hash.
| Member | Type | What it is |
|---|---|---|
rows | number | |
hash | string |
Lookinterfaceno test imports itinterface LookWhat to read of a table to know it is as it was: the columns, and the ones that put the rows in a fixed order.
| Member | Type | What it is |
|---|---|---|
table | Table<never, never, string> | |
columns | readonly string[] | |
orderBy | readonly string[] | |
except? | { readonly column: string; readonly value: string; } | Rows whose column is not this value only: the rows that a load did not write. |
fingerprintfunctionfunction fingerprint(on: Runner, look: Look): Promise<Fingerprint>A fingerprint of a table: the count and the hash of the columns asked for, in the order asked for.
Drilledinterfaceno test imports itinterface DrilledWhat a drill found.
| Member | Type | What it is |
|---|---|---|
drillId | string | |
outcome | "failed" | "passed" | |
seconds | number | |
before | Fingerprint | |
after | Fingerprint | |
problems | readonly string[] | Where a drill failed, what did not hold. |
RollbackDrillinterfaceno test imports itinterface RollbackDrillWhat a rollback drill is told.
| Member | Type | What it is |
|---|---|---|
drillId | string | |
loadId | string | |
tables | readonly Written[] | |
looks | Readonly<Record<string, Omit<Look, "except" | "table">>> | What to read of each table that is not the load's, which must be as it was: by the table's name. |
rollbackDrillfunctionfunction rollbackDrill(on: Runner, d: RollbackDrill): Promise<Drilled>Drills the rollback of a load by its id: counts what it wrote, does the rollback, and shows that the count of what it removed is the count it wrote, that none of its rows remain, and that the rows of other loads in each table are what they were. The drill is kept, whatever it found.
StateDrillinterfaceno test imports itinterface StateDrillWhat a replay or a restore drill is told.
| Member | Type | What it is |
|---|---|---|
drillId | string | |
kind | "replay" | "restore" | |
scope | string | |
look | Look | What to read of the table whose state must come back. |
steps | () => Promise<void> | What is done: a replay of a load from the raw copy, or a backup, a loss and a restore. |
expect? | "different" | "same" | Whether the state must be what it was (a replay that is idempotent, a restore), or may change (a replay of a corrected load): the default is that it is the same. |
stateDrillfunctionfunction stateDrill(on: Runner, d: StateDrill): Promise<Drilled>Drills a replay or a restore: reads the table, does the steps, reads it again, and says whether it is as it was. The drill is kept, whatever it found.
Calculationinterfaceno test imports itinterface Calculation<I, O>A calculation to run and record.
| Member | Type | What it is |
|---|---|---|
calculationRunId | string | |
calculation | string | |
version | string | |
parameters? | unknown | |
inputs | readonly I[] | |
run | (inputs: readonly I[]) => readonly O[] | |
loadId? | string | |
at? | Date |
recordCalculationfunctionfunction recordCalculation<I, O>(on: Runner, c: Calculation<I, O>): Promise<{ readonly outputs: readonly O[]; readonly inputHash: string; readonly outputHash: string; }>Runs a calculation and records the hash of its inputs and outputs, the parameters and the version.
Reproducedtypeno test imports ittype Reproduced = { readonly kind: "reproduced" } | { readonly kind: "different-inputs" } | { readonly kind: "different-outputs"; readonly recorded: string; readonly now: string } | { readonly kind: "unknown-run" };What reproducing a run found.
reproducefunctionfunction reproduce<I, O>(on: Runner, calculationRunId: string, inputs: readonly I[], run: (inputs: readonly I[]) => readonly O[]): Promise<Reproduced>Runs a calculation again on the inputs given and says whether its outputs are what the recorded run gave: the inputs must be the recorded ones, or it says they are not.
CaseResultinterfaceno test imports itinterface CaseResultWhat running a reference case found.
| Member | Type | What it is |
|---|---|---|
referenceCaseId | string | |
expected | number | |
got | number | |
difference | number | |
within | boolean |
runReferenceCasesfunctionfunction runReferenceCases(on: Runner, calculation: string, calc: (inputs: unknown) => number): Promise<readonly CaseResult[]>Runs each reference case of a calculation (its inputs are the JSON the case holds) and says whether the value is within the tolerance of the one expected.
Reachedinterfaceno test imports itinterface ReachedA dataset a lineage walk reached.
| Member | Type | What it is |
|---|---|---|
dataset | string | |
depth | number | |
via | string |
lineagefunctionfunction lineage(on: Runner, dataset: string, direction: "downstream" | "upstream"): Promise<readonly Reached[]>Walks the lineage from a dataset upstream (what it is made from) or downstream (what a change to it reaches), nearest first, each dataset once, and where there is a cycle, without looping.
edgesOffunctionfunction edgesOf(on: Runner): Promise<readonly { readonly from: string; readonly to: string; }[]>The edges of the lineage as labelViolations of the contract model reads them.
Record_interfaceno test imports itinterface Record_A record of a source, to be linked.
| Member | Type | What it is |
|---|---|---|
source | string | |
key | string | |
fields | Readonly<Record<string, string | null | undefined>> |
LinkRuletypeno test imports ittype LinkRule =
| { readonly rule: "exact"; readonly fields: readonly string[] }
| { readonly rule: "normalized"; readonly fields: readonly string[] }
| { readonly rule: "similar"; readonly fields: readonly string[]; readonly threshold: number };A rule to link records by, tried in order.
normalizeconstconst normalize: (v: string) => stringA value written in one way: accents removed, lower case, only letters and digits.
similarityfunctionfunction similarity(a: string, b: string): numberThe Sørensen–Dice similarity of two values by their bigrams, from 0 to 1.
linkRecordsfunctionfunction linkRecords(records: readonly Record_[], rules: readonly LinkRule[]): readonly Link[]Links records of different sources to entities, by rules tried in order: a key they share as it is, a key they share once it is written in one way, and a similarity above a threshold. A record joins an entity by the first rule that links it, and the weakest rule that joined it to the group is kept with it; a record nothing links is its own entity. The entity's id is that of the first of its records, so it does not change when the records come in another order. Records of the same source are not linked by similar, which is for crossing sources.
saveLinksfunctionfunction saveLinks(on: Runner, links: readonly Link[]): Promise<void>Keeps links in the table of the proof model; a singleton, which no rule linked, is kept as an exact link to itself.