IF Commons GitHub

sql/proof/mod

Written to: 7 requirements: 7 proved, 0 partly, 0 failing, 0 with no test recorded. 0064-recovery-proof-identity FR-007, 0064-recovery-proof-identity FR-008, 0064-recovery-proof-identity FR-009, 0064-recovery-proof-identity FR-010, 0064-recovery-proof-identity FR-011, 0064-recovery-proof-identity FR-012 and 1 more

Recovery, proof and identity at work (commons data 0064-recovery-proof-identity): the code that works on the tables of the proof model (models/proof). A drill puts a recovery to the test and keeps what it found: a rollback of a load by its id is first counted and then done, and what other loads wrote must be what it was; a replay that is idempotent leaves a table as it was; a restore brings back what was destroyed. A calculation is run with the hash of its inputs and outputs so that it can be reproduced; a reference case holds it to a value; lineage says what a dataset is made from and what a change reaches; and records of different sources are linked to one entity by rules that are written down.

Source: commons/data/sql/proof/mod.ts

13 of 24 exports are imported by a test of the run. A test that imports an export may not call it or check what it returns.

hashRowsconst
const hashRows: (rows: readonly unknown[]) => string

The SHA-256 of rows, in the order given: the same rows in the same order always give the same hash.

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

Fingerprintinterfaceno test imports it
interface Fingerprint

What a table held, as a count and a hash.

MemberTypeWhat it is
rowsnumber
hashstring
Lookinterfaceno test imports it
interface Look

What to read of a table to know it is as it was: the columns, and the ones that put the rows in a fixed order.

MemberTypeWhat it is
tableTable<never, never, string>
columnsreadonly string[]
orderByreadonly string[]
except?{ readonly column: string; readonly value: string; }Rows whose column is not this value only: the rows that a load did not write.
fingerprintfunction
function fingerprint(on: Runner, look: Look): Promise<Fingerprint>

A fingerprint of a table: the count and the hash of the columns asked for, in the order asked for.

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

Drilledinterfaceno test imports it
interface Drilled

What a drill found.

MemberTypeWhat it is
drillIdstring
outcome"failed" | "passed"
secondsnumber
beforeFingerprint
afterFingerprint
problemsreadonly string[]Where a drill failed, what did not hold.
RollbackDrillinterfaceno test imports it
interface RollbackDrill

What a rollback drill is told.

MemberTypeWhat it is
drillIdstring
loadIdstring
tablesreadonly Written[]
looksReadonly<Record<string, Omit<Look, "except" | "table">>>What to read of each table that is not the load's, which must be as it was: by the table's name.
rollbackDrillfunction
function rollbackDrill(on: Runner, d: RollbackDrill): Promise<Drilled>

Drills the rollback of a load by its id: counts what it wrote, does the rollback, and shows that the count of what it removed is the count it wrote, that none of its rows remain, and that the rows of other loads in each table are what they were. The drill is kept, whatever it found.

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

StateDrillinterfaceno test imports it
interface StateDrill

What a replay or a restore drill is told.

MemberTypeWhat it is
drillIdstring
kind"replay" | "restore"
scopestring
lookLookWhat to read of the table whose state must come back.
steps() => Promise<void>What is done: a replay of a load from the raw copy, or a backup, a loss and a restore.
expect?"different" | "same"Whether the state must be what it was (a replay that is idempotent, a restore), or may change (a replay of a corrected load): the default is that it is the same.
stateDrillfunction
function stateDrill(on: Runner, d: StateDrill): Promise<Drilled>

Drills a replay or a restore: reads the table, does the steps, reads it again, and says whether it is as it was. The drill is kept, whatever it found.

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

Calculationinterfaceno test imports it
interface Calculation<I, O>

A calculation to run and record.

MemberTypeWhat it is
calculationRunIdstring
calculationstring
versionstring
parameters?unknown
inputsreadonly I[]
run(inputs: readonly I[]) => readonly O[]
loadId?string
at?Date
recordCalculationfunction
function recordCalculation<I, O>(on: Runner, c: Calculation<I, O>): Promise<{ readonly outputs: readonly O[]; readonly inputHash: string; readonly outputHash: string; }>

Runs a calculation and records the hash of its inputs and outputs, the parameters and the version.

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

Reproducedtypeno test imports it
type Reproduced = { readonly kind: "reproduced" } | { readonly kind: "different-inputs" } | { readonly kind: "different-outputs"; readonly recorded: string; readonly now: string } | { readonly kind: "unknown-run" };

What reproducing a run found.

reproducefunction
function reproduce<I, O>(on: Runner, calculationRunId: string, inputs: readonly I[], run: (inputs: readonly I[]) => readonly O[]): Promise<Reproduced>

Runs a calculation again on the inputs given and says whether its outputs are what the recorded run gave: the inputs must be the recorded ones, or it says they are not.

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

CaseResultinterfaceno test imports it
interface CaseResult

What running a reference case found.

MemberTypeWhat it is
referenceCaseIdstring
expectednumber
gotnumber
differencenumber
withinboolean
runReferenceCasesfunction
function runReferenceCases(on: Runner, calculation: string, calc: (inputs: unknown) => number): Promise<readonly CaseResult[]>

Runs each reference case of a calculation (its inputs are the JSON the case holds) and says whether the value is within the tolerance of the one expected.

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

Reachedinterfaceno test imports it
interface Reached

A dataset a lineage walk reached.

MemberTypeWhat it is
datasetstring
depthnumber
viastring
lineagefunction
function lineage(on: Runner, dataset: string, direction: "downstream" | "upstream"): Promise<readonly Reached[]>

Walks the lineage from a dataset upstream (what it is made from) or downstream (what a change to it reaches), nearest first, each dataset once, and where there is a cycle, without looping.

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

edgesOffunction
function edgesOf(on: Runner): Promise<readonly { readonly from: string; readonly to: string; }[]>

The edges of the lineage as labelViolations of the contract model reads them.

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

Record_interfaceno test imports it
interface Record_

A record of a source, to be linked.

MemberTypeWhat it is
sourcestring
keystring
fieldsReadonly<Record<string, string | null | undefined>>
LinkRuletypeno test imports it
type LinkRule =
  | { readonly rule: "exact"; readonly fields: readonly string[] }
  | { readonly rule: "normalized"; readonly fields: readonly string[] }
  | { readonly rule: "similar"; readonly fields: readonly string[]; readonly threshold: number };

A rule to link records by, tried in order.

normalizeconst
const normalize: (v: string) => string

A value written in one way: accents removed, lower case, only letters and digits.

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

similarityfunction
function similarity(a: string, b: string): number

The Sørensen–Dice similarity of two values by their bigrams, from 0 to 1.

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

linkRecordsfunction
function linkRecords(records: readonly Record_[], rules: readonly LinkRule[]): readonly Link[]

Links records of different sources to entities, by rules tried in order: a key they share as it is, a key they share once it is written in one way, and a similarity above a threshold. A record joins an entity by the first rule that links it, and the weakest rule that joined it to the group is kept with it; a record nothing links is its own entity. The entity's id is that of the first of its records, so it does not change when the records come in another order. Records of the same source are not linked by similar, which is for crossing sources.

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

saveLinksfunction
function saveLinks(on: Runner, links: readonly Link[]): Promise<void>

Keeps links in the table of the proof model; a singleton, which no rule linked, is kept as an exact link to itself.

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

Titles, headings, exports, tables, columns and requirements. Esc closes.