IF Commons GitHub

sql/flow/mod

Written to: 6 requirements: 6 proved, 0 partly, 0 failing, 0 with no test recorded. 0060-flow-ledger FR-006, 0060-flow-ledger FR-007, 0060-flow-ledger FR-008, 0060-flow-ledger FR-009, 0060-flow-ledger FR-010, 0060-flow-ledger FR-011

The flow ledger at work (commons data 0060-flow-ledger): the code that writes and reads the tables of the flow model (models/flow). A load is opened and its id is carried on every row and effect; each stage's confirmation is a row of the ledger, only ever added; the intake of a delivery says whether it is the first, a duplicate or a conflict, never overwriting; a gap query finds what vanished between two stages; the funnel of a manifest says how many entities are unexplained and at which boundary between stages; and a load is undone by its id across the tables it wrote.

Source: commons/data/sql/flow/mod.ts

11 of 18 exports are imported by a test of the run. A test that imports an export may not call it or check what it returns.

STAGESconst
const STAGES: readonly ["received", "captured", "parsed", "mapped", "validated", "forwarded", "dest_verdict", "usable"]

The stages in the order of a pipeline.

Imported by 5 tests of the run (5 passed, 0 failed): the library's results

Stagetypeno test imports it
type Stage = (typeof STAGES)[number];

A stage.

LoadTypetypeno test imports it
type LoadType = "delivery" | "backfill" | "incremental" | "replay";

Why a load was made.

OpenLoadinterfaceno test imports it
interface OpenLoad

What opening a load is told.

MemberTypeWhat it is
loadTypeLoadType
sourcestring
tenant?string
loadId?stringThe id, where the caller gives it; made where not.
now?(() => Date)
openLoadfunction
function openLoad(on: Runner, spec: OpenLoad): Promise<string>

Opens a load: an id given at the first stage, to be carried on every row and effect it makes.

Returns: the load's id

Imported by 5 tests of the run (5 passed, 0 failed): the library's results

Confirmationinterface
interface Confirmation

What one stage confirmed of one thing.

MemberTypeWhat it is
loadIdstring
subjectstring
stage"captured" | "dest_verdict" | "forwarded" | "mapped" | "parsed" | "received" | "usable" | "validated"
outcomeOutcome
grain?"batch" | "entity" | "message"
entities?number
reason?stringThe code of the reason, where it did not pass.
at?string

Imported by 5 tests of the run (5 passed, 0 failed): the library's results

confirmfunction
function confirm(on: Runner, rows: readonly Confirmation[]): Promise<void>

Adds rows to the ledger: what each stage confirmed. A row is only ever added; the id of a row is its load, subject and stage, so that the same confirmation twice is refused by the database.

ParameterWhat it is
rowsthe confirmations

Imported by 5 tests of the run (5 passed, 0 failed): the library's results

Intakeinterfaceno test imports it
interface Intake

What the intake of a delivery found.

MemberTypeWhat it is
disposition"conflict" | "duplicate" | "first"
deliveryCountnumberHow many times this message has been delivered with the content it first had.
intakefunction
function intake(on: Runner, d: Delivery): Promise<Intake>

The idempotent intake of a delivery. The first time a message is seen it is first and the caller makes its effect; the same message with the same content again is duplicate, counted and no effect; the same id with other content is a conflict, queued and never overwriting the original. The database's key is what keeps two racing deliveries to one first. Call it outside a transaction (a refused insert ends one on some engines).

Imported by 5 tests of the run (5 passed, 0 failed): the library's results

LastConfirmedinterfaceno test imports it
interface LastConfirmed

The last stage that confirmed a subject.

MemberTypeWhat it is
subjectstring
stage"captured" | "dest_verdict" | "forwarded" | "mapped" | "parsed" | "received" | "usable" | "validated"
outcomeOutcome
loadIdstring
lastConfirmedfunction
function lastConfirmed(on: Runner, loadId?: string | undefined): Promise<readonly LastConfirmed[]>

For each subject, the last stage that confirmed it and what it said: the answer to "which stage last confirmed this message".

ParameterWhat it is
loadIdonly this load's subjects; every load's where not given

Imported by 5 tests of the run (5 passed, 0 failed): the library's results

gapsfunction
function gaps(on: Runner, after?: "captured" | "dest_verdict" | "forwarded" | "mapped" | "parsed" | "received" | "usable" | "validated", loadId?: string | undefined): Promise<readonly string[]>

The subjects that vanished: whose last confirmation is a stage that passed and which have no row after it, although the stage is not the last of the pipeline and nothing said they were refused.

ParameterWhat it is
afterthe stage after which a subject is expected to have a row (the last stage a pipeline always confirms), captured where not given

Imported by 5 tests of the run (5 passed, 0 failed): the library's results

Funnelinterfaceno test imports it
interface Funnel

The funnel of one batch: from what the sender said to what was accepted, with where it went missing.

MemberTypeWhat it is
manifestIdstring
manifestCountnumber
acceptednumberThe entities the destination accepted.
lostnumberThe entities that stopped, with a reason.
unexplainednumberWhat is neither accepted nor lost with a reason: it must be 0.
boundariesreadonly { readonly before: "captured" | "dest_verdict" | "forwarded" | "manifest" | "mapped" | "parsed" | "received" | "usable" | "validated"; readonly after: "captured" | "dest_verdict" | "forwarded" | "mapped" | "parsed" | "received" | "usable" | "validated"; readonly missing: number; }[]The boundary between two stages at which entities went missing without a reason, with how many.
funnelfunction
function funnel(on: Runner, manifests?: readonly string[] | undefined, acceptedAt?: "captured" | "dest_verdict" | "forwarded" | "mapped" | "parsed" | "received" | "usable" | "validated"): Promise<readonly Funnel[]>

Reconciles batches to what their senders say: for each manifest, the entities accepted at the stage of the destination's verdict and the entities lost with a reason must add up to the manifest, and where they do not the entities that went missing are attributed to the boundary between the two stages where the count fell without a loss to account for it.

ParameterWhat it is
manifeststhe batches to reconcile; every batch where not given
acceptedAtthe stage whose count is what was accepted; dest_verdict where not given

Imported by 5 tests of the run (5 passed, 0 failed): the library's results

Rollbackinterfaceno test imports it
interface Rollback

What undoing a load did.

MemberTypeWhat it is
loadIdstring
removedReadonly<Record<string, number>>The rows removed from each table, by the table's name.
writtenconst
const written: <R, I, N extends string>(table: Table<R, I, N>, column: keyof R & string) => Written

A table a load wrote to and its column of the load's id (a key of its row).

Imported by 12 tests of the run (12 passed, 0 failed): the library's results

rollbackLoadfunction
function rollbackLoad(on: Runner, loadId: string, tables: readonly Written[], now?: () => Date): Promise<Rollback>

Undoes a load by its id: the rows it wrote are removed from each table that carries its id, the rows of other loads are not touched, and the load is marked as rolled back. The ledger is only ever added to, so what the load confirmed stays as the record that it was made and undone.

Imported by 5 tests of the run (5 passed, 0 failed): the library's results

loadsconst
const loads: (on: Runner) => Promise<readonly RowOfProjection<{ loadId: Column<string>; loadType: Column<"backfill" | "delivery" | "incremental" | "replay">; source: Column<string>; rolledBackAt: Column<Timestamp | null>; }>[]>

The loads, newest first, with whether they were undone.

Imported by 5 tests of the run (5 passed, 0 failed): the library's results

Titles, headings, exports, tables, columns and requirements. Esc closes.