STAGESconstconst STAGES: readonly ["received", "captured", "parsed", "mapped", "validated", "forwarded", "dest_verdict", "usable"]The stages in the order of a pipeline.
The flow ledger at work (commons data 0060-flow-ledger): the code that writes and reads the tables of the flow model (models/flow). A load is opened and its id is carried on every row and effect; each stage's confirmation is a row of the ledger, only ever added; the intake of a delivery says whether it is the first, a duplicate or a conflict, never overwriting; a gap query finds what vanished between two stages; the funnel of a manifest says how many entities are unexplained and at which boundary between stages; and a load is undone by its id across the tables it wrote.
STAGESconstconst STAGES: readonly ["received", "captured", "parsed", "mapped", "validated", "forwarded", "dest_verdict", "usable"]The stages in the order of a pipeline.
Stagetypeno test imports ittype Stage = (typeof STAGES)[number];A stage.
LoadTypetypeno test imports ittype LoadType = "delivery" | "backfill" | "incremental" | "replay";Why a load was made.
OpenLoadinterfaceno test imports itinterface OpenLoadWhat opening a load is told.
| Member | Type | What it is |
|---|---|---|
loadType | LoadType | |
source | string | |
tenant? | string | |
loadId? | string | The id, where the caller gives it; made where not. |
now? | (() => Date) |
openLoadfunctionfunction openLoad(on: Runner, spec: OpenLoad): Promise<string>Opens a load: an id given at the first stage, to be carried on every row and effect it makes.
Returns: the load's id
Confirmationinterfaceinterface ConfirmationWhat one stage confirmed of one thing.
| Member | Type | What it is |
|---|---|---|
loadId | string | |
subject | string | |
stage | "captured" | "dest_verdict" | "forwarded" | "mapped" | "parsed" | "received" | "usable" | "validated" | |
outcome | Outcome | |
grain? | "batch" | "entity" | "message" | |
entities? | number | |
reason? | string | The code of the reason, where it did not pass. |
at? | string |
confirmfunctionfunction confirm(on: Runner, rows: readonly Confirmation[]): Promise<void>Adds rows to the ledger: what each stage confirmed. A row is only ever added; the id of a row is its load, subject and stage, so that the same confirmation twice is refused by the database.
| Parameter | What it is |
|---|---|
rows | the confirmations |
Intakeinterfaceno test imports itinterface IntakeWhat the intake of a delivery found.
| Member | Type | What it is |
|---|---|---|
disposition | "conflict" | "duplicate" | "first" | |
deliveryCount | number | How many times this message has been delivered with the content it first had. |
intakefunctionfunction intake(on: Runner, d: Delivery): Promise<Intake>The idempotent intake of a delivery. The first time a message is seen it is first and the caller makes its effect; the same message with the same content again is duplicate, counted and no effect; the same id with other content is a conflict, queued and never overwriting the original. The database's key is what keeps two racing deliveries to one first. Call it outside a transaction (a refused insert ends one on some engines).
LastConfirmedinterfaceno test imports itinterface LastConfirmedThe last stage that confirmed a subject.
| Member | Type | What it is |
|---|---|---|
subject | string | |
stage | "captured" | "dest_verdict" | "forwarded" | "mapped" | "parsed" | "received" | "usable" | "validated" | |
outcome | Outcome | |
loadId | string |
lastConfirmedfunctionfunction lastConfirmed(on: Runner, loadId?: string | undefined): Promise<readonly LastConfirmed[]>For each subject, the last stage that confirmed it and what it said: the answer to "which stage last confirmed this message".
| Parameter | What it is |
|---|---|
loadId | only this load's subjects; every load's where not given |
gapsfunctionfunction gaps(on: Runner, after?: "captured" | "dest_verdict" | "forwarded" | "mapped" | "parsed" | "received" | "usable" | "validated", loadId?: string | undefined): Promise<readonly string[]>The subjects that vanished: whose last confirmation is a stage that passed and which have no row after it, although the stage is not the last of the pipeline and nothing said they were refused.
| Parameter | What it is |
|---|---|
after | the stage after which a subject is expected to have a row (the last stage a pipeline always confirms), captured where not given |
Funnelinterfaceno test imports itinterface FunnelThe funnel of one batch: from what the sender said to what was accepted, with where it went missing.
| Member | Type | What it is |
|---|---|---|
manifestId | string | |
manifestCount | number | |
accepted | number | The entities the destination accepted. |
lost | number | The entities that stopped, with a reason. |
unexplained | number | What is neither accepted nor lost with a reason: it must be 0. |
boundaries | readonly { readonly before: "captured" | "dest_verdict" | "forwarded" | "manifest" | "mapped" | "parsed" | "received" | "usable" | "validated"; readonly after: "captured" | "dest_verdict" | "forwarded" | "mapped" | "parsed" | "received" | "usable" | "validated"; readonly missing: number; }[] | The boundary between two stages at which entities went missing without a reason, with how many. |
funnelfunctionfunction funnel(on: Runner, manifests?: readonly string[] | undefined, acceptedAt?: "captured" | "dest_verdict" | "forwarded" | "mapped" | "parsed" | "received" | "usable" | "validated"): Promise<readonly Funnel[]>Reconciles batches to what their senders say: for each manifest, the entities accepted at the stage of the destination's verdict and the entities lost with a reason must add up to the manifest, and where they do not the entities that went missing are attributed to the boundary between the two stages where the count fell without a loss to account for it.
| Parameter | What it is |
|---|---|
manifests | the batches to reconcile; every batch where not given |
acceptedAt | the stage whose count is what was accepted; dest_verdict where not given |
Rollbackinterfaceno test imports itinterface RollbackWhat undoing a load did.
| Member | Type | What it is |
|---|---|---|
loadId | string | |
removed | Readonly<Record<string, number>> | The rows removed from each table, by the table's name. |
writtenconstconst written: <R, I, N extends string>(table: Table<R, I, N>, column: keyof R & string) => WrittenA table a load wrote to and its column of the load's id (a key of its row).
rollbackLoadfunctionfunction rollbackLoad(on: Runner, loadId: string, tables: readonly Written[], now?: () => Date): Promise<Rollback>Undoes a load by its id: the rows it wrote are removed from each table that carries its id, the rows of other loads are not touched, and the load is marked as rolled back. The ledger is only ever added to, so what the load confirmed stays as the record that it was made and undone.
loadsconstconst loads: (on: Runner) => Promise<readonly RowOfProjection<{ loadId: Column<string>; loadType: Column<"backfill" | "delivery" | "incremental" | "replay">; source: Column<string>; rolledBackAt: Column<Timestamp | null>; }>[]>The loads, newest first, with whether they were undone.