IF Commons GitHub

flow

How data moves through a pipeline, and the proof that nothing was lost on the way

Applies to: Pipelines and data flow. How data moves through a pipeline and the proof that nothing was lost on the way.

Written in commons/data/models/flow/mod.ts:194. Requires nothing. 9 concepts, 7 code lists.

manifest firstLoad delivery load load reason manifest reason load batch_stage manifest delivery load delivery_conflict ledger_entry reason loss
How the concepts of flow relate
Diagram source
flowchart LR
  batch_stage -->|manifest| manifest
  delivery -->|firstLoad| load
  delivery_conflict -->|delivery| delivery
  delivery_conflict -->|load| load
  ledger_entry -->|load| load
  ledger_entry -->|reason| reason
  loss -->|manifest| manifest
  loss -->|reason| reason
  manifest -->|load| load

Concepts

ConceptDefinitionAttributesWritten in
batchStageHow many entities of a batch a stage confirmed: a step of the funnel from the manifest to what was accepted3commons/data/models/flow/mod.ts:176
deliveryA message as it first arrived, with the hash of its content and how many times it has been delivered: the intake that makes a redelivery safe6commons/data/models/flow/mod.ts:137
deliveryConflictA delivery that had the id of one already taken and other content: the original is kept and this is queued to be decided4commons/data/models/flow/mod.ts:153
ledgerEntryWhat one stage confirmed of one message, batch or entity, with the load it was in: a row of the stage ledger, only ever added7commons/data/models/flow/mod.ts:120
loadOne load: an id given at the first stage and carried on every row and effect it makes, so that it can be traced and undone as one6commons/data/models/flow/mod.ts:86
lossEntities of a batch that did not go on, with the stage they stopped at and the reason: what makes a loss explained3commons/data/models/flow/mod.ts:185
manifestWhat a sender says a batch holds: the count to be accounted for, from which every stage and every loss is reconciled4commons/data/models/flow/mod.ts:167
reasonA reason code that says why a stage stopped, rejected or lost something, so that no loss is unexplained4commons/data/models/flow/mod.ts:77
stageContractWhat a stage guarantees to the stage after it, the outcome codes it may give and how its failure is detected: the stage-contract table of a pipeline8commons/data/models/flow/mod.ts:102

Code lists

Code listDefinitionCodes
conflictStatusWhere a conflict is3
dispositionWhat the intake of a delivery found3
grainWhat a ledger row is about3
loadTypeWhat kind of load it is4
outcomeWhat a stage confirmed of a message, a batch or an entity5
producerWho produces a reason code3
stageA stage of a pipeline, which reports success in its own terms8

Titles, headings, exports, tables, columns and requirements. Esc closes.