flow
How data moves through a pipeline, and the proof that nothing was lost on the way
Applies to: Pipelines and data flow. How data moves through a pipeline and the proof that nothing was lost on the way.
Written in commons/data/models/flow/mod.ts:194. Requires nothing. 9 concepts, 7 code lists.
Diagram source
flowchart LR
batch_stage -->|manifest| manifest
delivery -->|firstLoad| load
delivery_conflict -->|delivery| delivery
delivery_conflict -->|load| load
ledger_entry -->|load| load
ledger_entry -->|reason| reason
loss -->|manifest| manifest
loss -->|reason| reason
manifest -->|load| loadConcepts
| Concept | Definition | Attributes | Written in |
|---|---|---|---|
batchStage | How many entities of a batch a stage confirmed: a step of the funnel from the manifest to what was accepted | 3 | commons/data/models/flow/mod.ts:176 |
delivery | A message as it first arrived, with the hash of its content and how many times it has been delivered: the intake that makes a redelivery safe | 6 | commons/data/models/flow/mod.ts:137 |
deliveryConflict | A delivery that had the id of one already taken and other content: the original is kept and this is queued to be decided | 4 | commons/data/models/flow/mod.ts:153 |
ledgerEntry | What one stage confirmed of one message, batch or entity, with the load it was in: a row of the stage ledger, only ever added | 7 | commons/data/models/flow/mod.ts:120 |
load | One load: an id given at the first stage and carried on every row and effect it makes, so that it can be traced and undone as one | 6 | commons/data/models/flow/mod.ts:86 |
loss | Entities of a batch that did not go on, with the stage they stopped at and the reason: what makes a loss explained | 3 | commons/data/models/flow/mod.ts:185 |
manifest | What a sender says a batch holds: the count to be accounted for, from which every stage and every loss is reconciled | 4 | commons/data/models/flow/mod.ts:167 |
reason | A reason code that says why a stage stopped, rejected or lost something, so that no loss is unexplained | 4 | commons/data/models/flow/mod.ts:77 |
stageContract | What a stage guarantees to the stage after it, the outcome codes it may give and how its failure is detected: the stage-contract table of a pipeline | 8 | commons/data/models/flow/mod.ts:102 |
Code lists
| Code list | Definition | Codes |
|---|---|---|
conflictStatus | Where a conflict is | 3 |
disposition | What the intake of a delivery found | 3 |
grain | What a ledger row is about | 3 |
loadType | What kind of load it is | 4 |
outcome | What a stage confirmed of a message, a batch or an entity | 5 |
producer | Who produces a reason code | 3 |
stage | A stage of a pipeline, which reports success in its own terms | 8 |