IF Commons GitHub

meaning-registers

0061Active

0061-meaning-registers · owner if · domain data

Requirements

FR-001checkpassing

The model MUST be a module of governance named meaning that requires nothing, with a mapping as a record of the source system, field and value, the canonical field and value, how the two relate, the rule it follows, its status, who approved it and when, and the dates between which it holds.

Enforced (check): .github: cd commons/data && bun run verify — suite/meaning.test.ts: the meaning model has mappings with how they relate and when they hold, variances that expire, null rules, and closed code lists for absence, null policy, relationship, mapping status and variance status

Proved by: data/suite/meaning.test.ts

Written in: data/models/meaning/mod.ts

Tests: passed: suite/meaning.test.ts the meaning model has mappings with how they relate and when they hold, variances that exp

FR-002checkpassing

How a source value relates to the canonical one MUST be a closed code list (equivalent, broader, narrower, related, unmapped) and a mapping's status MUST be a closed code list (draft, approved, retired).

Enforced (check): .github: cd commons/data && bun run verify — suite/meaning.test.ts: the meaning model has mappings with how they relate and when they hold, variances that expire, null rules, and closed code lists for absence, null policy, relationship, mapping status and variance status

Proved by: data/suite/meaning.test.ts

Written in: data/models/meaning/mod.ts

Tests: passed: suite/meaning.test.ts the meaning model has mappings with how they relate and when they hold, variances that exp

FR-003checkpassing

A variance MUST be a record of the source and field it is about, what differs, its status (open, accepted, resolved, expired), who answers for it, who decided and when, and when the acceptance ends.

Enforced (check): .github: cd commons/data && bun run verify — suite/meaning.test.ts: the meaning model has mappings with how they relate and when they hold, variances that expire, null rules, and closed code lists for absence, null policy, relationship, mapping status and variance status

Proved by: data/suite/meaning.test.ts

Written in: data/models/meaning/mod.ts

Tests: passed: suite/meaning.test.ts the meaning model has mappings with how they relate and when they hold, variances that exp

FR-004checkpassing

A null rule MUST be a record of the table and column it is about (unique), what is done with a missing value (reject, allow absent, default to), the kind of absence it stands for, the default and why.

Enforced (check): .github: cd commons/data && bun run verify — suite/meaning.test.ts: the meaning model has mappings with how they relate and when they hold, variances that expire, null rules, and closed code lists for absence, null policy, relationship, mapping status and variance status

Proved by: data/suite/meaning.test.ts

Written in: data/models/meaning/mod.ts

Tests: passed: suite/meaning.test.ts the meaning model has mappings with how they relate and when they hold, variances that exp

FR-005checkpassing

The kinds of absence MUST be a closed code list (not collected, not applicable, unknown, not yet known, refused, suppressed, invalid) so that one null is not taken for another; every concept and code list MUST be defined.

Enforced (check): .github: cd commons/data && bun run verify — suite/meaning.test.ts: the meaning model has mappings with how they relate and when they hold, variances that expire, null rules, and closed code lists for absence, null policy, relationship, mapping status and variance status

Proved by: data/suite/meaning.test.ts

Written in: data/models/meaning/mod.ts

Tests: passed: suite/meaning.test.ts the meaning model has mappings with how they relate and when they hold, variances that exp

FR-006checkpassing

translate MUST give the one approved mapping that holds at the time asked (now where not given) with its target, how it relates and its id; unmapped for a value that no approved mapping that holds covers, whether it is a draft, retired or ended; and ambiguous with the ids where two hold, choosing none.

Enforced (check): .github: cd commons/data && bun run verify — suite/meaning.test.ts: translate gives the approved mapping that holds at the time, nothing for a value no mapping covers or one that has ended or is only a draft, and says ambiguous where two hold; coverage names the unmapped values most seen first

Proved by: data/suite/meaning.test.ts

Written in: data/sql/meaning/mod.ts

Tests: passed: suite/meaning.test.ts translate gives the approved mapping that holds at the time, nothing for a value no mappin

FR-007checkpassing

coverage MUST give how many values observed in a field there were (a missing value is not counted), how many a mapping that holds covers, the share, and the values nothing maps with how many times each was seen, the most seen first.

Enforced (check): .github: cd commons/data && bun run verify — suite/meaning.test.ts: translate gives the approved mapping that holds at the time, nothing for a value no mapping covers or one that has ended or is only a draft, and says ambiguous where two hold; coverage names the unmapped values most seen first

Proved by: data/suite/meaning.test.ts

Written in: data/sql/meaning/mod.ts

Tests: passed: suite/meaning.test.ts translate gives the approved mapping that holds at the time, nothing for a value no mappin

FR-008checkpassing

resolve MUST keep a value that is there, and for one that is missing (null, undefined or only spaces) MUST reject, keep the kind of absence, or replace it with the default as the rule's policy says, rejecting a replacing rule that has no default; ruleOf MUST read the rule of a table and column.

Enforced (check): .github: cd commons/data && bun run verify — suite/meaning.test.ts: a null rule keeps a value that is there, rejects, keeps the kind of absence or replaces with a default as the policy says, and an empty string is as missing as a null

Proved by: data/suite/meaning.test.ts

Written in: data/sql/meaning/mod.ts

Tests: passed: suite/meaning.test.ts a null rule keeps a value that is there, rejects, keeps the kind of absence or replaces wi

FR-009checkpassing

dueVariances MUST give the variances that are accepted and past the date they end, and those that are open, and no others.

Enforced (check): .github: cd commons/data && bun run verify — suite/meaning.test.ts: the variance register finds what is accepted past its date and what is open, and leaves accepted ones that have not expired and resolved ones

Proved by: data/suite/meaning.test.ts

Written in: data/sql/meaning/mod.ts

Tests: passed: suite/meaning.test.ts the variance register finds what is accepted past its date and what is open, and leaves ac

FR-010checkpassing

profile MUST give for each column named the rows, the nulls, the distinct values and the lowest and highest, as counts and extremes only, on every engine.

Enforced (check): .github: cd commons/data && bun run verify — suite/meaning.test.ts: profile gives rows, nulls, distinct values and the lowest and highest of a column with counts, on every engine

Proved by: data/suite/meaning.test.ts

Written in: data/sql/meaning/mod.ts

Tests: passed: suite/meaning.test.ts profile gives rows, nulls, distinct values and the lowest and highest of a column with cou

FR-011checkpassing

synthetic MUST make for a seed, a number of rows and a spec of columns (sequence, integer range, weighted choice, date range, pattern) the same rows every time, other rows for another seed, a rate of absence for a column and a rate of duplicate rows as asked.

Enforced (check): .github: cd commons/data && bun run verify — suite/meaning.test.ts: the generator makes the same rows for the same seed, other rows for another, the rate of absence and of duplicates asked for, and no value that is not synthetic

Proved by: data/suite/meaning.test.ts

Written in: data/sql/meaning/mod.ts

Tests: passed: suite/meaning.test.ts the generator makes the same rows for the same seed, other rows for another, the rate of a

Out of scope

  • Reading a mapping from a message and the codecs that use it: commons/integrations's, which will satisfy a structural port of translate.
  • Seeding a table from a code list or a profile of a distribution of a real source: a later stage.
  • Linkage of records (matching one entity across sources): an optional later extension (0067).

Edge cases

  • A value that is observed as null is not counted by coverage, per FR-007.
  • Two mappings that hold at once are an error of the register and translate says so, per FR-006.
  • A mapping whose validTo is the time asked has ended: the end is not included, per FR-006.

Assumptions

  • The tables are made by bind from the model on any of SQLite, DuckDB and PostgreSQL; the fixtures of the tests are synthetic.

Open questions

OQ-1open question

Whether the registers of mappings and variances should be loadable from a spreadsheet by runLoad, which they can be today as any table.

Key entities

  • Mapping - a source value, the canonical value, how they relate and when they hold.
  • Variance - a decided difference between a source and the canonical form, with an end.
  • Null rule - what a missing value of a field may be.

Success criteria

SC-001success criterion

For any source value, which canonical value it means at a time, who approved that and what is not covered is a query; and what a missing value stands for is a rule and not a guess.

Review & acceptance checklist

  • [x] Every requirement is testable (MUST / MUST NOT), not aspirational
  • [x] No company fact is asserted here
  • [x] Every open item is marked, not silently decided

Cited by

Data, models/meaning/mod, sql/meaning/mod

Titles, headings, exports, tables, columns and requirements. Esc closes.