IF Commons GitHub

sql/federate/merge

Written to: 3 requirements: 3 proved, 0 partly, 0 failing, 0 with no test recorded. 0059-sql-federate FR-005, 0059-sql-federate FR-006, 0059-sql-federate FR-007

Merging (commons data 0059-sql-federate): the rows of a fan-out are loaded, tagged with the target they came from, into tables of an engine the caller gives (DuckDB is the one meant for it), where the merge is SQL: a fleet_rows table with a column for each column of the answers (the type they all agree on, or text where they do not), the target, when it was collected and a key that is the same for the same row; and a fleet_targets table with what each target answered. Counting by a column and finding the targets that differ are given as queries over those tables.

Source: commons/data/sql/federate/merge.ts

4 of 10 exports are imported by a test of the run. A test that imports an export may not call it or check what it returns.

columnsOffunction
function columnsOf(result: FanOut): readonly MergedColumn[]

The columns of the answers and the type each has: the one every non-null value has, decimal where numbers are whole and not, and text where they disagree.

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

ColumnTypetypeno test imports it
type ColumnType = "text" | "integer" | "decimal" | "boolean";

The type a column of the merged rows has.

countByfunction
function countBy(m: Merged, column: string): Promise<readonly { readonly value: unknown; readonly rows: number; readonly targets: number; }[]>

How many of the rows have each value of a column, and in how many targets: the value, the rows and the targets.

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

Differenceinterfaceno test imports it
interface Difference

What differs between a target and the baseline: a row the baseline has and the target lacks, or the target has and the baseline lacks.

MemberTypeWhat it is
targetstring
kind"extra" | "missing"
rowReadonly<Record<string, unknown>>
differencesfunction
function differences(m: Merged, baseline?: "majority" | { readonly target: string; }): Promise<readonly Difference[]>

The targets that differ from a baseline, row by row. The baseline is the rows that at least half of the targets that answered have (majority, the default), or the rows of one target.

ParameterWhat it is
mthe merge
baselinemajority, or the id of a target

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

Mergedinterfaceno test imports it
interface Merged

The tables of a merge.

MemberTypeWhat it is
columnsreadonly MergedColumn[]
rowsTable<Record<string, unknown>, Record<string, unknown>, "fleet_rows">fleet_rows: the rows of every target that answered.
targetsTable<TargetRow, TargetRow, "fleet_targets">fleet_targets: what each target answered.
onRunnerRuns SQL on the engine the merge is in.
MergedColumninterfaceno test imports it
interface MergedColumn

A column of the merged rows.

MemberTypeWhat it is
namestring
typeColumnType
mixedbooleanWhether the targets gave it in more than one type, and it is text for that.
mergeIntofunction
function mergeInto(result: FanOut, on: Runner): Promise<Merged>

Loads a fan-out into fleet_rows and fleet_targets of an engine; tables of those names that are there are replaced.

ParameterWhat it is
resultthe fan-out
onthe engine and its dialect (DuckDB, where the merge is analytical)

Imported by 7 tests of the run (7 passed, 0 failed): the library's results

tablesOfconstno test imports it
const tablesOf: (m: Merged) => { readonly rows: Table<Record<string, unknown>, Record<string, unknown>, "fleet_rows">; readonly targets: Table<TargetRow, TargetRow, "fleet_targets">; }

A merge's tables as typed tables for a query of the caller's own (0053): from(tablesOf(m).rows)....

TargetRowinterfaceno test imports it
interface TargetRow

A row of fleet_targets.

MemberTypeWhat it is
targetstring
statusstring
errorstring | null
rowsnumber
durationMsnumber
collectedAtstring

Titles, headings, exports, tables, columns and requirements. Esc closes.