IF Commons GitHub

Data loads, extracts and ETL/ELT: connectors and native bulk paths are left

highin-progress

Built (0054-sql-loads, `sql/loads`): typed sources (rows, CSV with typed cells and rejected lines, JSON lines, a select of another database), loads in four modes (append, skip-existing, upsert, replace) in one transaction with a ledger of runs, ELT through a staging table gated by SQL-DQ tests, a high-water mark for incremental loads, extracts to CSV and JSON lines, and a copy between any two of SQLite, DuckDB and PostgreSQL. Left, each a connector on `Source` and `Load`: dialect-native bulk paths (PostgreSQL COPY, DuckDB read_csv, read_parquet and COPY TO, SQLite .import), Parquet and Arrow, HTTP and API sources with paging, object stores, queues, change data capture, and orchestration (schedules, dependencies, retries). The ledger of loads is a table made by the load; whether it becomes a model is OQ-1 of the spec.

LibraryData
Severityhigh
PriorityP1
Statusin-progress by claude
Sizexl
GoalsG1, G2, G5
Raised2026-11-19

Requirements it is about

Evidence

Titles, headings, exports, tables, columns and requirements. Esc closes.