Data loads, extracts and ETL/ELT: connectors and native bulk paths are left
highin-progressBuilt (0054-sql-loads, `sql/loads`): typed sources (rows, CSV with typed cells and rejected lines, JSON lines, a select of another database), loads in four modes (append, skip-existing, upsert, replace) in one transaction with a ledger of runs, ELT through a staging table gated by SQL-DQ tests, a high-water mark for incremental loads, extracts to CSV and JSON lines, and a copy between any two of SQLite, DuckDB and PostgreSQL. Left, each a connector on `Source` and `Load`: dialect-native bulk paths (PostgreSQL COPY, DuckDB read_csv, read_parquet and COPY TO, SQLite .import), Parquet and Arrow, HTTP and API sources with paging, object stores, queues, change data capture, and orchestration (schedules, dependencies, retries). The ledger of loads is a table made by the load; whether it becomes a model is OQ-1 of the spec.
| Library | Data |
| Severity | high |
| Priority | P1 |
| Status | in-progress by claude |
| Size | xl |
| Goals | G1, G2, G5 |
| Raised | 2026-11-19 |
Requirements it is about
Evidence
- file
sql/loads/load.ts - specification
0054-sql-loads