The tokens MUST be kept as parallel columns (kind, start, end, id), each with a kind from a closed set (word, quoted identifier, number, string, parameter, operator, the punctuation, error, end of text), the last token always the end of text; a word that the dialect lists as a keyword MUST carry the keyword's id, found without regard to case, and a word with a character outside ASCII MUST NOT be a keyword.
Enforced (check): .github: cd commons/lang-sql && bun run verify — suite/lexer.test.ts: tokens are columns of a closed set of kinds that end with the end of the text, a keyword is found without regard to case, and a word with a character outside ASCII is never a keyword
Proved by: lang-sql/suite/lexer.test.ts
Tests: passed: suite/lexer.test.ts tokens are columns of a closed set of kinds that end with the end of the text, a keyword i
Cited by: Lang Sql, lexer, token