IF Commons GitHub

lexer

The lexer of SQL (commons lang-sql 0001-sql-lang): one pass over the character codes of a text, by the tables of a compiled lexical, that returns the tokens, the comments and the errors, so that every character is in exactly one token, one comment or white space. It does not throw, does not judge where a word is a keyword, and names no dialect: what a dialect writes is its lexical.

The hot path reads each character once: white space, then a comment (the one table lookup that says whether a character may begin one), a word with its keyword found by a hash of its upper-cased characters, a number, a quoted string or identifier, a parameter, punctuation, then an operator, the longest of the dialect's first. A token is four pushes onto arrays of small integers; its text is cut from the source only when someone asks (textAt).

const { tokens, comments, errors } = lex("select 1 -- one", ansi.compiled);
errors.length; // 0

Spec: 0001-sql-lang FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-018

Source: commons/lang-sql/src/lexer.ts

2 of 4 exports are imported by a test of the run. A test that imports an export may not call it or check what it returns.

describeLexErrorfunction
function describeLexError(e: LexError): string

One sentence for an error of the lexer.

Imported by 5 tests of the run (5 passed, 0 failed): the library's results

LexErrortypeno test imports it
type LexError =
  | { readonly kind: "unterminated-comment"; readonly span: Span }
  | { readonly kind: "unterminated-string"; readonly span: Span }
  | { readonly kind: "unterminated-identifier"; readonly span: Span }
  | { readonly kind: "unexpected-character"; readonly span: Span; readonly char: string };

What the lexer found that the dialect cannot read.

Lexedinterfaceno test imports it
interface Lexed

The result of lexing a text.

MemberTypeWhat it is
textstring
tokensTokens
commentsComments
errorsreadonly LexError[]
lexfunction
function lex(text: string, lx: CompiledLexical, from?: number, to?: number): Lexed

Lexes a text.

ParameterWhat it is
textSQL, in any dialect
lxhow that dialect is written, compiled
fromwhere to begin, 0 by default; the tokens, comments and errors have offsets of the whole text
towhere to end, the end of the text by default

Returns: its tokens (the last is the end of the text), its comments and what it holds that the dialect cannot read

Imported by 31 tests of the run (31 passed, 0 failed): the library's results

Titles, headings, exports, tables, columns and requirements. Esc closes.