IF Commons GitHub

token

The tokens of SQL text (commons lang-sql 0001-sql-lang FR-005): a closed set of kinds and the parallel columns the lexer fills, so that a token costs no object, no string and no allocation of its own. A column is read with its accessor, which answers the end of the text past the last token, so that a parser may look ahead without a bounds check.

The columns are arrays of small integers, which the engine keeps packed; they carry the types of the kinds, so that no cast is needed to read one.

const t = lex("select a", ansi.compiled).tokens;
kindAt(t, 0) === Tok.Word; // true
textAt(t, "select a", 1);  // "a"

Spec: 0001-sql-lang FR-005

Source: commons/lang-sql/src/token.ts

4 of 9 exports are imported by a test of the run. A test that imports an export may not call it or check what it returns.

endAtfunctionno test imports it
function endAt(t: Tokens, i: number): number

Where the token at i ends; the length of the text past the last.

idAtfunction
function idAt(t: Tokens, i: number): number

The id of the token at i (see Tokens); 0 past the last.

Imported by 5 tests of the run (5 passed, 0 failed): the library's results

kindAtfunction
function kindAt(t: Tokens, i: number): TokKind

The kind of the token at i; the end of the text past the last.

Imported by 21 tests of the run (21 passed, 0 failed): the library's results

startAtfunctionno test imports it
function startAt(t: Tokens, i: number): number

Where the token at i begins; the length of the text past the last.

textAtfunctionno test imports it
function textAt(t: Tokens, source: string, i: number): string

The text of the token at i, cut from the source only now.

TokKindtypeno test imports it
type TokKind = (typeof Tok)[keyof typeof Tok];

A kind of token.

Tokconst
const Tok: { readonly Eof: 0; readonly Word: 1; readonly QuotedIdent: 2; readonly Number: 3; readonly String: 4; readonly Param: 5; readonly Op: 6; readonly LParen: 7; readonly RParen: 8; readonly LBracket: 9; readonly RBracket: 10; readonly LBrace: 11; readonly RBrace: 12; readonly Comma: 13; readonly Semicolon: 14; readonly Dot: 15; readonly Error: 16; }

The kinds of token.

Imported by 21 tests of the run (21 passed, 0 failed): the library's results

tokenCountfunction
function tokenCount(t: Tokens): number

How many tokens there are, the end of the text included.

Imported by 5 tests of the run (5 passed, 0 failed): the library's results

Tokensinterfaceno test imports it
interface Tokens

The tokens of a text, as columns. id is, by kind: for a word the keyword's id (0 where it is none); for an operator the operator's id; for a parameter its style (ParamStyleId); for a string the index of its syntax plus one (0 for a dollar quote); for a quoted identifier the index of its syntax; otherwise 0.

MemberTypeWhat it is
kindreadonly TokKind[]
startreadonly number[]
endreadonly number[]
idreadonly number[]

Titles, headings, exports, tables, columns and requirements. Esc closes.