IF Commons GitHub

lexical

How a dialect is written (commons lang-sql 0001-sql-lang FR-015): a Lexical is data (its comments, quoted identifiers, strings and their prefixes, parameters, operators, keywords and numbers), and compileLexical turns it, once, into the tables the lexer reads: a class for each ASCII character, the rules that can begin at each character, the longest operators first, and a hash table of the keywords. A lexical that cannot be compiled is an error value that names the field.

Every opener, marker and operator must begin with an ASCII character, so that the lexer chooses the rules at a character by one table lookup; a word may hold any character above ASCII.

const r = compileLexical({ ...ANSI_LEXICAL, comments: [{ kind: "line", marker: "#" }] });
r.kind === "ok"; // true

Spec: 0001-sql-lang FR-007, FR-008, FR-009, FR-010, FR-011, FR-015

Source: commons/lang-sql/src/lexical.ts

6 of 14 exports are imported by a test of the run. A test that imports an export may not call it or check what it returns.

CommentSyntaxtypeno test imports it
type CommentSyntax = | { readonly kind: "line"; readonly marker: string; /** Whether the marker is a… (767 characters)
type CommentSyntax =
  | {
      readonly kind: "line";
      readonly marker: string;
      /** Whether the marker is a comment only when white space, a control character or the end of the text follows it (MySQL's `--`). */
      readonly spaceAfter?: boolean;
    }
  | {
      readonly kind: "block";
      readonly open: string;
      readonly close: string;
      /** Whether a comment inside the comment must be closed too (PostgreSQL), or the first `close` ends it (SQLite). */
      readonly nested: boolean;
      /** Whether a comment that begins `open` then `+` is an optimizer hint. */
      readonly hints?: boolean;
      /** Whether a comment that begins `open` then `!` is a conditional comment (MySQL). */
      readonly conditional?: boolean;
    };

A kind of comment: a line comment runs from its marker to the end of the line, a block comment from open to close.

CompiledLexicalinterfaceno test imports it
interface CompiledLexical

A lexical made into the tables the lexer reads.

MemberTypeWhat it is
lexicalLexical
clsUint8Array<ArrayBufferLike>The class flags of each ASCII character (0 to 127).
commentsByFirstreadonly (readonly CommentRule[])[]
stringsByFirstreadonly (readonly StringRule[])[]
identsByFirstreadonly (readonly IdentRule[])[]
prefixesReadonlyMap<string, Escape | undefined>The prefixes (upper case) with the escape each gives, undefined where it gives none.
hasUnicodePrefixboolean
opsByFirstreadonly (readonly OperatorRule[])[]The operators of two or more characters that begin at each ASCII character, the longest first.
singleOpUint16Array<ArrayBufferLike>The id of the one-character operator at each ASCII character, 0 where it is none.
operatorsreadonly string[]Every operator's text; the id of the operator at index i is i + 1.
paramBitsnumberThe bit of each style in ParamStyleId that the dialect has: 1 << id.
dollarQuotesboolean
kwTableInt32Array<ArrayBufferLike>The keyword hash table: the keyword's id (its index plus one), 0 where a slot is empty.
kwMasknumber
kwWordsreadonly string[]
kwMinnumber
kwMaxnumber
radixboolean
underscoresboolean
numberSuffixesreadonly string[]
qQuotingboolean
compileLexicalfunction
function compileLexical(lexical: Lexical): Result<CompiledLexical, LexicalError>

Compiles a lexical.

ParameterWhat it is
lexicalhow a dialect is written

Returns: the tables the lexer reads, or what is wrong with the lexical

Imported by 13 tests of the run (13 passed, 0 failed): the library's results

describeLexicalErrorfunction
function describeLexicalError(e: LexicalError): string

One sentence for a lexical's error.

Imported by 3 tests of the run (3 passed, 0 failed): the library's results

Escapetypeno test imports it
type Escape = "double" | "backslash" | "both" | "none";

How a quote is written inside a string: by writing it twice, by a backslash before it, either, or not at all.

IdentSyntaxinterfaceno test imports it
interface IdentSyntax

A quoted identifier: what opens it and what closes it; the close is written twice inside.

MemberTypeWhat it is
openstring
closestring
keywordIdfunction
function keywordId(lx: CompiledLexical, word: string): number | undefined

The id of a keyword in a compiled lexical, or undefined where the dialect does not list it; the id is what a word token carries.

Imported by 5 tests of the run (5 passed, 0 failed): the library's results

Lexicalinterface
interface Lexical

What a dialect's text is made of.

MemberTypeWhat it is
commentsreadonly CommentSyntax[]
identifiersreadonly IdentSyntax[]
stringsreadonly StringSyntax[]
prefixesreadonly StringPrefix[]
dollarQuotesbooleanWhether $$ ... $$ and $tag$ ... $tag$ are strings.
paramsreadonly ParamStyle[]
operatorsreadonly string[]The operators of two or more characters; each of one character is read as an operator too.
keywordsreadonly string[]The words the dialect knows; the lexer tells a parser which words these are and never judges where they are keywords.
wordCharsstringASCII characters, besides letters, digits and _, that a word may hold after its first character ($ in PostgreSQL).
numbers{ readonly radix: boolean; readonly underscores: boolean; }
numberSuffixesreadonly string[]Upper-case letters after a number that belong to it (L, BD in Hive: 1L, 1.5BD).
wordStartCharsstringASCII characters, besides letters and _, that may begin a word (# for T-SQL's #temp).
qQuotingbooleanWhether q'[...]' is a string whose delimiter is the character after the quote (Oracle).

Imported by 8 tests of the run (8 passed, 0 failed): the library's results

LexicalErrortypeno test imports it
type LexicalError =
  | { readonly kind: "empty"; readonly field: string }
  | { readonly kind: "not-ascii"; readonly field: string; readonly value: string }
  | { readonly kind: "duplicate"; readonly field: string; readonly value: string }
  | { readonly kind: "invalid"; readonly field: string; readonly value: string };

What is wrong with a lexical, one sentence each (see describeLexicalError).

operatorIdfunction
function operatorId(lx: CompiledLexical, text: string): number | undefined

The id of an operator in a compiled lexical, or undefined where it is not one; the id is what an operator token carries.

Imported by 5 tests of the run (5 passed, 0 failed): the library's results

ParamStyletypeno test imports it
type ParamStyle = "question" | "dollar-number" | "dollar-name" | "colon-name" | "at-name";

The styles of a parameter.

ParamStyleIdconst
const ParamStyleId: { readonly Question: 1; readonly DollarNumber: 2; readonly DollarName: 3; readonly ColonName: 4; readonly AtName: 5; }

The id a parameter token carries for its style.

Imported by 5 tests of the run (5 passed, 0 failed): the library's results

StringPrefixinterfaceno test imports it
interface StringPrefix

A letter or letters before a string's opening quote, such as E, and the escape it gives the string, where it changes it.

MemberTypeWhat it is
prefixstring
escape?Escape
StringSyntaxinterfaceno test imports it
interface StringSyntax

A string: what opens it, what closes it, and how a quote is written inside.

MemberTypeWhat it is
openstring
closestring
escapeEscape

Titles, headings, exports, tables, columns and requirements. Esc closes.