php-dextrin

Lexer
in package

FinalYes

Hand-written `.dxn` tokenizer, directly off `dxn/DXN.md` ยง1.1's lexical grammar. No parser-generator dependency -- the grammar is small and fully normative.

ws = { " " | "\t" | "\n" | "\r" | "," } is "fully insignificant, everywhere" per the spec -- this lexer skips it (and # comments) between every token rather than emitting whitespace tokens at all.

A few sigils bind tighter than plain "insignificant whitespace everywhere" would suggest:

  • ~D[/~T[/~U[/~r/ are each lexed as one compound token -- no whitespace may appear between ~ and the letter that follows.
  • identifier ":" (the map-entry colon-shorthand key) is lexed as one MAP_KEY token when the : immediately follows the identifier with no whitespace between them; otherwise the : starts a fresh KEYWORD token instead (:name in value position), and the identifier is left as a separate token.
  • keyword (:name / :"...") similarly requires no whitespace between : and what follows.

Operates on the source split into an array of single-codepoint strings (see Token's own doc) rather than raw byte-indexed PHP strings -- \p{ID_Start}/\p{ID_Continue} regex tests then always run against one well-formed codepoint at a time.

Table of Contents

Constants

DIGIT  : mixed = '/^[0-9]$/'
HEX_DIGIT  : mixed = '/^[0-9a-fA-F]$/'
ID_CONTINUE  : mixed = '/^[\p{ID_Continue}\-?!]$/u'
ID_START  : mixed = '/^[\p{ID_Start}_]$/u'
REGEX_FLAG  : mixed = '/^[imsuxfr]$/'
RESERVED  : mixed = ['nil' => true, 'true' => true, 'false' => true...

Properties

$cp  : array<int, string>
$i  : int
$len  : int

Methods

tokenize()  : array<int, Token>
__construct()  : mixed
arrow()  : Token
isDigit()  : bool
isIdContinue()  : bool
isIdStart()  : bool
looksLikeExponent()  : bool
nextToken()  : Token
readChar()  : Token
readColon()  : Token
readIdentifierOrKeywordLike()  : Token
readIdentifierRun()  : string
`identifier = (letter | "_"), {ident_char}, ["/", (letter | "_"), {ident_char}]`. First char already validated present at $this->i.
readNumber()  : Token
readSigil()  : Token
readString()  : Token
readUntil()  : string
Reads a delimited sigil body up to (not including) `$close`, honoring `\$close` as an escaped literal delimiter.
reservedToken()  : Token
run()  : array<int, Token>
single()  : Token
skipTrivia()  : void
slice()  : string
startsWith()  : bool

Constants

DIGIT

private mixed DIGIT = '/^[0-9]$/'

HEX_DIGIT

private mixed HEX_DIGIT = '/^[0-9a-fA-F]$/'

ID_CONTINUE

private mixed ID_CONTINUE = '/^[\p{ID_Continue}\-?!]$/u'

ID_START

private mixed ID_START = '/^[\p{ID_Start}_]$/u'

REGEX_FLAG

private mixed REGEX_FLAG = '/^[imsuxfr]$/'

RESERVED

private mixed RESERVED = ['nil' => true, 'true' => true, 'false' => true, 'NaN' => true, 'Infinity' => true]

Properties

$cp read-only

private array<int, string> $cp

$len read-only

private int $len

Methods

tokenize()

public static tokenize(string $src) : array<int, Token>
Parameters
$src : string
Return values
array<int, Token>

__construct()

private __construct(string $src) : mixed
Parameters
$src : string

isDigit()

private isDigit(string|null $c) : bool
Parameters
$c : string|null
Return values
bool

isIdContinue()

private isIdContinue(string|null $c) : bool
Parameters
$c : string|null
Return values
bool

isIdStart()

private isIdStart(string $c) : bool
Parameters
$c : string
Return values
bool

looksLikeExponent()

private looksLikeExponent(int $at) : bool
Parameters
$at : int
Return values
bool

readIdentifierOrKeywordLike()

private readIdentifierOrKeywordLike(int $start) : Token
Parameters
$start : int
Return values
Token

readIdentifierRun()

`identifier = (letter | "_"), {ident_char}, ["/", (letter | "_"), {ident_char}]`. First char already validated present at $this->i.

private readIdentifierRun() : string
Return values
string

readUntil()

Reads a delimited sigil body up to (not including) `$close`, honoring `\$close` as an escaped literal delimiter.

private readUntil(string $close) : string
Parameters
$close : string
Return values
string

reservedToken()

private reservedToken(string $name, int $start, int $end) : Token
Parameters
$name : string
$start : int
$end : int
Return values
Token

single()

private single(string $type) : Token
Parameters
$type : string
Return values
Token

skipTrivia()

private skipTrivia() : void

slice()

private slice(int $start, int $end) : string
Parameters
$start : int
$end : int
Return values
string

startsWith()

private startsWith(int $at, string $literal) : bool
Parameters
$at : int
$literal : string
Return values
bool
On this page

Search results