Module dlib.text.lexer
General-purpose non-allocating lexical analyzer.
Description
Breaks the input string to a stream of lexemes according to a given delimiter dictionary. Delimiters are symbols that separate sequences of characters (e.g. operators). Lexemes are slices of the input string. Assumes UTF-8 input. Treats \r\n as a single \n.
Classes
| Name | Description |
|---|---|
Lexer
|
Lexical analyzer class |