Description:
Breaks the input string to a stream of lexemes according to a given delimiter dictionary.
Delimiters are symbols that separate sequences of characters (e.g. operators).
Lexemes are slices of the input string.
Assumes UTF-8 input.
Treats \r\n as a single \n.
General-purpose non-allocating lexical analyzer.
Description: Breaks the input string to a stream of lexemes according to a given delimiter dictionary. Delimiters are symbols that separate sequences of characters (e.g. operators). Lexemes are slices of the input string. Assumes UTF-8 input. Treats \r\n as a single \n.