`finesse.script.tokenizer`¶

Tokenization for kat script.

This differs from a typical tokenizer in that it tokenizes whitespace and newline characters, which have important meanings in kat script and help maintain formatting in later model serialization.

Inspired by the Python standard library tokenize module and lex.py from David Beazley’s SLY (https://github.com/dabeaz/sly/).