finesse.script.tokenizer module

Tokenization for kat script.

This differs from a typical tokenizer in that it tokenizes whitespace and newline characters, which have important meanings in kat script and help maintain formatting in later model serialization.

Inspired by the Python standard library tokenize module and lex.py from David Beazley’s SLY (https://github.com/dabeaz/sly/).

Sean Leavey <sean.leavey@ligo.org>

class finesse.script.tokenizer.KatTokenizer

Bases: object

Kat script token generator.

Regular expressions define patterns to match prototypes. Prototypes are then either emitted as tokens as-is, or modified by callback methods defined in this class.

Parameters

graphKatGraph, optional

The parse graph to add parsed tokens to. Defaults to a new KatGraph.

on_LBRACKET(token)

on_LPAREN(token)

on_NEWLINE(token)

on_RBRACKET(token)

on_RPAREN(token)

tokenize(string)

Tokenize specified string.

Parameters

stringstr

The string to tokenize kat script from.

Yields

KatToken

The next token read from string.

tokenize_file(fobj)

Generate tokens from the given file.

Parameters

fobjio.FileIO

The file object to tokenize kat script from. This should be opened in text mode.

Yields

KatToken

The next token read from fobj.