finesse.script.highlighter.KatScriptPygmentsLexer.get_tokens_unprocessed

KatScriptPygmentsLexer.get_tokens_unprocessed(text)

This method should process the text and return an iterable of (index, tokentype, value) tuples where index is the starting position of the token within the input text.

It must be overridden by subclasses. It is recommended to implement it as a generator to maximize effectiveness.