A default tokenizer for handling syntaxes that are not explicitly handled elsewhere. It simply yields the given text as a single token.

Methods
Public Instance methods
tokenize( text ) {|Token.new( text, :normal )| ...}

Yield the given text as a single token.

[ show source ]

    # File lib/syntax.rb, line 10
10:     def tokenize( text )
11:       yield Token.new( text, :normal )
12:     end