A single token extracted by a tokenizer. It is simply the lexeme itself, decorated with a ‘group’ attribute to identify the type of the lexeme.

Methods
Attributes
[R] group the type of the lexeme that was extracted.
[R] instruction the instruction associated with this token (:none, :region_open, or :region_close)
Public Class methods
new( text, group, instruction = :none )

Create a new Token representing the given text, and belonging to the given group.

[ show source ]

    # File lib/syntax/common.rb, line 19
19:     def initialize( text, group, instruction = :none )
20:       super text
21:       @group = group
22:       @instruction = instruction
23:     end