Sha256: 0514af6f6abbc1fa3e7595aff8cf71b0867738b65fb7e99274f29537b2ffe688
Contents?: true
Size: 1.5 KB
Versions: 9
Compression:
Stored size: 1.5 KB
Contents
module Rley # This module is used as a namespace
module Lexical # This module is used as a namespace
# In Rley, a (lexical) token is an object created by a lexer (tokenizer)
# and passed to the parser. Such token an object is created when a lexer
# detects that a sequence of characters(a lexeme) from the input stream
# is an instance of a terminal grammar symbol.
# Say, that in a particular language, the lexeme 'foo' is an occurrence
# of the terminal symbol IDENTIFIER. Then the lexer will return a Token
# object that states the fact that 'foo' is indeed an IDENTIFIER. Basically,
# a Token is a pair (lexeme, terminal): it asserts that a given lexeme
# is an instance of given terminal symbol.
class Token
# The sequence of character(s) from the input stream that is an occurrence
# of the related terminal symbol.
# @return [String] Input substring that is an instance of the terminal.
attr_reader(:lexeme)
# @return [Syntax::Terminal] Terminal symbol corresponding to the lexeme.
attr_reader(:terminal)
# Constructor.
# @param theLexeme [String] the lexeme (= piece of text from input)
# @param aTerminal [Syntax::Terminal]
# The terminal symbol corresponding to the lexeme.
def initialize(theLexeme, aTerminal)
raise 'Internal error: nil terminal symbol detected' if aTerminal.nil?
@lexeme = theLexeme
@terminal = aTerminal
end
end # class
end # module
end # module
# End of file
Version data entries
9 entries across 9 versions & 1 rubygems