Contents

module TextRank
  module Tokenizer
    ##
    # A tokenizer regex that preserves single whitespace characters as a token. Use
    # this if one or more of your TokenFilter classes need whitespace in order to
    # make decisions.
    ##
    Whitespace = %r{\s}

  end
end

Version data entries

6 entries across 6 versions & 1 rubygems

Version	Path
text_rank-1.2.3	lib/text_rank/tokenizer/whitespace.rb
text_rank-1.2.2	lib/text_rank/tokenizer/whitespace.rb
text_rank-1.2.0	lib/text_rank/tokenizer/whitespace.rb
text_rank-1.1.7	lib/text_rank/tokenizer/whitespace.rb
text_rank-1.1.6	lib/text_rank/tokenizer/whitespace.rb
text_rank-1.1.5	lib/text_rank/tokenizer/whitespace.rb