Contents

module TextRank
  module Tokenizer
    ##
    # Tokenizer to split on any whitespace
    #
    # = Example
    #
    #  Whitespace.new.tokenize("i should:like to know:which is worse.")
    #  => ["i", "should:like", "to", "know:which", "is", "worse."]
    ##
    class Whitespace < Regex

      def initialize
        super(/\s+/)
      end

    end
  end
end

Version data entries

2 entries across 2 versions & 1 rubygems

Version	Path
text_rank-1.1.1	lib/text_rank/tokenizer/whitespace.rb
text_rank-1.1.0	lib/text_rank/tokenizer/whitespace.rb