Contents

module OpenNlp
  class Tokenizer < Tool
    self.java_class = Java::opennlp.tools.tokenize.TokenizerME

    def tokenize(str)
      raise ArgumentError, "str must be a String" unless str.is_a?(String)
      @j_instance.tokenize(str).to_ary
    end

    private
    def get_last_probabilities
      @j_instance.getTokenProbabilities.to_ary
    end
  end
end

Version data entries

5 entries across 5 versions & 1 rubygems

Version	Path
open_nlp-0.0.7-java	lib/open_nlp/tokenizer.rb
open_nlp-0.0.6-java	lib/open_nlp/tokenizer.rb
open_nlp-0.0.5-java	lib/open_nlp/tokenizer.rb
open_nlp-0.0.4-java	lib/open_nlp/tokenizer.rb
open_nlp-0.0.3-java	lib/open_nlp/tokenizer.rb