Contents

module ActsAsIndexed
  class TokenNormalizer

    # Takes an array of tokens.
    # - Downcases the tokens when :normalize_case option is passed.
    # - Removes tokens of :min_token_length when option is passed.
    # Returns the resulting array of tokens.
    def self.process(arr, options={})
      if options[:normalize_case]
        arr = arr.map{ |t| t.downcase }
      end

      if options[:min_token_length]
        arr = arr.reject{ |w| w.size < options[:min_token_length] }
      end

      arr
    end

  end
end

Version data entries

5 entries across 5 versions & 1 rubygems

Version	Path
acts_as_indexed-0.9.0	lib/acts_as_indexed/token_normalizer.rb
acts_as_indexed-0.8.3	lib/acts_as_indexed/token_normalizer.rb
acts_as_indexed-0.8.2	lib/acts_as_indexed/token_normalizer.rb
acts_as_indexed-0.8.1	lib/acts_as_indexed/token_normalizer.rb
acts_as_indexed-0.8.0	lib/acts_as_indexed/token_normalizer.rb