Sha256: 531e68f9879f3bf8c4fa17fcdebea8b32eb054478cdb501829527c3e3cf09fe4
Contents?: true
Size: 429 Bytes
Versions: 70
Compression:
Stored size: 429 Bytes
Contents
.. -*- rst -*- .. highlightlang:: none Tokenizers ========== TODO: Write me. Here are the list of built-in tokenizers: * TokenBigram * TokenBigramSplitSymbol * TokenBigramSplitSymbolAlpha * TokenBigramSplitSymbolAlphaDigit * TokenBigramIgnoreBlank * TokenBigramIgnoreBlankSplitSymbol * TokenBigramIgnoreBlankSplitAlpha * TokenBigramIgnoreBlankSplitAlphaDigit * TokenDelimit * TokenDelimitNull * TokenTrigram * TokenUnigram
Version data entries
70 entries across 35 versions & 1 rubygems