8.7. Tokenizers
TODO: Write me.
Here are the list of built-in tokenizers:
- TokenBigram
- TokenBigramSplitSymbol
- TokenBigramSplitSymbolAlpha
- TokenBigramSplitSymbolAlphaDigit
- TokenBigramIgnoreBlank
- TokenBigramIgnoreBlankSplitSymbol
- TokenBigramIgnoreBlankSplitAlpha
- TokenBigramIgnoreBlankSplitAlphaDigit
- TokenDelimit
- TokenDelimitNull
- TokenTrigram
- TokenUnigram
日本語版はこちら