8.7. Tokenizers¶

TODO: Write me.

Here are the list of built-in tokenizers:

TokenBigram
TokenBigramSplitSymbol
TokenBigramSplitSymbolAlpha
TokenBigramSplitSymbolAlphaDigit
TokenBigramIgnoreBlank
TokenBigramIgnoreBlankSplitSymbol
TokenBigramIgnoreBlankSplitAlpha
TokenBigramIgnoreBlankSplitAlphaDigit
TokenDelimit
TokenDelimitNull
TokenTrigram
TokenUnigram