.. -*- rst -*-

.. highlightlang:: none

.. groonga-command
.. database: commands_table_tokenize

``table_tokenize``
==================

Summary
-------

``table_tokenize`` command tokenizes text by the specified table's tokenizer.

Syntax
------

``table_tokenize`` command has required parameters and optional parameters.
``table`` and ``string`` are required parameters. Others are
optional::

  table_tokenize table
                 string
                 [flags=NONE]
                 [mode=GET]

Usage
-----

Here is a simple example.

.. groonga-command
.. include:: ../../example/reference/commands/table_tokenize/simple_example.log
.. register token_filters/stop_word
.. table_create Terms TABLE_PAT_KEY ShortText   --default_tokenizer TokenBigram   --normalizer NormalizerAuto   --token_filters TokenFilterStopWord
.. column_create Terms is_stop_word COLUMN_SCALAR Bool
.. load --table Terms
.. [
.. {"_key": "and", "is_stop_word": true}
.. ]
.. table_tokenize Terms "Hello and Good-bye" --mode GET

``Terms`` table is set ``TokenBigram`` tokenizer, ``NormalizerAuto`` normalizer,
``TokenFilterStopWord`` token filter. It returns tokens that is
generated by tokenizeing ``"Hello and Good-bye"`` with ``TokenBigram`` tokenizer.
It is normalized by ``NormalizerAuto`` normalizer.
``and`` token is removed with ``TokenFilterStopWord`` token filter.

Parameters
----------

This section describes all parameters. Parameters are categorized.

Required parameters
^^^^^^^^^^^^^^^^^^^

There are required parameters, ``table`` and ``string``.

``table``
"""""""""

It specifies the lexicon table. ``table_tokenize`` command uses the
tokenizer, the normalizer, the token filters that is set the
lexicon table.

``string``
""""""""""

It specifies any string which you want to tokenize.

See :ref:`tokenize-string` option in :doc:`/reference/commands/tokenize` about details.

Optional parameters
^^^^^^^^^^^^^^^^^^^

There are optional parameters.

``flags``
"""""""""

It specifies a tokenization customize options. You can specify
multiple options separated by "``|``".

The default value is ``NONE``.

See :ref:`tokenize-flags` option in :doc:`/reference/commands/tokenize` about details.

``mode``
""""""""

It specifies a tokenize mode.

The default value is ``GET``.

See :ref:`tokenize-mode` option in :doc:`/reference/commands/tokenize` about details.

Return value
------------

``table_tokenize`` command returns tokenized tokens.

See :ref:`tokenize-return-value` option in :doc:`/reference/commands/tokenize` about details.

See also
--------

* :doc:`/reference/tokenizers`
* :doc:`/reference/commands/tokenize`