README.md in roseflow-tiktoken-0.1.0 vs README.md in roseflow-tiktoken-0.2.0
- old
+ new
@@ -1,10 +1,10 @@
# Tiktoken tokenizer for Roseflow
[tiktoken](https://github.com/openai/tiktoken) is a fast BPE tokenizer for use with OpenAI's models. `roseflow-tiktoken` gem helps you use the tokenizer in Ruby, especially with (Roseflow)[https://github.com/ljuti/roseflow].
-Currently, this gem wraps the (`tiktoken` Python module)[https://github.com/openai/tiktoken] for convenient use in Roseflow.
+This gem wraps the (`tiktoken_ruby` gem)[https://github.com/IAPark/tiktoken_ruby] for convenient use in Roseflow.
## Installation
Install the gem and add to the application's Gemfile by executing:
@@ -42,9 +42,11 @@
| ----------------------- | ------------------------------------------------------------------------- |
| `cl100k_base` | ChatGPT models, `text-embedding-ada-002` |
| `p50k_base` | Code models, `text-davinci-002`, `text-davinci-003` |
| `p50k_edit` | Use for edit models like `text-davinci-edit-001`, `code-davinci-edit-001` |
| `r50k_base` (or `gpt2`) | GPT-3 models like `davinci` |
+
+If a model is not provided or is unknown to the library, it will default to `cl100k_base` encoding.
## Development
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.