README.md in roseflow-tiktoken-0.1.0 vs README.md in roseflow-tiktoken-0.2.0

- old
+ new

@@ -1,10 +1,10 @@ # Tiktoken tokenizer for Roseflow [tiktoken](https://github.com/openai/tiktoken) is a fast BPE tokenizer for use with OpenAI's models. `roseflow-tiktoken` gem helps you use the tokenizer in Ruby, especially with (Roseflow)[https://github.com/ljuti/roseflow]. -Currently, this gem wraps the (`tiktoken` Python module)[https://github.com/openai/tiktoken] for convenient use in Roseflow. +This gem wraps the (`tiktoken_ruby` gem)[https://github.com/IAPark/tiktoken_ruby] for convenient use in Roseflow. ## Installation Install the gem and add to the application's Gemfile by executing: @@ -42,9 +42,11 @@ | ----------------------- | ------------------------------------------------------------------------- | | `cl100k_base` | ChatGPT models, `text-embedding-ada-002` | | `p50k_base` | Code models, `text-davinci-002`, `text-davinci-003` | | `p50k_edit` | Use for edit models like `text-davinci-edit-001`, `code-davinci-edit-001` | | `r50k_base` (or `gpt2`) | GPT-3 models like `davinci` | + +If a model is not provided or is unknown to the library, it will default to `cl100k_base` encoding. ## Development After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.