Sha256: f7a64f283ac73e11086940383f682b2d145b287623343eccae1d30f6fb9f1af7
Contents?: true
Size: 1.72 KB
Versions: 9
Compression:
Stored size: 1.72 KB
Contents
[](https://badge.fury.io/rb/tiktoken_ruby)
# tiktoken_ruby
[Tiktoken](https://github.com/openai/tiktoken) is BPE tokenizer from OpenAI used with their GPT models.
This is a wrapper around it aimed primarily at enabling accurate counts of GPT model tokens used.
## Installation
Install the gem and add to the application's Gemfile by executing:
$ bundle add tiktoken_ruby
If bundler is not being used to manage dependencies, install the gem by executing:
$ gem install tiktoken_ruby
## Usage
Usage should be very similar to the python library. here's a simple example
Encode and decode text
```ruby
require 'tiktoken_ruby'
enc = Tiktoken.get_encoding("cl100k_base")
enc.decode(enc.encode("hello world")) #=> "hello world"
```
Encoders can also be retrieved by model name
```ruby
require 'tiktoken_ruby'
enc = Tiktoken.encoding_for_model("gpt-4")
enc.encode("hello world").length #=> 2
```
## Development
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
## Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/iapark/tiktoken_ruby.
## License
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
Version data entries
9 entries across 9 versions & 1 rubygems