Sha256: a0c038e86cd2c946b7e9184fba07fff4af63c14f2306d173fa9678568a10a071
Contents?: true
Size: 1.89 KB
Versions: 44
Compression:
Stored size: 1.89 KB
Contents
[](https://badge.fury.io/rb/tiktoken_ruby)
# tiktoken_ruby
[Tiktoken](https://github.com/openai/tiktoken) is BPE tokenizer from OpenAI used with their GPT models.
This is a wrapper around it aimed primarily at enabling accurate counts of GPT model tokens used.
## Installation
Install the gem and add to the application's Gemfile by executing:
$ bundle add tiktoken_ruby
If bundler is not being used to manage dependencies, install the gem by executing:
$ gem install tiktoken_ruby
## Usage
Usage should be very similar to the python library. Here's a simple example
Encode and decode text
```ruby
require 'tiktoken_ruby'
enc = Tiktoken.get_encoding("cl100k_base")
enc.decode(enc.encode("hello world")) #=> "hello world"
```
Encoders can also be retrieved by model name
```ruby
require 'tiktoken_ruby'
enc = Tiktoken.encoding_for_model("gpt-4")
enc.encode("hello world").length #=> 2
```
## Development
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
## Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/iapark/tiktoken_ruby.
To get started with development:
```sh
git clone https://github.com/IAPark/tiktoken_ruby.git
cd tiktoken_ruby
bundle install
bundle exec rake compile
bundle exec rake spec
```
## License
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
Version data entries
44 entries across 44 versions & 2 rubygems