README.md in json-streamer-2.0.1 vs README.md in json-streamer-2.1.0

- old
+ new

@@ -1,34 +1,47 @@ # Json::Streamer -#### Ruby utility that supports JSON streaming allowing you to get data based on various criteria (key, nesting level, etc). +#### Ruby gem for getting data from JSON streams based on various criteria (key, nesting level, etc). +Status and support + +- &#x2714; stable +- &#x2714; supported +- &#x2716; no ongoing development + +<!--- Version informartion --> +*You are viewing the README of version [v2.1.0](https://github.com/thisismydesign/json-streamer/releases/tag/v2.1.0). You can find other releases [here](https://github.com/thisismydesign/json-streamer/releases).* +<!--- Version informartion end --> + | Branch | Status | | ------ | ------ | | Release | [![Build Status](https://travis-ci.org/thisismydesign/json-streamer.svg?branch=release)](https://travis-ci.org/thisismydesign/json-streamer) [![Coverage Status](https://coveralls.io/repos/github/thisismydesign/json-streamer/badge.svg?branch=release)](https://coveralls.io/github/thisismydesign/json-streamer?branch=release) [![Gem Version](https://badge.fury.io/rb/json-streamer.svg)](https://badge.fury.io/rb/json-streamer) [![Total Downloads](http://ruby-gem-downloads-badge.herokuapp.com/json-streamer?type=total)](https://rubygems.org/gems/json-streamer) | | Development | [![Build Status](https://travis-ci.org/thisismydesign/json-streamer.svg?branch=master)](https://travis-ci.org/thisismydesign/json-streamer) [![Coverage Status](https://coveralls.io/repos/github/thisismydesign/json-streamer/badge.svg?branch=master)](https://coveralls.io/github/thisismydesign/json-streamer?branch=master) | -*If you've tried JSON streaming with other Ruby libraries before (e.g. [JSON::Stream](https://github.com/dgraham/json-stream), [Yajl::FFI](https://github.com/dgraham/yajl-ffi)):* +#### If you've tried JSON streaming with other Ruby libraries before (e.g. [JSON::Stream](https://github.com/dgraham/json-stream), [Yajl::FFI](https://github.com/dgraham/yajl-ffi)) This gem will basically spare you the need to define your own callbacks (i.e. implement an actual JSON parser using `start_object`, `end_object`, `key`, `value`, etc.). +#### If you're new to this -*If you're new to this:* - Streaming is useful for - big files that do not fit in the memory (or you'd rather avoid the risk) - files read in chunks (e.g. arriving over network) - cases where you expect some issue with the file (e.g. losing connection to source, invalid data at some point) but would like to get as much data as possible anyway This gem is aimed at making streaming as easy and convenient as possible. -*Performance:* +#### Performance -The gem uses JSON::Stream's events in the background. It was chosen because it's a pure Ruby parser. -A similar implementation can be done using the ~10 times faster Yajl::FFI gem that is dependent on the native YAJL library. +Highly depends on the event generator. Out of the box the gem uses [JSON::Stream](https://github.com/dgraham/json-stream). It was chosen because it's a pure Ruby parser with no runtime dependencies. You can use any custom event generator, such as [Yajl::FFI](https://github.com/dgraham/yajl-ffi) which is dependent on the native YAJL library and is [~10 times faster](https://github.com/dgraham/yajl-ffi#performance). See the [Custom event generators](#custom-event-generators) chapter. + I did not measure the performance of my implementation on top of these libraries. +#### Dependencies + +The gem's single runtime dependency is [JSON::Stream](https://github.com/dgraham/json-stream). It is only loaded if the default event generator is used. + ## Installation Add this line to your application's Gemfile: ```ruby @@ -43,22 +56,18 @@ $ gem install json-streamer ## Usage -Check the unit tests for more examples ([spec/streamer_spec.rb](spec/json/streamer/json_streamer_spec.rb)). - -One `streamer` object handles one set of conditions. For multiple conditions create multiple streamers. For more details see [this discussion](https://github.com/thisismydesign/json-streamer/issues/9). - ```ruby require 'json/streamer' ``` -### v1.2 (and above) API - #### Passing IO upfront +Since [v1.2.0](https://github.com/thisismydesign/json-streamer/releases/tag/v1.2.0) + ```ruby file_stream = File.open('data.json', 'r') chunk_size = 500 # defaults to 1000 streamer = Json::Streamer.parser(file_io: file_stream, chunk_size: chunk_size) @@ -179,14 +188,28 @@ def receive_data(data) streamer << data end ``` +#### Custom event generators + +Since [v2.1.0](https://github.com/thisismydesign/json-streamer/releases/tag/v2.1.0) + +```ruby +require "yajl/ffi" + +Json::Streamer.parser(event_generator: Yajl::FFI::Parser.new) +``` + +Any parser can be used that provides the right events. The gem is tested with [Yajl::FFI](https://github.com/dgraham/yajl-ffi) and [JSON::Stream](https://github.com/dgraham/json-stream). + #### Custom yield conditions -[v2.0.0](https://github.com/thisismydesign/json-streamer/releases/tag/v2.0.0) introduces custom conditions which provide ultimate control over what to yield. +Since [v2.0.0](https://github.com/thisismydesign/json-streamer/releases/tag/v2.0.0) +Custom conditions provide ultimate control over what to yield. + The Conditions API exposes 3 callbacks: - `yield_value` - `yield_array` - `yield_object` @@ -244,12 +267,66 @@ ```ruby {"key1"=>"value"} {"key2"=>"value"} ``` -### Legacy API (pre-v1.2) +#### Get an Enumerable when not passing a block +Since [v2.1.0](https://github.com/thisismydesign/json-streamer/releases/tag/v2.1.0) + +When _not_ passed a block both `get` and `get_with_conditions` return an enumerator of the requested objects. When passed a block they return an empty enumerator. This means that **when _not_ passed a block the requested objects will accumulate in memory**. + +Without block + +```ruby +objects = streamer.get(nesting_level:1) +p objects +``` + +Input: +```json +{ + "object1": "first_level_value", + "object2": {} +} +``` + +Output: +```ruby +["first_level_value", {}] +``` + +With block + +```ruby +unyielded_objects = streamer.get(nesting_level:1) { |object| do_something(object) } +p unyielded_objects +``` + +Input: +```json +{ + "object1": "first_level_value", + "object2": {} +} +``` + +Output: +```ruby +[] +``` + +#### Other usage information + +Check the unit tests for more examples ([spec/streamer_spec.rb](spec/json/streamer/json_streamer_spec.rb)). + +One `streamer` object handles one set of conditions. For multiple conditions create multiple streamers. For more details see [this discussion](https://github.com/thisismydesign/json-streamer/issues/9). + +#### Deprecated API + +Pre [v1.2.0](https://github.com/thisismydesign/json-streamer/releases/tag/v1.2.0) + This functionality is deprecated but kept for compatibility reasons. ```ruby # Same as Json::Streamer.parser streamer = Json::Streamer::JsonStreamer.new @@ -258,30 +335,24 @@ ```ruby # Same as streamer << data streamer.parser << data ``` -## Feedback +## Contribution and feedback -Any feedback is much appreciated. +This project is built around known use-cases. If have one that isn't covered don't hesitate to open an issue and start a discussion. -I can only tailor this project to fit use-cases I know about - which are usually my own ones. If you find that this might be the right direction to solve your problem too but you find that it's suboptimal or lacks features don't hesitate to contact me. +Bug reports and pull requests are welcome on GitHub at https://github.com/thisismydesign/json-streamer. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct. -Please let me know if you make use of this project so that I can prioritize further efforts. - ## Conventions This gem is developed using the following conventions: - [Bundler's guide for developing a gem](http://bundler.io/v1.14/guides/creating_gem.html) - [Better Specs](http://www.betterspecs.org/) - [Semantic versioning](http://semver.org/) - [RubyGems' guide on gem naming](http://guides.rubygems.org/name-your-gem/) - [RFC memo about key words used to Indicate Requirement Levels](https://tools.ietf.org/html/rfc2119) - [Bundler improvements](https://github.com/thisismydesign/bundler-improvements) - -## Contributing - -Bug reports and pull requests are welcome on GitHub at https://github.com/thisismydesign/json-streamer. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct. ## License The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).