= IRB Reboot:\nModernize Implementation and Features

:author
  ITOYANAGI Sakura
:theme
  .
:allotted-time
  38m
: content-source
  RubyKaigi 2018

= Greeting

1st day's morning,

it was cloudy.

= Greeting

2nd day's morning,

it was rainy.

= Greeting

But today...

= Greeting

It's a beautiful day outside.

= Greeting

Birds are singing, flowers are blooming...

= Greeting

On days like these...

= IRB Reboot:\nModernize Implementation and Features

= Let me introduce myself

:name
  ITOYANAGI Sakura
:GitHub
  aycabta
:maintainer
  RDoc

= Community: Asakusa.rb

  # image
  # src = asakusarb.jpg
  # relative-height = 80
  # caption = Asakusa.rb every Ruby Tuesday
  # relative-padding-top = 0
  # relative-padding-bottom = 0
  # relative-padding-right = 0
  # relative-padding-left = 0

= Company:\nSpace Pirates, LLC.

  # image
  # src = space-pirates-logo.svg
  # relative-height = 80
  # caption = Space Pirates, LLC.
  # relative-padding-top = 0
  # relative-padding-bottom = 0
  # relative-padding-right = 0
  # relative-padding-left = 0

= Hobby: Climbing

I planed to climb Mt. Zao, it's the highest mountain in Miyagi. The highest mountain is the nearest place to space. It fits for Space Pirates.

= Hobby: Climbing

But it's so far from this venue, so I went to ((*gorge*)) near here.

= Hobby: Climbing

The ((*gorge*)) means narrow river between escarpments.

= Hobby: Climbing

The Tohoku University official web site provides digging points map for fossils.

= Hobby: Climbing

  # image
  # src = locmap.gif
  # relative-height = 75
  # caption = http://www.museum.tohoku.ac.jp/exhibition_info/mini/fosss/locality/locmap.html
  # relative-padding-top = 0
  # relative-padding-bottom = 0
  # relative-padding-right = 0
  # relative-padding-left = 0

= Hobby: Climbing

  # image
  # src = locmap.gif
  # relative-height = 75
  # caption = A dozen pink points are fossils digging points.
  # relative-padding-top = 0
  # relative-padding-bottom = 0
  # relative-padding-right = 0
  # relative-padding-left = 0

= Hobby: Climbing

  # image
  # src = locmap_venue.png
  # relative-height = 75
  # caption = This venue is hemmed in by many fossils digging points.
  # relative-padding-top = 0
  # relative-padding-bottom = 0
  # relative-padding-right = 0
  # relative-padding-left = 0

= Hobby: Climbing

I went to some digging points.

= Hobby: Climbing

  # image
  # src = waterfall_0.jpg
  # relative-height = 90
  # caption = Waterfall
  # relative-padding-top = 0
  # relative-padding-bottom = 0
  # relative-padding-right = 0
  # relative-padding-left = 0

= Hobby: Climbing

  # image
  # src = waterfall_1.jpg
  # relative-height = 90
  # caption = Waterfall
  # relative-padding-top = 0
  # relative-padding-bottom = 0
  # relative-padding-right = 0
  # relative-padding-left = 0

= Hobby: Climbing

  # image
  # src = gorge_0.jpg
  # relative-height = 90
  # caption = Gorge
  # relative-padding-top = 0
  # relative-padding-bottom = 0
  # relative-padding-right = 0
  # relative-padding-left = 0

= Hobby: Climbing

  # image
  # src = gorge_1.jpg
  # relative-height = 90
  # caption = Gorge
  # relative-padding-top = 0
  # relative-padding-bottom = 0
  # relative-padding-right = 0
  # relative-padding-left = 0

= Hobby: Climbing

I burst through from 4m to 10m many waterfalls into several kilometers gorge.

= Hobby: Climbing

  # image
  # src = bonfire.jpg
  # relative-height = 60
  # caption = I bivouacked in the gorge with a bonfire for cooking rice and miso soup.
  # relative-padding-top = 0
  # relative-padding-bottom = 0
  # relative-padding-right = 0
  # relative-padding-left = 0

= Hobby: Climbing

When I was cooking rice and miso soup early morning by bonfire I was given notice "Today, we have Asakusa.rb" so I escaped the gorge quickly and went to Asakusa.rb by bullet train and joined it and went back to Sendai by midnight highway bus,

= Hobby: Climbing

in a 24 hours period.

And joined pre-party of RubyKaigi.

= Hobby: Climbing

It was the hardest experience of this RubyKaigi.

= Hobby: Climbing

In gorge, I didn't find a fossil of the aimed whale, but found so many ((*shell beds(dense shell fossils)*)) and leaf's fossils.

= Hobby: Climbing

Shell...?

= Today's topic

IRB Reboot:\nModernize Implementation and Features

= Recent years keiju-san's

In the several past RubyKaigis, keiju-san who is godfather of Ruby and the author of IRB talked about old Ruby.

= Recent years keiju-san's

He said

((*"The first language design of Ruby was like shell."*))

in "Ruby Archaeology"\nat RubyKaigi 2013.

= Recent years keiju-san's

After that, keiju-san carries on talking about Ruby and shell.

= Recent years keiju-san's

* 2014: Reish, an unix shell for rubyist.
* 2015: Usage and implementation of Reish which is an Unix shell for Rubyist
* 2017: Irb 20th anniversary memorial session: Reish and Irb2

= Recent years keiju-san's

Next session of Hagi(#rubykaigiC) is keiju-san's one, don't miss it.

= Today's topic

Let's back up a minute.

= Today's topic

I sent 2 patches to IRB.

* #14683 IRB with Ripper
* #14787 Show documents when completion

= #14683 IRB with Ripper\n...How use RDoc use Ripper

I talked about RDoc with Ripper,

"Ruby Parser\n
In IRB 20th Anniversary...\n
Now Let Time Resume"

at RubyKaigi 2017.

= #14683 IRB with Ripper\n...How use RDoc use Ripper

Ruby syntax is very complex.

= #14683 IRB with Ripper\n...How use RDoc use Ripper

Ruby's parser is spaghetti.

* (('wait'))Lexical analyzer is tightly coupled with parser
* (('wait'))parse.y has over 11,000 lines
* (('wait'))The overwhelming weight of ((*syntax*)) to come and the uncertainty of (({((*lex_state*))})) hung over us.

= #14683 IRB with Ripper\n...How use RDoc use Ripper

Ruby's syntax is\nvery dirty.

= #14683 IRB with Ripper\n...How use RDoc use Ripper

Ruby's syntax is\nvery 🙅(('del:dirty'))🙅.

= #14683 IRB with Ripper\n...How use RDoc use Ripper

Ruby's syntax is\nvery 🙆complex🙆.

= #14683 IRB with Ripper\n...How use RDoc use Ripper

Ruby's syntax\n😉abrades😉 parser developer.

= #14683 IRB with Ripper\n...How use RDoc use Ripper

But, the abradable syntax for parser developer\n((*is gentle for Ruby users*))\nby matz.

It's great point of Ruby.

= #14683 IRB with Ripper\n...How use RDoc use Ripper

So RDoc had very many bugs in parsing Ruby code.

= #14683 IRB with Ripper\n...How use RDoc use Ripper

I fixed so many bugs of RDoc, and replaced it fixed RDoc with Ripper version.

= #14683 IRB with Ripper\n...How use RDoc use Ripper

Ripper is one of Ruby's standard libraries of lexical analysis by parse.y.

= #14683 IRB with Ripper\n...How use RDoc use Ripper

I think that Ripper is best way for parsing Ruby code, for following latest Ruby syntax.

= #14683 IRB with Ripper

But IRB implement pure Ruby parser. It's hard to support Ruby's new syntax.

= #14683 IRB with Ripper

I thought that Ripper makes IRB's source code parsing better.

= #14683 IRB with Ripper

I discussed Ruby's REPL with matz, and matz said "I implemented mruby's REPL(mirb), learning from IRB's history".

= #14683 IRB with Ripper

REPL needs when code block will end(close) because REPL evaluates it at the timing.

= #14683 IRB with Ripper

The mirb uses

* token's (({lex_state}))
  * (({parser->lstate}))
* syntax error messages
  * (({parser->error_buffer[0].message}))

(The (({parser})) is a\n(({struct mrb_parser_state})))

= #14683 IRB with Ripper

In CRuby,

* token's (({lex_state}))
  * (({Ripper}))
* syntax error messages
  * (({RubyVM::InstructionSequence}))

= #14683 IRB with Ripper

I ported mirb's implementation to IRB.

= #14683 IRB with Ripper

IRB has some prompt features, (({PROMPT_N})), (({PROMPT_S})), and (({%NNi})).

= #14683 IRB with Ripper

* (({PROMPT_N}))
  * (('wait'))Prompt when\nthe code line is continued
* (({PROMPT_S}))
  * (('wait'))Prompt when\nthe code block is in literal
* (({%NNi}))
  * (('wait'))Nesting level of\nthe code block

= #14683 IRB with Ripper

* (({PROMPT_N}))
  * ((*Ripper*))\n 
* (({PROMPT_S}))
  * ((*Ripper*))\n 
* (({%NNi}))
  * ((*Ripper*))\n 

= #14683 IRB with Ripper

In CRuby, I could resolve the parameters of prompt by Ripper.

= #14683 IRB with Ripper

The (({PROMPT_N})) is a part of "when the code block is ended" logic.

= #14683 IRB with Ripper

Inside "splitted sentence", IRB uses (({PROMPT_N})) prompt.

Like:

  # enscript ruby
  method(a,
         b,
         c)

= #14683 IRB with Ripper

The (({PROMPT_S})) is implemented by checking corresponding open and close tokens of literals.

= #14683 IRB with Ripper

Literal tokens:

* (({"}))
* (({'}))
* percent literals
  * (({%q{})) and (({}}))
  * (({%w{})) and (({}}))
  * blah blah blah
* here-document

= #14683 IRB with Ripper

Example:

  # enscript ruby
  "This
   is
   multiline
   string"

= #14683 IRB with Ripper

Example:

  # enscript ruby
  %w{
    array
    of
    strings
  }

= #14683 IRB with Ripper

Inside String or other literal, IRB uses (({PROMPT_S})) prompt.

= #14683 IRB with Ripper
	
The (({%NNi})) is implemented by count corresponding name space open and close tokens.

= #14683 IRB with Ripper

* Increase nesting level when takes open token
  * (({if})), (({unless})), (({while})), (({until})), (({rescue}))
    * skip post-fix version (it doesn't need (({end})))
  * (({def})), (({do})), (({case})), (({for})), (({begin})), (({class})), (({module}))
  * (({[})), (({{})), (({(}))

= #14683 IRB with Ripper

* Decrease down nesting level when takes open token
  * (({end}))
  * (({]})), (({}})), (({)}))

= #14683 IRB with Ripper

  # enscript ruby
                   # nesting level is 0
  class C          # increase nesting level to 1
    def m          # increase nesting level to 2
      if true      # increase nesting level to 3
        1 if true  # skip (post-fix if)
      end          # decrease nesting level to 2
    end            # decrease nesting level to 1
  end              # decrease nesting level to 0
  # <=== evaluation!

= #14683 IRB with Ripper

Actual example by default:

  # enscript ruby
                ↓ %NNi (nesting level)
  irb(main):001:0> def foo(a,
  irb(main):002:2*         b)  # PROMPT_N
  irb(main):003:1>   <<-EOM
  irb(main):004:1" Hello,      # PROMPT_S
  irb(main):005:1" World!      # PROMPT_S
  irb(main):006:1"   EOM       # PROMPT_S
  irb(main):007:1> end
  => :foo
  irb(main):008:0> 

= ((*#14683*)) IRB with Ripper

https://bugs.ruby-lang.org/issues/14683

= ((*#14683*)) IRB with Ripper

This removes

* lib/irb/slex.rb (283 lines)
* lib/irb/ruby-token.rb (268 lines)

= ((*#14683*)) IRB with Ripper

This simplifies

* lib/irb/ruby-lex.rb (1181 to 287 lines)

= ((*#14683*)) IRB with Ripper

Ruby parser of IRB was shrunk from total 1732 lines to 287 lines.

= ((*#14683*)) IRB with Ripper

The simple implementation is best, because Ruby syntax is complex.

The simple implementation is easy to support and keep the gentleness of Ruby.

= #14787 Show documents when completion

This is second patch for IRB.

= #14787 Show documents when completion

I talked about this Q&A time at RubyKaigi 2017.

= #14787 Show documents when completion

First, RDoc's RI binary files are installed to Ruby's directory.

= #14787 Show documents when completion

CRuby:

  $ tar xvzf ruby-2.5.1.tar.gz
  $ cd ruby-2.5.1
  $ autoconf
  $ ./configure
  $ make
  $ make install # <=== RDoc runs inside

= #14787 Show documents when completion

CRuby:

  $ rbenv install 2.5.1 # <=== RDoc runs inside

= #14787 Show documents when completion

RubyGems:

  $ gem install rails # <=== RDoc runs inside

= #14787 Show documents when completion

But many users set:

  $ gem install rails --no-document

= #14787 Show documents when completion

Many blogs recommend:

  $ cat ~/.gemrc
  install: --no-document
  update: --no-document

= #14787 Show documents when completion

Unfortunately many users don't need documents data, but I understand it.

= #14787 Show documents when completion

Because it's ((*just*)) for RI((({ri})) command).

= #14787 Show documents when completion

Usage of class:

  $ ri 'String'

= #14787 Show documents when completion

Usage of instance method:

  $ ri 'String#gsub'

= #14787 Show documents when completion

Usage of class method:

  $ ri 'String.new'

= #14787 Show documents when completion

Bothersome.

= #14787 Show documents when completion

I wrote on the ticket:

  # blockquote
  # title = #14787
  RDoc installs all documents to Ruby's directory by default.

= #14787 Show documents when completion

  # blockquote
  # title = #14787
  Many users never use it because it's just for RI("ri" command).

= #14787 Show documents when completion

  # blockquote
  # title = #14787
  I think that it is a reason of that many users don't attach importance to documentation.

= #14787 Show documents when completion

I want to improve the importance of RDoc's data.

= #14787 Show documents when completion

shevegen (Robert A. Heiler) replied to the ticket:

  # blockquote
  # title = #14787 shevegen
  I also do not use "ri" on the commandline.

= #14787 Show documents when completion

  # blockquote
  # title = #14787 shevegen
  I would not know why, because I myself simply do not use local look-up ways for documentation normally.

= #14787 Show documents when completion

It's the same opinion of me.

= #14787 Show documents when completion

And the continuation of shevegen's comment:

= #14787 Show documents when completion

  # blockquote
  # title = #14787 shevegen
  I really "just google".

= #14787 Show documents when completion

  # blockquote
  # title = #14787 shevegen
  And using the browser is about 100x more convenient for me as well.

= #14787 Show documents when completion

It's the exact same opinion of me.

= #14787 Show documents when completion

Perfect.

= #14787 Show documents when completion

I'm actually sad.

= #14787 Show documents when completion

I want to improve the importance of RDoc's data(2).

= #14787 Show documents when completion

IRB(with Readline) completes namespace such as classes, modules, methods and so on when it caught TAB key.

= #14787 Show documents when completion

In the patch of this ticket, I use RDoc as a library.

= #14787 Show documents when completion

When you press TAB key one more just after that namespace is exact matched, RI document is shown.

= #14787 Show documents when completion

Demonstration

= #14787 Show documents when completion

This is just an aside, I want Ruby's documentation ((*design*)).

= #14787 Show documents when completion

In the ticket, I talked about language documentation design.

= #14787 Show documents when completion

  # blockquote
  # title = #14787
  Perl has "perldoc" feature and users easily access documents of modules by "perldoc" command.

= #14787 Show documents when completion

  # blockquote
  # title = #14787
  Python has "docstring" feature and users can access it on REPL.

= #14787 Show documents when completion

  # blockquote
  # title = #14787
  Those are each language's design of importance.

= #14787 Show documents when completion

  # blockquote
  # title = #14787
  Users use the language on the documentation design, so library developers write documents on the documentation design.

= #14787 Show documents when completion

  # blockquote
  # title = #14787
  Ruby doesn't have documentation design like Perl and Python.

= #14787 Show documents when completion

  # blockquote
  # title = #14787
  Ruby just has RDoc, IRB, and any other supports, but these are just fragmented features, these are not a documentation design.

= Documentation design

I want to improve Ruby's documentation design.

= Documentation design

This ticket is a slice of my documentation design.

= Documentation design

I want to improve\n((*the gentleness*))\nof Ruby's documentation design.

= Documentation design

I think that\n((*the gentleness for users*))\nis very important in Ruby.

= Documentation design

Please remove\n(({--no-document}))\nfor\nimprovement documentation at Ruby 2.6 or later.

= Thank you for your attention

Please write documents!