testing.txt in rubylexer-0.7.0 vs testing.txt in rubylexer-0.7.1

@@ -1,85 +1,74 @@
-
 Running the tests:
-the simplest thing to do is run "ruby -Ilib test/code/locatetest.rb". 
-this will use locate to find as much ruby code on your system and test 
+The simplest thing to do is run "ruby -Ilib test/code/regression.rb". This 
+tests against a list of known ruby expressions. It will take several minutes 
+to run. Currently, there are 5 failures.
+
+If you're ambitious, try this command: "ruby -Ilib test/code/locatetest.rb". 
+This will use locate to find as much ruby code on your system and test 
 each specimen to see if it can be tokenized correctly (by feeding it to 
 testcode/rubylexervsruby.rb, the operation of which is outlined below 
 under 'testing strategy').
 
 Interpreting the output of rubylexervsruby.rb (and locatetest):
-in rubylexervsruby, i've tried to follow the philosophy that the test program
-doesn't print anything unless there's an error. perhaps i haven't followed
+In rubylexervsruby, I've tried to follow the philosophy that the test program
+doesn't print anything unless there's an error. Perhaps I haven't followed
 this far enough; every run of rubylexervsruby produces a little output, and 
 sometimes a run will produce output that doesn't actually indicate a problem, 
-or only a low-priority problem. (since locatetest, torment, and test all run
-rubylexervsruby over and over, they all produce lots of (mostly harmless) 
-output. sorry.)
+or only a low-priority problem. (Since locatetest runs rubylexervsruby over 
+and over, it produces lots of (mostly harmless) output. Sorry.)
 
-the following types of output should be ignored:
+The following types of output should be ignored:
 
 diff file or chunk headers
 
 lines that look like this:
   executing: ruby testcode/tokentest.rb ...     #normal, 1 for every file
 or this:
-  warning moved from 24 to 22: ambiguous first argument; put parentheses or even spaces
-or this:
   Created warning(s) in new file, line 85: useless use of <=> in void context
 or this:
   Removed warning(s) from old file (?!), line 85: useless use of <=> in void context
-indicate that a warning was added deleted, or moved. ultimately, these should
+indicate that a warning was added or deleted. Ultimately, these should
 go away, but right now it's a low-priority issue.
 
-if you ever see ruby stack dump in rubylexervsruby output, that's certainly
-an error (if the input ruby code is valid). 
+If you ever see ruby stack dump in rubylexervsruby output, that's certainly
+an error. 
 
-something that looks like a unidiff chunk body (not header) may indicate 
-an error as well. the problem is that sometimes those morpheaous warnings 
-sneak through my filter (which is supposed to condense them into a single 
-line like those above), so you will see diff chunks where the only real
-difference is a warning. here are some examples of the kind of diff chunks 
-that should NOT cause alarm:
+Something that looks like a unidiff chunk body (not header) may indicate 
+an error as well. To understand more about how the unidiff output is 
+created, see the section on testing strategy below.
 
---:89: warning: useless use of <=> in void context
---:92: warning: useless use of <=> in void context
-+-:90: warning: useless use of <=> in void context
- Stack now 0 2 62 300 5 110 365 544
+htree/template.rb should be ok now.
 
- Shifting token tIDENTIFIER, Entering state 34
--Reading a token: -:318: warning: ambiguous first argument; put parentheses or even spaces
--Next token is token tINTEGER ()
-+Reading a token: Next token is token tINTEGER ()
- Reducing stack by rule 476 (line 2382), tIDENTIFIER -> operation
+currently, lots of warnings are printed about token offsets being off by 1,
+particularly the AssignmentRhsListToken. This is a problem, but for now I'm
+ignoring it. 
 
-if you look closely, (and are experienced in reading unidiff output), you'll
-see that the only difference is a warning. to understand more about how the
-unidiff output is created, see the section on testing strategy below.
+Diff chunks like this indicate a minor problem with the placement of (empty)
+string fragments. Ignore it for now:
 
-htree/template.rb:
-testing this file prints a small unidiff chunk. analysis indicates that the
-problem is because ruby's lexer generates an extra (empty) string content 
-token at this point, which mine omits. there's no actual semantic difference
-between the two tokenizations, so there's nothing to be concerned about. in
-a future release, when my lexer supports the notion of string contents and
-string delimiters as separate token types, i'll try to emulate ruby more 
-closely. the same case is replicated in p.rb.
-(in other words, ignore the error in this file and the identical one in p.rb.)
+@@ -13,2 +13,3 @@
+ Shifting token tSTRING_BEG ()
++Shifting token tSTRING_CONTENT ()
+ Shifting token tSTRING_DBEG ()
+@@ -19,2 +20,2 @@
+ Shifting token '\n' ()
+@@ -13,2 +13,3 @@
+ Shifting token tSTRING_BEG ()
++Shifting token tSTRING_CONTENT ()
+ Shifting token tSTRING_DBEG ()
 
 
+
 if you find any output that doesn't look like one of the above exceptions, 
 and the input file was valid ruby, please send it to me so that i can add it
 to my arsenal of tests.
 
 there are a number of 'ruby' files that i know of out there that actually 
 contain syntax errors:
 rpcd.rb from freeride -- missing an end
 sample1.rb from 1.6 version of tcltk -- not legal in ruby 1.8
 bdb.rb from libdb2, 3, and 4 -- not how you declare [] method in ruby
-
-testdata/p.rb (my menagerie of weird test cases) is one of the worst 
-offenders; it prints lots of output when tested, but all of the problems
-are harmless or minor.
 
 only the 10 first lines of each failing file are printed. the rest, as well
 as other intermediate files are kept in the testresults directory. the test
 output files are named *.prs.diff. beware: this directory is never cleaned,
 and can get quite large. after a large test run, you'll want to empty this