The implementation of the position computation
was surprisingly broken. Implemented fixes and
added extra test cases.
There is a slight interface change: Calling
Pos() returns the current position; but if
called before Scan() that position may not
be the position of the next token returned
by Scan() (depending on the scan settings
and the source text) - this in contrast to
the original comment.
However, after calling Scan(), the Scanner's
Position field reports the position of the
scanned token, as before.
Fixes#1327.
R=rsc
CC=golang-dev
https://golang.org/cl/3972047
Init may report an error on the first character and
thus one needs an ability to set the error handler
for Init. Was a design bug.
Added corresponding test cases and better documentation.
Also: Fixed a subtle infinite loop exposed by one of the
new test cases.
Fixes#1380.
R=rsc, gri
CC=golang-dev
https://golang.org/cl/4094041
Emphasis on minimal interface and fast scanning.
Recognizes all Go literals by default. Easily
configurable to recognize different whitespace
characters and tokens.
Provides detailed position information for each
token.
R=rsc, r
CC=golang-dev
https://golang.org/cl/181160
