2009-07-13 11:10:56 -06:00
|
|
|
// Copyright 2009 The Go Authors. All rights reserved.
|
|
|
|
// Use of this source code is governed by a BSD-style
|
|
|
|
// license that can be found in the LICENSE file.
|
|
|
|
|
|
|
|
package ebnf
|
|
|
|
|
|
|
|
import (
|
2009-12-15 16:27:16 -07:00
|
|
|
"go/scanner"
|
|
|
|
"go/token"
|
|
|
|
"os"
|
|
|
|
"strconv"
|
2009-07-13 11:10:56 -06:00
|
|
|
)
|
|
|
|
|
|
|
|
|
|
|
|
type parser struct {
|
go/ast: use token.Pos instead of token.Position; adjust all dependent code
Specifically:
* lib/godoc:
- provide file set (FSet) argument to formatters where needed
* src/cmd:
- cgo, ebnflint, godoc, gofmt, goinstall: provide file set (fset) where needed
- godoc: remove local binary search with sort.Search (change by rsc),
extract file set for formatters
* src/pkg:
- exp/eval: remove embedded token.Position fields from nodes and replace
with named token.Pos fields; add corresponding Pos() accessor methods
- go/token: added file.Line(), changed signature of File.Position()
* test/fixedbugs/:
- bug206.go: change test to not rely on token.Pos details
* added various extra comments
* Runs all.bash
* gofmt formats all of src, misc w/o changes
* godoc runs
* performance:
- The new version of godoc consumes about the same space after indexing
has completed, but indexing is half the speed. Significant space savings
are expected from smaller ASTs, but since they are thrown away after a
file has been indexed, this is not visible anymore. The slower indexing
time is due to the much more expensive computation of line information.
However, with the new compressed position information, indexing can be
rewritten and simplified. Furthermore, computing the line info can be
done more efficiently.
New godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
44381 godoc 0.0% 0:38.00 4 19 149 145M 184K 148M 176M
2010/12/03 17:58:35 index updated (39.231s, 18505 unique words, 386387 spots)
2010/12/03 17:58:35 bytes=90858456 footprint=199182584
2010/12/03 17:58:36 bytes=47858568 footprint=167295224
Old godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
23167 godoc 0.0% 0:22.02 4 17 132 129M 184K 132M 173M
2010/12/03 14:51:32 index updated (24.892s, 18765 unique words, 393830 spots)
2010/12/03 14:51:32 bytes=66404528 footprint=163907832
2010/12/03 14:51:32 bytes=46282224 footprint=163907832
The different numbers for unique words/spots stem from the fact the the
two workspaces are not exactly identical. The new godoc maintains a large
file set data structure during indexing which (probably) is the reason
for the larger heap (90858456 vs 66404528) before garbage collection.
R=rsc, r
CC=golang-dev
https://golang.org/cl/3050041
2010-12-06 15:23:18 -07:00
|
|
|
fset *token.FileSet
|
2009-12-15 16:27:16 -07:00
|
|
|
scanner.ErrorVector
|
|
|
|
scanner scanner.Scanner
|
go/ast: use token.Pos instead of token.Position; adjust all dependent code
Specifically:
* lib/godoc:
- provide file set (FSet) argument to formatters where needed
* src/cmd:
- cgo, ebnflint, godoc, gofmt, goinstall: provide file set (fset) where needed
- godoc: remove local binary search with sort.Search (change by rsc),
extract file set for formatters
* src/pkg:
- exp/eval: remove embedded token.Position fields from nodes and replace
with named token.Pos fields; add corresponding Pos() accessor methods
- go/token: added file.Line(), changed signature of File.Position()
* test/fixedbugs/:
- bug206.go: change test to not rely on token.Pos details
* added various extra comments
* Runs all.bash
* gofmt formats all of src, misc w/o changes
* godoc runs
* performance:
- The new version of godoc consumes about the same space after indexing
has completed, but indexing is half the speed. Significant space savings
are expected from smaller ASTs, but since they are thrown away after a
file has been indexed, this is not visible anymore. The slower indexing
time is due to the much more expensive computation of line information.
However, with the new compressed position information, indexing can be
rewritten and simplified. Furthermore, computing the line info can be
done more efficiently.
New godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
44381 godoc 0.0% 0:38.00 4 19 149 145M 184K 148M 176M
2010/12/03 17:58:35 index updated (39.231s, 18505 unique words, 386387 spots)
2010/12/03 17:58:35 bytes=90858456 footprint=199182584
2010/12/03 17:58:36 bytes=47858568 footprint=167295224
Old godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
23167 godoc 0.0% 0:22.02 4 17 132 129M 184K 132M 173M
2010/12/03 14:51:32 index updated (24.892s, 18765 unique words, 393830 spots)
2010/12/03 14:51:32 bytes=66404528 footprint=163907832
2010/12/03 14:51:32 bytes=46282224 footprint=163907832
The different numbers for unique words/spots stem from the fact the the
two workspaces are not exactly identical. The new godoc maintains a large
file set data structure during indexing which (probably) is the reason
for the larger heap (90858456 vs 66404528) before garbage collection.
R=rsc, r
CC=golang-dev
https://golang.org/cl/3050041
2010-12-06 15:23:18 -07:00
|
|
|
pos token.Pos // token position
|
|
|
|
tok token.Token // one token look-ahead
|
|
|
|
lit []byte // token literal
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
func (p *parser) next() {
|
2009-12-15 16:27:16 -07:00
|
|
|
p.pos, p.tok, p.lit = p.scanner.Scan()
|
2009-07-13 11:10:56 -06:00
|
|
|
if p.tok.IsKeyword() {
|
|
|
|
// TODO Should keyword mapping always happen outside scanner?
|
|
|
|
// Or should there be a flag to scanner to enable keyword mapping?
|
2009-11-09 13:07:39 -07:00
|
|
|
p.tok = token.IDENT
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
|
go/ast: use token.Pos instead of token.Position; adjust all dependent code
Specifically:
* lib/godoc:
- provide file set (FSet) argument to formatters where needed
* src/cmd:
- cgo, ebnflint, godoc, gofmt, goinstall: provide file set (fset) where needed
- godoc: remove local binary search with sort.Search (change by rsc),
extract file set for formatters
* src/pkg:
- exp/eval: remove embedded token.Position fields from nodes and replace
with named token.Pos fields; add corresponding Pos() accessor methods
- go/token: added file.Line(), changed signature of File.Position()
* test/fixedbugs/:
- bug206.go: change test to not rely on token.Pos details
* added various extra comments
* Runs all.bash
* gofmt formats all of src, misc w/o changes
* godoc runs
* performance:
- The new version of godoc consumes about the same space after indexing
has completed, but indexing is half the speed. Significant space savings
are expected from smaller ASTs, but since they are thrown away after a
file has been indexed, this is not visible anymore. The slower indexing
time is due to the much more expensive computation of line information.
However, with the new compressed position information, indexing can be
rewritten and simplified. Furthermore, computing the line info can be
done more efficiently.
New godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
44381 godoc 0.0% 0:38.00 4 19 149 145M 184K 148M 176M
2010/12/03 17:58:35 index updated (39.231s, 18505 unique words, 386387 spots)
2010/12/03 17:58:35 bytes=90858456 footprint=199182584
2010/12/03 17:58:36 bytes=47858568 footprint=167295224
Old godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
23167 godoc 0.0% 0:22.02 4 17 132 129M 184K 132M 173M
2010/12/03 14:51:32 index updated (24.892s, 18765 unique words, 393830 spots)
2010/12/03 14:51:32 bytes=66404528 footprint=163907832
2010/12/03 14:51:32 bytes=46282224 footprint=163907832
The different numbers for unique words/spots stem from the fact the the
two workspaces are not exactly identical. The new godoc maintains a large
file set data structure during indexing which (probably) is the reason
for the larger heap (90858456 vs 66404528) before garbage collection.
R=rsc, r
CC=golang-dev
https://golang.org/cl/3050041
2010-12-06 15:23:18 -07:00
|
|
|
func (p *parser) error(pos token.Pos, msg string) {
|
|
|
|
p.Error(p.fset.Position(pos), msg)
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
func (p *parser) errorExpected(pos token.Pos, msg string) {
|
2009-12-15 16:27:16 -07:00
|
|
|
msg = "expected " + msg
|
go/ast: use token.Pos instead of token.Position; adjust all dependent code
Specifically:
* lib/godoc:
- provide file set (FSet) argument to formatters where needed
* src/cmd:
- cgo, ebnflint, godoc, gofmt, goinstall: provide file set (fset) where needed
- godoc: remove local binary search with sort.Search (change by rsc),
extract file set for formatters
* src/pkg:
- exp/eval: remove embedded token.Position fields from nodes and replace
with named token.Pos fields; add corresponding Pos() accessor methods
- go/token: added file.Line(), changed signature of File.Position()
* test/fixedbugs/:
- bug206.go: change test to not rely on token.Pos details
* added various extra comments
* Runs all.bash
* gofmt formats all of src, misc w/o changes
* godoc runs
* performance:
- The new version of godoc consumes about the same space after indexing
has completed, but indexing is half the speed. Significant space savings
are expected from smaller ASTs, but since they are thrown away after a
file has been indexed, this is not visible anymore. The slower indexing
time is due to the much more expensive computation of line information.
However, with the new compressed position information, indexing can be
rewritten and simplified. Furthermore, computing the line info can be
done more efficiently.
New godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
44381 godoc 0.0% 0:38.00 4 19 149 145M 184K 148M 176M
2010/12/03 17:58:35 index updated (39.231s, 18505 unique words, 386387 spots)
2010/12/03 17:58:35 bytes=90858456 footprint=199182584
2010/12/03 17:58:36 bytes=47858568 footprint=167295224
Old godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
23167 godoc 0.0% 0:22.02 4 17 132 129M 184K 132M 173M
2010/12/03 14:51:32 index updated (24.892s, 18765 unique words, 393830 spots)
2010/12/03 14:51:32 bytes=66404528 footprint=163907832
2010/12/03 14:51:32 bytes=46282224 footprint=163907832
The different numbers for unique words/spots stem from the fact the the
two workspaces are not exactly identical. The new godoc maintains a large
file set data structure during indexing which (probably) is the reason
for the larger heap (90858456 vs 66404528) before garbage collection.
R=rsc, r
CC=golang-dev
https://golang.org/cl/3050041
2010-12-06 15:23:18 -07:00
|
|
|
if pos == p.pos {
|
2009-07-13 11:10:56 -06:00
|
|
|
// the error happened at the current position;
|
|
|
|
// make the error message more specific
|
2009-12-15 16:27:16 -07:00
|
|
|
msg += ", found '" + p.tok.String() + "'"
|
2009-07-13 11:10:56 -06:00
|
|
|
if p.tok.IsLiteral() {
|
2009-11-09 22:13:17 -07:00
|
|
|
msg += " " + string(p.lit)
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
}
|
go/ast: use token.Pos instead of token.Position; adjust all dependent code
Specifically:
* lib/godoc:
- provide file set (FSet) argument to formatters where needed
* src/cmd:
- cgo, ebnflint, godoc, gofmt, goinstall: provide file set (fset) where needed
- godoc: remove local binary search with sort.Search (change by rsc),
extract file set for formatters
* src/pkg:
- exp/eval: remove embedded token.Position fields from nodes and replace
with named token.Pos fields; add corresponding Pos() accessor methods
- go/token: added file.Line(), changed signature of File.Position()
* test/fixedbugs/:
- bug206.go: change test to not rely on token.Pos details
* added various extra comments
* Runs all.bash
* gofmt formats all of src, misc w/o changes
* godoc runs
* performance:
- The new version of godoc consumes about the same space after indexing
has completed, but indexing is half the speed. Significant space savings
are expected from smaller ASTs, but since they are thrown away after a
file has been indexed, this is not visible anymore. The slower indexing
time is due to the much more expensive computation of line information.
However, with the new compressed position information, indexing can be
rewritten and simplified. Furthermore, computing the line info can be
done more efficiently.
New godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
44381 godoc 0.0% 0:38.00 4 19 149 145M 184K 148M 176M
2010/12/03 17:58:35 index updated (39.231s, 18505 unique words, 386387 spots)
2010/12/03 17:58:35 bytes=90858456 footprint=199182584
2010/12/03 17:58:36 bytes=47858568 footprint=167295224
Old godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
23167 godoc 0.0% 0:22.02 4 17 132 129M 184K 132M 173M
2010/12/03 14:51:32 index updated (24.892s, 18765 unique words, 393830 spots)
2010/12/03 14:51:32 bytes=66404528 footprint=163907832
2010/12/03 14:51:32 bytes=46282224 footprint=163907832
The different numbers for unique words/spots stem from the fact the the
two workspaces are not exactly identical. The new godoc maintains a large
file set data structure during indexing which (probably) is the reason
for the larger heap (90858456 vs 66404528) before garbage collection.
R=rsc, r
CC=golang-dev
https://golang.org/cl/3050041
2010-12-06 15:23:18 -07:00
|
|
|
p.error(pos, msg)
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
|
go/ast: use token.Pos instead of token.Position; adjust all dependent code
Specifically:
* lib/godoc:
- provide file set (FSet) argument to formatters where needed
* src/cmd:
- cgo, ebnflint, godoc, gofmt, goinstall: provide file set (fset) where needed
- godoc: remove local binary search with sort.Search (change by rsc),
extract file set for formatters
* src/pkg:
- exp/eval: remove embedded token.Position fields from nodes and replace
with named token.Pos fields; add corresponding Pos() accessor methods
- go/token: added file.Line(), changed signature of File.Position()
* test/fixedbugs/:
- bug206.go: change test to not rely on token.Pos details
* added various extra comments
* Runs all.bash
* gofmt formats all of src, misc w/o changes
* godoc runs
* performance:
- The new version of godoc consumes about the same space after indexing
has completed, but indexing is half the speed. Significant space savings
are expected from smaller ASTs, but since they are thrown away after a
file has been indexed, this is not visible anymore. The slower indexing
time is due to the much more expensive computation of line information.
However, with the new compressed position information, indexing can be
rewritten and simplified. Furthermore, computing the line info can be
done more efficiently.
New godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
44381 godoc 0.0% 0:38.00 4 19 149 145M 184K 148M 176M
2010/12/03 17:58:35 index updated (39.231s, 18505 unique words, 386387 spots)
2010/12/03 17:58:35 bytes=90858456 footprint=199182584
2010/12/03 17:58:36 bytes=47858568 footprint=167295224
Old godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
23167 godoc 0.0% 0:22.02 4 17 132 129M 184K 132M 173M
2010/12/03 14:51:32 index updated (24.892s, 18765 unique words, 393830 spots)
2010/12/03 14:51:32 bytes=66404528 footprint=163907832
2010/12/03 14:51:32 bytes=46282224 footprint=163907832
The different numbers for unique words/spots stem from the fact the the
two workspaces are not exactly identical. The new godoc maintains a large
file set data structure during indexing which (probably) is the reason
for the larger heap (90858456 vs 66404528) before garbage collection.
R=rsc, r
CC=golang-dev
https://golang.org/cl/3050041
2010-12-06 15:23:18 -07:00
|
|
|
func (p *parser) expect(tok token.Token) token.Pos {
|
2009-12-15 16:27:16 -07:00
|
|
|
pos := p.pos
|
2009-07-13 11:10:56 -06:00
|
|
|
if p.tok != tok {
|
2009-11-09 22:13:17 -07:00
|
|
|
p.errorExpected(pos, "'"+tok.String()+"'")
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
2009-12-15 16:27:16 -07:00
|
|
|
p.next() // make progress in any case
|
|
|
|
return pos
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
func (p *parser) parseIdentifier() *Name {
|
2009-12-15 16:27:16 -07:00
|
|
|
pos := p.pos
|
|
|
|
name := string(p.lit)
|
|
|
|
p.expect(token.IDENT)
|
|
|
|
return &Name{pos, name}
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
func (p *parser) parseToken() *Token {
|
2009-12-15 16:27:16 -07:00
|
|
|
pos := p.pos
|
|
|
|
value := ""
|
2009-07-13 11:10:56 -06:00
|
|
|
if p.tok == token.STRING {
|
2009-12-15 16:27:16 -07:00
|
|
|
value, _ = strconv.Unquote(string(p.lit))
|
2009-07-13 11:10:56 -06:00
|
|
|
// Unquote may fail with an error, but only if the scanner found
|
|
|
|
// an illegal string in the first place. In this case the error
|
|
|
|
// has already been reported.
|
2009-12-15 16:27:16 -07:00
|
|
|
p.next()
|
2009-07-13 11:10:56 -06:00
|
|
|
} else {
|
2009-11-09 13:07:39 -07:00
|
|
|
p.expect(token.STRING)
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
2009-12-15 16:27:16 -07:00
|
|
|
return &Token{pos, value}
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
func (p *parser) parseTerm() (x Expression) {
|
2009-12-15 16:27:16 -07:00
|
|
|
pos := p.pos
|
2009-07-13 11:10:56 -06:00
|
|
|
|
|
|
|
switch p.tok {
|
|
|
|
case token.IDENT:
|
2009-11-09 13:07:39 -07:00
|
|
|
x = p.parseIdentifier()
|
2009-07-13 11:10:56 -06:00
|
|
|
|
|
|
|
case token.STRING:
|
2009-12-15 16:27:16 -07:00
|
|
|
tok := p.parseToken()
|
|
|
|
x = tok
|
2009-07-13 11:10:56 -06:00
|
|
|
if p.tok == token.ELLIPSIS {
|
2009-12-15 16:27:16 -07:00
|
|
|
p.next()
|
|
|
|
x = &Range{tok, p.parseToken()}
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
case token.LPAREN:
|
2009-12-15 16:27:16 -07:00
|
|
|
p.next()
|
|
|
|
x = &Group{pos, p.parseExpression()}
|
|
|
|
p.expect(token.RPAREN)
|
2009-07-13 11:10:56 -06:00
|
|
|
|
|
|
|
case token.LBRACK:
|
2009-12-15 16:27:16 -07:00
|
|
|
p.next()
|
|
|
|
x = &Option{pos, p.parseExpression()}
|
|
|
|
p.expect(token.RBRACK)
|
2009-07-13 11:10:56 -06:00
|
|
|
|
|
|
|
case token.LBRACE:
|
2009-12-15 16:27:16 -07:00
|
|
|
p.next()
|
|
|
|
x = &Repetition{pos, p.parseExpression()}
|
|
|
|
p.expect(token.RBRACE)
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
|
2009-12-15 16:27:16 -07:00
|
|
|
return x
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
func (p *parser) parseSequence() Expression {
|
2010-10-28 22:23:24 -06:00
|
|
|
var list Sequence
|
2009-07-13 11:10:56 -06:00
|
|
|
|
|
|
|
for x := p.parseTerm(); x != nil; x = p.parseTerm() {
|
2010-10-28 22:23:24 -06:00
|
|
|
list = append(list, x)
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
// no need for a sequence if list.Len() < 2
|
2010-10-28 22:23:24 -06:00
|
|
|
switch len(list) {
|
2009-07-13 11:10:56 -06:00
|
|
|
case 0:
|
2009-11-09 13:07:39 -07:00
|
|
|
return nil
|
2009-07-13 11:10:56 -06:00
|
|
|
case 1:
|
2010-10-28 22:23:24 -06:00
|
|
|
return list[0]
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
|
2010-10-28 22:23:24 -06:00
|
|
|
return list
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
func (p *parser) parseExpression() Expression {
|
2010-10-28 22:23:24 -06:00
|
|
|
var list Alternative
|
2009-07-13 11:10:56 -06:00
|
|
|
|
|
|
|
for {
|
2010-10-28 22:23:24 -06:00
|
|
|
if x := p.parseSequence(); x != nil {
|
|
|
|
list = append(list, x)
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
if p.tok != token.OR {
|
2009-11-09 13:07:39 -07:00
|
|
|
break
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
2009-12-15 16:27:16 -07:00
|
|
|
p.next()
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
// no need for an Alternative node if list.Len() < 2
|
2010-10-28 22:23:24 -06:00
|
|
|
switch len(list) {
|
2009-07-13 11:10:56 -06:00
|
|
|
case 0:
|
2009-11-09 13:07:39 -07:00
|
|
|
return nil
|
2009-07-13 11:10:56 -06:00
|
|
|
case 1:
|
2010-10-28 22:23:24 -06:00
|
|
|
return list[0]
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
|
2010-10-28 22:23:24 -06:00
|
|
|
return list
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
func (p *parser) parseProduction() *Production {
|
2009-12-15 16:27:16 -07:00
|
|
|
name := p.parseIdentifier()
|
|
|
|
p.expect(token.ASSIGN)
|
|
|
|
expr := p.parseExpression()
|
|
|
|
p.expect(token.PERIOD)
|
|
|
|
return &Production{name, expr}
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
|
go/ast: use token.Pos instead of token.Position; adjust all dependent code
Specifically:
* lib/godoc:
- provide file set (FSet) argument to formatters where needed
* src/cmd:
- cgo, ebnflint, godoc, gofmt, goinstall: provide file set (fset) where needed
- godoc: remove local binary search with sort.Search (change by rsc),
extract file set for formatters
* src/pkg:
- exp/eval: remove embedded token.Position fields from nodes and replace
with named token.Pos fields; add corresponding Pos() accessor methods
- go/token: added file.Line(), changed signature of File.Position()
* test/fixedbugs/:
- bug206.go: change test to not rely on token.Pos details
* added various extra comments
* Runs all.bash
* gofmt formats all of src, misc w/o changes
* godoc runs
* performance:
- The new version of godoc consumes about the same space after indexing
has completed, but indexing is half the speed. Significant space savings
are expected from smaller ASTs, but since they are thrown away after a
file has been indexed, this is not visible anymore. The slower indexing
time is due to the much more expensive computation of line information.
However, with the new compressed position information, indexing can be
rewritten and simplified. Furthermore, computing the line info can be
done more efficiently.
New godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
44381 godoc 0.0% 0:38.00 4 19 149 145M 184K 148M 176M
2010/12/03 17:58:35 index updated (39.231s, 18505 unique words, 386387 spots)
2010/12/03 17:58:35 bytes=90858456 footprint=199182584
2010/12/03 17:58:36 bytes=47858568 footprint=167295224
Old godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
23167 godoc 0.0% 0:22.02 4 17 132 129M 184K 132M 173M
2010/12/03 14:51:32 index updated (24.892s, 18765 unique words, 393830 spots)
2010/12/03 14:51:32 bytes=66404528 footprint=163907832
2010/12/03 14:51:32 bytes=46282224 footprint=163907832
The different numbers for unique words/spots stem from the fact the the
two workspaces are not exactly identical. The new godoc maintains a large
file set data structure during indexing which (probably) is the reason
for the larger heap (90858456 vs 66404528) before garbage collection.
R=rsc, r
CC=golang-dev
https://golang.org/cl/3050041
2010-12-06 15:23:18 -07:00
|
|
|
func (p *parser) parse(fset *token.FileSet, filename string, src []byte) Grammar {
|
2009-07-13 11:10:56 -06:00
|
|
|
// initialize parser
|
go/ast: use token.Pos instead of token.Position; adjust all dependent code
Specifically:
* lib/godoc:
- provide file set (FSet) argument to formatters where needed
* src/cmd:
- cgo, ebnflint, godoc, gofmt, goinstall: provide file set (fset) where needed
- godoc: remove local binary search with sort.Search (change by rsc),
extract file set for formatters
* src/pkg:
- exp/eval: remove embedded token.Position fields from nodes and replace
with named token.Pos fields; add corresponding Pos() accessor methods
- go/token: added file.Line(), changed signature of File.Position()
* test/fixedbugs/:
- bug206.go: change test to not rely on token.Pos details
* added various extra comments
* Runs all.bash
* gofmt formats all of src, misc w/o changes
* godoc runs
* performance:
- The new version of godoc consumes about the same space after indexing
has completed, but indexing is half the speed. Significant space savings
are expected from smaller ASTs, but since they are thrown away after a
file has been indexed, this is not visible anymore. The slower indexing
time is due to the much more expensive computation of line information.
However, with the new compressed position information, indexing can be
rewritten and simplified. Furthermore, computing the line info can be
done more efficiently.
New godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
44381 godoc 0.0% 0:38.00 4 19 149 145M 184K 148M 176M
2010/12/03 17:58:35 index updated (39.231s, 18505 unique words, 386387 spots)
2010/12/03 17:58:35 bytes=90858456 footprint=199182584
2010/12/03 17:58:36 bytes=47858568 footprint=167295224
Old godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
23167 godoc 0.0% 0:22.02 4 17 132 129M 184K 132M 173M
2010/12/03 14:51:32 index updated (24.892s, 18765 unique words, 393830 spots)
2010/12/03 14:51:32 bytes=66404528 footprint=163907832
2010/12/03 14:51:32 bytes=46282224 footprint=163907832
The different numbers for unique words/spots stem from the fact the the
two workspaces are not exactly identical. The new godoc maintains a large
file set data structure during indexing which (probably) is the reason
for the larger heap (90858456 vs 66404528) before garbage collection.
R=rsc, r
CC=golang-dev
https://golang.org/cl/3050041
2010-12-06 15:23:18 -07:00
|
|
|
p.fset = fset
|
2009-12-15 16:27:16 -07:00
|
|
|
p.ErrorVector.Reset()
|
2011-01-13 18:20:26 -07:00
|
|
|
p.scanner.Init(fset.AddFile(filename, fset.Base(), len(src)), src, p, 0)
|
2009-12-15 16:27:16 -07:00
|
|
|
p.next() // initializes pos, tok, lit
|
2009-07-13 11:10:56 -06:00
|
|
|
|
2009-12-15 16:27:16 -07:00
|
|
|
grammar := make(Grammar)
|
2009-07-13 11:10:56 -06:00
|
|
|
for p.tok != token.EOF {
|
2009-12-15 16:27:16 -07:00
|
|
|
prod := p.parseProduction()
|
|
|
|
name := prod.Name.String
|
2009-09-15 10:41:59 -06:00
|
|
|
if _, found := grammar[name]; !found {
|
2009-11-09 13:07:39 -07:00
|
|
|
grammar[name] = prod
|
2009-07-13 11:10:56 -06:00
|
|
|
} else {
|
go/ast: use token.Pos instead of token.Position; adjust all dependent code
Specifically:
* lib/godoc:
- provide file set (FSet) argument to formatters where needed
* src/cmd:
- cgo, ebnflint, godoc, gofmt, goinstall: provide file set (fset) where needed
- godoc: remove local binary search with sort.Search (change by rsc),
extract file set for formatters
* src/pkg:
- exp/eval: remove embedded token.Position fields from nodes and replace
with named token.Pos fields; add corresponding Pos() accessor methods
- go/token: added file.Line(), changed signature of File.Position()
* test/fixedbugs/:
- bug206.go: change test to not rely on token.Pos details
* added various extra comments
* Runs all.bash
* gofmt formats all of src, misc w/o changes
* godoc runs
* performance:
- The new version of godoc consumes about the same space after indexing
has completed, but indexing is half the speed. Significant space savings
are expected from smaller ASTs, but since they are thrown away after a
file has been indexed, this is not visible anymore. The slower indexing
time is due to the much more expensive computation of line information.
However, with the new compressed position information, indexing can be
rewritten and simplified. Furthermore, computing the line info can be
done more efficiently.
New godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
44381 godoc 0.0% 0:38.00 4 19 149 145M 184K 148M 176M
2010/12/03 17:58:35 index updated (39.231s, 18505 unique words, 386387 spots)
2010/12/03 17:58:35 bytes=90858456 footprint=199182584
2010/12/03 17:58:36 bytes=47858568 footprint=167295224
Old godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
23167 godoc 0.0% 0:22.02 4 17 132 129M 184K 132M 173M
2010/12/03 14:51:32 index updated (24.892s, 18765 unique words, 393830 spots)
2010/12/03 14:51:32 bytes=66404528 footprint=163907832
2010/12/03 14:51:32 bytes=46282224 footprint=163907832
The different numbers for unique words/spots stem from the fact the the
two workspaces are not exactly identical. The new godoc maintains a large
file set data structure during indexing which (probably) is the reason
for the larger heap (90858456 vs 66404528) before garbage collection.
R=rsc, r
CC=golang-dev
https://golang.org/cl/3050041
2010-12-06 15:23:18 -07:00
|
|
|
p.error(prod.Pos(), name+" declared already")
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2009-12-15 16:27:16 -07:00
|
|
|
return grammar
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
// Parse parses a set of EBNF productions from source src.
|
|
|
|
// It returns a set of productions. Errors are reported
|
|
|
|
// for incorrect syntax and if a production is declared
|
go/ast: use token.Pos instead of token.Position; adjust all dependent code
Specifically:
* lib/godoc:
- provide file set (FSet) argument to formatters where needed
* src/cmd:
- cgo, ebnflint, godoc, gofmt, goinstall: provide file set (fset) where needed
- godoc: remove local binary search with sort.Search (change by rsc),
extract file set for formatters
* src/pkg:
- exp/eval: remove embedded token.Position fields from nodes and replace
with named token.Pos fields; add corresponding Pos() accessor methods
- go/token: added file.Line(), changed signature of File.Position()
* test/fixedbugs/:
- bug206.go: change test to not rely on token.Pos details
* added various extra comments
* Runs all.bash
* gofmt formats all of src, misc w/o changes
* godoc runs
* performance:
- The new version of godoc consumes about the same space after indexing
has completed, but indexing is half the speed. Significant space savings
are expected from smaller ASTs, but since they are thrown away after a
file has been indexed, this is not visible anymore. The slower indexing
time is due to the much more expensive computation of line information.
However, with the new compressed position information, indexing can be
rewritten and simplified. Furthermore, computing the line info can be
done more efficiently.
New godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
44381 godoc 0.0% 0:38.00 4 19 149 145M 184K 148M 176M
2010/12/03 17:58:35 index updated (39.231s, 18505 unique words, 386387 spots)
2010/12/03 17:58:35 bytes=90858456 footprint=199182584
2010/12/03 17:58:36 bytes=47858568 footprint=167295224
Old godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
23167 godoc 0.0% 0:22.02 4 17 132 129M 184K 132M 173M
2010/12/03 14:51:32 index updated (24.892s, 18765 unique words, 393830 spots)
2010/12/03 14:51:32 bytes=66404528 footprint=163907832
2010/12/03 14:51:32 bytes=46282224 footprint=163907832
The different numbers for unique words/spots stem from the fact the the
two workspaces are not exactly identical. The new godoc maintains a large
file set data structure during indexing which (probably) is the reason
for the larger heap (90858456 vs 66404528) before garbage collection.
R=rsc, r
CC=golang-dev
https://golang.org/cl/3050041
2010-12-06 15:23:18 -07:00
|
|
|
// more than once. Position information is recorded relative
|
|
|
|
// to the file set fset.
|
2009-07-13 11:10:56 -06:00
|
|
|
//
|
go/ast: use token.Pos instead of token.Position; adjust all dependent code
Specifically:
* lib/godoc:
- provide file set (FSet) argument to formatters where needed
* src/cmd:
- cgo, ebnflint, godoc, gofmt, goinstall: provide file set (fset) where needed
- godoc: remove local binary search with sort.Search (change by rsc),
extract file set for formatters
* src/pkg:
- exp/eval: remove embedded token.Position fields from nodes and replace
with named token.Pos fields; add corresponding Pos() accessor methods
- go/token: added file.Line(), changed signature of File.Position()
* test/fixedbugs/:
- bug206.go: change test to not rely on token.Pos details
* added various extra comments
* Runs all.bash
* gofmt formats all of src, misc w/o changes
* godoc runs
* performance:
- The new version of godoc consumes about the same space after indexing
has completed, but indexing is half the speed. Significant space savings
are expected from smaller ASTs, but since they are thrown away after a
file has been indexed, this is not visible anymore. The slower indexing
time is due to the much more expensive computation of line information.
However, with the new compressed position information, indexing can be
rewritten and simplified. Furthermore, computing the line info can be
done more efficiently.
New godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
44381 godoc 0.0% 0:38.00 4 19 149 145M 184K 148M 176M
2010/12/03 17:58:35 index updated (39.231s, 18505 unique words, 386387 spots)
2010/12/03 17:58:35 bytes=90858456 footprint=199182584
2010/12/03 17:58:36 bytes=47858568 footprint=167295224
Old godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
23167 godoc 0.0% 0:22.02 4 17 132 129M 184K 132M 173M
2010/12/03 14:51:32 index updated (24.892s, 18765 unique words, 393830 spots)
2010/12/03 14:51:32 bytes=66404528 footprint=163907832
2010/12/03 14:51:32 bytes=46282224 footprint=163907832
The different numbers for unique words/spots stem from the fact the the
two workspaces are not exactly identical. The new godoc maintains a large
file set data structure during indexing which (probably) is the reason
for the larger heap (90858456 vs 66404528) before garbage collection.
R=rsc, r
CC=golang-dev
https://golang.org/cl/3050041
2010-12-06 15:23:18 -07:00
|
|
|
func Parse(fset *token.FileSet, filename string, src []byte) (Grammar, os.Error) {
|
2009-12-15 16:27:16 -07:00
|
|
|
var p parser
|
go/ast: use token.Pos instead of token.Position; adjust all dependent code
Specifically:
* lib/godoc:
- provide file set (FSet) argument to formatters where needed
* src/cmd:
- cgo, ebnflint, godoc, gofmt, goinstall: provide file set (fset) where needed
- godoc: remove local binary search with sort.Search (change by rsc),
extract file set for formatters
* src/pkg:
- exp/eval: remove embedded token.Position fields from nodes and replace
with named token.Pos fields; add corresponding Pos() accessor methods
- go/token: added file.Line(), changed signature of File.Position()
* test/fixedbugs/:
- bug206.go: change test to not rely on token.Pos details
* added various extra comments
* Runs all.bash
* gofmt formats all of src, misc w/o changes
* godoc runs
* performance:
- The new version of godoc consumes about the same space after indexing
has completed, but indexing is half the speed. Significant space savings
are expected from smaller ASTs, but since they are thrown away after a
file has been indexed, this is not visible anymore. The slower indexing
time is due to the much more expensive computation of line information.
However, with the new compressed position information, indexing can be
rewritten and simplified. Furthermore, computing the line info can be
done more efficiently.
New godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
44381 godoc 0.0% 0:38.00 4 19 149 145M 184K 148M 176M
2010/12/03 17:58:35 index updated (39.231s, 18505 unique words, 386387 spots)
2010/12/03 17:58:35 bytes=90858456 footprint=199182584
2010/12/03 17:58:36 bytes=47858568 footprint=167295224
Old godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
23167 godoc 0.0% 0:22.02 4 17 132 129M 184K 132M 173M
2010/12/03 14:51:32 index updated (24.892s, 18765 unique words, 393830 spots)
2010/12/03 14:51:32 bytes=66404528 footprint=163907832
2010/12/03 14:51:32 bytes=46282224 footprint=163907832
The different numbers for unique words/spots stem from the fact the the
two workspaces are not exactly identical. The new godoc maintains a large
file set data structure during indexing which (probably) is the reason
for the larger heap (90858456 vs 66404528) before garbage collection.
R=rsc, r
CC=golang-dev
https://golang.org/cl/3050041
2010-12-06 15:23:18 -07:00
|
|
|
grammar := p.parse(fset, filename, src)
|
2009-12-15 16:27:16 -07:00
|
|
|
return grammar, p.GetError(scanner.Sorted)
|
2009-07-13 11:10:56 -06:00
|
|
|
}
|