On big corpuses, the indexer was spending most of its time waiting
for filesystem operations (especially with network filesystems)
and not actually indexing. This keeps the filesystem busy and indexer
running in different goroutines.
Also, add a hook to let godoc hosts disable indexing of certain
directories.
And finally, start adding tests for godoc, which required
fleshing out (and testing) the mapfs code.
R=golang-dev, adg, bgarcia
CC=golang-dev
https://golang.org/cl/21520045
cmd/godoc/godoc.go is now merged into main.go, which is now
only 530 lines.
App Engine mode is still broken, but should be easy to fix up.
(just needs a global *godoc.Presentation created in init)
R=golang-dev, adg
CC=golang-dev
https://golang.org/cl/11498044
