On big corpuses, the indexer was spending most of its time waiting
for filesystem operations (especially with network filesystems)
and not actually indexing. This keeps the filesystem busy and indexer
running in different goroutines.
Also, add a hook to let godoc hosts disable indexing of certain
directories.
And finally, start adding tests for godoc, which required
fleshing out (and testing) the mapfs code.
R=golang-dev, adg, bgarcia
CC=golang-dev
https://golang.org/cl/21520045
The exported Server becomes handlerServer, and part of Presentation
now. Presentation is also now an http.Handler with its own
internal mux (a detail, which might go away).
main.go becomes ever simpler.
R=golang-dev, adg
CC=golang-dev
https://golang.org/cl/11505043
