On big corpuses, the indexer was spending most of its time waiting
for filesystem operations (especially with network filesystems)
and not actually indexing. This keeps the filesystem busy and indexer
running in different goroutines.
Also, add a hook to let godoc hosts disable indexing of certain
directories.
And finally, start adding tests for godoc, which required
fleshing out (and testing) the mapfs code.
R=golang-dev, adg, bgarcia
CC=golang-dev
https://golang.org/cl/21520045
