Previously we would spin up a new goroutine for every directory
we needed to parse. Since parent directories cannot exit until
children have exited, in large GOPATH directories we could have 8000+
goroutines evaluating directories concurrently. The race detector uses
tsan, and tsan can handle a maximum of 8192 concurrent threads at one
time.
Limit the number of concurrent threads. It is difficult to do this
in a way that does not involve a child being blocked on a parent
completing to spin up goroutines for itself to complete. Solve this by
completing work on the main thread if there are no additional workers
available to help complete work.
I was expecting limiting the concurrency to hurt performance, but it
actually significantly improves it. Benchmarks were performed using CL
94904 at tip, and with goroutines at 2, 4, and 8 times the number of
CPU's.
$ benchstat tip.benchmark gated-2.benchmark gated-4.benchmark gated-8.benchmark
name \ time/op tip.benchmark gated-2.benchmark gated-4.benchmark gated-8.benchmark
NewDirectory-4 293ms ± 2% 262ms ± 4% 252ms ± 4% 253ms ± 2%
name \ alloc/op tip.benchmark gated-2.benchmark gated-4.benchmark gated-8.benchmark
NewDirectory-4 218MB ± 0% 218MB ± 0% 218MB ± 0% 218MB ± 0%
name \ allocs/op tip.benchmark gated-2.benchmark gated-4.benchmark gated-8.benchmark
NewDirectory-4 513k ± 0% 508k ± 0% 509k ± 0% 510k ± 0%
Fixesgolang/go#22110.
Change-Id: If01f78f1fc53cd195e4f8f6988c3c39b3c275992
Reviewed-on: https://go-review.googlesource.com/94955
Reviewed-by: Yury Smolsky <yury@smolsky.by>
Reviewed-by: Kevin Burke <kev@inburke.com>
When building the corpus of local packages, a "too many open files"
error would cause some directories inside GOPATH to be skipped.
Further, the error would not be reported because it was masked by a
"file not found" error from the GOROOT VFS layer.
This change adds a rate limit around parsing files when buildling
the directory tree, error reporting when godoc is run with -v, and
fixes the masked error issue in the vfs package.
It's possible that the rate limiting could be put into the
godoc/vfs/gatefs package, but I tried making the gate account for
open files (not just individual open/close/read/write operations)
but then godoc just hard locks (it wasn't designed to only open 20
files at once).
Change-Id: I925d120b53d9a86430b6977cb90eb143785ecc48
Reviewed-on: https://go-review.googlesource.com/24060
Reviewed-by: Dave Day <djd@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
To avoid breaking URLs, we redirect /src/pkg/* to /src/*.
The URL /pkg is now the "directory" /src, which triggers the
"Packages" index.
All other references to "src/pkg" are now gone,
except a number in the namespace documentation which are
probably still illustrative.
Tested: go test cmd/godoc godoc
Manual inspection of src and src/pkg pages.
with GOROOT and GOPATH packages
-analysis
/AUTHORS file URL still works
LGTM=bradfitz, adg
R=bradfitz, adg
CC=golang-codereviews
https://golang.org/cl/141770044
This lets godoc implementations provide a more efficient means
of getting this information, without reading files and parsing the
package docs.
This is especially important when the files themselves don't
actually exist and the VFS is synthesizing them on demand
(e.g. protocol buffer files -> their generated *.pb.go files).
This means corpus.Init can run quickly, without generating
every protocol file in a large corpus (or fetching it from a
cache).
In the future, this hook could also be used for caching the summaries of
regular packages.
R=golang-dev, adg
CC=golang-dev
https://golang.org/cl/19440043
Massive win for high-latency network filesystems.
Also benefits the local disk/ssd case too, though.
R=golang-dev, bgarcia
CC=golang-dev
https://golang.org/cl/18650043
