This is based off Michael's notes.
Updates #47694
Change-Id: I6e7944f85b776e8481829a2fafd177a49557c6ca
Reviewed-on: https://go-review.googlesource.com/c/go/+/368156
Trust: Jeremy Faller <jeremy@golang.org>
Run-TryBot: Jeremy Faller <jeremy@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
This description is based on https://golang.org/cl/321490.
Updates #47694
Change-Id: I48656cd487d2fccf0b0d3390f350f1bc6f2b0080
Reviewed-on: https://go-review.googlesource.com/c/go/+/365738
Trust: Jeremy Faller <jeremy@golang.org>
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Jeremy Faller <jeremy@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
For #47694.
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Change-Id: I38c2fd9b57fbbacf220a2bc679f67e2dfdcc7cb1
Reviewed-on: https://go-review.googlesource.com/c/go/+/367514
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Fixes#49214.
For #47694.
Change-Id: Iba68ed17bfd81890309b6a6732087f87a03e1350
Reviewed-on: https://go-review.googlesource.com/c/go/+/366274
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Also mention local types restriction.
We probably want to say more at some point, this is just a
placeholder to start.
Update #47631
Change-Id: I828e451e1e8504d21cb55c7132e9cb330b160a54
Reviewed-on: https://go-review.googlesource.com/c/go/+/364134
Trust: Keith Randall <khr@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
For #47694
Change-Id: I39594c273aeb038702457587ee1c46e4b3920bb6
Reviewed-on: https://go-review.googlesource.com/c/go/+/363359
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Trust: Jeremy Faller <jeremy@golang.org>
This PR adds a note into the Go 1.18 changelog for CL 330852.
Updates #46923.
Change-Id: I99150e9275ce23fcf3697d6a22ac216818223c74
GitHub-Last-Rev: b2772ce68b
GitHub-Pull-Request: golang/go#49258
Reviewed-on: https://go-review.googlesource.com/c/go/+/360297
Trust: Roland Shoemaker <roland@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
For #47327
Change-Id: I50418c0d017c4e90a2c13d26945ee639079e4e33
Reviewed-on: https://go-review.googlesource.com/c/go/+/363174
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Paschalis Tsilias <paschalistsilias@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Updates #48367.
Change-Id: Ib8fc6d9dd7c3c6a70fefe077615f51a71d9c42ed
Reviewed-on: https://go-review.googlesource.com/c/go/+/361899
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Trust: Dmitri Shuralyov <dmitshur@golang.org>
A small touchup after CL 361894.
For #47694.
Change-Id: Ifc161516f897f727195d21351a3c8eda7b6e327e
Reviewed-on: https://go-review.googlesource.com/c/go/+/361895
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Carlos Amedee <carlos@golang.org>
For #47694.
Change-Id: Ia80a1859bd0fc6f08d27293f519c22fd9a804fd2
Reviewed-on: https://go-review.googlesource.com/c/go/+/361894
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Trust: Carlos Amedee <carlos@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
In modules that specify 'go 1.17' or higher, the go.mod file
explicitly requires modules for all packages transitively imported by
the main module. Users tend to use 'go mod download' to prepare for
testing the main module itself, so we should only download those
relevant modules.
In 'go 1.16' and earlier modules, we continue to download all modules
in the module graph (because we cannot in general tell which ones are
relevant without loading the full package import graph).
'go mod download all' continues to download every module in
'go list all', as it did before.
Fixes#44435
Change-Id: I3f286c0e2549d6688b3832ff116e6cd77a19401c
Reviewed-on: https://go-review.googlesource.com/c/go/+/357310
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
This reverts CL 359096.
Updates #47788.
Reason for revert: -buildmode=shared may have actually been working in a few very specific cases. We should not remove -buildmode=shared until we have implemented an alternative to support those few cases.
Change-Id: Ia962b06abacc11f6f29fc29d092773be175e32f1
Reviewed-on: https://go-review.googlesource.com/c/go/+/359575
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
Change-Id: I032791a3cda1916099b0fd2955dcca4e69763660
Reviewed-on: https://go-review.googlesource.com/c/go/+/357958
Trust: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
It never worked in module mode (or with a read-only GOROOT).
A proposal to drop it was filed (and approved) in
https://golang.org/issue/47788.
Fixes#47788
Change-Id: I0c12f38eb0c5dfe9384fbdb49ed202301fa4273d
Reviewed-on: https://go-review.googlesource.com/c/go/+/359096
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Mention future versions will require the COMPAT_FREEBSD12 kernel option
set in the kernel.
For #47694.
Change-Id: Ia94c4f9dbb38c68025d3c1d12dd2e241a5480a6b
Reviewed-on: https://go-review.googlesource.com/c/go/+/354971
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
For #43684
Change-Id: I9ce47de82203ec87e7d3683f56e6c6d61ae255f5
Reviewed-on: https://go-review.googlesource.com/c/go/+/352151
Trust: Jay Conrod <jayconrod@google.com>
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
gofmt is pretty heavily CPU-bound, since parsing and formatting 1MiB
of Go code takes much longer than reading that amount of bytes from
disk. However, parsing and manipulating a large Go source file is very
difficult to parallelize, so we continue to process each file in its
own goroutine.
A Go module may contain a large number of Go source files, so we need
to bound the amount of work in flight. However, because the
distribution of sizes for Go source files varies widely — from tiny
doc.go files containing a single package comment all the way up to
massive API wrappers generated by automated tools — the amount of
time, work, and memory overhead needed to process each file also
varies. To account for this variability, we limit the in-flight work
by bytes of input rather than by number of files. That allows us to
make progress on many small files while we wait for work on a handful
of large files to complete.
The gofmt tool has a well-defined output format on stdout, which was
previously deterministic. We keep it deterministic by printing the
results of each file in order, using a lazily-synchronized io.Writer
(loosly inspired by Haskell's IO monad). After a file has been
formatted in memory, we keep it in memory (again, limited by the
corresponding number of input bytes) until the output for all previous
files has been flushed. This adds a bit of latency compared to
emitting the output in nondeterministic order, but a little extra
latency seems worth the cost to preserve output stability.
This change is based on Daniel Martí's work in CL 284139, but using a
weighted semaphore and ephemeral goroutines instead of a worker pool
and batches. Benchmark results are similar, and I find the concurrency
in this approach a bit easier to reason about.
In the batching-based approach, the batch size allows us to "look
ahead" to find large files and start processing them early. To keep
the CPUs saturated and prevent stragglers, we would need to tune the
batch size to be about the same as the largest input files. If the
batch size is set too high, a large batch of small files could turn
into a straggler, but if the batch size is set too low, the largest
files in the data set won't be started early enough and we'll end up
with a large-file straggler.
One possible alternative would be to sort by file size instead of
batching: identify all of the files to be processed, sort from largest
to smallest, and then process the largest files first so that the
"tail" of processing covers the smallest files. However, that approach
would still fail to saturate available CPU when disk latency is high,
would require buffering an arbitrary amount of metadata in order to
sort by size, and (perhaps most importantly!) would not allow the
`gofmt` binary to preserve the same (deterministic) output order that
it has today.
In contrast, with a semaphore we can produce the same deterministic
output as ever using only one tuning parameter: the memory footprint,
expressed as a rough lower bound on the amount of RAM available per
thread. While we're below the memory limit, we can run arbitrarily
many disk operations arbitrarily far ahead, and process the results of
those operations whenever they become avaliable. Then it's up to the
kernel (not us) to schedule the disk operations for throughput and
latency, and it's up to the runtime (not us) to schedule the
goroutines so that they complete quickly.
In practice, even a modest assumption of a few megabytes per thread
seems to provide a nice speedup, and it should scale reasonably even
to machines with vastly different ratios of CPU to disk. (In practice,
I expect that most 'gofmt' invocations will work with files on at most
one physical disk, so the CPU:disk ratio should vary more-or-less
directly with the thread count, whereas the CPU:memory ratio is
more-or-less independent of thread count.)
name \ time/op baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 11.9s ± 2% 2.7s ± 3% 2.8s ± 5%
name \ user-time/op baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 13.5s ± 2% 14.4s ± 1% 14.7s ± 1%
name \ sys-time/op baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 465ms ± 8% 229ms ±28% 232ms ±31%
name \ peak-RSS-bytes baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 77.7MB ± 4% 162.2MB ±10% 192.9MB ±15%
For #43566
Change-Id: I4ba251eb4d188a3bd1901039086be57f0b341910
Reviewed-on: https://go-review.googlesource.com/c/go/+/317975
Trust: Bryan C. Mills <bcmills@google.com>
Trust: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
This CL adds a new syscall.SyscallN API.
The proposal discussion also suggests the API should not only for
Windows but other platforms. However, the existing API set already
contain differences between platforms, hence the CL only implements
the Windows platform.
Moreover, although the API offers variadic parameters, the permitted
parameters remains up to a limit, which is selected as 42, and arguably
large enough.
Fixes#46552
Change-Id: I66b49988a304d9fc178c7cd5de46d0b75e167a4f
Reviewed-on: https://go-review.googlesource.com/c/go/+/336550
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Trust: Matthew Dempsky <mdempsky@google.com>
Trust: Jason A. Donenfeld <Jason@zx2c4.com>
This template is based on CL 295209 and previous ones like it.
Continue to eagerly include often-used sections, and clarify that
the TODO is about completing the section, or removing if it turns
out not to be needed.
Move the Go 1.17 release notes to x/website, since that's the new
home for past Go release notes as of CL 291711. They're added to
x/website in CL 342089.
For #47694
Updates #44513
Change-Id: I24962eed800d1509bdf71b7d7f819a683eb96f8f
Reviewed-on: https://go-review.googlesource.com/c/go/+/342070
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
