For #43684
Change-Id: I9ce47de82203ec87e7d3683f56e6c6d61ae255f5
Reviewed-on: https://go-review.googlesource.com/c/go/+/352151
Trust: Jay Conrod <jayconrod@google.com>
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
gofmt is pretty heavily CPU-bound, since parsing and formatting 1MiB
of Go code takes much longer than reading that amount of bytes from
disk. However, parsing and manipulating a large Go source file is very
difficult to parallelize, so we continue to process each file in its
own goroutine.
A Go module may contain a large number of Go source files, so we need
to bound the amount of work in flight. However, because the
distribution of sizes for Go source files varies widely — from tiny
doc.go files containing a single package comment all the way up to
massive API wrappers generated by automated tools — the amount of
time, work, and memory overhead needed to process each file also
varies. To account for this variability, we limit the in-flight work
by bytes of input rather than by number of files. That allows us to
make progress on many small files while we wait for work on a handful
of large files to complete.
The gofmt tool has a well-defined output format on stdout, which was
previously deterministic. We keep it deterministic by printing the
results of each file in order, using a lazily-synchronized io.Writer
(loosly inspired by Haskell's IO monad). After a file has been
formatted in memory, we keep it in memory (again, limited by the
corresponding number of input bytes) until the output for all previous
files has been flushed. This adds a bit of latency compared to
emitting the output in nondeterministic order, but a little extra
latency seems worth the cost to preserve output stability.
This change is based on Daniel Martí's work in CL 284139, but using a
weighted semaphore and ephemeral goroutines instead of a worker pool
and batches. Benchmark results are similar, and I find the concurrency
in this approach a bit easier to reason about.
In the batching-based approach, the batch size allows us to "look
ahead" to find large files and start processing them early. To keep
the CPUs saturated and prevent stragglers, we would need to tune the
batch size to be about the same as the largest input files. If the
batch size is set too high, a large batch of small files could turn
into a straggler, but if the batch size is set too low, the largest
files in the data set won't be started early enough and we'll end up
with a large-file straggler.
One possible alternative would be to sort by file size instead of
batching: identify all of the files to be processed, sort from largest
to smallest, and then process the largest files first so that the
"tail" of processing covers the smallest files. However, that approach
would still fail to saturate available CPU when disk latency is high,
would require buffering an arbitrary amount of metadata in order to
sort by size, and (perhaps most importantly!) would not allow the
`gofmt` binary to preserve the same (deterministic) output order that
it has today.
In contrast, with a semaphore we can produce the same deterministic
output as ever using only one tuning parameter: the memory footprint,
expressed as a rough lower bound on the amount of RAM available per
thread. While we're below the memory limit, we can run arbitrarily
many disk operations arbitrarily far ahead, and process the results of
those operations whenever they become avaliable. Then it's up to the
kernel (not us) to schedule the disk operations for throughput and
latency, and it's up to the runtime (not us) to schedule the
goroutines so that they complete quickly.
In practice, even a modest assumption of a few megabytes per thread
seems to provide a nice speedup, and it should scale reasonably even
to machines with vastly different ratios of CPU to disk. (In practice,
I expect that most 'gofmt' invocations will work with files on at most
one physical disk, so the CPU:disk ratio should vary more-or-less
directly with the thread count, whereas the CPU:memory ratio is
more-or-less independent of thread count.)
name \ time/op baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 11.9s ± 2% 2.7s ± 3% 2.8s ± 5%
name \ user-time/op baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 13.5s ± 2% 14.4s ± 1% 14.7s ± 1%
name \ sys-time/op baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 465ms ± 8% 229ms ±28% 232ms ±31%
name \ peak-RSS-bytes baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 77.7MB ± 4% 162.2MB ±10% 192.9MB ±15%
For #43566
Change-Id: I4ba251eb4d188a3bd1901039086be57f0b341910
Reviewed-on: https://go-review.googlesource.com/c/go/+/317975
Trust: Bryan C. Mills <bcmills@google.com>
Trust: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Thanks to @bodar (Github) for finding this.
Fixes#48422.
Change-Id: I031c3d82a02db1d204e2b86b494d89784d37f073
Reviewed-on: https://go-review.googlesource.com/c/go/+/350409
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
CL 85215 added prose to provide some minimal intuition for the
definition of a "terminating statement". While the original definition
was perfectly fine, the added prose was actually incorrect: If the
terminating statement is a goto, it might jump to a labeled statement
following that goto in the same block (it could be the very next
statement), and thus a terminating statement does not in fact
"prevent execution of all statements that lexically appear after
it in the same block".
Rather than explaining the special case for gotos with targets that
are lexically following the goto in the same block, this CL opts for
a simpler approach.
Thanks to @3bodar (Github) for finding this.
Fixes#48323.
Change-Id: I8031346250341d038938a1ce6a75d3e687d32c37
Reviewed-on: https://go-review.googlesource.com/c/go/+/349172
Trust: Robert Griesemer <gri@golang.org>
Trust: Emmanuel Odeke <emmanuel@orijtech.com>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
Reviewed-by: Rob Pike <r@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Change-Id: I33bde4835d3b83fafd55beea483f6236c4c62840
Reviewed-on: https://go-review.googlesource.com/c/go/+/338990
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Trust: Than McIntosh <thanm@google.com>
So it's clear to the reader that if "M" is a promoted method from
embedded field "T", then "x.M" will be expanded to "x.T.M" during the
evaluation of the method value.
Fixes#47863
Change-Id: Id3b82127a2054584b6842c487f6e15c3102dc9fe
Reviewed-on: https://go-review.googlesource.com/c/go/+/344209
Trust: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Run-TryBot: Cuong Manh Le <cuong.manhle.vn@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
This CL adds a new syscall.SyscallN API.
The proposal discussion also suggests the API should not only for
Windows but other platforms. However, the existing API set already
contain differences between platforms, hence the CL only implements
the Windows platform.
Moreover, although the API offers variadic parameters, the permitted
parameters remains up to a limit, which is selected as 42, and arguably
large enough.
Fixes#46552
Change-Id: I66b49988a304d9fc178c7cd5de46d0b75e167a4f
Reviewed-on: https://go-review.googlesource.com/c/go/+/336550
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Trust: Matthew Dempsky <mdempsky@google.com>
Trust: Jason A. Donenfeld <Jason@zx2c4.com>
This template is based on CL 295209 and previous ones like it.
Continue to eagerly include often-used sections, and clarify that
the TODO is about completing the section, or removing if it turns
out not to be needed.
Move the Go 1.17 release notes to x/website, since that's the new
home for past Go release notes as of CL 291711. They're added to
x/website in CL 342089.
For #47694
Updates #44513
Change-Id: I24962eed800d1509bdf71b7d7f819a683eb96f8f
Reviewed-on: https://go-review.googlesource.com/c/go/+/342070
Trust: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
The current text is slightly inaccurate. Make it more correct.
Change-Id: Iebe0051b74649d13982d7eefe3697f9e69c9b75d
Reviewed-on: https://go-review.googlesource.com/c/go/+/340449
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Fixes#44513.
Change-Id: I82c44a681b1fa67df123af86ee02a980b13acdc8
Reviewed-on: https://go-review.googlesource.com/c/go/+/341673
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Trust: Carlos Amedee <carlos@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
For #22446
Change-Id: I71a30761a28e81c50b7089d5a28be99c736c2dc8
Reviewed-on: https://go-review.googlesource.com/c/go/+/341332
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
The current document uses the "top" and "bottom" when talking
about the address within a frame, which may easily lead to
misunderstandings. This patch directly uses "high address"
and "low address" to make the expression clearer.
Change-Id: I7469330bbdc158672d7f0314fe6680ebdd9ab79a
Reviewed-on: https://go-review.googlesource.com/c/go/+/339369
Trust: fannie zhang <Fannie.Zhang@arm.com>
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Writing CL 333629 clarified my thinking about the behavioral changes
associated with lazy loading. There are really two interrelated
changes — graph pruning, and lazy loading proper — that are both made
possible by the added redundancy in the go.mod file.
(I had initially approached the whole cluster of features as “lazy
loading” because that was the starting point for the design. Graph
pruning came into the picture when we looked at how to bound the
worst-case behavior of lazy loading, but it is really the more
important of the two aspects of the design.)
Note that this change adds links to doc anchors added in CL 333629.
Fixes#36460Fixes#47397
Change-Id: I0ef4af57f647bf5ee210ea7099191fb4befa2cc1
Reviewed-on: https://go-review.googlesource.com/c/go/+/335135
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
There is an example for nil slice already, so adding example for non-nil
zero length slice, too, clarifying to the reader that the result is also
non-nil and different from nil slice case.
Updates #395
Change-Id: I019db1b1a1c0c621161ecaaacab5a4d888764b1a
Reviewed-on: https://go-review.googlesource.com/c/go/+/336890
Trust: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Cuong Manh Le <cuong.manhle.vn@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
Updates #44513
Change-Id: Ia0c6b48bde2719f3a99cb216b6166d82159198d1
Reviewed-on: https://go-review.googlesource.com/c/go/+/336930
Trust: Alberto Donizetti <alb.donizetti@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
For #395
For #46746
Change-Id: I4bfc094cf1cecd27ce48e31f92384cf470f371a6
Reviewed-on: https://go-review.googlesource.com/c/go/+/334669
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
For #46229
Change-Id: I54d01d90f2b0c892d76121f1350c0e8cf4b2772f
Reviewed-on: https://go-review.googlesource.com/c/go/+/334729
Trust: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
For #19367
Change-Id: If0ff8ddba3b6b48e2e198cf3653e73284c7572a3
Reviewed-on: https://go-review.googlesource.com/c/go/+/332409
Trust: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Robert Griesemer <gri@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
These types already implemented the Image interface. They should also
implement the RGBA64Image interface (new in Go 1.17)
Updates #44808
Change-Id: I9a2b13e305997088ae874efb95ad9e1648f94812
Reviewed-on: https://go-review.googlesource.com/c/go/+/331570
Trust: Nigel Tao <nigeltao@golang.org>
Run-TryBot: Nigel Tao <nigeltao@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Add unsafe.Add and unsafe.Slice to the list of built-in functions
which are not permitted in statement context. The compiler and
type checker already enforce this restriction, this just fixes
a documentation oversight.
For #19367.
For #40481.
Change-Id: Iabc63a8db048eaf40a5f5b5573fdf00b79d54119
Reviewed-on: https://go-review.googlesource.com/c/go/+/329925
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Rob Pike <r@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
For #46366
Change-Id: I8417e6e4dbb8cb56ff7afc16893a01b7bb938217
Reviewed-on: https://go-review.googlesource.com/c/go/+/329529
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Documents the mode added in CL 306149 to skip object resolution.
Fixes#46298
Change-Id: I6a14aaa00790f9f7e4e4ba17033355f5e878d74b
Reviewed-on: https://go-review.googlesource.com/c/go/+/329009
Trust: Robert Findley <rfindley@google.com>
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Findley <rfindley@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
The new RGBA64At method is equivalent to the existing At method (and the
new SetRGBA64 method is equivalent to the existing Set method in the
image/draw package), but they can avoid allocations from converting
concrete color types to the color.Color interface type.
Also update api/go1.17.txt and doc/go1.17.html
Fixes#44808
Change-Id: I8671f3144512b1200fa373840ed6729a5d61bc35
Reviewed-on: https://go-review.googlesource.com/c/go/+/311129
Trust: Nigel Tao <nigeltao@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
"Results" may sound like the results from the previous sentence.
Reword to "function return values" for clarity.
Suggested by Tobias Kohlbau.
Change-Id: Ie78df36b5b191b06e98b802f9a46db6d067a8ea0
Reviewed-on: https://go-review.googlesource.com/c/go/+/327774
Trust: Cherry Mui <cherryyz@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Tobias Kohlbau <tobias@kohlbau.de>
Simplify and remove forward-compatibility reference, as OpenBSD 6.9 has
already been released (1st of May 2021).
Updates #44513
Change-Id: I0a1abbb397f31d15c80a970edaa9723f894cafa9
Reviewed-on: https://go-review.googlesource.com/c/go/+/327652
Trust: Joel Sing <joel@sing.id.au>
Reviewed-by: Cherry Mui <cherryyz@google.com>
It was the only h3 in <code>, and it lacked <p> around its content.
It looked like it was part of the prior section:
https://tip.golang.org/doc/go1.17#gofmt
Change-Id: I7e9ef70e9a03474225833f44420aabd70dab3cd5
Reviewed-on: https://go-review.googlesource.com/c/go/+/326774
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Trust: Brad Fitzpatrick <bradfitz@golang.org>
As far as I can tell the Core Library section is complete. Remove its
TODO.
Change-Id: Ia84c6656fac045e25fae1a7ce8b488a3a26fd250
Reviewed-on: https://go-review.googlesource.com/c/go/+/326469
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
In 1.17, //go:build lines are fully supported. This entails changes to
the go command, vet, and gofmt. Document all of them.
I'm not Russ, but this is a significant change, it slipped under the
radar, and we're trying to get the release out. So here's what I got.
I wasn't sure where to put the go command change. On the one hand,
it's pretty significant. On the other, it certainly affects fewer
people than lazy loading. So it probably shouldn't be first, but I also
didn't want to bury it the middle of all the other module changes. Open
to suggestions.
Change-Id: Ia1a96bcfb1977973c5b0b0a6b18a9242a745af12
Reviewed-on: https://go-review.googlesource.com/c/go/+/326209
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
I'm not aware of anything more to mention for ports.
Change-Id: I686df8a230a55ad7f4c5eae43ca27f85fad9dd84
Reviewed-on: https://go-review.googlesource.com/c/go/+/326409
Trust: Cherry Mui <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
