2021-08-13 12:42:49 -06:00
|
|
|
<!--{
|
|
|
|
|
"Title": "Go 1.18 Release Notes",
|
|
|
|
|
"Path": "/doc/go1.18"
|
|
|
|
|
}-->
|
|
|
|
|
|
|
|
|
|
<!--
|
|
|
|
|
NOTE: In this document and others in this directory, the convention is to
|
|
|
|
|
set fixed-width phrases with non-fixed-width spaces, as in
|
|
|
|
|
<code>hello</code> <code>world</code>.
|
|
|
|
|
Do not send CLs removing the interior tags from such phrases.
|
|
|
|
|
-->
|
|
|
|
|
|
|
|
|
|
<style>
|
|
|
|
|
main ul li { margin: 0.5em 0; }
|
|
|
|
|
</style>
|
|
|
|
|
|
|
|
|
|
<h2 id="introduction">DRAFT RELEASE NOTES — Introduction to Go 1.18</h2>
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
<strong>
|
|
|
|
|
Go 1.18 is not yet released. These are work-in-progress
|
|
|
|
|
release notes. Go 1.18 is expected to be released in February 2022.
|
|
|
|
|
</strong>
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<h2 id="language">Changes to the language</h2>
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
TODO: complete this section
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<h2 id="ports">Ports</h2>
|
|
|
|
|
|
2021-10-10 15:05:09 -06:00
|
|
|
<p id="freebsd">
|
|
|
|
|
Go 1.18 is the last release that is supported on FreeBSD 11.x, which has
|
|
|
|
|
already reached end-of-life. Go 1.19 will require FreeBSD 12.2+ or FreeBSD
|
|
|
|
|
13.0+.
|
|
|
|
|
FreeBSD 13.0+ will require a kernel with the COMPAT_FREEBSD12 option set (this is the default).
|
2021-08-13 12:42:49 -06:00
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<h2 id="tools">Tools</h2>
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
TODO: complete this section, or delete if not needed
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<h3 id="go-command">Go command</h3>
|
|
|
|
|
|
2021-09-24 14:54:52 -06:00
|
|
|
<p><!-- golang.org/issue/43684 -->
|
|
|
|
|
<code>go</code> <code>get</code> no longer builds or installs packages in
|
|
|
|
|
module-aware mode. <code>go</code> <code>get</code> is now dedicated to
|
|
|
|
|
adjusting dependencies in <code>go.mod</code>. Effectively, the
|
|
|
|
|
<code>-d</code> flag is always enabled. To install the latest version
|
|
|
|
|
of an executable outside the context of the current module, use
|
|
|
|
|
<a href="https://golang.org/ref/mod#go-install"><code>go</code>
|
|
|
|
|
<code>install</code> <code>example.com/cmd@latest</code></a>. Any
|
|
|
|
|
<a href="https://golang.org/ref/mod#version-queries">version query</a>
|
|
|
|
|
may be used instead of <code>latest</code>. This form of <code>go</code>
|
|
|
|
|
<code>install</code> was added in Go 1.16, so projects supporting older
|
|
|
|
|
versions may need to provide install instructions for both <code>go</code>
|
|
|
|
|
<code>install</code> and <code>go</code> <code>get</code>. <code>go</code>
|
|
|
|
|
<code>get</code> now reports an error when used outside a module, since there
|
|
|
|
|
is no <code>go.mod</code> file to update. In GOPATH mode (with
|
|
|
|
|
<code>GO111MODULE=off</code>), <code>go</code> <code>get</code> still builds
|
|
|
|
|
and installs packages, as before.
|
|
|
|
|
</p>
|
|
|
|
|
|
2021-10-14 16:40:44 -06:00
|
|
|
<p><!-- golang.org/issue/37475 -->
|
|
|
|
|
The <code>go</code> command now embeds version control information in
|
2021-10-22 01:41:41 -06:00
|
|
|
binaries including the currently checked-out revision, commit time, and a
|
|
|
|
|
flag indicating whether edited or untracked files are present. Version
|
|
|
|
|
control information is embedded if the <code>go</code> command is invoked in
|
|
|
|
|
a directory within a Git, Mercurial, Fossil, or Bazaar repository, and the
|
|
|
|
|
<code>main</code> package and its containing main module are in the same
|
|
|
|
|
repository. This information may be omitted using the flag
|
|
|
|
|
<code>-buildvcs=false</code>.
|
2021-10-14 16:40:44 -06:00
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p><!-- golang.org/issue/37475 -->
|
|
|
|
|
Additionally, the <code>go</code> command embeds information about the build
|
|
|
|
|
including build and tool tags (set with <code>-tags</code>), compiler,
|
|
|
|
|
assembler, and linker flags (like <code>-gcflags</code>), whether cgo was
|
|
|
|
|
enabled, and if it was, the values of the cgo environment variables
|
|
|
|
|
(like <code>CGO_CFLAGS</code>). This information may be omitted using the
|
|
|
|
|
flag <code>-buildinfo=false</code>. Both VCS and build information may be
|
|
|
|
|
read together with module information using <code>go</code>
|
|
|
|
|
<code>version</code> <code>-m</code> <code>file</code> or
|
|
|
|
|
<code>runtime/debug.ReadBuildInfo</code> (for the currently running binary)
|
|
|
|
|
or the new <a href="#debug/buildinfo"><code>debug/buildinfo</code></a>
|
|
|
|
|
package.
|
|
|
|
|
</p>
|
|
|
|
|
|
2021-10-20 12:02:24 -06:00
|
|
|
<p><!-- https://golang.org/issue/44435 -->
|
|
|
|
|
If the main module's <code>go.mod</code> file
|
|
|
|
|
specifies <a href="/ref/mod#go-mod-file-go"><code>go</code> <code>1.17</code></a>
|
|
|
|
|
or higher, <code>go</code> <code>mod</code> <code>download</code> without
|
|
|
|
|
arguments now downloads source code for only the modules
|
|
|
|
|
explicitly <a href="/ref/mod#go-mod-file-require">required</a> in the main
|
|
|
|
|
module's <code>go.mod</code> file. (In a <code>go</code> <code>1.17</code> or
|
|
|
|
|
higher module, that set already includes all dependencies needed to build the
|
|
|
|
|
packages and tests in the main module.)
|
|
|
|
|
To also download source code for transitive dependencies, use
|
|
|
|
|
<code>go</code> <code>mod</code> <code>download</code> <code>all</code>.
|
|
|
|
|
</p>
|
|
|
|
|
|
2021-08-13 12:42:49 -06:00
|
|
|
<p>
|
|
|
|
|
TODO: complete this section, or delete if not needed
|
|
|
|
|
</p>
|
|
|
|
|
|
cmd/gofmt: format files in parallel
gofmt is pretty heavily CPU-bound, since parsing and formatting 1MiB
of Go code takes much longer than reading that amount of bytes from
disk. However, parsing and manipulating a large Go source file is very
difficult to parallelize, so we continue to process each file in its
own goroutine.
A Go module may contain a large number of Go source files, so we need
to bound the amount of work in flight. However, because the
distribution of sizes for Go source files varies widely — from tiny
doc.go files containing a single package comment all the way up to
massive API wrappers generated by automated tools — the amount of
time, work, and memory overhead needed to process each file also
varies. To account for this variability, we limit the in-flight work
by bytes of input rather than by number of files. That allows us to
make progress on many small files while we wait for work on a handful
of large files to complete.
The gofmt tool has a well-defined output format on stdout, which was
previously deterministic. We keep it deterministic by printing the
results of each file in order, using a lazily-synchronized io.Writer
(loosly inspired by Haskell's IO monad). After a file has been
formatted in memory, we keep it in memory (again, limited by the
corresponding number of input bytes) until the output for all previous
files has been flushed. This adds a bit of latency compared to
emitting the output in nondeterministic order, but a little extra
latency seems worth the cost to preserve output stability.
This change is based on Daniel Martí's work in CL 284139, but using a
weighted semaphore and ephemeral goroutines instead of a worker pool
and batches. Benchmark results are similar, and I find the concurrency
in this approach a bit easier to reason about.
In the batching-based approach, the batch size allows us to "look
ahead" to find large files and start processing them early. To keep
the CPUs saturated and prevent stragglers, we would need to tune the
batch size to be about the same as the largest input files. If the
batch size is set too high, a large batch of small files could turn
into a straggler, but if the batch size is set too low, the largest
files in the data set won't be started early enough and we'll end up
with a large-file straggler.
One possible alternative would be to sort by file size instead of
batching: identify all of the files to be processed, sort from largest
to smallest, and then process the largest files first so that the
"tail" of processing covers the smallest files. However, that approach
would still fail to saturate available CPU when disk latency is high,
would require buffering an arbitrary amount of metadata in order to
sort by size, and (perhaps most importantly!) would not allow the
`gofmt` binary to preserve the same (deterministic) output order that
it has today.
In contrast, with a semaphore we can produce the same deterministic
output as ever using only one tuning parameter: the memory footprint,
expressed as a rough lower bound on the amount of RAM available per
thread. While we're below the memory limit, we can run arbitrarily
many disk operations arbitrarily far ahead, and process the results of
those operations whenever they become avaliable. Then it's up to the
kernel (not us) to schedule the disk operations for throughput and
latency, and it's up to the runtime (not us) to schedule the
goroutines so that they complete quickly.
In practice, even a modest assumption of a few megabytes per thread
seems to provide a nice speedup, and it should scale reasonably even
to machines with vastly different ratios of CPU to disk. (In practice,
I expect that most 'gofmt' invocations will work with files on at most
one physical disk, so the CPU:disk ratio should vary more-or-less
directly with the thread count, whereas the CPU:memory ratio is
more-or-less independent of thread count.)
name \ time/op baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 11.9s ± 2% 2.7s ± 3% 2.8s ± 5%
name \ user-time/op baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 13.5s ± 2% 14.4s ± 1% 14.7s ± 1%
name \ sys-time/op baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 465ms ± 8% 229ms ±28% 232ms ±31%
name \ peak-RSS-bytes baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 77.7MB ± 4% 162.2MB ±10% 192.9MB ±15%
For #43566
Change-Id: I4ba251eb4d188a3bd1901039086be57f0b341910
Reviewed-on: https://go-review.googlesource.com/c/go/+/317975
Trust: Bryan C. Mills <bcmills@google.com>
Trust: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2021-01-16 11:03:31 -07:00
|
|
|
<h3 id="gofmt"><code>gofmt</code></h3>
|
|
|
|
|
|
|
|
|
|
<p><!-- https://golang.org/issue/43566 -->
|
|
|
|
|
<code>gofmt</code> now reads and formats input files concurrently, with a
|
|
|
|
|
memory limit proportional to <code>GOMAXPROCS</code>. On a machine with
|
2021-10-20 05:13:29 -06:00
|
|
|
multiple CPUs, <code>gofmt</code> should now be significantly faster.
|
cmd/gofmt: format files in parallel
gofmt is pretty heavily CPU-bound, since parsing and formatting 1MiB
of Go code takes much longer than reading that amount of bytes from
disk. However, parsing and manipulating a large Go source file is very
difficult to parallelize, so we continue to process each file in its
own goroutine.
A Go module may contain a large number of Go source files, so we need
to bound the amount of work in flight. However, because the
distribution of sizes for Go source files varies widely — from tiny
doc.go files containing a single package comment all the way up to
massive API wrappers generated by automated tools — the amount of
time, work, and memory overhead needed to process each file also
varies. To account for this variability, we limit the in-flight work
by bytes of input rather than by number of files. That allows us to
make progress on many small files while we wait for work on a handful
of large files to complete.
The gofmt tool has a well-defined output format on stdout, which was
previously deterministic. We keep it deterministic by printing the
results of each file in order, using a lazily-synchronized io.Writer
(loosly inspired by Haskell's IO monad). After a file has been
formatted in memory, we keep it in memory (again, limited by the
corresponding number of input bytes) until the output for all previous
files has been flushed. This adds a bit of latency compared to
emitting the output in nondeterministic order, but a little extra
latency seems worth the cost to preserve output stability.
This change is based on Daniel Martí's work in CL 284139, but using a
weighted semaphore and ephemeral goroutines instead of a worker pool
and batches. Benchmark results are similar, and I find the concurrency
in this approach a bit easier to reason about.
In the batching-based approach, the batch size allows us to "look
ahead" to find large files and start processing them early. To keep
the CPUs saturated and prevent stragglers, we would need to tune the
batch size to be about the same as the largest input files. If the
batch size is set too high, a large batch of small files could turn
into a straggler, but if the batch size is set too low, the largest
files in the data set won't be started early enough and we'll end up
with a large-file straggler.
One possible alternative would be to sort by file size instead of
batching: identify all of the files to be processed, sort from largest
to smallest, and then process the largest files first so that the
"tail" of processing covers the smallest files. However, that approach
would still fail to saturate available CPU when disk latency is high,
would require buffering an arbitrary amount of metadata in order to
sort by size, and (perhaps most importantly!) would not allow the
`gofmt` binary to preserve the same (deterministic) output order that
it has today.
In contrast, with a semaphore we can produce the same deterministic
output as ever using only one tuning parameter: the memory footprint,
expressed as a rough lower bound on the amount of RAM available per
thread. While we're below the memory limit, we can run arbitrarily
many disk operations arbitrarily far ahead, and process the results of
those operations whenever they become avaliable. Then it's up to the
kernel (not us) to schedule the disk operations for throughput and
latency, and it's up to the runtime (not us) to schedule the
goroutines so that they complete quickly.
In practice, even a modest assumption of a few megabytes per thread
seems to provide a nice speedup, and it should scale reasonably even
to machines with vastly different ratios of CPU to disk. (In practice,
I expect that most 'gofmt' invocations will work with files on at most
one physical disk, so the CPU:disk ratio should vary more-or-less
directly with the thread count, whereas the CPU:memory ratio is
more-or-less independent of thread count.)
name \ time/op baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 11.9s ± 2% 2.7s ± 3% 2.8s ± 5%
name \ user-time/op baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 13.5s ± 2% 14.4s ± 1% 14.7s ± 1%
name \ sys-time/op baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 465ms ± 8% 229ms ±28% 232ms ±31%
name \ peak-RSS-bytes baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 77.7MB ± 4% 162.2MB ±10% 192.9MB ±15%
For #43566
Change-Id: I4ba251eb4d188a3bd1901039086be57f0b341910
Reviewed-on: https://go-review.googlesource.com/c/go/+/317975
Trust: Bryan C. Mills <bcmills@google.com>
Trust: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2021-01-16 11:03:31 -07:00
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
|
2021-08-13 12:42:49 -06:00
|
|
|
<h2 id="runtime">Runtime</h2>
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
TODO: complete this section, or delete if not needed
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<h2 id="compiler">Compiler</h2>
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
TODO: complete this section, or delete if not needed
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<h2 id="linker">Linker</h2>
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
TODO: complete this section, or delete if not needed
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<h2 id="library">Core library</h2>
|
|
|
|
|
|
2021-11-01 23:25:55 -06:00
|
|
|
<h3>TODO</h3>
|
2021-08-13 12:42:49 -06:00
|
|
|
<p>
|
|
|
|
|
TODO: complete this section
|
|
|
|
|
</p>
|
|
|
|
|
|
2021-11-01 23:25:55 -06:00
|
|
|
<h3 id="netip">New <code>net/netip</code> package</h3>
|
|
|
|
|
<p>
|
|
|
|
|
The new <a href="/pkg/net/netip/"><code>net/netip</code></a>
|
2021-11-02 08:59:16 -06:00
|
|
|
package defines a new IP address type, <a href="/pkg/net/netip/#Addr"><code>Addr</code></a>.
|
|
|
|
|
Compared to the existing
|
2021-11-01 23:25:55 -06:00
|
|
|
<a href="/pkg/net/#IP"><code>net.IP</code></a> type, the <code>netip.Addr</code> type takes less
|
|
|
|
|
memory, is immutable, and is comparable so it supports <code>==</code>
|
|
|
|
|
and can be used as a map key.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
In addition to <code>Addr</code>, the package defines
|
|
|
|
|
<a href="/pkg/net/netip/#AddrPort"><code>AddrPort</code></a>, representing
|
|
|
|
|
an IP and port, and
|
|
|
|
|
<a href="/pkg/net/netip/#Prefix"><code>Prefix</code></a>, representing
|
|
|
|
|
a network CIDR prefix.
|
|
|
|
|
</p>
|
|
|
|
|
<p>
|
|
|
|
|
The <code>net</code> package now has methods to send and receive UDP packets
|
|
|
|
|
using <code>netip.Addr</code> values instead of the relatively heavy
|
|
|
|
|
<code>*net.UDPAddr</code> values.
|
|
|
|
|
</p>
|
|
|
|
|
|
2021-08-13 12:42:49 -06:00
|
|
|
<h3 id="minor_library_changes">Minor changes to the library</h3>
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
As always, there are various minor changes and updates to the library,
|
|
|
|
|
made with the Go 1 <a href="/doc/go1compat">promise of compatibility</a>
|
|
|
|
|
in mind.
|
|
|
|
|
</p>
|
|
|
|
|
|
|
|
|
|
<p>
|
|
|
|
|
TODO: complete this section
|
|
|
|
|
</p>
|
2021-07-22 09:50:42 -06:00
|
|
|
|
2021-10-14 16:40:44 -06:00
|
|
|
<dl id="debug/buildinfo"><dt><a href="/pkg/debug/buildinfo">debug/buildinfo</a></dt>
|
|
|
|
|
<dd>
|
|
|
|
|
<p><!-- golang.org/issue/39301 -->
|
|
|
|
|
This new package provides access to module versions, version control
|
|
|
|
|
information, and build flags embedded in executable files built by
|
|
|
|
|
the <code>go</code> command. The same information is also available via
|
|
|
|
|
<a href="/pkg/runtime/debug#ReadBuildInfo"><code>runtime/debug.ReadBuildInfo</code></a>
|
|
|
|
|
for the currently running binary and via <code>go</code>
|
|
|
|
|
<code>version</code> <code>-m</code> on the command line.
|
|
|
|
|
</p>
|
|
|
|
|
</dd>
|
|
|
|
|
</dl>
|
|
|
|
|
|
2021-08-05 00:11:28 -06:00
|
|
|
<dl id="image/draw"><dt><a href="/pkg/image/draw/">image/draw</a></dt>
|
|
|
|
|
<dd>
|
|
|
|
|
<p><!-- CL 340049 -->
|
|
|
|
|
The <code>Draw</code> and <code>DrawMask</code> fallback implementations
|
|
|
|
|
(used when the arguments are not the most common image types) are now
|
|
|
|
|
faster when those arguments implement the optional
|
|
|
|
|
<a href="/pkg/image/draw/#RGBA64Image"><code>draw.RGBA64Image</code></a>
|
|
|
|
|
and <a href="/pkg/image/#RGBA64Image"><code>image.RGBA64Image</code></a>
|
|
|
|
|
interfaces that were added in Go 1.17.
|
|
|
|
|
</p>
|
|
|
|
|
</dd>
|
|
|
|
|
</dl><!-- image/draw -->
|
|
|
|
|
|
2021-10-14 22:08:11 -06:00
|
|
|
<dl id="reflect"><dt><a href="/pkg/reflect/">reflect</a></dt>
|
|
|
|
|
<dd>
|
|
|
|
|
<p><!-- CL 356049, 320929 -->
|
|
|
|
|
The new
|
|
|
|
|
<a href="/pkg/reflect/#Value.SetIterKey"><code>Value.SetIterKey</code></a>
|
|
|
|
|
and <a href="/pkg/reflect/#Value.SetIterValue"><code>Value.SetIterValue</code></a>
|
|
|
|
|
methods set a Value using a map iterator as the source. They are equivalent to
|
|
|
|
|
<code>Value.Set(iter.Key())</code> and <code>Value.Set(iter.Value())</code> but
|
|
|
|
|
do fewer allocations.
|
|
|
|
|
</p>
|
|
|
|
|
</dd>
|
2021-10-18 10:54:02 -06:00
|
|
|
<dd>
|
|
|
|
|
<p><!-- CL 350691 -->
|
|
|
|
|
The new
|
|
|
|
|
<a href="/pkg/reflect/#Value.UnsafePointer"><code>Value.UnsafePointer</code></a>
|
|
|
|
|
method returns the Value's value as an <a href="/pkg/unsafe/#Pointer"><code>unsafe.Pointer</code></a>.
|
|
|
|
|
This allows callers to migrate from <a href="/pkg/reflect/#Value.UnsafeAddr"><code>Value.UnsafeAddr</code></a>
|
|
|
|
|
and <a href="/pkg/reflect/#Value.Pointer"><code>Value.Pointer</code></a>
|
|
|
|
|
to eliminate the need to perform uintptr to unsafe.Pointer conversions at the callsite (as unsafe.Pointer rules require).
|
|
|
|
|
</p>
|
|
|
|
|
</dd>
|
2021-10-14 22:08:11 -06:00
|
|
|
</dl><!-- reflect -->
|
|
|
|
|
|
2021-07-22 09:50:42 -06:00
|
|
|
<dl id="syscall"><dt><a href="/pkg/syscall/">syscall</a></dt>
|
|
|
|
|
<dd>
|
|
|
|
|
<p><!-- CL 336550 -->
|
|
|
|
|
The new function <a href="/pkg/syscall/?GOOS=windows#SyscallN"><code>SyscallN</code></a>
|
|
|
|
|
has been introduced for Windows, allowing for calls with arbitrary number
|
2021-10-14 22:08:11 -06:00
|
|
|
of arguments. As a result,
|
2021-07-22 09:50:42 -06:00
|
|
|
<a href="/pkg/syscall/?GOOS=windows#Syscall"><code>Syscall</code></a>,
|
|
|
|
|
<a href="/pkg/syscall/?GOOS=windows#Syscall6"><code>Syscall6</code></a>,
|
|
|
|
|
<a href="/pkg/syscall/?GOOS=windows#Syscall9"><code>Syscall9</code></a>,
|
|
|
|
|
<a href="/pkg/syscall/?GOOS=windows#Syscall12"><code>Syscall12</code></a>,
|
|
|
|
|
<a href="/pkg/syscall/?GOOS=windows#Syscall15"><code>Syscall15</code></a>, and
|
|
|
|
|
<a href="/pkg/syscall/?GOOS=windows#Syscall18"><code>Syscall18</code></a> are
|
|
|
|
|
deprecated in favor of <a href="/pkg/syscall/?GOOS=windows#SyscallN"><code>SyscallN</code></a>.
|
|
|
|
|
</p>
|
|
|
|
|
</dd>
|
cmd/gofmt: format files in parallel
gofmt is pretty heavily CPU-bound, since parsing and formatting 1MiB
of Go code takes much longer than reading that amount of bytes from
disk. However, parsing and manipulating a large Go source file is very
difficult to parallelize, so we continue to process each file in its
own goroutine.
A Go module may contain a large number of Go source files, so we need
to bound the amount of work in flight. However, because the
distribution of sizes for Go source files varies widely — from tiny
doc.go files containing a single package comment all the way up to
massive API wrappers generated by automated tools — the amount of
time, work, and memory overhead needed to process each file also
varies. To account for this variability, we limit the in-flight work
by bytes of input rather than by number of files. That allows us to
make progress on many small files while we wait for work on a handful
of large files to complete.
The gofmt tool has a well-defined output format on stdout, which was
previously deterministic. We keep it deterministic by printing the
results of each file in order, using a lazily-synchronized io.Writer
(loosly inspired by Haskell's IO monad). After a file has been
formatted in memory, we keep it in memory (again, limited by the
corresponding number of input bytes) until the output for all previous
files has been flushed. This adds a bit of latency compared to
emitting the output in nondeterministic order, but a little extra
latency seems worth the cost to preserve output stability.
This change is based on Daniel Martí's work in CL 284139, but using a
weighted semaphore and ephemeral goroutines instead of a worker pool
and batches. Benchmark results are similar, and I find the concurrency
in this approach a bit easier to reason about.
In the batching-based approach, the batch size allows us to "look
ahead" to find large files and start processing them early. To keep
the CPUs saturated and prevent stragglers, we would need to tune the
batch size to be about the same as the largest input files. If the
batch size is set too high, a large batch of small files could turn
into a straggler, but if the batch size is set too low, the largest
files in the data set won't be started early enough and we'll end up
with a large-file straggler.
One possible alternative would be to sort by file size instead of
batching: identify all of the files to be processed, sort from largest
to smallest, and then process the largest files first so that the
"tail" of processing covers the smallest files. However, that approach
would still fail to saturate available CPU when disk latency is high,
would require buffering an arbitrary amount of metadata in order to
sort by size, and (perhaps most importantly!) would not allow the
`gofmt` binary to preserve the same (deterministic) output order that
it has today.
In contrast, with a semaphore we can produce the same deterministic
output as ever using only one tuning parameter: the memory footprint,
expressed as a rough lower bound on the amount of RAM available per
thread. While we're below the memory limit, we can run arbitrarily
many disk operations arbitrarily far ahead, and process the results of
those operations whenever they become avaliable. Then it's up to the
kernel (not us) to schedule the disk operations for throughput and
latency, and it's up to the runtime (not us) to schedule the
goroutines so that they complete quickly.
In practice, even a modest assumption of a few megabytes per thread
seems to provide a nice speedup, and it should scale reasonably even
to machines with vastly different ratios of CPU to disk. (In practice,
I expect that most 'gofmt' invocations will work with files on at most
one physical disk, so the CPU:disk ratio should vary more-or-less
directly with the thread count, whereas the CPU:memory ratio is
more-or-less independent of thread count.)
name \ time/op baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 11.9s ± 2% 2.7s ± 3% 2.8s ± 5%
name \ user-time/op baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 13.5s ± 2% 14.4s ± 1% 14.7s ± 1%
name \ sys-time/op baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 465ms ± 8% 229ms ±28% 232ms ±31%
name \ peak-RSS-bytes baseline.txt 284139.txt simplified.txt
GofmtGorootCmd 77.7MB ± 4% 162.2MB ±10% 192.9MB ±15%
For #43566
Change-Id: I4ba251eb4d188a3bd1901039086be57f0b341910
Reviewed-on: https://go-review.googlesource.com/c/go/+/317975
Trust: Bryan C. Mills <bcmills@google.com>
Trust: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2021-01-16 11:03:31 -07:00
|
|
|
</dl><!-- syscall -->
|
