On most platforms newly-mapped memory is untouched, meaning the pages
backing the region haven't been faulted in yet. However, we mark this
memory as unscavenged which means the background scavenger
aggressively "returns" this memory to the OS if the heap is small.
The only platform where newly-mapped memory is actually unscavenged (and
counts toward the application's RSS) is on Windows, since
(*mheap).sysAlloc commits the reservation. Instead of making a special
case for Windows, I change the requirements a bit for a sysReserve'd
region. It must now be both sysMap'd and sysUsed'd, with sysMap being a
no-op on Windows. Comments about memory allocation have been updated to
include a more up-to-date mental model of which states a region of memory
may be in (at a very low level) and how to transition between these
states.
Now this means we can correctly mark newly-mapped heap memory as
scavenged on every platform, reducing the load on the background
scavenger early on in the application for small heaps. As a result,
heap-growth scavenging is no longer necessary, since any actual RSS
growth will be accounted for on the allocation codepath.
Finally, this change also cleans up grow a little bit to avoid
pretending that it's freeing an in-use span and just does the necessary
operations directly.
Fixes#32012.
Fixes#31966.
Updates #26473.
Change-Id: Ie06061eb638162e0560cdeb0b8993d94cfb4d290
Reviewed-on: https://go-review.googlesource.com/c/go/+/177097
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
For:
http2: track reused connections
https://golang.org/cl/176720 (updates golang/go#31982)
Some x/sys/unix updates come along for the ride too.
I filed #32031 for making the bundling process less difficult and
error-prone in the future.
Change-Id: Ic822080991ffa2d50352c5f613e45648a327cf16
Reviewed-on: https://go-review.googlesource.com/c/go/+/177037
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
The compiler appears to have a latent bug:
fusePlain calls invalidateCFG when it changes block structure,
but fuseIf does not.
Fix this by hoisting the call to invalidateCFG to the top level.
Change-Id: Ic960fb3ac963b15b4a225aad84863d58efa954e6
Reviewed-on: https://go-review.googlesource.com/c/go/+/177198
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
The CL 164718 mistyped the comparison flags. The rules for floating
point comparison should be GreaterThanF and GreaterEqualF. Fortunately,
the wrong optimizations were overwritten by other integer rules, so the
issue won't cause failure but just some performance impact.
The fixed CL optimizes the floating point test as follows.
source code: func foo(f float64) bool { return f > 4 || f < -4}
previous version: "FCMPD", "CSET\tGT", "CBZ"
fixed version: "FCMPD", BLE"
Add the test case.
Change-Id: Iea954fdbb8272b2d642dae0f816dc77286e6e1fa
Reviewed-on: https://go-review.googlesource.com/c/go/+/177121
Reviewed-by: Ben Shi <powerman1st@163.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Run-TryBot: Ben Shi <powerman1st@163.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This is CVE-2019-11888.
Previously, passing a nil environment but a non-nil token would result
in the new potentially unprivileged process inheriting the parent
potentially privileged environment, or would result in the new
potentially privileged process inheriting the parent potentially
unprivileged environment. Either way, it's bad. In the former case, it's
an infoleak. In the latter case, it's a possible EoP, since things like
PATH could be overwritten.
Not specifying an environment currently means, "use the existing
environment". This commit amends the behavior to be, "use the existing
environment of the token the process is being created for." The behavior
therefore stays the same when creating processes without specifying a
token. And it does the correct thing when creating processes when
specifying a token.
Fixes#32000
Change-Id: Ia57f6e89b97bdbaf7274d6a89c1d9948b6d40ef5
Reviewed-on: https://go-review.googlesource.com/c/go/+/176619
Run-TryBot: Jason Donenfeld <Jason@zx2c4.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
This is useful for debugging line number assignment and
also for making sense of pathological line number inputs.
Activated with
-gcflags=-d=ssa/number_lines/stats=1 (the bit matters)
-gcflags=-d=ssa/number_lines/debug
Stats:
"SUM_LINE_RANGE",
SUM for f in files {MAX line in f {line}-MIN line in f {line}}
"MAXMIN_LINE_RANGE",
MAX for f in files {MAX line in f {line}} -
MIN for f in files {MIN line in f {line}}
"MAXFILE", maxfile,
MAX for f in files {f}
"NFILES", len(entries)
| files |
Change-Id: I8a7336e6370452fe2e3a62de17606db9bd6a6fd3
Reviewed-on: https://go-review.googlesource.com/c/go/+/174947
Run-TryBot: David Chase <drchase@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
This cuts the time for 'go test -short testing' from 0.9s to < 0.1s.
Change-Id: Ib8402f80239e1e96ea5221dfd5cd0db08170d85b
Reviewed-on: https://go-review.googlesource.com/c/go/+/177420
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
CL 121936 added this diagnostic to avoid a panic accessing *short.
(Hence the "This shouldn't really be a panic" comment.)
That CL was right to produce a clearer error than a plain memory fault,
but I think wrong to print+exit instead of panicking. I just ran into
one of these in a real program, and there is no indication anywhere
of how the program reached this point. The panic will show that.
So change print+exit to a panic with a helpful message, in contrast
to the original panic with an unhelpful message and the current
helpful message without stack trace.
Change-Id: Ib2bae1dead4ccde92f00fa3a34c05241ff7690c6
Reviewed-on: https://go-review.googlesource.com/c/go/+/177419
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
This just makes all.bash a bit less chatty.
Change-Id: I7d2ecabf0c7d8df2065d7052718f611bb2907801
Reviewed-on: https://go-review.googlesource.com/c/go/+/177418
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
These tests take 20 seconds each to rebuild the entire world
in their respective modes. That's too much for short mode,
including all.bash on developer machines.
Keep doing it on builders and if someone runs 'go test' by hand
in that directory.
For #26473.
Change-Id: I3dc6955bc3aa7a20fd170efcde72a7d19b37bdbf
Reviewed-on: https://go-review.googlesource.com/c/go/+/177417
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Was 50 seconds on unloaded Mac laptop; now 27.
Still longer than I would like, but every little bit helps.
For #26473.
Change-Id: Id4be016ee1555cbc3512eca0ae10236d7f06bd02
Reviewed-on: https://go-review.googlesource.com/c/go/+/177398
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Add information about action and build IDs,
commands run, time spent in commands,
and overall time to -debug-actiongraph output.
Also avoid a terrible outcome in:
go build -debug-actiongraph x.go
Change-Id: I7a3a638f4848d75f2bdc997517f4ab23656da4f1
Reviewed-on: https://go-review.googlesource.com/c/go/+/177138
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
The work of running full vet on std and cmd during local development
has moved to go test, which of course runs during all.bash.
For errors in other GOOS/GOARCH combinations, the misc-compile
builders (running buildall.bash) also now run go vet std cmd.
The vetall builder need not do anything anymore.
Make it a no-op until it can be retired, and remove
cmd/vet/all and its special case in the go command.
Fixes#31916.
Change-Id: I8f30d184c382ea7c2c8f520e5618f680db633968
Reviewed-on: https://go-review.googlesource.com/c/go/+/176440
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Now that the main tree complies with 'go vet', enable all vet checks
during 'go test' in the main tree. This helps surface helpful errors
while developing, instead of having to wait for the misc-vet-vetall builder.
During 'go test', the additional vet checks are essentially free:
the vet invocations themselves take only 8 seconds total for the entire tree.
Also update buildall.bash (used by the misc-compile builders)
to run 'go vet std cmd' for each GOOS/GOARCH pair.
This is not as free, since in general it can require recompiling
packages with their tests included before invoking vet.
(That compilation was going on anyway in the 'go test' case.)
On my Mac laptop, ./buildall.bash freebsd used to take
68+16+17+18 = 119 seconds for make.bash and then
the builds of the three freebsd architectures.
Now it takes 68+16+23+17+23+18+24 = 189 seconds, 60% longer.
Some of this is spent doing unnecessary cgo work.
Still, this lets us shard the vet checks and match all.bash.
Fixes#20119.
For #31916.
Change-Id: I6b0c40bac47708a688463c7fca12c0fc23ab2751
Reviewed-on: https://go-review.googlesource.com/c/go/+/176439
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This updates the Go function declarations to match race_amd64.s.
Change-Id: I2b541a6b335ce732f4c31652aa615240ce7bb1c3
Reviewed-on: https://go-review.googlesource.com/c/go/+/177397
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
When fmt.Errorf is provided with a %w verb with an error operand,
return an error implementing an Unwrap method returning that operand.
It is invalid to use %w with other formatting functions, to use %w
multiple times in a format string, or to use %w with a non-error
operand. When the Errorf format string contains an invalid use of %w,
the returned error does not implement Unwrap.
Change-Id: I534e20d3b163ab22c2b137b1c9095906dc243221
Reviewed-on: https://go-review.googlesource.com/c/go/+/176998
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
This allows the use of CLONE_VFORK and CLONE_VM for fork/exec, preventing
"fork/exec ...: cannot allocate memory" failures from occuring when attempting
to execute commands from a Go process that has a large memory footprint.
Additionally, this should reduce the latency of fork/exec on these platforms.
The same problem was addressed on linux/amd64 via issue #5838.
Updates #31936
Change-Id: I7ae0fbbeaa29cab944a49a11272a380d497eb2d0
Reviewed-on: https://go-review.googlesource.com/c/go/+/175697
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Vet needs export data for the imports of the package it is analyzing.
Vet does not need export data for the package itself, since vet will
do its own type checking. Assuming that vet is just as good as the compiler
at detecting invalid programs, don't run the compiler unnecessarily.
This especially matters for tests without external test files or for
which the external test files do not import the test-augmented original
package. In that case, the test-augmented original package need not
be compiled at all.
Cuts time for 'go clean -cache && go vet -x cmd/compile/internal/ssa'
from 7.6r 24.3u 2.8s to 3.5r 8.5u 1.9s, by not running the compiler
on the augmented test package.
There is still more to be done here - if we do need to build a
test-augmented package, we rerun cgo unnecessarily.
But this is a big help.
Cuts time for 'go vet std cmd' by about 30%.
For #31916.
Change-Id: If6136b4d384f1da77aed90b43f1a6b95f09b5d86
Reviewed-on: https://go-review.googlesource.com/c/go/+/176438
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Now renameio package creates file use ioutil.TempFile, which calls
OpenFile with mode 0600, we should support creates a file with given
permission bits.
Fixes#31871
Change-Id: I0436e9f7081f2fce18bf9f3b14d55b02d4d995fb
Reviewed-on: https://go-review.googlesource.com/c/go/+/175958
Run-TryBot: Baokun Lee <nototon@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
The spec was not very precise as to what happens with respect to sharing
if a sliced operand is (a pointer to) an array. Added a small clarification
and a supporting example.
Fixes#31689.
Change-Id: Ic49351bec2033abd3f5428154ec3e9a7c2c9eaa5
Reviewed-on: https://go-review.googlesource.com/c/go/+/177139
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Rob Pike <r@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
This reduces allocations and also resolves some
lurking inliner/inlinee line-number match problems.
However, it does add about 1.5% to compile time.
This fixes compiler OOMs seen compiling some large protobuf-
derived inputs. For compiling the compiler itself,
compilebench -pkg cmd/compile/internal/ssa -memprofile withcl.prof
the numberlines-related memory consumption is reduced from 129MB
to 29MB (about a 5% overall reduction in allocation).
Additionally modified after going over changes with Austin
to remove unused code (nobody called size()) and correct
the cache-clearing code.
I've attempted to speed this up by not using maps, and have
not succeeded. I'd rather get correct code in now, speed it
up later if I can.
Updates #27739.
Fixes#29279.
Change-Id: I098005de4e45196a5f5b10c0886a49f88e9f8fd5
Reviewed-on: https://go-review.googlesource.com/c/go/+/154617
Run-TryBot: David Chase <drchase@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
If a slice's entries are sparse, we decide to initialize it dynamically
instead of statically. That's CL 151319.
But if we do initialize it dynamically, we still need to initialize
the static entries. Typically we do that, but the bug fixed here is
that we don't if the entry's value is itself an array or struct.
To fix, use initKindLocalCode to ensure that both static and
dynamic entries are initialized via code.
Fixes#31987
Change-Id: I1192ffdbfb5cd50445c1206c4a3d8253295201dd
Reviewed-on: https://go-review.googlesource.com/c/go/+/176904
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
In the reverseproxy, replace use (Header).Get, which returns only one value
of a multiple value header, with using the Header map directly. Also fixes
corresponding tests which hid the bug, and adds more tests.
Fixes#30303
Change-Id: Ic9094b5983043460697748759f6dfd95fc111db7
GitHub-Last-Rev: b41038143f
GitHub-Pull-Request: golang/go#30687
Reviewed-on: https://go-review.googlesource.com/c/go/+/166298
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The memory might not be synchronized in a thread being woken up after a
semasleep. Using atomic instructions in noteclear function will force
this synchronisation.
Fixes#30189
Change-Id: If7432f29b2a1a56288231822db52f3f8d1d6dbfe
Reviewed-on: https://go-review.googlesource.com/c/go/+/163624
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Fixes#14683
Change-Id: I62c429e4fcc2f20a94d3db8c1f0ca587252c07a7
Reviewed-on: https://go-review.googlesource.com/c/go/+/174130
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Some repositories include tags like 'v1.0.0-rc.1+oryOS.9'.
If we were to allow such tags, they could become ambiguous: semantic
versioning defines versions that differ only in metadata to have equal
precedence, so if someone added a tag 'v1.0.0-rc.1+other' at a
different commit, then the version 'v1.0.0-rc.1' would become
ambiguous.
However, we should still allow those tags to be used to resolve
versions, and since we can even parse the underlying semantic version,
we can at least use that as the basis for a unique (and well-ordered)
pseudo-version.
Fixes#31713
Change-Id: I5035f76d74ead6e786c04a368595cb5e42d36f91
Reviewed-on: https://go-review.googlesource.com/c/go/+/176905
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Before this commit, the two logically equivalent conditionals below would
produce different output:
{{ if not .NonEmptyInterfaceTypedNil }}OK{{ else }}{{ end }}
{{ if .NonEmptyInterfaceTypedNil }}{{ else }}OK{{ end }}
The functions `not`, `or`, and `and` all use the same `truth` function, which
unwraps any concrete interface value before passing it to `isTrue`.
`if` and `with` also use `isTrue` to establish truth, but was missing the
interface indirect call.
Fixes#30501
Change-Id: I9c49eed41e737d8f162e39bef1c3b82fd5518fed
GitHub-Last-Rev: 95fc2c82f2
GitHub-Pull-Request: golang/go#30534
Reviewed-on: https://go-review.googlesource.com/c/go/+/164958
Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
Add a small paragraph and example pointing out
the difference for the case where T is a slice
or map. This is a common error for Go novices.
Fixes#29425.
Change-Id: Icdb59f25361e9f6a09b190fbfcc9ae0c7d90077b
Reviewed-on: https://go-review.googlesource.com/c/go/+/176338
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
The very first paragraph on "Package initialization" stated that
"variables are initialized in declaration order, but after any
variables they might depend on". This phrasing was easily
misread as "declaration order is the first sorting criteria"
and then contradicted what the subsequent paragraphs spelled
out in precise detail.
Instead, variable initialization proceeds by repeatedly determining
a set of ready to initialize variables, and then selecting from that
set the variable declared earliest. That is, declaration order is the
second sorting criteria.
Also, for the purpose of variable initialization, declarations
introducing blank (_) variables are considered like any other
variables (their initialization expressions may have side-effects
and affect initialization order), even though blank identifiers
are not "declared".
This CL adds clarifying language regarding these two issues
and the supporting example.
Both gccgo and go/types implement this behavior. cmd/compile
has a long-standing issue (#22326).
The spec also did not state in which order multiple variables
initialized by a single (multi-value) initialization expression are
handled. This CL adds a clarifying paragraph: If any such variable
is initialized, all that declaration's variables are initialized at
the same time.
This behavior matches user expectation: We are not expecting to
observe partially initialized sets of variables in declarations
such as "var a, b, c = f()".
It also matches existing cmd/compile and go/types (but not gccgo)
behavior.
Finally, cmd/compile, gccgo, and go/types produce different
initialization orders in (esoteric) cases where hidden (not
detected with existing rules) dependencies exist. Added a
sentence and example clarifying how much leeway compilers have
in those situations. The goal is to preserve the ability to
use static initialization while at the same time maintain
the relative initialization order of variables with detected
dependencies.
Fixes #31292.
Updates #22326.
Change-Id: I0a369abff8cfce27afc975998db875f5c580caa2
Reviewed-on: https://go-review.googlesource.com/c/go/+/175980
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reduces the time on my machine for
go clean -cache; go test -short -count=1 cmd/compile/internal/gc
from 4.7s to 3.7s.
Updates #26473
Change-Id: I9f9573675ffd6519da63961f48f61260ae4717fd
Reviewed-on: https://go-review.googlesource.com/c/go/+/176937
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This speeds up
go test -short -count=1 cmd/fix
on my machine from about 8s to about 0.05s.
Updates #26473
Change-Id: I698ee20704ae0aee874ba642e7b0e070ddc99194
Reviewed-on: https://go-review.googlesource.com/c/go/+/176900
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This is a follow-up to CL 174099, fixing an important TODO.
The 'go help modget' documentation will be clarified in anotehr CL,
pending further discussion.
When invoked without -m, 'go get' will no longer match arguments
containing "..." against module paths. If a module's path matches a
pattern but no packages within the module match the pattern, the
module should not be upgraded. For example, if
golang.org/x/tools/playground and golang.org/x/tools are separate
modules, and only golang.org/x/tools is in the build list,
'go get golang.org/x/tools/playground/...' should add
golang.org/x/tools/playground to the build list and leave
golang.org/x/tools alone.
Updates #26902
Change-Id: I2bd18c7950db1aa7bd8527210c1baf2c7d174375
Reviewed-on: https://go-review.googlesource.com/c/go/+/176578
Run-TryBot: Jay Conrod <jayconrod@google.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Speeds up
go test -short -count=1 cmd/objdump
on my machine from 1.7s to 1.3s.
Not much, but as the backpacking saying goes,
take care of the ounces and the pounds will take care of themselves.
Updates #26473
Change-Id: I59fe9a179e48537c7d82cbba72cde9f92b42a029
Reviewed-on: https://go-review.googlesource.com/c/go/+/176901
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
