Because profile labels are copied from the goroutine into the tag
buffer by the signal handler, there's a carefully-crafted set of race
detector annotations to create the necessary happens-before edges
between setting a goroutine's profile label and retrieving it from the
profile tag buffer.
Given the constraints of the signal handler, we have to approximate
the true synchronization behavior. Currently, that approximation is
too weak.
Ideally, runtime_setProfLabel would perform a store-release on
&getg().labels and copying each label into the profile would perform a
load-acquire on &getg().labels. This would create the necessary
happens-before edges through each individual g.labels object.
Since we can't do this in the signal handler, we instead synchronize
on a "labelSync" global. The problem occurs with the following
sequence:
1. Goroutine 1 calls setProfLabel, which does a store-release on
labelSync.
2. Goroutine 2 calls setProfLabel, which does a store-release on
labelSync.
3. Goroutine 3 reads the profile, which does a load-acquire on
labelSync.
The problem is that the load-acquire only synchronizes with the *most
recent* store-release to labelSync, and the two store-releases don't
synchronize with each other. So, once goroutine 3 touches the label
set by goroutine 1, we report a race.
The solution is to use racereleasemerge. This is like a
read-modify-write, rather than just a store-release. Each RMW of
labelSync in runtime_setProfLabel synchronizes with the previous RMW
of labelSync, and this ultimately carries forward to the load-acquire,
so it synchronizes with *all* setProfLabel operations, not just the
most recent.
Change-Id: Iab58329b156122002fff12cfe64fbeacb31c9613
Reviewed-on: https://go-review.googlesource.com/56670
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Minor refactoring. This is a step towards specializing evacuate
for mapfast key types.
Change-Id: Icffe2759b7d38e5c008d03941918d5a912ce62f6
Reviewed-on: https://go-review.googlesource.com/56933
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
Since oldbucket == h.nevacuate, we can just increment h.nevacuate here.
This removes oldbucket from scope, which will be useful shortly.
Change-Id: I70f81ec3995f17845ebf5d77ccd20ea4338f23e6
Reviewed-on: https://go-review.googlesource.com/56932
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Avelino <t@avelino.xxx>
Reviewed-by: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
The number of times that alg has to be spilled
and restored makes it better to just reload it.
Change-Id: I2674752a889ecad59dab54da1d68fad03db1ca85
Reviewed-on: https://go-review.googlesource.com/56931
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
The new code is not quite equivalent to the old,
in that if newbit was very large it might have altered the new tophash.
The old behavior is unnecessary and probably undesirable.
Change-Id: I7fb3222520cb61081a857adcddfbb9078ead7122
Reviewed-on: https://go-review.googlesource.com/56930
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
After the key and value arrays, we have an overflow pointer.
So there's no way a past-the-end key or value pointer could point
past the end of the containing bucket.
So we don't need this additional protection.
Update #21459
Change-Id: I7726140033b06b187f7a7d566b3af8cdcaeab0b0
Reviewed-on: https://go-review.googlesource.com/56772
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Avelino <t@avelino.xxx>
Found with mvdan.cc/unindent. Prioritized the ones with the biggest wins
for now.
Change-Id: I2b032e45cdd559fc9ed5b1ee4c4de42c4c92e07b
Reviewed-on: https://go-review.googlesource.com/56470
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Use mallogc instead of newarray to save some overhead since
makechan already checks for _MaxMem constraints.
Flattens the if else construct that determines if buf and hchan struct
should be allocated in one mallocgc call and where buf should point to.
Uses maxSliceCap to avoid divisions similar to makeslice.
name old time/op new time/op delta
MakeChan/Byte 82.0ns ± 8% 81.4ns ± 7% ~ (p=0.643 n=10+10)
MakeChan/Int 97.9ns ± 2% 96.6ns ± 2% -1.40% (p=0.009 n=10+10)
MakeChan/Ptr 128ns ± 3% 120ns ± 1% -6.63% (p=0.000 n=10+10)
MakeChan/Struct/0 66.7ns ± 4% 66.4ns ± 2% ~ (p=0.697 n=10+10)
MakeChan/Struct/32 136ns ± 1% 130ns ± 0% -4.42% (p=0.000 n=10+10)
MakeChan/Struct/40 150ns ± 1% 150ns ± 1% ~ (p=0.725 n=10+10)
Change-Id: Ibb5675d0843a072aae2bfa58ecd39cf4cd926533
Reviewed-on: https://go-review.googlesource.com/55132
Run-TryBot: Martin Möhrmann <moehrmann@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
Stack allocated hmap structs are explicitly zeroed before being
passed by pointer to makemap.
Heap allocated hmap structs are created with newobject
which also zeroes on allocation.
Therefore, setting the hmap fields to 0 or nil is redundant
since they will have been zeroed when hmap was allocated.
Change-Id: I5fc55b75e9dc5ba69f5e3588d6c746f53b45ba66
Reviewed-on: https://go-review.googlesource.com/56291
Reviewed-by: Keith Randall <khr@golang.org>
We use a call to strncpy to work around a TSAN bug (wherein TSAN only
delivers asynchronous signals when the thread receiving the signal
calls a libc function). Unfortunately, GCC 7 inlines the call,
avoiding the TSAN libc trap entirely.
Per Ian's suggestion, use global variables as strncpy arguments: that
way, the compiler can't make any assumptions about the concrete values
and can't inline the call away.
fixes#21196
Change-Id: Ie95f1feaf9af1a8056f924f49c29cfc8515385d7
Reviewed-on: https://go-review.googlesource.com/55872
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The code was adding race.Errors to t.raceErrors before checking
Failed, but Failed was using t.raceErrors+race.Errors. We don't want
to change Failed, since that would affect tests themselves, so modify
the harness to not unnecessarily change t.raceErrors.
Updates #19851Fixes#21338
Change-Id: I7bfdf281f90e045146c92444f1370d55c45221d4
Reviewed-on: https://go-review.googlesource.com/54050
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Rather than emitting spaces and newlines for println
as we walk the expression, construct it all up front.
This enables further optimizations.
This requires using printstring instead of print in
the implementation of printsp and printnl,
on pain of infinite recursion.
That's ok; it's more efficient anyway, and just as simple.
While we're here, do it for other print routines as well.
Change-Id: I61d7df143810e00710c4d4d948d904007a7fd190
Reviewed-on: https://go-review.googlesource.com/55097
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Writing to selectdone on the stack of another goroutine meant a
pretty subtle dance between the select code and the stack copying
code. Instead move the selectdone variable into the g struct.
Change-Id: Id246aaf18077c625adef7ca2d62794afef1bdd1b
Reviewed-on: https://go-review.googlesource.com/53390
Reviewed-by: Austin Clements <austin@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
I noticed that we don't set an itab's function pointers at compile
time. Instead, we currently do it at executable startup.
Set the function pointers at compile time instead. This shortens
startup time. It has no effect on normal binary size. Object files
will have more relocations, but that isn't a big deal.
For PIE there are additional pointers that will need to be adjusted at
load time. There are already other pointers in an itab that need to be
adjusted, so the cache line will already be paged in. There might be
some binary size overhead to mark these pointers. The "go test -c
-buildmode=pie net/http" binary is 0.18% bigger.
Update #20505
Change-Id: I267c82489915b509ff66e512fc7319b2dd79b8f7
Reviewed-on: https://go-review.googlesource.com/44341
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
Currently, GC captures the start-the-world time stamp after
startTheWorldWithSema returns. This is problematic for two reasons:
1. It's possible to get preempted between startTheWorldWithSema
starting the world and calling nanotime.
2. startTheWorldWithSema does several clean-up tasks after the world
is up and running that on rare occasions can take upwards of 10ms.
Since the runtime uses the start-the-world time stamp to compute the
STW duration, both of these can significantly inflate the reported STW
duration.
Fix this by having startTheWorldWithSema itself call nanotime once the
world is started.
Change-Id: I114630234fb73c9dabae50a2ef1884661f2459db
Reviewed-on: https://go-review.googlesource.com/55410
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rick Hudson <rlh@golang.org>
struct32 and struct40 structs are already declared, remove them to
make runtime tests build.
Change-Id: I3814f2b850dcb15c4002a3aa22e2a9326e5a5e53
Reviewed-on: https://go-review.googlesource.com/55614
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
Instead of comparing if the number of elements will
not fit into memory check if the memory size of the
slices backing memory is higher then the memory limit.
This avoids a division or maxElems lookup.
With et.size > 0:
uintptr(newcap) > maxSliceCap(et.size)
-> uintptr(int(capmem / et.size)) > _MaxMem / et.size
-> capmem / et.size > _MaxMem / et.size
-> capmem > _MaxMem
Note that due to integer division from capmem > _MaxMem
it does not follow that uintptr(newcap) > maxSliceCap(et.size).
Consolidated runtime GrowSlice benchmarks by using sub-benchmarks and
added more struct sizes to show performance improvement when division
is avoided for element sizes larger than 32 bytes.
AMD64:
GrowSlice/Byte 38.9ns ± 2% 38.9ns ± 1% ~ (p=0.974 n=20+20)
GrowSlice/Int 58.3ns ± 3% 58.0ns ± 2% ~ (p=0.154 n=20+19)
GrowSlice/Ptr 95.7ns ± 2% 95.1ns ± 2% -0.60% (p=0.034 n=20+20)
GrowSlice/Struct/24 95.4ns ± 1% 93.9ns ± 1% -1.54% (p=0.000 n=19+19)
GrowSlice/Struct/32 110ns ± 1% 108ns ± 1% -1.76% (p=0.000 n=19+20)
GrowSlice/Struct/40 138ns ± 1% 128ns ± 1% -7.09% (p=0.000 n=20+20)
Change-Id: I1c37857c74ea809da373e668791caffb6a5cbbd3
Reviewed-on: https://go-review.googlesource.com/53471
Run-TryBot: Martin Möhrmann <moehrmann@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
We weren't initializing this field for dynamically-generated itabs.
Turns out it doesn't matter, as any time we use this field we also
generate a static itab for the interface type / concrete type pair.
But we should initialize it anyway, just to be safe.
Performance on the benchmarks in CL 44339:
benchmark old ns/op new ns/op delta
BenchmarkItabFew-12 1040585 26466 -97.46%
BenchmarkItabAll-12 228873499 4287696 -98.13%
Change-Id: I58ed2b31e6c98b584122bdaf844fee7268b58295
Reviewed-on: https://go-review.googlesource.com/44475
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
We don't use it any more, remove it.
Change-Id: I76ce1a4c2e7048fdd13a37d3718b5abf39ed9d26
Reviewed-on: https://go-review.googlesource.com/44474
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
Just use fun[0]==0 to indicate a bad itab.
Change-Id: I28ecb2d2d857090c1ecc40b1d1866ac24a844848
Reviewed-on: https://go-review.googlesource.com/44473
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
Keep itabs in a growable hash table.
Use a simple open-addressable hash table, quadratic probing, power
of two sized.
Synchronization gets a bit more tricky. The common read path now
has two atomic reads, one to get the table pointer and one to read
the entry out of the table.
I set the max load factor to 75%, kind of arbitrarily. There's a
space-speed tradeoff here, and I'm not sure where we should land.
Because we use open addressing the itab.link field is no longer needed.
I'll remove it in a separate CL.
Fixes#20505
Change-Id: Ifb3d9a337512d6cf968c1fceb1eeaf89559afebf
Reviewed-on: https://go-review.googlesource.com/44472
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
Last runtime use was removed in https://golang.org/cl/133700043,
September 2014.
Replace plan9 syscall uses with plan9-specific variable.
Change-Id: Ifb910c021c1419a7c782959f90b054ed600d9e19
Reviewed-on: https://go-review.googlesource.com/55450
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
Run-TryBot: Martin Möhrmann <moehrmann@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
The preceding cleanup made it clear that two cases
(have golden data, unreachable key) are handled identically.
Simplify the control flow to reflect that.
Simplifies the code and generates shorter machine code.
Change-Id: Id612e0da6679813e855506f47222c58ea6497d70
Reviewed-on: https://go-review.googlesource.com/55093
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
This change unifies the x and y cases.
It shrinks evacuate's machine code by ~25% and its stack size by ~15%.
It also eliminates a critical branch.
Whether an entry should go to x or y is designed to be unpredictable.
As a result, half of the branch predictions for useX were wrong.
Mispredicting that branch can easily incur an expensive cache miss.
Switching to an xy array allows elimination of that branch,
which in turn reduces cache misses.
Change-Id: Ie9cef53744b96c724c377ac0985b487fc50b49b1
Reviewed-on: https://go-review.googlesource.com/54653
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
Make the calculation of k and v a bit lazier.
None of the following code cares about indirect-vs-direct k,
and it happens on all code paths, so check t.indirectkey earlier.
Simplifies the code and reduces both machine code and stack size.
Change-Id: I5ea4c0772848d7a4b15383baedb9a1f7feb47201
Reviewed-on: https://go-review.googlesource.com/55092
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
This avoids division and multiplication.
Instrumentation suggests that this is a very common case.
Change-Id: I2d5d5012d4f4df4c4af1f9f85ca9c323c9889c0e
Reviewed-on: https://go-review.googlesource.com/54657
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
This avoids the never triggered capacity checks in newarray.
Change-Id: Ib72b204adcb9e3fd3ab963defe0cd40e22d5d492
Reviewed-on: https://go-review.googlesource.com/54731
Run-TryBot: Martin Möhrmann <moehrmann@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
This makes sure that its argument is marked live on entry.
We need its arg to be live so defers of KeepAlive get
scanned correctly by the GC.
Fixes#21402
Change-Id: I906813e433d0e9726ca46483723303338da5b4d7
Reviewed-on: https://go-review.googlesource.com/55150
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
This change replaces the current runtime capabilities check for ppc64x with the
new internal/cpu package. It also adds support for the new POWER9 ISA and
capabilities.
Updates #15403
Change-Id: I5b64a79e782f8da3603e5529600434f602986292
Reviewed-on: https://go-review.googlesource.com/53830
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
This generates better code.
Masking B in the return statement should be unnecessary,
but the compiler is understandably not yet clever enough to see that.
Someday, it'd also be nice for the compiler to generate
a CMOV for the saturation if statement.
Change-Id: Ie1c157b21f5212610da1f3c7823a93816b3b61b9
Reviewed-on: https://go-review.googlesource.com/54656
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
Combine conditions into a single if statement.
This is more readable.
It should generate identical machine code, but it doesn't.
The new code is shorter.
Change-Id: I9bf52f8f288b0df97a2b9b4e4183f6ca74175e8a
Reviewed-on: https://go-review.googlesource.com/54651
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
This reduces the wall time to run these benchmarks by about 30%.
Change-Id: I494a93c93e5acb1514510d85f65796f62e1629a5
Reviewed-on: https://go-review.googlesource.com/54650
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
Currently we only support finding symbols in the VDSO using the old
DT_HASH. These days everything uses DT_GNU_HASH instead. To keep up
with the times and future-proof against DT_HASH disappearing from the
VDSO in the future, this commit adds support for DT_GNU_HASH and
prefers it over DT_HASH.
Tested by making sure it found a DT_GNU_HASH section and all of the
expected symbols in it, and then disabling the DT_GNU_HASH path and
making sure the old DT_HASH path still found all of the symbols.
Fixes#19649.
Change-Id: I508c8b35a019330d2c32f04f3833b69cb2686f13
Reviewed-on: https://go-review.googlesource.com/45511
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The dieFromSignal runtime function attempts to forward crashing
signals to a signal handler registered before the runtime was
initialized, if any. However, on Darwin, a special signal handler
trampoline is invoked, even for non-Go signal handlers.
Clear the crashing signal's handlingSig entry to ensure sigtramp
forwards the signal.
Fixes the darwin/386 builder.
Updates #20392
Updates #19389
Change-Id: I441a3d30c672cdb21ed6d8f1e1322d7c0e5b9669
Reviewed-on: https://go-review.googlesource.com/55032
Run-TryBot: Elias Naur <elias.naur@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
According to http://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_key_create.html,
pthread_key_create return an error number which is greater than or equal
to 0. I don't know the scenario that pthread_setspecific would fail, but
also don't know the future. Add some error handlings just in case.
Change-Id: I0774b79ef658d67e300f4a9aab1f2e3879acc7ee
Reviewed-on: https://go-review.googlesource.com/54811
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
_main has an early check to verify if a binary is statically or dynamically
linked that depends on R0 being zero. R0 is not guaranteed to be zero at that
point and this was breaking Go on Alpine for ppc64le.
Change-Id: I4a1059ff7fd3db6fc489e7dcfe631c1814dd965b
Reviewed-on: https://go-review.googlesource.com/54730
Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com>
Reviewed-by: Lynn Boger <laboger@linux.vnet.ibm.com>
Although mincore is declared in stubs.go, mincore isn't used by any
OSes except linux. Move it to os_linux.go and clean up unused code.
Change-Id: I6cfb0fed85c0317a4d091a2722ac55fa79fc7c9a
Reviewed-on: https://go-review.googlesource.com/54910
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This computes the maximum possible waste in a size class due to both
internal and external fragmentation as a percent of the span size.
This parallels the reasoning about overhead in the comment at the top
of mksizeclasses.go and confirms that comment's assertion that (except
for the few smallest size classes), none of the size classes have
worst-case internal and external fragmentation simultaneously.
Change-Id: Idb66fe6c241d56f33d391831d4cd5a626955562b
Reviewed-on: https://go-review.googlesource.com/49370
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
