1
0
mirror of https://github.com/golang/go synced 2024-11-19 14:24:47 -07:00
Commit Graph

2811 Commits

Author SHA1 Message Date
Josh Bleecher Snyder
cf69867af4 runtime: speed up key copying in specialized evacuate routines
Similar to CL 59110.

Change-Id: Ia2858541c86a44b105eacbca9a46b1044632c5ca
Reviewed-on: https://go-review.googlesource.com/59134
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-28 18:55:47 +00:00
Josh Bleecher Snyder
0480bfe57f runtime: remove handling of indirect key types in evacuate_fastX
None of the mapfast key types are indirect.

Change-Id: I1fb2682257567ee69504082a6cdad63c99916671
Reviewed-on: https://go-review.googlesource.com/59133
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
2017-08-28 18:55:17 +00:00
Josh Bleecher Snyder
db9f5917fc runtime: remove handling of non-reflexive key types in evacuate_fastX
All of the mapfast key types are reflexive.

Change-Id: I8595aed2a9d945cda1b5d08e2067dce0f1c0d585
Reviewed-on: https://go-review.googlesource.com/59132
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
2017-08-28 18:54:54 +00:00
Josh Bleecher Snyder
3c182a12f7 runtime: replace t.keysize with fixed key size in evacuate_fastX
Change-Id: I89c3c3b21d7a4acbc49b14a52ac8d9a5861c0c39
Reviewed-on: https://go-review.googlesource.com/59131
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
2017-08-28 18:54:21 +00:00
Josh Bleecher Snyder
4cf7f909c6 runtime: add specialized copies of growWork and evacuate
The newly added routines are exact copies of the generic routines,
except for the function names and that growWork_fastX calls evacuate_fastX.

Actual optimization will happen in subsequent CLs.
This is intended to ease reviewing.

Change-Id: I52ef7dd40b2bdfc9cba2496544c0604e6e71cf7f
Reviewed-on: https://go-review.googlesource.com/59130
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-28 18:00:03 +00:00
Daniel Martí
99da8730b0 all: remove some double spaces from comments
Went mainly for the ones that make no sense, such as the ones
mid-sentence or after commas.

Change-Id: Ie245d2c19cc7428a06295635cf6a9482ade25ff0
Reviewed-on: https://go-review.googlesource.com/57293
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-08-26 15:09:09 +00:00
Ian Lance Taylor
6126384f6d runtime: unify sigTabT type across Unix systems
Change-Id: I8e8a3a118b1216f191c9076b70a88f6f3f19f79f
Reviewed-on: https://go-review.googlesource.com/59150
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-25 21:42:33 +00:00
Wei Xiao
c02fc1605a cmd/compile: memory clearing optimization for arm64
Use "STP (ZR, ZR), O(R)" instead of "MOVD ZR, O(R)" to implement memory clearing.
Also improve assembler supports to STP/LDP.
Results (A57@2GHzx8):

benchmark                   old ns/op     new ns/op     delta
BenchmarkClearFat8-8        1.00          1.00          +0.00%
BenchmarkClearFat12-8       1.01          1.01          +0.00%
BenchmarkClearFat16-8       1.01          1.01          +0.00%
BenchmarkClearFat24-8       1.52          1.52          +0.00%
BenchmarkClearFat32-8       3.00          2.02          -32.67%
BenchmarkClearFat40-8       3.50          2.52          -28.00%
BenchmarkClearFat48-8       3.50          3.03          -13.43%
BenchmarkClearFat56-8       4.00          3.50          -12.50%
BenchmarkClearFat64-8       4.25          4.00          -5.88%
BenchmarkClearFat128-8      8.01          8.01          +0.00%
BenchmarkClearFat256-8      16.1          16.0          -0.62%
BenchmarkClearFat512-8      32.1          32.0          -0.31%
BenchmarkClearFat1024-8     64.1          64.1          +0.00%

Change-Id: Ie5f5eac271ff685884775005825f206167a5c146
Reviewed-on: https://go-review.googlesource.com/55610
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2017-08-25 20:09:06 +00:00
Josh Bleecher Snyder
25bbb695c8 runtime: optimize storing new keys in mapassign_fastNN
Prior to this change, we use typedmemmove to write the key
value to its new location in mapassign_fast32 and mapassign_fast64.
(The use of typedmemmove was a last-minute fix in the 1.9 cycle;
see #21297 and CL 53414.)

This is significantly less inefficient than direct assignment or
calling writebarrierptr directly.

Fortunately, there aren't many cases to consider.

On systems with 32 bit pointers:

* A 32 bit AMEM value either is a single pointer or has no pointers.
* A 64 bit AMEM value may contain a pointer at the beginning,
  a pointer at 32 bits, or two pointers.

On systems with 64 bit pointers:

* A 32 bit AMEM value contains no pointers.
* A 64 bit AMEM value either is a single pointer or has no pointers.

All combinations except the 32 bit pointers / 64 bit AMEM value are
cheap and easy to handle, and the problematic case is likely rare.
The most popular map keys appear to be ints and pointers.

So we handle them exhaustively. The sys.PtrSize checks are constant branches
and are eliminated by the compiler.

An alternative fix would be to return a pointer to the key,
and have the calling code do the assignment, at which point the compiler
would have full type information.

Initial tests suggest that the performance difference between these
strategies is negligible, and this fix is considerably simpler,
and has much less impact on binary size.

Fixes #21321

Change-Id: Ib03200e89e2324dd3c76d041131447df66f22bfe
Reviewed-on: https://go-review.googlesource.com/59110
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-08-25 19:42:05 +00:00
Austin Clements
9d17e175e0 runtime: capture runtimeInitTime after nanotime is initialized
CL 36428 changed the way nanotime works so on Darwin and Windows it
now depends on runtime.startNano, which is computed at runtime.init
time. Unfortunately, the `runtimeInitTime = nanotime()` initialization
happened *before* runtime.init, so on these platforms runtimeInitTime
is set incorrectly. The one (and only) consequence of this is that the
start time printed in gctrace lines is bogus:

gc 1 18446653480.186s 0%: 0.092+0.47+0.038 ms clock, 0.37+0.15/0.81/1.8+0.15 ms cpu, 4->4->1 MB, 5 MB goal, 8 P

To fix this, this commit moves the runtimeInitTime initialization to
shortly after runtime.init, at which point nanotime is safe to use.

This also requires changing the condition in newproc1 that currently
uses runtimeInitTime != 0 simply to detect whether or not the main M
has started. Since runtimeInitTime could genuinely be 0 now, this
introduces a separate flag to newproc1.

Fixes #21554.

Change-Id: Id874a4b912d3fa3d22f58d01b31ffb3548266d3b
Reviewed-on: https://go-review.googlesource.com/58690
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rick Hudson <rlh@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-08-25 16:02:17 +00:00
Josh Bleecher Snyder
0b0cc4154e runtime: refactor walking of bucket overflows
This eliminates a nil check of b while evaluating b.tophash,
which is in the inner loop of many hot map functions.
It also makes the code a bit clearer.

Also remove some gotos in favor of labeled breaks.

On non-x86 architectures, this change introduces a pointless reg-reg move,
although the cause is well-understood (#21572).

Change-Id: Ib7ee58b59ea5463b92e1590c8b8f5c0ef87d410a
Reviewed-on: https://go-review.googlesource.com/58372
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-24 17:17:49 +00:00
Michael Munday
744ebfde04 cmd/compile: eliminate stores to unread auto variables
This is a crude compiler pass to eliminate stores to auto variables
that are only ever written to.

Eliminates an unnecessary store to x from the following code:

func f() int {
	var x := 1
	return *(&x)
}

Fixes #19765.

Change-Id: If2c63a8ae67b8c590b6e0cc98a9610939a3eeffa
Reviewed-on: https://go-review.googlesource.com/38746
Run-TryBot: Michael Munday <mike.munday@ibm.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-24 16:53:56 +00:00
Josh Bleecher Snyder
83ae9b07b8 runtime: convert more unsafe.Pointer arithmetic to add
Change-Id: Icfe24d5660666093f3e645f82d30b7687c8077be
Reviewed-on: https://go-review.googlesource.com/58370
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
2017-08-24 13:55:34 +00:00
Martin Möhrmann
f3e0d14313 runtime: avoid infinite loop in growslice
On 386 the below code triggered an infinite loop in growslice:
x = make([]byte, 1<<30-1, 1<<30-1)
x = append(x, x...)

Check for overflow when calculating the new slice capacity
and set the new capacity to the requested capacity when an overflow
is detected to avoid an infinite loop.

No automatic test added due to requiring to allocate 1GB of memory
on a 32bit plaform before use of append is able to trigger the
overflow check.

Fixes #21441

Change-Id: Ia871cc9f88479dacf2c7044531b233f83d2fcedf
Reviewed-on: https://go-review.googlesource.com/57950
Run-TryBot: Martin Möhrmann <moehrmann@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Marvin Stenger <marvin.stenger94@gmail.com>
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-24 12:11:14 +00:00
Josh Bleecher Snyder
f7aa454c58 runtime: mask shifts in map implementation on x86
This slightly improves the generated code on x86 architectures,
including on many hot paths.

It is a no-op on other architectures.

Change-Id: I86336fd846bc5805a27bbec572e8c73dcbd0d567
Reviewed-on: https://go-review.googlesource.com/57411
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-24 05:38:57 +00:00
Josh Bleecher Snyder
5df1fe52fe runtime: always rebuild in TestIntendedInlining
This is necessary when you aren't actively changing the runtime. Oops.

Also, run the tests on the builders, to avoid silent failures (#17472).

Change-Id: I1fc03790cdbddddb07026a772137a79919dcaac7
Reviewed-on: https://go-review.googlesource.com/58050
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-08-24 05:07:40 +00:00
Josh Bleecher Snyder
0c7fd56951 runtime: only clear pointer-containing memory during map delete
When deleting entries from a map, only clear the key and value
if they contain pointers. And use memclrHasPointers to do so.

While we're here, specialize key clearing in mapdelete_faststr,
and fix another missed usage of add in mapdelete.

Benchmarking impeded by #21546.

Change-Id: I3f6f924f738d6b899b722d6438e9e63f52359b84
Reviewed-on: https://go-review.googlesource.com/57630
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-23 13:53:00 +00:00
Josh Bleecher Snyder
29e6bdc69c runtime: strength reduce key pointer calculation in mapdelete_fast*
Move the tophash checks after the equality/length checks.

For fast32/fast64, since we've done a full equality check already,
just check whether tophash is empty instead of checking tophash.
This is cheaper and allows us to skip calculating tophash.

These changes are modeled on the changes in CL 57590,
which were polished based on benchmarking.
Benchmarking directly is impeded by #21546.

Change-Id: I0e17163028e34720310d1bf8f95c5ef42d223e00
Reviewed-on: https://go-review.googlesource.com/57611
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-23 13:49:52 +00:00
Josh Bleecher Snyder
61043d4671 runtime: use add in mapdelete*
This better matches the style of the rest of the runtime.

Change-Id: I6abb755df50eb3d9086678629c0d184177e1981f
Reviewed-on: https://go-review.googlesource.com/57610
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
2017-08-23 13:49:29 +00:00
Josh Bleecher Snyder
dad5d76e8f runtime: strength reduce key pointer calculations in mapaccess*_fast*
While we're here, check string length before checking b.tophash.

name                     old time/op  new time/op  delta
MapStringKeysEight_16-8  11.4ns ±10%   7.0ns ± 2%  -38.27%  (p=0.000 n=29+28)
MapStringKeysEight_32-8  10.9ns ± 2%   6.3ns ± 3%  -41.89%  (p=0.000 n=26+30)
MapStringKeysEight_64-8  10.8ns ± 3%   6.3ns ± 2%  -41.52%  (p=0.000 n=28+27)
MapStringKeysEight_1M-8  10.9ns ± 4%   6.3ns ± 2%  -41.91%  (p=0.000 n=29+29)
IntMap-8                 7.05ns ± 4%  6.77ns ± 3%   -3.94%  (p=0.000 n=29+30)


Change-Id: I0f3dc3301bdf550e4ac5250e1e64e7f2a0ffb269
Reviewed-on: https://go-review.googlesource.com/57590
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-23 13:47:14 +00:00
Martin Möhrmann
15d5fa0291 runtime: fix makemap64 function signature
During rebase of golang.org/cl/55152 the bucket argument
which was removed in golang.org/cl/56290 from makemap
was not removed from the argument list of makemap64.

This did lead to "pointer in unallocated span" errors
on 32bit platforms since the compiler did only generate
calls to makemap64 without the bucket argument.

Fixes #21568

Change-Id: Ia964a3c285837cd901297f4e16e40402148f8c1c
Reviewed-on: https://go-review.googlesource.com/57990
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-08-23 12:27:22 +00:00
Josh Bleecher Snyder
f6944c780f runtime: add TestIntendedInlining
The intent is to allow more aggressive refactoring
in the runtime without silent performance changes.

The test would be useful for many functions.
I've seeded it with the runtime functions tophash and add;
it will grow organically (or wither!) from here.

Updates #21536 and #17566

Change-Id: Ib26d9cfd395e7a8844150224da0856add7bedc42
Reviewed-on: https://go-review.googlesource.com/57410
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-08-22 23:07:40 +00:00
Martin Möhrmann
cbc4e5d9c4 cmd/compile: generate makemap calls with int arguments
Where possible generate calls to runtime makemap with int hint argument
during compile time instead of makemap with int64 hint argument.

This eliminates converting the hint argument for calls to makemap with
int64 hint argument for platforms where int64 values do not fit into
an argument of type int.

A similar optimization for makeslice was introduced in CL
golang.org/cl/27851.

386:
name         old time/op    new time/op    delta
NewEmptyMap    53.5ns ± 5%    41.9ns ± 5%  -21.56%  (p=0.000 n=10+10)
NewSmallMap     182ns ± 1%     165ns ± 1%   -8.92%  (p=0.000 n=10+10)

Change-Id: Ibd2b4c57b36f171b173bf7a0602b3a59771e6e44
Reviewed-on: https://go-review.googlesource.com/55142
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-22 20:28:21 +00:00
Martin Möhrmann
3216e0cefa cmd/compile: replace eqstring with memequal
eqstring is only called for strings with equal lengths.
Instead of pushing a pointer and length for each argument string
on the stack we can omit pushing one of the lengths on the stack.

Changing eqstrings signature to eqstring(*uint8, *uint8, int) bool
to implement the above optimization would make it very similar to the
existing memequal(*any, *any, uintptr) bool function.

Since string lengths are positive we can avoid code redundancy and
use memequal instead of using eqstring with an optimized signature.

go command binary size reduced by 4128 bytes on amd64.

name                          old time/op    new time/op    delta
CompareStringEqual              6.03ns ± 1%    5.71ns ± 1%   -5.23%  (p=0.000 n=19+18)
CompareStringIdentical          2.88ns ± 1%    3.22ns ± 7%  +11.86%  (p=0.000 n=20+20)
CompareStringSameLength         4.31ns ± 1%    4.01ns ± 1%   -7.17%  (p=0.000 n=19+19)
CompareStringDifferentLength    0.29ns ± 2%    0.29ns ± 2%     ~     (p=1.000 n=20+20)
CompareStringBigUnaligned       64.3µs ± 2%    64.1µs ± 3%     ~     (p=0.164 n=20+19)
CompareStringBig                61.9µs ± 1%    61.6µs ± 2%   -0.46%  (p=0.033 n=20+19)

Change-Id: Ice15f3b937c981f0d3bc8479a9ea0d10658ac8df
Reviewed-on: https://go-review.googlesource.com/53650
Run-TryBot: Martin Möhrmann <moehrmann@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-22 17:59:02 +00:00
Martin Möhrmann
06a78b5737 cmd/compile: pass stack allocated bucket to makemap inside hmap
name         old time/op    new time/op    delta
NewEmptyMap    53.2ns ± 7%    48.0ns ± 5%  -9.77%  (p=0.000 n=20+20)
NewSmallMap     111ns ± 1%     106ns ± 2%  -3.78%  (p=0.000 n=20+19)

Change-Id: I979d21ab16eae9f6893873becca517db57e054b5
Reviewed-on: https://go-review.googlesource.com/56290
Run-TryBot: Martin Möhrmann <moehrmann@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-08-22 06:01:59 +00:00
Josh Bleecher Snyder
3f972df4a7 runtime: don't clear pointer-free memory when growing maps
If there are no pointers, then clearing memory doesn't help GC,
and the memory is otherwise dead, so don't bother clearing it.

Change-Id: I953f4a3264939f2825e82292030eda2e835cbb97
Reviewed-on: https://go-review.googlesource.com/57350
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
2017-08-20 15:46:43 +00:00
Austin Clements
57584a0ee1 runtime: fix false positive race in profile label reading
Because profile labels are copied from the goroutine into the tag
buffer by the signal handler, there's a carefully-crafted set of race
detector annotations to create the necessary happens-before edges
between setting a goroutine's profile label and retrieving it from the
profile tag buffer.

Given the constraints of the signal handler, we have to approximate
the true synchronization behavior. Currently, that approximation is
too weak.

Ideally, runtime_setProfLabel would perform a store-release on
&getg().labels and copying each label into the profile would perform a
load-acquire on &getg().labels. This would create the necessary
happens-before edges through each individual g.labels object.

Since we can't do this in the signal handler, we instead synchronize
on a "labelSync" global. The problem occurs with the following
sequence:

1. Goroutine 1 calls setProfLabel, which does a store-release on
   labelSync.

2. Goroutine 2 calls setProfLabel, which does a store-release on
   labelSync.

3. Goroutine 3 reads the profile, which does a load-acquire on
   labelSync.

The problem is that the load-acquire only synchronizes with the *most
recent* store-release to labelSync, and the two store-releases don't
synchronize with each other. So, once goroutine 3 touches the label
set by goroutine 1, we report a race.

The solution is to use racereleasemerge. This is like a
read-modify-write, rather than just a store-release. Each RMW of
labelSync in runtime_setProfLabel synchronizes with the previous RMW
of labelSync, and this ultimately carries forward to the load-acquire,
so it synchronizes with *all* setProfLabel operations, not just the
most recent.

Change-Id: Iab58329b156122002fff12cfe64fbeacb31c9613
Reviewed-on: https://go-review.googlesource.com/56670
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
2017-08-18 21:40:37 +00:00
Josh Bleecher Snyder
776c33ee5e runtime: make evacDst a top level type
This will reduce duplication when evacuate is specialized.

Change-Id: I34cdfb7103442d3e0ea908c970fb46334b86d5c4
Reviewed-on: https://go-review.googlesource.com/56934
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Avelino <t@avelino.xxx>
Reviewed-by: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-08-18 18:21:10 +00:00
Josh Bleecher Snyder
e0b34e7be7 runtime: split advanceEvacuationMark from evacuate
Minor refactoring. This is a step towards specializing evacuate
for mapfast key types.

Change-Id: Icffe2759b7d38e5c008d03941918d5a912ce62f6
Reviewed-on: https://go-review.googlesource.com/56933
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-18 18:20:55 +00:00
Josh Bleecher Snyder
43d4c9f4f1 runtime: tiny refactor in evacuate
Since oldbucket == h.nevacuate, we can just increment h.nevacuate here.
This removes oldbucket from scope, which will be useful shortly.

Change-Id: I70f81ec3995f17845ebf5d77ccd20ea4338f23e6
Reviewed-on: https://go-review.googlesource.com/56932
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Avelino <t@avelino.xxx>
Reviewed-by: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-08-18 18:20:32 +00:00
Josh Bleecher Snyder
b5c4211159 runtime: don't cache t.key.alg in evacuate
The number of times that alg has to be spilled
and restored makes it better to just reload it.

Change-Id: I2674752a889ecad59dab54da1d68fad03db1ca85
Reviewed-on: https://go-review.googlesource.com/56931
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-08-18 18:20:10 +00:00
Josh Bleecher Snyder
8a9d4184e6 runtime: simplify evacuate's handling of NaNs
The new code is not quite equivalent to the old,
in that if newbit was very large it might have altered the new tophash.
The old behavior is unnecessary and probably undesirable.

Change-Id: I7fb3222520cb61081a857adcddfbb9078ead7122
Reviewed-on: https://go-review.googlesource.com/56930
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-18 18:19:47 +00:00
Keith Randall
77871cc664 runtime: no need to protect key/value increments against end of bucket
After the key and value arrays, we have an overflow pointer.
So there's no way a past-the-end key or value pointer could point
past the end of the containing bucket.

So we don't need this additional protection.

Update #21459

Change-Id: I7726140033b06b187f7a7d566b3af8cdcaeab0b0
Reviewed-on: https://go-review.googlesource.com/56772
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Avelino <t@avelino.xxx>
2017-08-18 14:52:25 +00:00
pvoicu
310be7be5c runtime: fix usleep by correctly setting nanoseconds parameter for pselect6
Fixes #21518

Change-Id: Idd67e3f0410d0ce991b34dcc0c8f15e0d5c529c9
Reviewed-on: https://go-review.googlesource.com/56850
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-08-18 13:35:47 +00:00
Daniel Martí
59413d34c9 all: unindent some big chunks of code
Found with mvdan.cc/unindent. Prioritized the ones with the biggest wins
for now.

Change-Id: I2b032e45cdd559fc9ed5b1ee4c4de42c4c92e07b
Reviewed-on: https://go-review.googlesource.com/56470
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-08-18 06:59:48 +00:00
Martin Möhrmann
455775dae6 runtime: improve makechan memory checks and allocation calls
Use mallogc instead of newarray to save some overhead since
makechan already checks for _MaxMem constraints.

Flattens the if else construct that determines if buf and hchan struct
should be allocated in one mallocgc call and where buf should point to.

Uses maxSliceCap to avoid divisions similar to makeslice.

name                old time/op  new time/op  delta
MakeChan/Byte       82.0ns ± 8%  81.4ns ± 7%    ~     (p=0.643 n=10+10)
MakeChan/Int        97.9ns ± 2%  96.6ns ± 2%  -1.40%  (p=0.009 n=10+10)
MakeChan/Ptr         128ns ± 3%   120ns ± 1%  -6.63%  (p=0.000 n=10+10)
MakeChan/Struct/0   66.7ns ± 4%  66.4ns ± 2%    ~     (p=0.697 n=10+10)
MakeChan/Struct/32   136ns ± 1%   130ns ± 0%  -4.42%  (p=0.000 n=10+10)
MakeChan/Struct/40   150ns ± 1%   150ns ± 1%    ~     (p=0.725 n=10+10)

Change-Id: Ibb5675d0843a072aae2bfa58ecd39cf4cd926533
Reviewed-on: https://go-review.googlesource.com/55132
Run-TryBot: Martin Möhrmann <moehrmann@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-17 20:24:15 +00:00
Martin Möhrmann
b6296426a0 runtime: avoid zeroing hmap fields in makemap twice
Stack allocated hmap structs are explicitly zeroed before being
passed by pointer to makemap.

Heap allocated hmap structs are created with newobject
which also zeroes on allocation.

Therefore, setting the hmap fields to 0 or nil is redundant
since they will have been zeroed when hmap was allocated.

Change-Id: I5fc55b75e9dc5ba69f5e3588d6c746f53b45ba66
Reviewed-on: https://go-review.googlesource.com/56291
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-17 20:10:23 +00:00
Bryan C. Mills
e0545faf27 runtime/cgo: defeat inlining in x_cgo_yield
We use a call to strncpy to work around a TSAN bug (wherein TSAN only
delivers asynchronous signals when the thread receiving the signal
calls a libc function). Unfortunately, GCC 7 inlines the call,
avoiding the TSAN libc trap entirely.

Per Ian's suggestion, use global variables as strncpy arguments: that
way, the compiler can't make any assumptions about the concrete values
and can't inline the call away.

fixes #21196

Change-Id: Ie95f1feaf9af1a8056f924f49c29cfc8515385d7
Reviewed-on: https://go-review.googlesource.com/55872
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-08-16 18:49:38 +00:00
Ilya Tocar
df70982825 cmd/compile/internal/ssa: use sse to zero on amd64
Use 16-byte stores instead of 8-byte stores to zero small blocks.
Also switch to duffzero for 65+ bytes only, because for each
duffzero call we also save/restore BP, so call requires 4 instructions
and replacing it with 4 sse stores doesn't cause code-bloat.
Also switch duffzero to use leaq, instead of addq to avoid clobbering flags.

ClearFat8-6     0.54ns ± 0%  0.54ns ± 0%     ~     (all equal)
ClearFat12-6    1.07ns ± 0%  1.07ns ± 0%     ~     (all equal)
ClearFat16-6    1.07ns ± 0%  0.69ns ± 0%  -35.51%  (p=0.001 n=8+9)
ClearFat24-6    1.61ns ± 1%  1.07ns ± 0%  -33.33%  (p=0.000 n=10+10)
ClearFat32-6    2.14ns ± 0%  1.07ns ± 0%  -50.00%  (p=0.001 n=8+9)
ClearFat40-6    2.67ns ± 1%  1.61ns ± 0%  -39.72%  (p=0.000 n=10+8)
ClearFat48-6    3.75ns ± 0%  2.68ns ± 0%  -28.59%  (p=0.000 n=9+9)
ClearFat56-6    4.29ns ± 0%  3.22ns ± 0%  -25.10%  (p=0.000 n=9+9)
ClearFat64-6    4.30ns ± 0%  3.22ns ± 0%  -25.15%  (p=0.000 n=8+8)
ClearFat128-6   7.50ns ± 1%  7.51ns ± 0%     ~     (p=0.767 n=10+9)
ClearFat256-6   13.9ns ± 1%  13.9ns ± 1%     ~     (p=0.257 n=10+10)
ClearFat512-6   26.8ns ± 0%  26.8ns ± 0%     ~     (p=0.467 n=8+8)
ClearFat1024-6  52.5ns ± 0%  52.5ns ± 0%     ~     (p=1.000 n=8+8)

Also shaves ~20kb from go tool:

go_old 10384994
go_new 10364514 [-20480 bytes]

section differences
global text (code) = -20585 bytes (-0.532047%)
read-only data = -302 bytes (-0.018101%)
Total difference -20887 bytes (-0.348731%)

Change-Id: I15854e87544545c1af24775df895e38e16e12694
Reviewed-on: https://go-review.googlesource.com/54410
Run-TryBot: Ilya Tocar <ilya.tocar@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-16 15:52:27 +00:00
Ian Lance Taylor
a1371756c3 testing: don't fail all tests after racy test failure
The code was adding race.Errors to t.raceErrors before checking
Failed, but Failed was using t.raceErrors+race.Errors. We don't want
to change Failed, since that would affect tests themselves, so modify
the harness to not unnecessarily change t.raceErrors.

Updates #19851
Fixes #21338
Change-Id: I7bfdf281f90e045146c92444f1370d55c45221d4
Reviewed-on: https://go-review.googlesource.com/54050
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-08-15 22:59:26 +00:00
Josh Bleecher Snyder
9de7990054 cmd/compile: insert ' ' and \n at beginning of walkprint
Rather than emitting spaces and newlines for println
as we walk the expression, construct it all up front.
This enables further optimizations.

This requires using printstring instead of print in
the implementation of printsp and printnl,
on pain of infinite recursion.
That's ok; it's more efficient anyway, and just as simple.
While we're here, do it for other print routines as well.

Change-Id: I61d7df143810e00710c4d4d948d904007a7fd190
Reviewed-on: https://go-review.googlesource.com/55097
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2017-08-15 21:54:30 +00:00
Daniel Morsing
32b94f13cf runtime: move selectdone into g
Writing to selectdone on the stack of another goroutine meant a
pretty subtle dance between the select code and the stack copying
code. Instead move the selectdone variable into the g struct.

Change-Id: Id246aaf18077c625adef7ca2d62794afef1bdd1b
Reviewed-on: https://go-review.googlesource.com/53390
Reviewed-by: Austin Clements <austin@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-08-15 19:18:00 +00:00
Keith Randall
89d74f5416 cmd/compile: set itab function pointers at compile time
I noticed that we don't set an itab's function pointers at compile
time. Instead, we currently do it at executable startup.

Set the function pointers at compile time instead. This shortens
startup time. It has no effect on normal binary size. Object files
will have more relocations, but that isn't a big deal.

For PIE there are additional pointers that will need to be adjusted at
load time. There are already other pointers in an itab that need to be
adjusted, so the cache line will already be paged in. There might be
some binary size overhead to mark these pointers. The "go test -c
-buildmode=pie net/http" binary is 0.18% bigger.

Update #20505

Change-Id: I267c82489915b509ff66e512fc7319b2dd79b8f7
Reviewed-on: https://go-review.googlesource.com/44341
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
2017-08-15 19:00:03 +00:00
Austin Clements
250a9610a4 runtime: make STW duration more accurate
Currently, GC captures the start-the-world time stamp after
startTheWorldWithSema returns. This is problematic for two reasons:

1. It's possible to get preempted between startTheWorldWithSema
starting the world and calling nanotime.

2. startTheWorldWithSema does several clean-up tasks after the world
is up and running that on rare occasions can take upwards of 10ms.

Since the runtime uses the start-the-world time stamp to compute the
STW duration, both of these can significantly inflate the reported STW
duration.

Fix this by having startTheWorldWithSema itself call nanotime once the
world is started.

Change-Id: I114630234fb73c9dabae50a2ef1884661f2459db
Reviewed-on: https://go-review.googlesource.com/55410
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rick Hudson <rlh@golang.org>
2017-08-15 18:47:08 +00:00
Austin Clements
03929984b9 runtime: fix getclosureptr doc
Change-Id: I1b42fca2107b06e6fc95728f7bf3d08d005c4cb4
Reviewed-on: https://go-review.googlesource.com/55810
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-08-15 17:59:16 +00:00
Muhammad Falak R Wani
58cb8a3c8f runtime: remove redeclared structs to make tests build
struct32 and struct40 structs are already declared, remove them to
make runtime tests build.

Change-Id: I3814f2b850dcb15c4002a3aa22e2a9326e5a5e53
Reviewed-on: https://go-review.googlesource.com/55614
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
2017-08-15 07:19:25 +00:00
Martin Möhrmann
8a6e51aede cmd/compile: generate makechan calls with int arguments
Where possible generate calls to runtime makechan with int arguments
during compile time instead of makechan with int64 arguments.

This eliminates converting arguments for calls to makechan with
int64 arguments for platforms where int64 values do not fit into
arguments of type int.

A similar optimization for makeslice was introduced in CL
golang.org/cl/27851.

386:
name                old time/op  new time/op  delta
MakeChan/Byte       52.4ns ± 6%  45.0ns ± 1%  -14.14%  (p=0.000 n=10+10)
MakeChan/Int        54.5ns ± 1%  49.1ns ± 1%   -9.87%  (p=0.000 n=10+10)
MakeChan/Ptr         150ns ± 1%   143ns ± 0%   -4.38%  (p=0.000 n=9+7)
MakeChan/Struct/0   49.2ns ± 2%  43.2ns ± 2%  -12.27%  (p=0.000 n=10+10)
MakeChan/Struct/32  81.7ns ± 2%  76.2ns ± 1%   -6.71%  (p=0.000 n=10+10)
MakeChan/Struct/40  88.4ns ± 2%  82.5ns ± 2%   -6.60%  (p=0.000 n=10+10)

AMD64:
name                old time/op  new time/op  delta
MakeChan/Byte       83.4ns ± 8%  80.8ns ± 3%    ~     (p=0.171 n=10+10)
MakeChan/Int         101ns ± 3%   101ns ± 2%    ~     (p=0.412 n=10+10)
MakeChan/Ptr         128ns ± 1%   128ns ± 1%    ~     (p=0.191 n=10+10)
MakeChan/Struct/0   67.6ns ± 3%  68.7ns ± 4%    ~     (p=0.224 n=10+10)
MakeChan/Struct/32   138ns ± 1%   139ns ± 1%    ~     (p=0.185 n=10+9)
MakeChan/Struct/40   154ns ± 1%   154ns ± 1%  -0.55%  (p=0.027 n=10+9)

Change-Id: Ie854cb066007232c5e9f71ea7d6fe27e81a9c050
Reviewed-on: https://go-review.googlesource.com/55140
Run-TryBot: Martin Möhrmann <moehrmann@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-15 05:54:24 +00:00
Martin Möhrmann
365594ad59 runtime: simplify memory capacity check in growslice
Instead of comparing if the number of elements will
not fit into memory check if the memory size of the
slices backing memory is higher then the memory limit.

This avoids a division or maxElems lookup.

With et.size > 0:
   uintptr(newcap)                > maxSliceCap(et.size)
-> uintptr(int(capmem / et.size)) > _MaxMem  /  et.size
->             capmem / et.size   > _MaxMem  /  et.size
->             capmem             > _MaxMem

Note that due to integer division from capmem > _MaxMem
it does not follow that uintptr(newcap) > maxSliceCap(et.size).

Consolidated runtime GrowSlice benchmarks by using sub-benchmarks and
added more struct sizes to show performance improvement when division
is avoided for element sizes larger than 32 bytes.

AMD64:
GrowSlice/Byte       38.9ns ± 2%  38.9ns ± 1%    ~     (p=0.974 n=20+20)
GrowSlice/Int        58.3ns ± 3%  58.0ns ± 2%    ~     (p=0.154 n=20+19)
GrowSlice/Ptr        95.7ns ± 2%  95.1ns ± 2%  -0.60%  (p=0.034 n=20+20)
GrowSlice/Struct/24  95.4ns ± 1%  93.9ns ± 1%  -1.54%  (p=0.000 n=19+19)
GrowSlice/Struct/32   110ns ± 1%   108ns ± 1%  -1.76%  (p=0.000 n=19+20)
GrowSlice/Struct/40   138ns ± 1%   128ns ± 1%  -7.09%  (p=0.000 n=20+20)

Change-Id: I1c37857c74ea809da373e668791caffb6a5cbbd3
Reviewed-on: https://go-review.googlesource.com/53471
Run-TryBot: Martin Möhrmann <moehrmann@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
2017-08-15 04:23:11 +00:00
Keith Randall
fcf445dce2 runtime: initialize itab.hash always
We weren't initializing this field for dynamically-generated itabs.
Turns out it doesn't matter, as any time we use this field we also
generate a static itab for the interface type / concrete type pair.
But we should initialize it anyway, just to be safe.

Performance on the benchmarks in CL 44339:
benchmark               old ns/op     new ns/op     delta
BenchmarkItabFew-12     1040585       26466         -97.46%
BenchmarkItabAll-12     228873499     4287696       -98.13%

Change-Id: I58ed2b31e6c98b584122bdaf844fee7268b58295
Reviewed-on: https://go-review.googlesource.com/44475
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-08-15 01:52:40 +00:00
Keith Randall
04d6f982ae runtime: remove link field from itab
We don't use it any more, remove it.

Change-Id: I76ce1a4c2e7048fdd13a37d3718b5abf39ed9d26
Reviewed-on: https://go-review.googlesource.com/44474
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2017-08-15 01:52:35 +00:00