1
0
mirror of https://github.com/golang/go synced 2024-10-04 19:31:21 -06:00
Commit Graph

61 Commits

Author SHA1 Message Date
Shenghou Ma
35e84546d7 runtime: check and fail early with a message if MMX is not available on 386
Fixes #12970.

Change-Id: Id0026e8274e071d65d47df63d65a93110abbec5d
Reviewed-on: https://go-review.googlesource.com/15998
Reviewed-by: Keith Randall <khr@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2015-12-05 17:43:51 +00:00
Shenghou Ma
3583a44ed2 runtime: check that masks and shifts are correct aligned
We need a runtime check because the original issue is encountered
when running cross compiled windows program from linux. It's better
to give a meaningful crash message earlier than to segfault later.

The added test should not impose any measurable overhead to Go
programs.

For #12415.

Change-Id: Ib4a24ef560c09c0585b351d62eefd157b6b7f04c
Reviewed-on: https://go-review.googlesource.com/14207
Reviewed-by: Keith Randall <khr@golang.org>
Run-TryBot: Minux Ma <minux@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2015-11-25 00:46:57 +00:00
Michael Hudson-Doyle
09d7de8d61 cmd/link, runtime: call addmoduledata when dynamically linking on linux/386
Change-Id: If1faa2bba28a4e9a8061693173797c4114a7d699
Reviewed-on: https://go-review.googlesource.com/16387
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2015-11-19 00:22:56 +00:00
Michael Hudson-Doyle
6056cc5df6 runtime: handle volatility of CX when dynamically linking on linux/386
Mostly by avoiding CX entirely, sometimes by reloading it.

I also vetted the assembly in other packages, it's all fine.

Change-Id: I50059669aaaa04efa303cf22ac228f9d14d83db0
Reviewed-on: https://go-review.googlesource.com/16386
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2015-11-18 01:36:44 +00:00
Matthew Dempsky
7bb38f6e47 runtime: replace tls0 with m0.tls
We're allocating TLS storage for m0 anyway, so might as well use it.

Change-Id: I7dc20bbea5320c8ab8a367f18a9540706751e771
Reviewed-on: https://go-review.googlesource.com/16890
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Michael Hudson-Doyle <michael.hudson@canonical.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2015-11-13 01:53:00 +00:00
Shenghou Ma
0adf6dce8a runtime: disable prefetching on 386
It doesn't seem to help on modern processors and it makes Go impossible to run
on Pentium MMX (which is the documented minimum hardware requirement.)

Old is with prefetch, new is w/o. Both are compiled with GO386=sse2.
Benchmarking is done on Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz.

name                     old time/op    new time/op    delta
BinaryTree17-4              2.89s ± 2%     2.87s ± 0%    ~     (p=0.061 n=11+10)
Fannkuch11-4                3.65s ± 0%     3.65s ± 0%    ~     (p=0.365 n=11+11)
FmtFprintfEmpty-4          52.1ns ± 0%    52.1ns ± 0%    ~      (p=0.065 n=10+9)
FmtFprintfString-4          168ns ± 0%     167ns ± 0%  -0.48%   (p=0.000 n=8+10)
FmtFprintfInt-4             167ns ± 0%     167ns ± 1%    ~      (p=0.591 n=9+10)
FmtFprintfIntInt-4          295ns ± 0%     292ns ± 0%  -0.99%   (p=0.000 n=9+10)
FmtFprintfPrefixedInt-4     327ns ± 0%     326ns ± 0%  -0.24%  (p=0.007 n=10+10)
FmtFprintfFloat-4           431ns ± 0%     431ns ± 0%  -0.07%  (p=0.000 n=10+11)
FmtManyArgs-4              1.13µs ± 0%    1.13µs ± 0%  -0.37%  (p=0.009 n=11+11)
GobDecode-4                9.36ms ± 1%    9.33ms ± 0%  -0.31%  (p=0.006 n=11+10)
GobEncode-4                7.38ms ± 1%    7.38ms ± 1%    ~     (p=0.797 n=11+11)
Gzip-4                      394ms ± 0%     395ms ± 1%    ~     (p=0.519 n=11+11)
Gunzip-4                   65.4ms ± 0%    65.4ms ± 0%    ~     (p=0.739 n=10+10)
HTTPClientServer-4         52.4µs ± 1%    52.5µs ± 1%    ~     (p=0.748 n=11+11)
JSONEncode-4               19.0ms ± 0%    19.0ms ± 0%    ~      (p=0.780 n=9+10)
JSONDecode-4               59.6ms ± 0%    59.6ms ± 0%    ~      (p=0.720 n=9+10)
Mandelbrot200-4            4.09ms ± 0%    4.09ms ± 0%    ~      (p=0.295 n=11+9)
GoParse-4                  3.45ms ± 1%    3.43ms ± 1%  -0.35%  (p=0.040 n=11+11)
RegexpMatchEasy0_32-4       101ns ± 1%     101ns ± 1%    ~     (p=1.000 n=11+11)
RegexpMatchEasy0_1K-4       796ns ± 0%     796ns ± 0%    ~      (p=0.954 n=10+8)
RegexpMatchEasy1_32-4       110ns ± 0%     110ns ± 1%    ~      (p=0.289 n=9+11)
RegexpMatchEasy1_1K-4       991ns ± 0%     991ns ± 0%    ~      (p=0.784 n=10+8)
RegexpMatchMedium_32-4      131ns ± 0%     130ns ± 0%  -0.42%   (p=0.004 n=11+9)
RegexpMatchMedium_1K-4     41.9µs ± 1%    41.6µs ± 0%    ~      (p=0.067 n=11+9)
RegexpMatchHard_32-4       2.34µs ± 0%    2.34µs ± 0%    ~     (p=0.208 n=11+11)
RegexpMatchHard_1K-4       70.9µs ± 0%    71.0µs ± 0%    ~      (p=0.968 n=9+10)
Revcomp-4                   819ms ± 0%     818ms ± 0%    ~     (p=0.251 n=10+11)
Template-4                 73.9ms ± 0%    73.8ms ± 0%  -0.25%  (p=0.013 n=10+11)
TimeParse-4                 414ns ± 0%     414ns ± 0%    ~     (p=0.809 n=11+10)
TimeFormat-4                485ns ± 0%     485ns ± 0%    ~      (p=0.404 n=11+7)

name                     old speed      new speed      delta
GobDecode-4              82.0MB/s ± 1%  82.3MB/s ± 0%  +0.31%  (p=0.007 n=11+10)
GobEncode-4               104MB/s ± 1%   104MB/s ± 1%    ~     (p=0.797 n=11+11)
Gzip-4                   49.2MB/s ± 0%  49.1MB/s ± 1%    ~     (p=0.507 n=11+11)
Gunzip-4                  297MB/s ± 0%   297MB/s ± 0%    ~     (p=0.670 n=10+10)
JSONEncode-4              102MB/s ± 0%   102MB/s ± 0%    ~      (p=0.794 n=9+10)
JSONDecode-4             32.6MB/s ± 0%  32.6MB/s ± 0%    ~       (p=0.334 n=9+9)
GoParse-4                16.8MB/s ± 1%  16.9MB/s ± 1%    ~     (p=0.052 n=11+11)
RegexpMatchEasy0_32-4     314MB/s ± 0%   314MB/s ± 1%    ~     (p=0.618 n=11+11)
RegexpMatchEasy0_1K-4    1.29GB/s ± 0%  1.29GB/s ± 0%    ~     (p=0.315 n=10+10)
RegexpMatchEasy1_32-4     290MB/s ± 1%   290MB/s ± 1%    ~     (p=0.667 n=10+11)
RegexpMatchEasy1_1K-4    1.03GB/s ± 0%  1.03GB/s ± 0%    ~      (p=0.829 n=10+8)
RegexpMatchMedium_32-4   7.63MB/s ± 0%  7.65MB/s ± 0%    ~     (p=0.142 n=11+11)
RegexpMatchMedium_1K-4   24.4MB/s ± 1%  24.6MB/s ± 0%    ~      (p=0.063 n=11+9)
RegexpMatchHard_32-4     13.7MB/s ± 0%  13.7MB/s ± 0%    ~     (p=0.302 n=11+11)
RegexpMatchHard_1K-4     14.4MB/s ± 0%  14.4MB/s ± 0%    ~      (p=0.784 n=9+10)
Revcomp-4                 310MB/s ± 0%   311MB/s ± 0%    ~     (p=0.243 n=10+11)
Template-4               26.2MB/s ± 0%  26.3MB/s ± 0%  +0.24%  (p=0.009 n=10+11)

Update #12970.

Change-Id: Id185080687a60c229a5cb2e5220e7ca1b53910e2
Reviewed-on: https://go-review.googlesource.com/15999
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
2015-11-13 01:09:36 +00:00
Michael Matloob
67faca7d9c runtime: break atomics out into package runtime/internal/atomic
This change breaks out most of the atomics functions in the runtime
into package runtime/internal/atomic. It adds some basic support
in the toolchain for runtime packages, and also modifies linux/arm
atomics to remove the dependency on the runtime's mutex. The mutexes
have been replaced with spinlocks.

all trybots are happy!
In addition to the trybots, I've tested on the darwin/arm64 builder,
on the darwin/arm builder, and on a ppc64le machine.

Change-Id: I6698c8e3cf3834f55ce5824059f44d00dc8e3c2f
Reviewed-on: https://go-review.googlesource.com/14204
Run-TryBot: Michael Matloob <matloob@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
2015-11-10 17:38:04 +00:00
Matthew Dempsky
8ee0fd8623 runtime: replace is{plan9,solaris,windows} with GOOS tests
Change-Id: I27589395f547c5837dc7536a0ab5bc7cc23a4ff6
Reviewed-on: https://go-review.googlesource.com/10872
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2015-10-23 18:11:17 +00:00
Keith Randall
91059de095 runtime: make aeshash more DOS-proof
Improve the aeshash implementation to make it harder to engineer collisions.

1) Scramble the seed before xoring with the input string.  This
   makes it harder to cancel known portions of the seed (like the size)
   because it mixes the per-table seed into those other parts.

2) Use table-dependent seeds for all stripes when hashing >16 byte strings.

For small strings this change uses 4 aesenc ops instead of 3, so it
is somewhat slower.  The first two can run in parallel, though, so
it isn't 33% slower.

benchmark                            old ns/op     new ns/op     delta
BenchmarkHash64-12                   10.2          11.2          +9.80%
BenchmarkHash16-12                   5.71          6.13          +7.36%
BenchmarkHash5-12                    6.64          7.01          +5.57%
BenchmarkHashBytesSpeed-12           30.3          31.9          +5.28%
BenchmarkHash65536-12                2785          2882          +3.48%
BenchmarkHash1024-12                 53.6          55.4          +3.36%
BenchmarkHashStringArraySpeed-12     54.9          56.5          +2.91%
BenchmarkHashStringSpeed-12          18.7          19.2          +2.67%
BenchmarkHashInt32Speed-12           14.8          15.1          +2.03%
BenchmarkHashInt64Speed-12           14.5          14.5          +0.00%

Change-Id: I59ea124b5cb92b1c7e8584008257347f9049996c
Reviewed-on: https://go-review.googlesource.com/14124
Reviewed-by: jcd . <jcd@golang.org>
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2015-10-08 16:43:03 +00:00
Keith Randall
731bdc5115 runtime: fix aeshash of empty string
Aeshash currently computes the hash of the empty string as
hash("", seed) = seed.  This is bad because the hash of a compound
object with empty strings in it doesn't include information about
where those empty strings were.  For instance [2]string{"", "foo"}
and [2]string{"foo", ""} might get the same hash.

Fix this by returning a scrambled seed instead of the seed itself.
With this fix, we can remove the scrambling done by the generated
array hash routines.

The test also rejects hash("", seed) = 0, if we ever thought
it would be a good idea to try that.

The fallback hash is already OK in this regard.

Change-Id: Iaedbaa5be8d6a246dc7e9383d795000e0f562037
Reviewed-on: https://go-review.googlesource.com/14129
Reviewed-by: jcd . <jcd@golang.org>
2015-09-15 17:51:23 +00:00
Michael Hudson-Doyle
d497eeb005 runtime: remove unused xchgp/xchgp1
I noticed that they were unimplemented on arm64 but then that they were
in fact not used at all.

Change-Id: Iee579feda2a5e374fa571bcc8c89e4ef607d50f6
Reviewed-on: https://go-review.googlesource.com/13951
Run-TryBot: Minux Ma <minux@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2015-08-27 00:28:35 +00:00
Austin Clements
f5d494bbdf runtime: ensure GC sees type-safe memory on weak machines
Currently its possible for the garbage collector to observe
uninitialized memory or stale heap bitmap bits on weakly ordered
architectures such as ARM and PPC. On such architectures, the stores
that zero newly allocated memory and initialize its heap bitmap may
move after a store in user code that makes the allocated object
observable by the garbage collector.

To fix this, add a "publication barrier" (also known as an "export
barrier") before returning from mallocgc. This is a store/store
barrier that ensures any write done by user code that makes the
returned object observable to the garbage collector will be ordered
after the initialization performed by mallocgc. No barrier is
necessary on the reading side because of the data dependency between
loading the pointer and loading the contents of the object.

Fixes one of the issues raised in #9984.

Change-Id: Ia3d96ad9c5fc7f4d342f5e05ec0ceae700cd17c8
Reviewed-on: https://go-review.googlesource.com/11083
Reviewed-by: Rick Hudson <rlh@golang.org>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: Minux Ma <minux@golang.org>
Reviewed-by: Martin Capitanio <capnm9@gmail.com>
Reviewed-by: Russ Cox <rsc@golang.org>
2015-06-19 15:29:50 +00:00
Alex Brainman
9d968cb47b runtime: rename cgocall_errno and asmcgocall_errno into cgocall and asmcgocall
Change-Id: I5917bea8bb35b0e725dcc56a68f3a70137cfc180
Reviewed-on: https://go-review.googlesource.com/9387
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2015-06-19 01:47:11 +00:00
Alex Brainman
2858b73843 runtime: remove cgocall and asmcgocall
In preparation for rename of cgocall_errno into cgocall and
asmcgocall_errno into asmcgocall in the fllowinng CL.
rsc requested CL 9387 to be split into two parts. This is first part.

Change-Id: I7434f0e4b44dd37017540695834bfcb1eebf0b2f
Reviewed-on: https://go-review.googlesource.com/11166
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2015-06-18 04:42:53 +00:00
Austin Clements
faa7a7e8ae runtime: implement GC stack barriers
This commit implements stack barriers to minimize the amount of
stack re-scanning that must be done during mark termination.

Currently the GC scans stacks of active goroutines twice during every
GC cycle: once at the beginning during root discovery and once at the
end during mark termination. The second scan happens while the world
is stopped and guarantees that we've seen all of the roots (since
there are no write barriers on writes to local stack
variables). However, this means pause time is proportional to stack
size. In particularly recursive programs, this can drive pause time up
past our 10ms goal (e.g., it takes about 150ms to scan a 50MB heap).

Re-scanning the entire stack is rarely necessary, especially for large
stacks, because usually most of the frames on the stack were not
active between the first and second scans and hence any changes to
these frames (via non-escaping pointers passed down the stack) were
tracked by write barriers.

To efficiently track how far a stack has been unwound since the first
scan (and, hence, how much needs to be re-scanned), this commit
introduces stack barriers. During the first scan, at exponentially
spaced points in each stack, the scan overwrites return PCs with the
PC of the stack barrier function. When "returned" to, the stack
barrier function records how far the stack has unwound and jumps to
the original return PC for that point in the stack. Then the second
scan only needs to proceed as far as the lowest barrier that hasn't
been hit.

For deeply recursive programs, this substantially reduces mark
termination time (and hence pause time). For the goscheme example
linked in issue #10898, prior to this change, mark termination times
were typically between 100 and 500ms; with this change, mark
termination times are typically between 10 and 20ms. As a result of
the reduced stack scanning work, this reduces overall execution time
of the goscheme example by 20%.

Fixes #10898.

The effect of this on programs that are not deeply recursive is
minimal:

name                   old time/op    new time/op    delta
BinaryTree17              3.16s ± 2%     3.26s ± 1%  +3.31%  (p=0.000 n=19+19)
Fannkuch11                2.42s ± 1%     2.48s ± 1%  +2.24%  (p=0.000 n=17+19)
FmtFprintfEmpty          50.0ns ± 3%    49.8ns ± 1%    ~     (p=0.534 n=20+19)
FmtFprintfString          173ns ± 0%     175ns ± 0%  +1.49%  (p=0.000 n=16+19)
FmtFprintfInt             170ns ± 1%     175ns ± 1%  +2.97%  (p=0.000 n=20+19)
FmtFprintfIntInt          288ns ± 0%     295ns ± 0%  +2.73%  (p=0.000 n=16+19)
FmtFprintfPrefixedInt     242ns ± 1%     252ns ± 1%  +4.13%  (p=0.000 n=18+18)
FmtFprintfFloat           324ns ± 0%     323ns ± 0%  -0.36%  (p=0.000 n=20+19)
FmtManyArgs              1.14µs ± 0%    1.12µs ± 1%  -1.01%  (p=0.000 n=18+19)
GobDecode                8.88ms ± 1%    8.87ms ± 0%    ~     (p=0.480 n=19+18)
GobEncode                6.80ms ± 1%    6.85ms ± 0%  +0.82%  (p=0.000 n=20+18)
Gzip                      363ms ± 1%     363ms ± 1%    ~     (p=0.077 n=18+20)
Gunzip                   90.6ms ± 0%    90.0ms ± 1%  -0.71%  (p=0.000 n=17+18)
HTTPClientServer         51.5µs ± 1%    50.8µs ± 1%  -1.32%  (p=0.000 n=18+18)
JSONEncode               17.0ms ± 0%    17.1ms ± 0%  +0.40%  (p=0.000 n=18+17)
JSONDecode               61.8ms ± 0%    63.8ms ± 1%  +3.11%  (p=0.000 n=18+17)
Mandelbrot200            3.84ms ± 0%    3.84ms ± 1%    ~     (p=0.583 n=19+19)
GoParse                  3.71ms ± 1%    3.72ms ± 1%    ~     (p=0.159 n=18+19)
RegexpMatchEasy0_32       100ns ± 0%     100ns ± 1%  -0.19%  (p=0.033 n=17+19)
RegexpMatchEasy0_1K       342ns ± 1%     331ns ± 0%  -3.41%  (p=0.000 n=19+19)
RegexpMatchEasy1_32      82.5ns ± 0%    81.7ns ± 0%  -0.98%  (p=0.000 n=18+18)
RegexpMatchEasy1_1K       505ns ± 0%     494ns ± 1%  -2.16%  (p=0.000 n=18+18)
RegexpMatchMedium_32      137ns ± 1%     137ns ± 1%  -0.24%  (p=0.048 n=20+18)
RegexpMatchMedium_1K     41.6µs ± 0%    41.3µs ± 1%  -0.57%  (p=0.004 n=18+20)
RegexpMatchHard_32       2.11µs ± 0%    2.11µs ± 1%  +0.20%  (p=0.037 n=17+19)
RegexpMatchHard_1K       63.9µs ± 2%    63.3µs ± 0%  -0.99%  (p=0.000 n=20+17)
Revcomp                   560ms ± 1%     522ms ± 0%  -6.87%  (p=0.000 n=18+16)
Template                 75.0ms ± 0%    75.1ms ± 1%  +0.18%  (p=0.013 n=18+19)
TimeParse                 358ns ± 1%     364ns ± 0%  +1.74%  (p=0.000 n=20+15)
TimeFormat                360ns ± 0%     372ns ± 0%  +3.55%  (p=0.000 n=20+18)

Change-Id: If8a9bfae6c128d15a4f405e02bcfa50129df82a2
Reviewed-on: https://go-review.googlesource.com/10314
Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2015-06-02 20:00:57 +00:00
Keith Randall
c526f3ac10 runtime: tail call into memeq/cmp body implementations
There's no need to call/ret to the body implementation.
It can write the result to the right place.  Just jump to
it and have it return to our caller.

Old:
  call body implementation
  compute result
  put result in a register
  return
  write register to result location
  return

New:
  load address of result location into a register
  jump to body implementation
  compute result
  write result to passed-in address
  return

It's a bit tricky on 386 because there is no free register
with which to pass the result location.  Free up a register
by keeping around blen-alen instead of both alen and blen.

Change-Id: If2cf0682a5bf1cc592bdda7c126ed4eee8944fba
Reviewed-on: https://go-review.googlesource.com/9202
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2015-04-29 04:46:25 +00:00
Russ Cox
92c826b1b2 cmd/internal/gc: inline runtime.getg
This more closely restores what the old C runtime did.
(In C, g was an 'extern register' with the same effective
implementation as in this CL.)

On a late 2012 MacBookPro10,2, best of 5 old vs best of 5 new:

benchmark                          old ns/op      new ns/op      delta
BenchmarkBinaryTree17              4981312777     4463426605     -10.40%
BenchmarkFannkuch11                3046495712     3006819428     -1.30%
BenchmarkFmtFprintfEmpty           89.3           79.8           -10.64%
BenchmarkFmtFprintfString          284            262            -7.75%
BenchmarkFmtFprintfInt             282            262            -7.09%
BenchmarkFmtFprintfIntInt          480            448            -6.67%
BenchmarkFmtFprintfPrefixedInt     382            358            -6.28%
BenchmarkFmtFprintfFloat           529            486            -8.13%
BenchmarkFmtManyArgs               1849           1773           -4.11%
BenchmarkGobDecode                 12835963       11794385       -8.11%
BenchmarkGobEncode                 10527170       10288422       -2.27%
BenchmarkGzip                      436109569      438422516      +0.53%
BenchmarkGunzip                    110121663      109843648      -0.25%
BenchmarkHTTPClientServer          81930          85446          +4.29%
BenchmarkJSONEncode                24638574       24280603       -1.45%
BenchmarkJSONDecode                93022423       85753546       -7.81%
BenchmarkMandelbrot200             4703899        4735407        +0.67%
BenchmarkGoParse                   5319853        5086843        -4.38%
BenchmarkRegexpMatchEasy0_32       151            151            +0.00%
BenchmarkRegexpMatchEasy0_1K       452            453            +0.22%
BenchmarkRegexpMatchEasy1_32       131            132            +0.76%
BenchmarkRegexpMatchEasy1_1K       761            722            -5.12%
BenchmarkRegexpMatchMedium_32      228            224            -1.75%
BenchmarkRegexpMatchMedium_1K      63751          64296          +0.85%
BenchmarkRegexpMatchHard_32        3188           3238           +1.57%
BenchmarkRegexpMatchHard_1K        95396          96756          +1.43%
BenchmarkRevcomp                   661587262      687107364      +3.86%
BenchmarkTemplate                  108312598      104008540      -3.97%
BenchmarkTimeParse                 453            459            +1.32%
BenchmarkTimeFormat                475            441            -7.16%

The garbage benchmark from the benchmarks subrepo gets 2.6% faster as well.

Change-Id: I320aeda332db81012688b26ffab23f6581c59cfa
Reviewed-on: https://go-review.googlesource.com/8460
Reviewed-by: Rick Hudson <rlh@golang.org>
Run-TryBot: Rick Hudson <rlh@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
2015-04-07 14:26:47 +00:00
Josh Bleecher Snyder
ad3600945a runtime: auto-generate duff routines
This makes it easier to experiment with alternative implementations.

While we're here, update the comments.

No functional changes. Passes toolstash -cmp.

Change-Id: I428535754908f0fdd7cc36c214ddb6e1e60f376e
Reviewed-on: https://go-review.googlesource.com/8310
Reviewed-by: Keith Randall <khr@golang.org>
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2015-04-02 02:37:59 +00:00
Michael Hudson-Doyle
f78dc1dac1 runtime: rename ·main·f to ·mainPC to avoid duplicate symbol
runtime·main·f is normalized by the linker to runtime.main.f, as is
the compiler-generated symbol runtime.main·f.  Change the former to
runtime·mainPC instead.

Fixes issue #9934

Change-Id: I656a6fa6422d45385fa2cc55bd036c6affa1abfe
Reviewed-on: https://go-review.googlesource.com/8234
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2015-03-30 18:52:14 +00:00
Russ Cox
631d6a33bf runtime: implement atomicand8 atomically
We're skating on thin ice, and things are finally starting to melt around here.
(I want to avoid the debugging session that will happen when someone
uses atomicand8 expecting it to be atomic with respect to other operations.)

Change-Id: I254f1582be4eb1f2d7fbba05335a91c6bf0c7f02
Reviewed-on: https://go-review.googlesource.com/7861
Reviewed-by: Minux Ma <minux@golang.org>
2015-03-20 04:45:29 +00:00
Shenghou Ma
3b00197017 runtime: add argument sizes for asm functions for bytes, strings
Also fixed a stack corruption bug for nacl/amd64p32.

Change-Id: I64b821b16999c296a159137d971af3870053c621
Signed-off-by: Shenghou Ma <minux@golang.org>
Reviewed-on: https://go-review.googlesource.com/7073
Reviewed-by: Dave Cheney <dave@cheney.net>
2015-03-07 06:02:40 +00:00
Dmitry Vyukov
894024f478 runtime: fix traceback from goexit1
We used to not call traceback from goexit1.
But now tracer does it and crashes on amd64p32:

runtime: unexpected return pc for runtime.getg called from 0x108a4240
goroutine 18 [runnable, locked to thread]:
runtime.traceGoEnd()
    src/runtime/trace.go:758 fp=0x10818fe0 sp=0x10818fdc
runtime.goexit1()
    src/runtime/proc1.go:1540 +0x20 fp=0x10818fe8 sp=0x10818fe0
runtime.getg(0x0)
    src/runtime/asm_386.s:2414 fp=0x10818fec sp=0x10818fe8
created by runtime/pprof_test.TestTraceStress
    src/runtime/pprof/trace_test.go:123 +0x500

Return PC from goexit1 points right after goexit (+0x6).
It happens to work most of the time somehow.

This change fixes traceback from goexit1 by adding an additional NOP to goexit.

Fixes #9931

Change-Id: Ied25240a181b0a2d7bc98127b3ed9068e9a1a13e
Reviewed-on: https://go-review.googlesource.com/5460
Reviewed-by: Russ Cox <rsc@golang.org>
2015-02-28 23:19:57 +00:00
Matthew Dempsky
7abdc90fe3 runtime: remove gogetcallerpc and gogetcallersp functions
Package runtime's Go code was converted to directly call getcallerpc
and getcallersp in https://golang.org/cl/138740043, but the assembly
implementations were not removed.

Change-Id: Ib2eaee674d594cbbe799925aae648af782a01c83
Reviewed-on: https://go-review.googlesource.com/5901
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
2015-02-25 09:34:58 +00:00
Dmitry Vyukov
6e70fddec0 runtime: fix cputicks on x86
See the following issue for context:
https://github.com/golang/go/issues/9729#issuecomment-74648287
In short, RDTSC can produce skewed results without preceding LFENCE/MFENCE.
Information on this matter is very scrappy in the internet.
But this is what linux kernel does (see rdtsc_barrier).
It also fixes the test program on my machine.

Update #9729

Change-Id: I3c1ffbf129fdfdd388bd5b7911b392b319248e68
Reviewed-on: https://go-review.googlesource.com/5033
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2015-02-20 16:52:13 +00:00
Josh Bleecher Snyder
135ef49fde runtime: speed up eqstring
eqstring does not need to check the length of the strings.

6g

benchmark                              old ns/op     new ns/op     delta
BenchmarkCompareStringEqual            7.03          6.14          -12.66%
BenchmarkCompareStringIdentical        3.36          3.04          -9.52%

5g

benchmark                                 old ns/op     new ns/op     delta
BenchmarkCompareStringEqual               238           232           -2.52%
BenchmarkCompareStringIdentical           90.8          80.7          -11.12%

The equivalent PPC changes are in a separate commit
because I don't have the hardware to test them.

Change-Id: I292874324b9bbd9d24f57a390cfff8b550cdd53c
Reviewed-on: https://go-review.googlesource.com/3955
Reviewed-by: Keith Randall <khr@golang.org>
2015-02-06 18:51:00 +00:00
Austin Clements
7a71726b1f runtime: check alignment of 8-byte atomic loads and stores on 386
Currently, if we do an atomic{load,store}64 of an unaligned address on
386, we'll simply get a non-atomic load/store.  This has been the
source of myriad bugs, so add alignment checks to these two
operations.  These checks parallel the equivalent checks in
sync/atomic.

The alignment check is not necessary in cas64 because it uses a locked
instruction.  The CPU will either execute this atomically or raise an
alignment fault (#AC)---depending on the alignment check flag---either
of which is fine.

This also fixes the two places in the runtime that trip the new
checks.  One is in the runtime self-test and shouldn't have caused
real problems.  The other is in tickspersecond and could, in
principle, have caused a misread of the ticks per second during
initialization.

Change-Id: If1796667012a6154f64f5e71d043c7f5fb3dd050
Reviewed-on: https://go-review.googlesource.com/3521
Reviewed-by: Russ Cox <rsc@golang.org>
2015-01-29 17:34:40 +00:00
Russ Cox
fd4dc91a96 strings: remove overengineered Compare implementation
The function is here ONLY for symmetry with package bytes.
This function should be used ONLY if it makes code clearer.
It is not here for performance. Remove any performance benefit.

If performance becomes an issue, the compiler should be fixed to
recognize the three-way compare (for all comparable types)
rather than encourage people to micro-optimize by using this function.

Change-Id: I71f4130bce853f7aef724c6044d15def7987b457
Reviewed-on: https://go-review.googlesource.com/3012
Reviewed-by: Rob Pike <r@golang.org>
2015-01-19 02:19:17 +00:00
Alan Donovan
90ce1936e3 strings: add Compare(x, y string) int, for symmetry with bytes.Compare
The implementation is the same assembly (or Go) routine.

Change-Id: Ib937c461c24ad2d5be9b692b4eed40d9eb031412
Reviewed-on: https://go-review.googlesource.com/2828
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2015-01-15 17:17:05 +00:00
Josh Bleecher Snyder
f7e43f14d3 runtime: remove stray commas in assembly
Change-Id: I4dc97ff8111bdc5ca6e4e3af06aaf4f768031c68
Reviewed-on: https://go-review.googlesource.com/2473
Reviewed-by: Minux Ma <minux@golang.org>
2015-01-07 22:41:54 +00:00
Keith Randall
d5e4c4061b runtime: remove size argument from hash and equal algorithms
The equal algorithm used to take the size
   equal(p, q *T, size uintptr) bool
With this change, it does not
   equal(p, q *T) bool
Similarly for the hash algorithm.

The size is rarely used, as most equal functions know the size
of the thing they are comparing.  For instance f32equal already
knows its inputs are 4 bytes in size.

For cases where the size is not known, we allocate a closure
(one for each size needed) that points to an assembly stub that
reads the size out of the closure and calls generic code that
has a size argument.

Reduces the size of the go binary by 0.07%.  Performance impact
is not measurable.

Change-Id: I6e00adf3dde7ad2974adbcff0ee91e86d2194fec
Reviewed-on: https://go-review.googlesource.com/2392
Reviewed-by: Russ Cox <rsc@golang.org>
2015-01-07 21:57:01 +00:00
Russ Cox
df027aceb9 reflect: add write barriers
Use typedmemmove, typedslicecopy, and adjust reflect.call
to execute the necessary write barriers.

Found with GODEBUG=wbshadow=2 mode.
Eventually that will run automatically, but right now
it still detects other missing write barriers.

Change-Id: Iec5b5b0c1be5589295e28e5228e37f1a92e07742
Reviewed-on: https://go-review.googlesource.com/2312
Reviewed-by: Keith Randall <khr@golang.org>
2015-01-06 00:28:31 +00:00
Russ Cox
e6d3511264 Revert "liblink, cmd/ld, runtime: remove stackguard1"
This reverts commit ab0535ae3f.

I think it will remain useful to distinguish code that must
run on a system stack from code that can run on either stack,
even if that distinction is no
longer based on the implementation language.

That is, I expect to add a //go:systemstack comment that,
in terms of the old implementation, tells the compiler,
to pretend this function was written in C.

Change-Id: I33d2ebb2f99ae12496484c6ec8ed07233d693275
Reviewed-on: https://go-review.googlesource.com/2275
Reviewed-by: Russ Cox <rsc@golang.org>
2015-01-05 16:29:56 +00:00
Josh Bleecher Snyder
17577e48c9 runtime: use SETEQ instead of JZ for cas
Change-Id: Ibabbca3988d39bdce584924173a912d45f50f0dd
Reviewed-on: https://go-review.googlesource.com/2243
Reviewed-by: Dave Cheney <dave@cheney.net>
Reviewed-by: Minux Ma <minux@golang.org>
2015-01-02 19:06:36 +00:00
Shenghou Ma
ab0535ae3f liblink, cmd/ld, runtime: remove stackguard1
Now that we've removed all the C code in runtime and the C compilers,
there is no need to have a separate stackguard field to check for C
code on Go stack.

Remove field g.stackguard1 and rename g.stackguard0 to g.stackguard.
Adjust liblink and cmd/ld as necessary.

Change-Id: I54e75db5a93d783e86af5ff1a6cd497d669d8d33
Reviewed-on: https://go-review.googlesource.com/2144
Reviewed-by: Keith Randall <khr@golang.org>
2014-12-29 07:36:07 +00:00
Russ Cox
7a524a1036 runtime: remove thunk.s
Replace with uses of //go:linkname in Go files, direct use of name in .s files.
The only one that really truly needs a jump is reflect.call; the jump is now
next to the runtime.reflectcall assembly implementations.

Change-Id: Ie7ff3020a8f60a8e4c8645fe236e7883a3f23f46
Reviewed-on: https://go-review.googlesource.com/1962
Reviewed-by: Austin Clements <austin@google.com>
2014-12-23 03:17:22 +00:00
Keith Randall
7a4a64e8f3 runtime: faster aeshash implementation
The aesenc instruction has high latency.  For hashing large objects,
hash several streams in parallel.

benchmark                         old ns/op     new ns/op     delta
BenchmarkHash5                    7.02          7.45          +6.13%
BenchmarkHash16                   6.53          6.94          +6.28%
BenchmarkHash32                   8.38          8.26          -1.43%
BenchmarkHash64                   12.6          12.0          -4.76%
BenchmarkHash1024                 247           62.9          -74.53%
BenchmarkHash65536                17335         2966          -82.89%
BenchmarkHashInt32Speed           14.7          14.9          +1.36%
BenchmarkHashInt64Speed           14.6          14.9          +2.05%
BenchmarkHashBytesSpeed           35.4          28.6          -19.21%
BenchmarkHashStringSpeed          22.0          20.4          -7.27%
BenchmarkHashStringArraySpeed     65.8          56.3          -14.44%

Change-Id: Ia8ba03063acc64a9066b8ab2d79f2c9aaac1770f
Reviewed-on: https://go-review.googlesource.com/1330
Reviewed-by: Russ Cox <rsc@golang.org>
2014-12-11 05:23:00 +00:00
Russ Cox
8c3f64022a [dev.garbage] runtime: add prefetcht0, prefetcht1, prefetcht2, prefetcht3, prefetchnta for GC
We don't know what we need yet, so add them all.
Add them even on x86 architectures (as no-ops) so that
the GC can refer to them unconditionally.

Eventually we'll know what we want and probably
have just one 'prefetch' with an appropriate meaning
on each architecture.

LGTM=rlh
R=rlh
CC=golang-codereviews
https://golang.org/cl/179160043
2014-11-21 15:57:10 -05:00
Russ Cox
3e804631d9 [dev.cc] all: merge dev.power64 (7667e41f3ced) into dev.cc
This is to reduce the delta between dev.cc and dev.garbage to just garbage collector changes.

These are the files that had merge conflicts and have been edited by hand:
        malloc.go
        mem_linux.go
        mgc.go
        os1_linux.go
        proc1.go
        panic1.go
        runtime1.go

LGTM=austin
R=austin
CC=golang-codereviews
https://golang.org/cl/174180043
2014-11-14 12:10:52 -05:00
Russ Cox
656be317d0 [dev.cc] runtime: delete scalararg, ptrarg; rename onM to systemstack
Scalararg and ptrarg are not "signal safe".
Go code filling them out can be interrupted by a signal,
and then the signal handler runs, and if it also ends up
in Go code that uses scalararg or ptrarg, now the old
values have been smashed.
For the pieces of code that do need to run in a signal handler,
we introduced onM_signalok, which is really just onM
except that the _signalok is meant to convey that the caller
asserts that scalarg and ptrarg will be restored to their old
values after the call (instead of the usual behavior, zeroing them).

Scalararg and ptrarg are also untyped and therefore error-prone.

Go code can always pass a closure instead of using scalararg
and ptrarg; they were only really necessary for C code.
And there's no more C code.

For all these reasons, delete scalararg and ptrarg, converting
the few remaining references to use closures.

Once those are gone, there is no need for a distinction between
onM and onM_signalok, so replace both with a single function
equivalent to the current onM_signalok (that is, it can be called
on any of the curg, g0, and gsignal stacks).

The name onM and the phrase 'm stack' are misnomers,
because on most system an M has two system stacks:
the main thread stack and the signal handling stack.

Correct the misnomer by naming the replacement function systemstack.

Fix a few references to "M stack" in code.

The main motivation for this change is to eliminate scalararg/ptrarg.
Rick and I have already seen them cause problems because
the calling sequence m.ptrarg[0] = p is a heap pointer assignment,
so it gets a write barrier. The write barrier also uses onM, so it has
all the same problems as if it were being invoked by a signal handler.
We worked around this by saving and restoring the old values
and by calling onM_signalok, but there's no point in keeping this nice
home for bugs around any longer.

This CL also changes funcline to return the file name as a result
instead of filling in a passed-in *string. (The *string signature is
left over from when the code was written in and called from C.)
That's arguably an unrelated change, except that once I had done
the ptrarg/scalararg/onM cleanup I started getting false positives
about the *string argument escaping (not allowed in package runtime).
The compiler is wrong, but the easiest fix is to write the code like
Go code instead of like C code. I am a bit worried that the compiler
is wrong because of some use of uninitialized memory in the escape
analysis. If that's the reason, it will go away when we convert the
compiler to Go. (And if not, we'll debug it the next time.)

LGTM=khr
R=r, khr
CC=austin, golang-codereviews, iant, rlh
https://golang.org/cl/174950043
2014-11-12 14:54:31 -05:00
Russ Cox
15ced2d008 [dev.cc] runtime: convert assembly files for C to Go transition
The main change is that #include "zasm_GOOS_GOARCH.h"
is now #include "go_asm.h" and/or #include "go_tls.h".

Also, because C StackGuard is now Go _StackGuard,
the assembly name changes from const_StackGuard to
const__StackGuard.

In asm_$GOARCH.s, add new function getg, formerly
implemented in C.

The renamed atomics now have Go wrappers, to get
escape analysis annotations right. Those wrappers
are in CL 174860043.

LGTM=r, aram
R=r, aram
CC=austin, dvyukov, golang-codereviews, iant, khr
https://golang.org/cl/168510043
2014-11-11 17:06:22 -05:00
Austin Clements
31b1207fde [dev.power64] all: merge default into dev.power64
Trivial merge except for src/runtime/asm_power64x.s and
src/runtime/signal_power64x.c

LGTM=rsc
R=rsc
CC=golang-codereviews
https://golang.org/cl/168950044
2014-11-03 10:53:11 -05:00
Russ Cox
a5a0733144 runtime: change top-most return PC from goexit to goexit+PCQuantum
If you get a stack of PCs from Callers, it would be expected
that every PC is immediately after a call instruction, so to find
the line of the call, you look up the line for PC-1.
CL 163550043 now explicitly documents that.

The most common exception to this is the top-most return PC
on the stack, which is the entry address of the runtime.goexit
function. Subtracting 1 from that PC will end up in a different
function entirely.

To remove this special case, make the top-most return PC
goexit+PCQuantum and then implement goexit in assembly
so that the first instruction can be skipped.

Fixes #7690.

LGTM=r
R=r
CC=golang-codereviews
https://golang.org/cl/170720043
2014-10-29 20:37:44 -04:00
Russ Cox
599199fd9f [dev.power64] all: merge default (dd5014ed9b01) into dev.power64
Still passes on amd64.

LGTM=austin
R=austin
CC=golang-codereviews
https://golang.org/cl/165110043
2014-10-29 11:45:01 -04:00
Russ Cox
c4efaac15d runtime: fix unrecovered panic on external thread
Fixes #8588.

LGTM=austin
R=austin
CC=golang-codereviews, khr
https://golang.org/cl/159700044
2014-10-28 21:53:09 -04:00
Russ Cox
b55791e200 [dev.power64] cmd/5a, cmd/6a, cmd/8a, cmd/9a: make labels function-scoped
I removed support for jumping between functions years ago,
as part of doing the instruction layout for each function separately.

Given that, it makes sense to treat labels as function-scoped.
This lets each function have its own 'loop' label, for example.

Makes the assembly much cleaner and removes the last
reason anyone would reach for the 123(PC) form instead.

Note that this is on the dev.power64 branch, but it changes all
the assemblers. The change will ship in Go 1.5 (perhaps after
being ported into the new assembler).

Came up as part of CL 167730043.

LGTM=r
R=r
CC=austin, dave, golang-codereviews, minux
https://golang.org/cl/159670043
2014-10-28 21:50:16 -04:00
Keith Randall
b60d5e12e9 runtime: warn that cputicks() might not be monotonic.
Get rid of gocputicks(), it is no longer used.

LGTM=bradfitz, dave
R=golang-codereviews, bradfitz, dave, minux
CC=golang-codereviews
https://golang.org/cl/161110044
2014-10-21 14:46:07 -07:00
Russ Cox
cb6f5ac0b0 runtime: remove hand-generated ptr bitmaps for reflectcall
A Go prototype can be used instead now, and the compiler
will do a better job than we will doing it by hand.
(We got it wrong in amd64p32, causing the current build
breakage.)

The auto-prototype-matching only applies to functions
without an explicit package path, so the TEXT lines for
reflectcall and callXX are s/runtime·/·/.

LGTM=khr
R=khr
CC=golang-codereviews, iant, r
https://golang.org/cl/153600043
2014-10-15 13:12:16 -04:00
Russ Cox
2b1659b57d runtime: change Windows M.thread from void* to uintptr
It appears to be an opaque bit pattern more than a pointer.
The Go garbage collector has discovered that for m0
it is set to 0x4c.

Should fix Windows build.

TBR=brainman
CC=golang-codereviews
https://golang.org/cl/149640043
2014-10-07 23:27:25 -04:00
Keith Randall
1aa65fe8d4 runtime: add load_g call in arm callback.
Need to restore the g register.  Somehow this line vaporized from
CL 144130043.  Also cgo_topofstack -> _cgo_topofstack, that vaporized also.

TBR=rsc
CC=golang-codereviews
https://golang.org/cl/150940044
2014-09-25 08:37:04 -07:00
Keith Randall
1b6807bb06 cgo: adjust return value location to account for stack copies.
During a cgo call, the stack can be copied.  This copy invalidates
the pointer that cgo has into the return value area.  To fix this
problem, pass the address of the location containing the stack
top value (which is in the G struct).  For cgo functions which
return values, read the stktop before and after the cgo call to
compute the adjustment necessary to write the return value.

Fixes #8771

LGTM=iant, rsc
R=iant, rsc, khr
CC=golang-codereviews
https://golang.org/cl/144130043
2014-09-25 07:59:01 -07:00