Now there's only one copy of the flow graph construction
and dominator computation, and different optimizations
can attach different annotations to the instructions.
R=ken2
CC=golang-dev
https://golang.org/cl/12797045
Code in gc/popt.c is compiled as part of 5g, 6g, and 8g,
meaning it can use arch-specific headers but there's
just one copy of the code.
This is the same arrangement we use for the portable
code generation logic in gc/pgen.c.
Move fixjmp and noreturn there to get the ball rolling.
R=ken2
CC=golang-dev
https://golang.org/cl/12789043
On entry to a function, zero the results and zero the pointer
section of the local variables.
This is an intermediate step on the way to precise collection
of Go frames.
This can incur a significant (up to 30%) slowdown, but it also ensures
that the garbage collector never looks at a word in a Go frame
and sees a stale pointer value that could cause a space leak.
(C frames and assembly frames are still possibly problematic.)
This CL is required to start making collection of interface values
as precise as collection of pointer values are today.
Since we have to dereference the interface type to understand
whether the value is a pointer, it is critical that the type field be
initialized.
A future CL by Carl will make the garbage collection pointer
bitmaps context-sensitive. At that point it will be possible to
remove most of the zeroing. The only values that will still need
zeroing are values whose addresses escape the block scoping
of the function but do not escape to the heap.
benchmark old ns/op new ns/op delta
BenchmarkBinaryTree17 4420289180 4331060459 -2.02%
BenchmarkFannkuch11 3442469663 3277706251 -4.79%
BenchmarkFmtFprintfEmpty 100 142 +42.00%
BenchmarkFmtFprintfString 262 310 +18.32%
BenchmarkFmtFprintfInt 213 281 +31.92%
BenchmarkFmtFprintfIntInt 355 431 +21.41%
BenchmarkFmtFprintfPrefixedInt 321 383 +19.31%
BenchmarkFmtFprintfFloat 444 533 +20.05%
BenchmarkFmtManyArgs 1380 1559 +12.97%
BenchmarkGobDecode 10240054 11794915 +15.18%
BenchmarkGobEncode 17350274 19970478 +15.10%
BenchmarkGzip 455179460 460699139 +1.21%
BenchmarkGunzip 114271814 119291574 +4.39%
BenchmarkHTTPClientServer 89051 89894 +0.95%
BenchmarkJSONEncode 40486799 52691558 +30.15%
BenchmarkJSONDecode 94193361 112428781 +19.36%
BenchmarkMandelbrot200 4747060 4748043 +0.02%
BenchmarkGoParse 6363798 6675098 +4.89%
BenchmarkRegexpMatchEasy0_32 129 171 +32.56%
BenchmarkRegexpMatchEasy0_1K 365 395 +8.22%
BenchmarkRegexpMatchEasy1_32 106 152 +43.40%
BenchmarkRegexpMatchEasy1_1K 952 1245 +30.78%
BenchmarkRegexpMatchMedium_32 198 283 +42.93%
BenchmarkRegexpMatchMedium_1K 79006 101097 +27.96%
BenchmarkRegexpMatchHard_32 3478 5115 +47.07%
BenchmarkRegexpMatchHard_1K 110245 163582 +48.38%
BenchmarkRevcomp 777384355 793270857 +2.04%
BenchmarkTemplate 136713089 157093609 +14.91%
BenchmarkTimeParse 1511 1761 +16.55%
BenchmarkTimeFormat 535 850 +58.88%
benchmark old MB/s new MB/s speedup
BenchmarkGobDecode 74.95 65.07 0.87x
BenchmarkGobEncode 44.24 38.43 0.87x
BenchmarkGzip 42.63 42.12 0.99x
BenchmarkGunzip 169.81 162.67 0.96x
BenchmarkJSONEncode 47.93 36.83 0.77x
BenchmarkJSONDecode 20.60 17.26 0.84x
BenchmarkGoParse 9.10 8.68 0.95x
BenchmarkRegexpMatchEasy0_32 247.24 186.31 0.75x
BenchmarkRegexpMatchEasy0_1K 2799.20 2591.93 0.93x
BenchmarkRegexpMatchEasy1_32 299.31 210.44 0.70x
BenchmarkRegexpMatchEasy1_1K 1074.71 822.45 0.77x
BenchmarkRegexpMatchMedium_32 5.04 3.53 0.70x
BenchmarkRegexpMatchMedium_1K 12.96 10.13 0.78x
BenchmarkRegexpMatchHard_32 9.20 6.26 0.68x
BenchmarkRegexpMatchHard_1K 9.29 6.26 0.67x
BenchmarkRevcomp 326.95 320.40 0.98x
BenchmarkTemplate 14.19 12.35 0.87x
R=cshapiro
CC=golang-dev
https://golang.org/cl/12616045
Prior to this change, pointer maps encoded the disposition of
a word using a single bit. A zero signaled a non-pointer
value and a one signaled a pointer value. Interface values,
which are a effectively a union type, were conservatively
labeled as a pointer.
This change widens the logical element size of the pointer map
to two bits per word. As before, zero signals a non-pointer
value and one signals a pointer value. Additionally, a two
signals an iface pointer and a three signals an eface pointer.
Following other changes to the runtime, values two and three
will allow a type information to drive interpretation of the
subsequent word so only those interface values containing a
pointer value will be scanned.
R=golang-dev, rsc
CC=golang-dev
https://golang.org/cl/12689046
I moved the pointer block from one end of the frame
to the other toward the end of working on the last CL,
and of course that made the optimization no longer work.
Now it works again:
0030 (bug361.go:12) DATA gclocals·0+0(SB)/4,$4
0030 (bug361.go:12) DATA gclocals·0+4(SB)/4,$3
0030 (bug361.go:12) GLOBL gclocals·0+0(SB),8,$8
Fixes arm build (this time for sure!).
TBR=golang-dev
CC=cshapiro, golang-dev, iant
https://golang.org/cl/12627044
Sort non-pointer-containing data to the low end of the
stack frame, and make the bitmaps only cover the
pointer-containing top end.
Generates significantly less garbage collection bitmap
for programs with large byte buffers on the stack.
Only 2% shorter for godoc, but 99.99998% shorter
in some test cases.
Fixes arm build.
TBR=golang-dev
CC=cshapiro, golang-dev, iant
https://golang.org/cl/12541047
Individual variables bigger than 10 MB are now
moved to the heap, as if they had escaped on
their own.
This avoids ridiculous stacks for programs that
do things like
x := [1<<30]byte{}
... use x ...
If 10 MB is too small, we can raise the limit.
Fixes#6077.
R=ken2
CC=golang-dev
https://golang.org/cl/12650045
Previously, all word aligned locations in the local variables
area were scanned as conservative roots. With this change, a
bitmap is generated describing the locations of pointer values
in local variables.
With this change the argument bitmap information has been
changed to only store information about arguments. The locals
member, has been removed. In its place, the bitmap data for
local variables is now used to store the size of locals. If
the size is negative, the magnitude indicates the size of the
local variables area.
R=rsc
CC=golang-dev
https://golang.org/cl/12328044
For normal slices a[i:j] we're generating 3 bounds
checks: j<={len(string),cap(slice)}, j<=j (!), and i<=j.
Somehow snuck in as part of the [i:j:k] implementation
where the second check does something.
Remove the second check when we don't need it.
R=rsc, r
CC=golang-dev
https://golang.org/cl/12311046
Backends do not exactly expect receiving binary operators with
constant operands or use workarounds to move them to
register/stack in order to handle them.
Fixes#5841.
R=golang-dev, daniel.morsing, rsc
CC=golang-dev
https://golang.org/cl/11107044
Phrases like "returns whether or not the image is opaque" could be
describing what the function does (it always returns, regardless of
the opacity) or what it returns (a boolean indicating the opacity).
Even when the "or not" is missing, the phrasing is bizarre.
Go with "reports whether", which is still clunky but at least makes
it clear we're talking about the return value.
These were edited by hand. A few were cleaned up in other ways.
R=golang-dev, dsymonds
CC=golang-dev
https://golang.org/cl/11699043
This CL introduces a FUNCDATA number for runtime-specific
garbage collection metadata, changes the C and Go compilers
to emit that metadata, and changes the runtime to expect it.
The old pseudo-instructions that carried this information
are gone, as is the linker code to process them.
R=golang-dev, dvyukov, cshapiro
CC=golang-dev
https://golang.org/cl/11406044
Design at http://golang.org/s/go12symtab.
This enables some cleanup of the garbage collector metadata
that will be done in future CLs.
This CL does not move the old symtab and pclntab back into
an unmapped section of the file. That's a bit tricky and will be
done separately.
Fixes#4020.
R=golang-dev, dave, cshapiro, iant, r
CC=golang-dev, nigeltao
https://golang.org/cl/11085043
Race instrumentation can allocate, switch stacks, preempt, etc.
All that is not allowed in between fork and exec.
Fixes#4840.
R=golang-dev, daniel.morsing, dave
CC=golang-dev
https://golang.org/cl/11324044
A type switch on a value with map index expressions,
could get a spurious instrumentation from a OTYPESW node.
These nodes do not need instrumentation because after
walk the type switch has been turned into a sequence
of ifs.
Fixes#5890.
R=golang-dev, dvyukov
CC=golang-dev
https://golang.org/cl/11308043
"M requires pointer receiver" can be misinterpreted to
mean that method M should have a pointer receiver but
does not. In fact the message means "M has a pointer
receiver" (and you don't have a pointer).
Fixes#5891.
R=ken2
CC=golang-dev
https://golang.org/cl/11313043
Escape analysis needs the right curfn value on a dclfunc node, otherwise it will not analyze the function.
When generating method value wrappers, we forgot to set the curfn correctly.
Fixes#5753.
R=golang-dev, rsc
CC=golang-dev
https://golang.org/cl/10383048
A struct with a single field was considered as equivalent to the
field type, which is incorrect is the field is blank.
Fields with padding could make the compiler think some
types are comparable when they are not.
Fixes#5698.
R=rsc, golang-dev, daniel.morsing, bradfitz, gri, r
CC=golang-dev
https://golang.org/cl/10271046
Design doc at golang.org/s/go12slice.
This is an experimental feature and may not be included in the release.
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/10743046
Exported inlined functions that perform a string conversion
using a non-exported named type may miss it in export data.
Fixes#5755.
R=rsc, golang-dev, ality, r
CC=golang-dev
https://golang.org/cl/10464043
Functions without bodies were excluded from the ordering logic,
because when I wrote the ordering logic there was no reason to
analyze them.
But then we added //go:noescape tags that need analysis, and we
didn't update the ordering logic.
So in the absence of good ordering, //go:noescape only worked
if it appeared before the use in the source code.
Fixes#5773.
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/10570043
(By not using the tail-call wrappers when the race
detector is enabled.)
R=golang-dev, minux.ma, dvyukov, daniel.morsing
CC=golang-dev
https://golang.org/cl/10227043
The previous implementation would only record access to
the address of the array but the memory access to the whole
memory range must be recorded instead.
R=golang-dev, dvyukov, r
CC=golang-dev
https://golang.org/cl/8053044
Instrumentation of ntest expression should go to ntest->init.
Same for nincr.
Fixes#5340.
R=golang-dev, daniel.morsing
CC=golang-dev
https://golang.org/cl/10026046
This avoids problems with inlining in genwrappers, which
occurs after functions have been compiled. Compiling a
function may cause some unused local vars to be removed from
the list. Since a local var may be unused due to
optimization, it is possible that a removed local var winds up
beingused in the inlined version, in which case hilarity
ensues.
Fixes#5515.
R=golang-dev, khr, dave
CC=golang-dev
https://golang.org/cl/10210043
It was off in the old implementation (because there was no high-level
description of the function at all). Maybe some day the race detector
should be fixed to handle the wrapper and then enabled for it, but there's
no reason that has to be today.
R=golang-dev
TBR=dvyukov
CC=golang-dev
https://golang.org/cl/10037045
Requires adding new linker instruction
RET f(SB)
meaning return but then immediately call f.
This is what you'd use to implement a tail call after
fiddling with the arguments, but the compiler only
uses it in genwrapper.
This CL eliminates the copy-and-paste genembedtramp
functions from 5g/8g/6g and makes the code run on ARM
for the first time. It removes a small special case for function
generation, which should help Carl a bit, but at the same time
it does not bother to implement general tail call optimization,
which we do not want anyway.
Fixes#5627.
R=ken2
CC=golang-dev
https://golang.org/cl/10057044
Escape analysis already gives that the underlying array
does not escape but the result was ignored.
Fixes#5484.
R=golang-dev, dave, daniel.morsing
CC=golang-dev
https://golang.org/cl/9662046
A nosplits was assumed to have no argument information and no
pointer map. However, nosplits created by the linker often
have both. This change uses the pointer map size as an
alternate source of argument size when processing a nosplit.
In addition, the symbol table construction pointer map size
and argument size consistency check is strengthened. If a
nptrs is greater than 0 it must be equal to the number of
argument words.
R=golang-dev, khr, khr
CC=golang-dev
https://golang.org/cl/9666047
With this change the compiler emits a bitmap for each function
covering its stack frame arguments area. If an argument word
is known to contain a pointer, a bit is set. The garbage
collector reads this information when scanning the stack by
frames and uses it to ignores locations known to not contain a
pointer.
R=golang-dev, bradfitz, daniel.morsing, dvyukov, khr, khr, iant, cshapiro
CC=golang-dev
https://golang.org/cl/9223046
It contains the LHS of the range clause and gets
instrumented by racewalk, but it doesn't have any meaning.
Fixes#5446.
R=golang-dev, dvyukov, daniel.morsing, r
CC=golang-dev
https://golang.org/cl/9560044
Some 64-bit fields were run through 32-bit words, some counts were
not checked for overflow, and relocations must fit in 32 bits.
Tests to follow.
R=golang-dev, dsymonds
CC=golang-dev
https://golang.org/cl/9033043
They caused internal compiler errors and they're expensive enough that inlining them doesn't make sense.
Fixes#5259.
R=golang-dev, r, iant, remyoudompheng
CC=golang-dev
https://golang.org/cl/8636043
The offset of an embedded field s.X must be relative to s
and not to the implicit s.Field of which X is a direct field.
Moreover, no indirections may happen on the path.
Fixes#4909.
R=nigeltao, ality, daniel.morsing, iant, gri, r
CC=golang-dev
https://golang.org/cl/8287043
A package file may begin as either "package foo" or
"package foo safe". The latter is relevant when using -u.
https://golang.org/cl/6903059 resulted in the distinction
being dropped when a package was read for the second or later time.
This CL records whether that "safe" tag was present,
and includes it in the dummy statement generated for the lexer.
R=golang-dev, r, minux.ma, daniel.morsing, iant
CC=golang-dev
https://golang.org/cl/8255044
Doing grow work on reads is not multithreaded safe.
Changed code to do grow work only on inserts & deletes.
This is a short-term fix, eventually we'll want to do
grow work in parallel to recover the space of the old
table.
Fixes#5120.
R=bradfitz, khr
CC=golang-dev
https://golang.org/cl/8242043
Reusing it when multiple comparisons occurred in the same
function call led to bad overwriting.
Fixes#5162.
R=golang-dev, daniel.morsing
CC=golang-dev
https://golang.org/cl/8174047
src/cmd/gc/closure.c:133 param declared and not used: nowrap
src/cmd/gc/const.c:1139 set and not used: t1
src/cmd/ld/data.c:652 format mismatch #llx INT, arg 7
src/cmd/ld/data.c:652 format mismatch #llx INT, arg 8
src/cmd/ld/data.c:1230 set and not used: datsize
R=dave, golang-dev, lucio.dere, remyoudompheng, bradfitz
CC=golang-dev
https://golang.org/cl/8182043
A HMUL node appears in some constant divisions, but
to observe a false negative in race detector the divisor must be
suitably chosen to make sure the only memory access is
done for HMUL.
R=dvyukov
CC=golang-dev
https://golang.org/cl/7935045
Handle interface comparison correctly,
add a few more tests, mark more nodes as impossible.
R=dvyukov, golang-dev
CC=golang-dev
https://golang.org/cl/7942045
The right operand of a && and || is only executed conditionnally,
so the instrumentation must be more careful. In particular
it should not turn nodes assumed to be cheap after walk into
expensive ones.
Update #4228
R=dvyukov, golang-dev
CC=golang-dev
https://golang.org/cl/7986043
Add missing CLOSUREVAR in switch.
Mark MAKE, string conversion nodes as impossible.
Control statements do not need instrumentation.
Instrument COM and LROT nodes.
Instrument map length.
Update #4228
R=dvyukov, golang-dev
CC=golang-dev
https://golang.org/cl/7504047
Hashtable is arranged as an array of
8-entry buckets with chained overflow.
Each bucket has 8 extra hash bits
per key to provide quick lookup within
a bucket. Table is grown incrementally.
Update #3885
Go time drops from 0.51s to 0.34s.
R=r, rsc, m3b, dave, bradfitz, khr, ugorji, remyoudompheng
CC=golang-dev
https://golang.org/cl/7504044
Usually, there is no esc info when inlining, but there will be when generating inlined wrapper functions.
If we don't use this information, we get invalid addresses on the stack.
Fixes#5056.
R=golang-dev, rsc
CC=golang-dev, remyoudompheng
https://golang.org/cl/7850045
It used to not mark parameters as escaping if only one of the
fields it points to leaks out of the function. This causes
problems when importing from another package.
Fixes#4964.
R=rsc, lvd, dvyukov, daniel.morsing
CC=golang-dev
https://golang.org/cl/7648045
valgrind complained that under some circumstances,
*nr = *nc
was being called when nr and nc were the same *Node. The suggestion my Rémy was to introduce a tmp node to avoid the potential for aliasing in subnode.
R=remyoudompheng, minux.ma, rsc
CC=golang-dev
https://golang.org/cl/7780044
Method calls on interfaces with large stored values
will call the pointer receiver method which may be
a wrapper over a method with value receiver.
This is particularly inefficient for very small bodies.
Inlining the wrapped method body saves a potentially expensive
function call.
benchmark old ns/op new ns/op delta
BenchmarkSortString1K 802295 641387 -20.06%
BenchmarkSortInt1K 359914 238234 -33.81%
BenchmarkSortInt64K 35764226 22803078 -36.24%
Fixes#4707.
R=golang-dev, daniel.morsing, rsc
CC=golang-dev
https://golang.org/cl/7214044
If all locals are optimized away, the type instructions would stay in the instruction stream. Call fixautoused to scrub the output.
Fixes#4915.
R=rsc
CC=golang-dev
https://golang.org/cl/7385055
