2009-01-26 18:37:05 -07:00
|
|
|
|
// Copyright 2009 The Go Authors. All rights reserved.
|
|
|
|
|
// Use of this source code is governed by a BSD-style
|
|
|
|
|
// license that can be found in the LICENSE file.
|
|
|
|
|
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
// Garbage collector (GC).
|
|
|
|
|
//
|
|
|
|
|
// GC is:
|
|
|
|
|
// - mark&sweep
|
|
|
|
|
// - mostly precise (with the exception of some C-allocated objects, assembly frames/arguments, etc)
|
|
|
|
|
// - parallel (up to MaxGcproc threads)
|
|
|
|
|
// - partially concurrent (mark is stop-the-world, while sweep is concurrent)
|
|
|
|
|
// - non-moving/non-compacting
|
|
|
|
|
// - full (non-partial)
|
|
|
|
|
//
|
|
|
|
|
// GC rate.
|
|
|
|
|
// Next GC is after we've allocated an extra amount of memory proportional to
|
|
|
|
|
// the amount already in use. The proportion is controlled by GOGC environment variable
|
|
|
|
|
// (100 by default). If GOGC=100 and we're using 4M, we'll GC again when we get to 8M
|
|
|
|
|
// (this mark is tracked in next_gc variable). This keeps the GC cost in linear
|
|
|
|
|
// proportion to the allocation cost. Adjusting GOGC just changes the linear constant
|
|
|
|
|
// (and also the amount of extra memory used).
|
|
|
|
|
//
|
|
|
|
|
// Concurrent sweep.
|
|
|
|
|
// The sweep phase proceeds concurrently with normal program execution.
|
|
|
|
|
// The heap is swept span-by-span both lazily (when a goroutine needs another span)
|
|
|
|
|
// and concurrently in a background goroutine (this helps programs that are not CPU bound).
|
|
|
|
|
// However, at the end of the stop-the-world GC phase we don't know the size of the live heap,
|
|
|
|
|
// and so next_gc calculation is tricky and happens as follows.
|
|
|
|
|
// At the end of the stop-the-world phase next_gc is conservatively set based on total
|
|
|
|
|
// heap size; all spans are marked as "needs sweeping".
|
|
|
|
|
// Whenever a span is swept, next_gc is decremented by GOGC*newly_freed_memory.
|
|
|
|
|
// The background sweeper goroutine simply sweeps spans one-by-one bringing next_gc
|
|
|
|
|
// closer to the target value. However, this is not enough to avoid over-allocating memory.
|
|
|
|
|
// Consider that a goroutine wants to allocate a new span for a large object and
|
|
|
|
|
// there are no free swept spans, but there are small-object unswept spans.
|
|
|
|
|
// If the goroutine naively allocates a new span, it can surpass the yet-unknown
|
|
|
|
|
// target next_gc value. In order to prevent such cases (1) when a goroutine needs
|
|
|
|
|
// to allocate a new small-object span, it sweeps small-object spans for the same
|
|
|
|
|
// object size until it frees at least one object; (2) when a goroutine needs to
|
|
|
|
|
// allocate large-object span from heap, it sweeps spans until it frees at least
|
|
|
|
|
// that many pages into heap. Together these two measures ensure that we don't surpass
|
|
|
|
|
// target next_gc value by a large margin. There is an exception: if a goroutine sweeps
|
|
|
|
|
// and frees two nonadjacent one-page spans to the heap, it will allocate a new two-page span,
|
|
|
|
|
// but there can still be other one-page unswept spans which could be combined into a two-page span.
|
|
|
|
|
// It's critical to ensure that no operations proceed on unswept spans (that would corrupt
|
|
|
|
|
// mark bits in GC bitmap). During GC all mcaches are flushed into the central cache,
|
|
|
|
|
// so they are empty. When a goroutine grabs a new span into mcache, it sweeps it.
|
|
|
|
|
// When a goroutine explicitly frees an object or sets a finalizer, it ensures that
|
|
|
|
|
// the span is swept (either by sweeping it, or by waiting for the concurrent sweep to finish).
|
|
|
|
|
// The finalizer goroutine is kicked off only when all spans are swept.
|
|
|
|
|
// When the next GC starts, it sweeps all not-yet-swept spans (if any).
|
2009-01-26 18:37:05 -07:00
|
|
|
|
|
|
|
|
|
#include "runtime.h"
|
2011-12-16 13:33:58 -07:00
|
|
|
|
#include "arch_GOARCH.h"
|
2009-01-26 18:37:05 -07:00
|
|
|
|
#include "malloc.h"
|
runtime: stack split + garbage collection bug
The g->sched.sp saved stack pointer and the
g->stackbase and g->stackguard stack bounds
can change even while "the world is stopped",
because a goroutine has to call functions (and
therefore might split its stack) when exiting a
system call to check whether the world is stopped
(and if so, wait until the world continues).
That means the garbage collector cannot access
those values safely (without a race) for goroutines
executing system calls. Instead, save a consistent
triple in g->gcsp, g->gcstack, g->gcguard during
entersyscall and have the garbage collector refer
to those.
The old code was occasionally seeing (because of
the race) an sp and stk that did not correspond to
each other, so that stk - sp was not the number of
stack bytes following sp. In that case, if sp < stk
then the call scanblock(sp, stk - sp) scanned too
many bytes (anything between the two pointers,
which pointed into different allocation blocks).
If sp > stk then stk - sp wrapped around.
On 32-bit, stk - sp is a uintptr (uint32) converted
to int64 in the call to scanblock, so a large (~4G)
but positive number. Scanblock would try to scan
that many bytes and eventually fault accessing
unmapped memory. On 64-bit, stk - sp is a uintptr (uint64)
promoted to int64 in the call to scanblock, so a negative
number. Scanblock would not scan anything, possibly
causing in-use blocks to be freed.
In short, 32-bit platforms would have seen either
ineffective garbage collection or crashes during garbage
collection, while 64-bit platforms would have seen
either ineffective or incorrect garbage collection.
You can see the invalid arguments to scanblock in the
stack traces in issue 1620.
Fixes #1620.
Fixes #1746.
R=iant, r
CC=golang-dev
https://golang.org/cl/4437075
2011-04-27 21:21:12 -06:00
|
|
|
|
#include "stack.h"
|
2012-12-16 17:32:12 -07:00
|
|
|
|
#include "mgc0.h"
|
reflect, runtime: fix crash in GC due to reflect.call + precise GC
Given
type Outer struct {
*Inner
...
}
the compiler generates the implementation of (*Outer).M dispatching to
the embedded Inner. The implementation is logically:
func (p *Outer) M() {
(p.Inner).M()
}
but since the only change here is the replacement of one pointer
receiver with another, the actual generated code overwrites the
original receiver with the p.Inner pointer and then jumps to the M
method expecting the *Inner receiver.
During reflect.Value.Call, we create an argument frame and the
associated data structures to describe it to the garbage collector,
populate the frame, call reflect.call to run a function call using
that frame, and then copy the results back out of the frame. The
reflect.call function does a memmove of the frame structure onto the
stack (to set up the inputs), runs the call, and the memmoves the
stack back to the frame structure (to preserve the outputs).
Originally reflect.call did not distinguish inputs from outputs: both
memmoves were for the full stack frame. However, in the case where the
called function was one of these wrappers, the rewritten receiver is
almost certainly a different type than the original receiver. This is
not a problem on the stack, where we use the program counter to
determine the type information and understand that during (*Outer).M
the receiver is an *Outer while during (*Inner).M the receiver in the
same memory word is now an *Inner. But in the statically typed
argument frame created by reflect, the receiver is always an *Outer.
Copying the modified receiver pointer off the stack into the frame
will store an *Inner there, and then if a garbage collection happens
to scan that argument frame before it is discarded, it will scan the
*Inner memory as if it were an *Outer. If the two have different
memory layouts, the collection will intepret the memory incorrectly.
Fix by only copying back the results.
Fixes #7725.
LGTM=khr
R=khr
CC=dave, golang-codereviews
https://golang.org/cl/85180043
2014-04-08 09:11:35 -06:00
|
|
|
|
#include "chan.h"
|
2012-10-07 12:05:32 -06:00
|
|
|
|
#include "race.h"
|
2013-01-10 13:45:46 -07:00
|
|
|
|
#include "type.h"
|
|
|
|
|
#include "typekind.h"
|
2013-07-19 14:04:09 -06:00
|
|
|
|
#include "funcdata.h"
|
2014-09-04 21:05:18 -06:00
|
|
|
|
#include "textflag.h"
|
2009-01-26 18:37:05 -07:00
|
|
|
|
|
|
|
|
|
enum {
|
2014-07-29 01:01:02 -06:00
|
|
|
|
Debug = 0,
|
|
|
|
|
ConcurrentSweep = 1,
|
|
|
|
|
PreciseScan = 1,
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
WorkbufSize = 4*1024,
|
2014-01-30 02:28:19 -07:00
|
|
|
|
FinBlockSize = 4*1024,
|
2014-01-21 02:06:57 -07:00
|
|
|
|
RootData = 0,
|
|
|
|
|
RootBss = 1,
|
|
|
|
|
RootFinalizers = 2,
|
2014-07-29 01:01:02 -06:00
|
|
|
|
RootSpans = 3,
|
2014-01-21 02:06:57 -07:00
|
|
|
|
RootFlushCaches = 4,
|
|
|
|
|
RootCount = 5,
|
2009-01-26 18:37:05 -07:00
|
|
|
|
};
|
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
#define ScanConservatively ((byte*)1)
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
|
|
|
|
|
// Initialized from $GOGC. GOGC=off means no gc.
|
2014-08-18 06:42:24 -06:00
|
|
|
|
extern int32 runtime·gcpercent;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
|
2012-02-22 19:45:01 -07:00
|
|
|
|
// Holding worldsema grants an M the right to try to stop the world.
|
|
|
|
|
// The procedure is:
|
|
|
|
|
//
|
|
|
|
|
// runtime·semacquire(&runtime·worldsema);
|
|
|
|
|
// m->gcing = 1;
|
|
|
|
|
// runtime·stoptheworld();
|
|
|
|
|
//
|
|
|
|
|
// ... do stuff ...
|
|
|
|
|
//
|
|
|
|
|
// m->gcing = 0;
|
|
|
|
|
// runtime·semrelease(&runtime·worldsema);
|
|
|
|
|
// runtime·starttheworld();
|
|
|
|
|
//
|
|
|
|
|
uint32 runtime·worldsema = 1;
|
|
|
|
|
|
2011-02-02 21:03:47 -07:00
|
|
|
|
typedef struct Workbuf Workbuf;
|
|
|
|
|
struct Workbuf
|
2010-09-07 07:57:22 -06:00
|
|
|
|
{
|
2014-07-29 01:01:02 -06:00
|
|
|
|
LFNode node; // must be first
|
|
|
|
|
uintptr nobj;
|
|
|
|
|
byte* obj[(WorkbufSize-sizeof(LFNode)-sizeof(uintptr))/PtrSize];
|
2010-09-07 07:57:22 -06:00
|
|
|
|
};
|
|
|
|
|
|
2014-08-27 18:15:05 -06:00
|
|
|
|
extern byte runtime·data[];
|
|
|
|
|
extern byte runtime·edata[];
|
|
|
|
|
extern byte runtime·bss[];
|
|
|
|
|
extern byte runtime·ebss[];
|
2009-01-26 18:37:05 -07:00
|
|
|
|
|
2014-08-27 18:15:05 -06:00
|
|
|
|
extern byte runtime·gcdata[];
|
|
|
|
|
extern byte runtime·gcbss[];
|
2012-12-16 17:32:12 -07:00
|
|
|
|
|
2014-08-28 14:23:10 -06:00
|
|
|
|
Mutex runtime·finlock; // protects the following variables
|
|
|
|
|
G* runtime·fing; // goroutine that runs finalizers
|
|
|
|
|
FinBlock* runtime·finq; // list of finalizers that are to be executed
|
|
|
|
|
FinBlock* runtime·finc; // cache of free blocks
|
2014-03-26 05:11:36 -06:00
|
|
|
|
bool runtime·fingwait;
|
|
|
|
|
bool runtime·fingwake;
|
2014-08-28 14:23:10 -06:00
|
|
|
|
static FinBlock *allfin; // list of all blocks
|
|
|
|
|
|
2014-08-25 12:38:19 -06:00
|
|
|
|
BitVector runtime·gcdatamask;
|
|
|
|
|
BitVector runtime·gcbssmask;
|
2014-03-26 05:11:36 -06:00
|
|
|
|
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
extern Mutex runtime·gclock;
|
2010-04-07 21:38:02 -06:00
|
|
|
|
|
2011-02-02 21:03:47 -07:00
|
|
|
|
static Workbuf* getempty(Workbuf*);
|
|
|
|
|
static Workbuf* getfull(Workbuf*);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
static void putempty(Workbuf*);
|
|
|
|
|
static Workbuf* handoff(Workbuf*);
|
2013-03-21 02:48:02 -06:00
|
|
|
|
static void gchelperstart(void);
|
2014-01-21 02:06:57 -07:00
|
|
|
|
static void flushallmcaches(void);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
static bool scanframe(Stkframe *frame, void *unused);
|
|
|
|
|
static void scanstack(G *gp);
|
2014-08-25 12:38:19 -06:00
|
|
|
|
static BitVector unrollglobgcprog(byte *prog, uintptr size);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
void runtime·bgsweep(void);
|
|
|
|
|
static FuncVal bgsweepv = {runtime·bgsweep};
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
static struct {
|
2012-05-24 00:55:50 -06:00
|
|
|
|
uint64 full; // lock-free list of full blocks
|
|
|
|
|
uint64 empty; // lock-free list of empty blocks
|
|
|
|
|
byte pad0[CacheLineSize]; // prevents false-sharing between full/empty and nproc/nwait
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
uint32 nproc;
|
2014-01-16 01:54:46 -07:00
|
|
|
|
int64 tstart;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
volatile uint32 nwait;
|
|
|
|
|
volatile uint32 ndone;
|
|
|
|
|
Note alldone;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
ParFor* markfor;
|
2014-08-24 02:05:07 -06:00
|
|
|
|
|
|
|
|
|
// Copy of mheap.allspans for marker or sweeper.
|
|
|
|
|
MSpan** spans;
|
|
|
|
|
uint32 nspan;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
} work;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
|
2012-12-16 17:32:12 -07:00
|
|
|
|
// scanblock scans a block of n bytes starting at pointer b for references
|
|
|
|
|
// to other objects, scanning any it finds recursively until there are no
|
|
|
|
|
// unscanned objects left. Instead of using an explicit recursion, it keeps
|
|
|
|
|
// a work list in the Workbuf* structures and loops in the main function
|
|
|
|
|
// body. Keeping an explicit work list is easier on the stack allocator and
|
|
|
|
|
// more efficient.
|
|
|
|
|
static void
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanblock(byte *b, uintptr n, byte *ptrmask)
|
2012-12-16 17:32:12 -07:00
|
|
|
|
{
|
2014-08-19 07:38:00 -06:00
|
|
|
|
byte *obj, *p, *arena_start, *arena_used, **wp, *scanbuf[8], *ptrbitp, *bitp, bits, xbits, shift, cached;
|
|
|
|
|
uintptr i, nobj, size, idx, x, off, scanbufpos;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
intptr ncached;
|
|
|
|
|
Workbuf *wbuf;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
Iface *iface;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
Eface *eface;
|
|
|
|
|
Type *typ;
|
|
|
|
|
MSpan *s;
|
cmd/cc, runtime: preserve C runtime type names in generated Go
uintptr or uint64 in the runtime C were turning into uint in the Go,
bool was turning into uint8, and so on. Fix that.
Also delete Go wrappers for C functions.
The C functions can be called directly now
(but still eventually need to be converted to Go).
LGTM=bradfitz, minux, iant
R=golang-codereviews, bradfitz, iant, minux
CC=golang-codereviews, khr, r
https://golang.org/cl/138740043
2014-08-27 19:59:49 -06:00
|
|
|
|
pageID k;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
bool keepworking;
|
2012-12-16 17:32:12 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// Cache memory arena parameters in local vars.
|
2013-05-28 12:14:47 -06:00
|
|
|
|
arena_start = runtime·mheap.arena_start;
|
|
|
|
|
arena_used = runtime·mheap.arena_used;
|
2012-12-16 17:32:12 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
wbuf = getempty(nil);
|
|
|
|
|
nobj = wbuf->nobj;
|
|
|
|
|
wp = &wbuf->obj[nobj];
|
|
|
|
|
keepworking = b == nil;
|
|
|
|
|
scanbufpos = 0;
|
|
|
|
|
for(i = 0; i < nelem(scanbuf); i++)
|
|
|
|
|
scanbuf[i] = nil;
|
|
|
|
|
|
2014-08-19 07:38:00 -06:00
|
|
|
|
ptrbitp = nil;
|
|
|
|
|
cached = 0;
|
|
|
|
|
ncached = 0;
|
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// ptrmask can have 3 possible values:
|
|
|
|
|
// 1. nil - obtain pointer mask from GC bitmap.
|
|
|
|
|
// 2. ScanConservatively - don't use any mask, scan conservatively.
|
|
|
|
|
// 3. pointer to a compact mask (for stacks and data).
|
|
|
|
|
if(b != nil)
|
|
|
|
|
goto scanobj;
|
2012-12-16 17:32:12 -07:00
|
|
|
|
for(;;) {
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(nobj == 0) {
|
|
|
|
|
// Out of work in workbuf.
|
|
|
|
|
// First, see is there is any work in scanbuf.
|
|
|
|
|
for(i = 0; i < nelem(scanbuf); i++) {
|
|
|
|
|
b = scanbuf[scanbufpos];
|
|
|
|
|
scanbuf[scanbufpos++] = nil;
|
|
|
|
|
if(scanbufpos == nelem(scanbuf))
|
|
|
|
|
scanbufpos = 0;
|
|
|
|
|
if(b != nil) {
|
|
|
|
|
n = arena_used - b; // scan until bitBoundary or BitsDead
|
|
|
|
|
ptrmask = nil; // use GC bitmap for pointer info
|
|
|
|
|
goto scanobj;
|
2013-04-08 14:36:35 -06:00
|
|
|
|
}
|
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(!keepworking) {
|
|
|
|
|
putempty(wbuf);
|
|
|
|
|
return;
|
2013-01-18 14:56:17 -07:00
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// Refill workbuf from global queue.
|
|
|
|
|
wbuf = getfull(wbuf);
|
|
|
|
|
if(wbuf == nil)
|
|
|
|
|
return;
|
|
|
|
|
nobj = wbuf->nobj;
|
|
|
|
|
wp = &wbuf->obj[nobj];
|
2013-01-10 13:45:46 -07:00
|
|
|
|
}
|
2012-12-16 17:32:12 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// If another proc wants a pointer, give it some.
|
|
|
|
|
if(work.nwait > 0 && nobj > 4 && work.full == 0) {
|
|
|
|
|
wbuf->nobj = nobj;
|
|
|
|
|
wbuf = handoff(wbuf);
|
|
|
|
|
nobj = wbuf->nobj;
|
|
|
|
|
wp = &wbuf->obj[nobj];
|
|
|
|
|
}
|
2013-03-15 14:07:52 -06:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
wp--;
|
|
|
|
|
nobj--;
|
|
|
|
|
b = *wp;
|
|
|
|
|
n = arena_used - b; // scan until next bitBoundary or BitsDead
|
|
|
|
|
ptrmask = nil; // use GC bitmap for pointer info
|
|
|
|
|
|
|
|
|
|
scanobj:
|
|
|
|
|
if(!PreciseScan) {
|
|
|
|
|
if(ptrmask == nil) {
|
|
|
|
|
// Heap obj, obtain real size.
|
|
|
|
|
if(!runtime·mlookup(b, &p, &n, nil))
|
|
|
|
|
continue; // not an allocated obj
|
|
|
|
|
if(b != p)
|
|
|
|
|
runtime·throw("bad heap object");
|
2013-03-15 14:07:52 -06:00
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
ptrmask = ScanConservatively;
|
|
|
|
|
}
|
2014-08-19 07:38:00 -06:00
|
|
|
|
// Find bits of the beginning of the object.
|
|
|
|
|
if(ptrmask == nil) {
|
|
|
|
|
off = (uintptr*)b - (uintptr*)arena_start;
|
|
|
|
|
ptrbitp = arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = (off % wordsPerBitmapByte) * gcBits;
|
|
|
|
|
cached = *ptrbitp >> shift;
|
|
|
|
|
cached &= ~bitBoundary;
|
|
|
|
|
ncached = (8 - shift)/gcBits;
|
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
for(i = 0; i < n; i += PtrSize) {
|
|
|
|
|
obj = nil;
|
|
|
|
|
// Find bits for this word.
|
|
|
|
|
if(ptrmask == nil) {
|
|
|
|
|
// Check is we have reached end of span.
|
|
|
|
|
if((((uintptr)b+i)%PageSize) == 0 &&
|
|
|
|
|
runtime·mheap.spans[(b-arena_start)>>PageShift] != runtime·mheap.spans[(b+i-arena_start)>>PageShift])
|
|
|
|
|
break;
|
|
|
|
|
// Consult GC bitmap.
|
|
|
|
|
if(ncached <= 0) {
|
|
|
|
|
// Refill cache.
|
2014-08-19 07:38:00 -06:00
|
|
|
|
cached = *--ptrbitp;
|
|
|
|
|
ncached = 2;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
bits = cached;
|
|
|
|
|
cached >>= gcBits;
|
|
|
|
|
ncached--;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
if((bits&bitBoundary) != 0)
|
2014-07-29 01:01:02 -06:00
|
|
|
|
break; // reached beginning of the next object
|
|
|
|
|
bits = (bits>>2)&BitsMask;
|
|
|
|
|
if(bits == BitsDead)
|
|
|
|
|
break; // reached no-scan part of the object
|
|
|
|
|
} else if(ptrmask != ScanConservatively) // dense mask (stack or data)
|
|
|
|
|
bits = (ptrmask[(i/PtrSize)/4]>>(((i/PtrSize)%4)*BitsPerPointer))&BitsMask;
|
|
|
|
|
else
|
|
|
|
|
bits = BitsPointer;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(bits == BitsScalar || bits == BitsDead)
|
2013-03-15 14:07:52 -06:00
|
|
|
|
continue;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(bits == BitsPointer) {
|
|
|
|
|
obj = *(byte**)(b+i);
|
|
|
|
|
goto markobj;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
}
|
2014-08-25 12:38:19 -06:00
|
|
|
|
|
|
|
|
|
// With those three out of the way, must be multi-word.
|
|
|
|
|
if(bits != BitsMultiWord)
|
|
|
|
|
runtime·throw("unexpected garbage collection bits");
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// Find the next pair of bits.
|
|
|
|
|
if(ptrmask == nil) {
|
|
|
|
|
if(ncached <= 0) {
|
2014-08-19 07:38:00 -06:00
|
|
|
|
// Refill cache.
|
|
|
|
|
cached = *--ptrbitp;
|
|
|
|
|
ncached = 2;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
bits = (cached>>2)&BitsMask;
|
|
|
|
|
} else
|
|
|
|
|
bits = (ptrmask[((i+PtrSize)/PtrSize)/4]>>((((i+PtrSize)/PtrSize)%4)*BitsPerPointer))&BitsMask;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
switch(bits) {
|
2014-08-25 12:38:19 -06:00
|
|
|
|
default:
|
|
|
|
|
runtime·throw("unexpected garbage collection bits");
|
2014-07-29 01:01:02 -06:00
|
|
|
|
case BitsIface:
|
|
|
|
|
iface = (Iface*)(b+i);
|
|
|
|
|
if(iface->tab != nil) {
|
|
|
|
|
typ = iface->tab->type;
|
2014-08-18 19:13:11 -06:00
|
|
|
|
if(!(typ->kind&KindDirectIface) || !(typ->kind&KindNoPointers))
|
2014-07-29 01:01:02 -06:00
|
|
|
|
obj = iface->data;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
break;
|
|
|
|
|
case BitsEface:
|
|
|
|
|
eface = (Eface*)(b+i);
|
|
|
|
|
typ = eface->type;
|
|
|
|
|
if(typ != nil) {
|
2014-08-18 19:13:11 -06:00
|
|
|
|
if(!(typ->kind&KindDirectIface) || !(typ->kind&KindNoPointers))
|
2014-07-29 01:01:02 -06:00
|
|
|
|
obj = eface->data;
|
2012-12-16 17:32:12 -07:00
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
break;
|
2012-12-16 17:32:12 -07:00
|
|
|
|
}
|
2013-01-10 13:45:46 -07:00
|
|
|
|
|
2014-08-25 12:38:19 -06:00
|
|
|
|
i += PtrSize;
|
|
|
|
|
cached >>= gcBits;
|
|
|
|
|
ncached--;
|
2012-12-16 17:32:12 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
markobj:
|
|
|
|
|
// At this point we have extracted the next potential pointer.
|
|
|
|
|
// Check if it points into heap.
|
|
|
|
|
if(obj == nil || obj < arena_start || obj >= arena_used)
|
2013-03-19 12:51:03 -06:00
|
|
|
|
continue;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// Mark the object.
|
|
|
|
|
off = (uintptr*)obj - (uintptr*)arena_start;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
bitp = arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = (off % wordsPerBitmapByte) * gcBits;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
xbits = *bitp;
|
|
|
|
|
bits = (xbits >> shift) & bitMask;
|
2014-08-13 10:42:55 -06:00
|
|
|
|
if((bits&bitBoundary) == 0) {
|
2014-08-19 07:38:00 -06:00
|
|
|
|
// Not a beginning of a block, consult span table to find the block beginning.
|
2014-07-29 01:01:02 -06:00
|
|
|
|
k = (uintptr)obj>>PageShift;
|
|
|
|
|
x = k;
|
|
|
|
|
x -= (uintptr)arena_start>>PageShift;
|
|
|
|
|
s = runtime·mheap.spans[x];
|
|
|
|
|
if(s == nil || k < s->start || obj >= s->limit || s->state != MSpanInUse)
|
2013-03-19 12:51:03 -06:00
|
|
|
|
continue;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
p = (byte*)((uintptr)s->start<<PageShift);
|
|
|
|
|
if(s->sizeclass != 0) {
|
|
|
|
|
size = s->elemsize;
|
|
|
|
|
idx = ((byte*)obj - p)/size;
|
|
|
|
|
p = p+idx*size;
|
2013-03-19 12:51:03 -06:00
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(p == obj) {
|
2014-08-13 10:42:55 -06:00
|
|
|
|
runtime·printf("runtime: failed to find block beginning for %p s=%p s->limit=%p\n",
|
|
|
|
|
p, s->start*PageSize, s->limit);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
runtime·throw("failed to find block beginning");
|
|
|
|
|
}
|
|
|
|
|
obj = p;
|
|
|
|
|
goto markobj;
|
2013-03-19 12:51:03 -06:00
|
|
|
|
}
|
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// Now we have bits, bitp, and shift correct for
|
|
|
|
|
// obj pointing at the base of the object.
|
2014-08-13 10:42:55 -06:00
|
|
|
|
// Only care about not marked objects.
|
|
|
|
|
if((bits&bitMarked) != 0)
|
2014-07-29 01:01:02 -06:00
|
|
|
|
continue;
|
2014-08-14 11:38:24 -06:00
|
|
|
|
// If obj size is greater than 8, then each byte of GC bitmap
|
|
|
|
|
// contains info for at most one object. In such case we use
|
|
|
|
|
// non-atomic byte store to mark the object. This can lead
|
|
|
|
|
// to double enqueue of the object for scanning, but scanning
|
|
|
|
|
// is an idempotent operation, so it is OK. This cannot lead
|
|
|
|
|
// to bitmap corruption because the single marked bit is the
|
|
|
|
|
// only thing that can change in the byte.
|
|
|
|
|
// For 8-byte objects we use non-atomic store, if the other
|
|
|
|
|
// quadruple is already marked. Otherwise we resort to CAS
|
|
|
|
|
// loop for marking.
|
2014-08-19 07:38:00 -06:00
|
|
|
|
if((xbits&(bitMask|(bitMask<<gcBits))) != (bitBoundary|(bitBoundary<<gcBits)) ||
|
2014-08-15 23:07:55 -06:00
|
|
|
|
work.nproc == 1)
|
2014-08-19 07:38:00 -06:00
|
|
|
|
*bitp = xbits | (bitMarked<<shift);
|
|
|
|
|
else
|
|
|
|
|
runtime·atomicor8(bitp, bitMarked<<shift);
|
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(((xbits>>(shift+2))&BitsMask) == BitsDead)
|
|
|
|
|
continue; // noscan object
|
|
|
|
|
|
|
|
|
|
// Queue the obj for scanning.
|
|
|
|
|
PREFETCH(obj);
|
|
|
|
|
obj = (byte*)((uintptr)obj & ~(PtrSize-1));
|
|
|
|
|
p = scanbuf[scanbufpos];
|
|
|
|
|
scanbuf[scanbufpos++] = obj;
|
|
|
|
|
if(scanbufpos == nelem(scanbuf))
|
|
|
|
|
scanbufpos = 0;
|
|
|
|
|
if(p == nil)
|
|
|
|
|
continue;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// If workbuf is full, obtain an empty one.
|
|
|
|
|
if(nobj >= nelem(wbuf->obj)) {
|
|
|
|
|
wbuf->nobj = nobj;
|
|
|
|
|
wbuf = getempty(wbuf);
|
|
|
|
|
nobj = wbuf->nobj;
|
|
|
|
|
wp = &wbuf->obj[nobj];
|
|
|
|
|
}
|
|
|
|
|
*wp = p;
|
|
|
|
|
wp++;
|
|
|
|
|
nobj++;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
}
|
2012-12-16 17:32:12 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(Debug && ptrmask == nil) {
|
|
|
|
|
// For heap objects ensure that we did not overscan.
|
|
|
|
|
n = 0;
|
|
|
|
|
p = nil;
|
|
|
|
|
if(!runtime·mlookup(b, &p, &n, nil) || b != p || i > n) {
|
|
|
|
|
runtime·printf("runtime: scanned (%p,%p), heap object (%p,%p)\n", b, i, p, n);
|
|
|
|
|
runtime·throw("scanblock: scanned invalid object");
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
}
|
2009-01-26 18:37:05 -07:00
|
|
|
|
}
|
2011-02-02 21:03:47 -07:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2012-05-24 00:55:50 -06:00
|
|
|
|
static void
|
|
|
|
|
markroot(ParFor *desc, uint32 i)
|
|
|
|
|
{
|
2014-01-21 02:06:57 -07:00
|
|
|
|
FinBlock *fb;
|
2014-08-24 02:05:07 -06:00
|
|
|
|
MSpan *s;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
uint32 spanidx, sg;
|
2014-01-21 02:06:57 -07:00
|
|
|
|
G *gp;
|
2014-01-24 11:35:11 -07:00
|
|
|
|
void *p;
|
2014-08-27 09:15:47 -06:00
|
|
|
|
uint32 status;
|
2014-09-03 10:06:36 -06:00
|
|
|
|
bool restart;
|
2012-12-16 17:32:12 -07:00
|
|
|
|
|
2012-05-24 00:55:50 -06:00
|
|
|
|
USED(&desc);
|
2014-03-25 16:09:49 -06:00
|
|
|
|
// Note: if you add a case here, please also update heapdump.c:dumproots.
|
2014-01-21 02:06:57 -07:00
|
|
|
|
switch(i) {
|
|
|
|
|
case RootData:
|
2014-08-27 18:15:05 -06:00
|
|
|
|
scanblock(runtime·data, runtime·edata - runtime·data, (byte*)runtime·gcdatamask.data);
|
2014-01-21 02:06:57 -07:00
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
case RootBss:
|
2014-08-27 18:15:05 -06:00
|
|
|
|
scanblock(runtime·bss, runtime·ebss - runtime·bss, (byte*)runtime·gcbssmask.data);
|
2014-01-21 02:06:57 -07:00
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
case RootFinalizers:
|
|
|
|
|
for(fb=allfin; fb; fb=fb->alllink)
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanblock((byte*)fb->fin, fb->cnt*sizeof(fb->fin[0]), ScanConservatively);
|
2014-01-21 02:06:57 -07:00
|
|
|
|
break;
|
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
case RootSpans:
|
|
|
|
|
// mark MSpan.specials
|
2014-08-24 02:05:07 -06:00
|
|
|
|
sg = runtime·mheap.sweepgen;
|
|
|
|
|
for(spanidx=0; spanidx<work.nspan; spanidx++) {
|
2014-01-21 02:06:57 -07:00
|
|
|
|
Special *sp;
|
|
|
|
|
SpecialFinalizer *spf;
|
|
|
|
|
|
2014-08-24 02:05:07 -06:00
|
|
|
|
s = work.spans[spanidx];
|
undo CL 101570044 / 2c57aaea79c4
redo stack allocation. This is mostly the same as
the original CL with a few bug fixes.
1. add racemalloc() for stack allocations
2. fix poolalloc/poolfree to terminate free lists correctly.
3. adjust span ref count correctly.
4. don't use cache for sizes >= StackCacheSize.
Should fix bugs and memory leaks in original changelist.
««« original CL description
undo CL 104200047 / 318b04f28372
Breaks windows and race detector.
TBR=rsc
««« original CL description
runtime: stack allocator, separate from mallocgc
In order to move malloc to Go, we need to have a
separate stack allocator. If we run out of stack
during malloc, malloc will not be available
to allocate a new stack.
Stacks are the last remaining FlagNoGC objects in the
GC heap. Once they are out, we can get rid of the
distinction between the allocated/blockboundary bits.
(This will be in a separate change.)
Fixes #7468
Fixes #7424
LGTM=rsc, dvyukov
R=golang-codereviews, dvyukov, khr, dave, rsc
CC=golang-codereviews
https://golang.org/cl/104200047
»»»
TBR=rsc
CC=golang-codereviews
https://golang.org/cl/101570044
»»»
LGTM=dvyukov
R=dvyukov, dave, khr, alex.brainman
CC=golang-codereviews
https://golang.org/cl/112240044
2014-07-17 15:41:46 -06:00
|
|
|
|
if(s->state != MSpanInUse)
|
|
|
|
|
continue;
|
2014-02-13 09:10:31 -07:00
|
|
|
|
if(s->sweepgen != sg) {
|
|
|
|
|
runtime·printf("sweep %d %d\n", s->sweepgen, sg);
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
runtime·throw("gc: unswept span");
|
2014-02-13 09:10:31 -07:00
|
|
|
|
}
|
2014-01-21 02:06:57 -07:00
|
|
|
|
for(sp = s->specials; sp != nil; sp = sp->next) {
|
|
|
|
|
if(sp->kind != KindSpecialFinalizer)
|
|
|
|
|
continue;
|
|
|
|
|
// don't mark finalized object, but scan it so we
|
|
|
|
|
// retain everything it points to.
|
|
|
|
|
spf = (SpecialFinalizer*)sp;
|
2014-01-24 11:35:11 -07:00
|
|
|
|
// A finalizer can be set for an inner byte of an object, find object beginning.
|
2014-08-07 07:00:02 -06:00
|
|
|
|
p = (void*)((s->start << PageShift) + spf->special.offset/s->elemsize*s->elemsize);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanblock(p, s->elemsize, nil);
|
|
|
|
|
scanblock((void*)&spf->fn, PtrSize, ScanConservatively);
|
2014-01-21 02:06:57 -07:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
case RootFlushCaches:
|
|
|
|
|
flushallmcaches();
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
default:
|
|
|
|
|
// the rest is scanning goroutine stacks
|
|
|
|
|
if(i - RootCount >= runtime·allglen)
|
|
|
|
|
runtime·throw("markroot: bad index");
|
|
|
|
|
gp = runtime·allg[i - RootCount];
|
|
|
|
|
// remember when we've first observed the G blocked
|
|
|
|
|
// needed only to output in traceback
|
2014-08-27 09:15:47 -06:00
|
|
|
|
status = runtime·readgstatus(gp);
|
|
|
|
|
if((status == Gwaiting || status == Gsyscall) && gp->waitsince == 0)
|
2014-01-21 02:06:57 -07:00
|
|
|
|
gp->waitsince = work.tstart;
|
2014-08-07 02:55:28 -06:00
|
|
|
|
// Shrink a stack if not much of it is being used.
|
|
|
|
|
runtime·shrinkstack(gp);
|
2014-09-03 10:06:36 -06:00
|
|
|
|
if(runtime·readgstatus(gp) == Gdead)
|
|
|
|
|
gp->gcworkdone = true;
|
|
|
|
|
else
|
|
|
|
|
gp->gcworkdone = false;
|
|
|
|
|
restart = runtime·stopg(gp);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanstack(gp);
|
2014-09-03 10:06:36 -06:00
|
|
|
|
if(restart)
|
|
|
|
|
runtime·restartg(gp);
|
2014-01-21 02:06:57 -07:00
|
|
|
|
break;
|
|
|
|
|
}
|
2012-05-24 00:55:50 -06:00
|
|
|
|
}
|
|
|
|
|
|
2011-02-02 21:03:47 -07:00
|
|
|
|
// Get an empty work buffer off the work.empty list,
|
|
|
|
|
// allocating new buffers as needed.
|
|
|
|
|
static Workbuf*
|
|
|
|
|
getempty(Workbuf *b)
|
|
|
|
|
{
|
2014-08-05 15:50:37 -06:00
|
|
|
|
MCache *c;
|
|
|
|
|
|
2012-05-24 00:55:50 -06:00
|
|
|
|
if(b != nil)
|
|
|
|
|
runtime·lfstackpush(&work.full, &b->node);
|
2014-08-05 15:50:37 -06:00
|
|
|
|
b = nil;
|
|
|
|
|
c = g->m->mcache;
|
|
|
|
|
if(c->gcworkbuf != nil) {
|
|
|
|
|
b = c->gcworkbuf;
|
|
|
|
|
c->gcworkbuf = nil;
|
|
|
|
|
}
|
|
|
|
|
if(b == nil)
|
|
|
|
|
b = (Workbuf*)runtime·lfstackpop(&work.empty);
|
2014-07-21 15:56:01 -06:00
|
|
|
|
if(b == nil)
|
|
|
|
|
b = runtime·persistentalloc(sizeof(*b), CacheLineSize, &mstats.gc_sys);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
b->nobj = 0;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
return b;
|
|
|
|
|
}
|
|
|
|
|
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
static void
|
|
|
|
|
putempty(Workbuf *b)
|
|
|
|
|
{
|
2014-08-05 15:50:37 -06:00
|
|
|
|
MCache *c;
|
|
|
|
|
|
|
|
|
|
c = g->m->mcache;
|
|
|
|
|
if(c->gcworkbuf == nil) {
|
|
|
|
|
c->gcworkbuf = b;
|
|
|
|
|
return;
|
|
|
|
|
}
|
2012-05-24 00:55:50 -06:00
|
|
|
|
runtime·lfstackpush(&work.empty, &b->node);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
}
|
|
|
|
|
|
2014-08-05 15:50:37 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·gcworkbuffree(void *b)
|
|
|
|
|
{
|
|
|
|
|
if(b != nil)
|
|
|
|
|
putempty(b);
|
|
|
|
|
}
|
|
|
|
|
|
2011-02-02 21:03:47 -07:00
|
|
|
|
// Get a full work buffer off the work.full list, or return nil.
|
|
|
|
|
static Workbuf*
|
|
|
|
|
getfull(Workbuf *b)
|
|
|
|
|
{
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
int32 i;
|
|
|
|
|
|
2012-05-24 00:55:50 -06:00
|
|
|
|
if(b != nil)
|
|
|
|
|
runtime·lfstackpush(&work.empty, &b->node);
|
|
|
|
|
b = (Workbuf*)runtime·lfstackpop(&work.full);
|
|
|
|
|
if(b != nil || work.nproc == 1)
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
return b;
|
|
|
|
|
|
|
|
|
|
runtime·xadd(&work.nwait, +1);
|
|
|
|
|
for(i=0;; i++) {
|
2012-05-24 00:55:50 -06:00
|
|
|
|
if(work.full != 0) {
|
|
|
|
|
runtime·xadd(&work.nwait, -1);
|
|
|
|
|
b = (Workbuf*)runtime·lfstackpop(&work.full);
|
|
|
|
|
if(b != nil)
|
|
|
|
|
return b;
|
|
|
|
|
runtime·xadd(&work.nwait, +1);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
}
|
|
|
|
|
if(work.nwait == work.nproc)
|
|
|
|
|
return nil;
|
2012-04-05 10:48:28 -06:00
|
|
|
|
if(i < 10) {
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->gcstats.nprocyield++;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
runtime·procyield(20);
|
2012-04-05 10:48:28 -06:00
|
|
|
|
} else if(i < 20) {
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->gcstats.nosyield++;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
runtime·osyield();
|
2012-04-05 10:48:28 -06:00
|
|
|
|
} else {
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->gcstats.nsleep++;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
runtime·usleep(100);
|
2012-04-05 10:48:28 -06:00
|
|
|
|
}
|
2009-01-26 18:37:05 -07:00
|
|
|
|
}
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static Workbuf*
|
|
|
|
|
handoff(Workbuf *b)
|
|
|
|
|
{
|
|
|
|
|
int32 n;
|
|
|
|
|
Workbuf *b1;
|
|
|
|
|
|
|
|
|
|
// Make new buffer with half of b's pointers.
|
|
|
|
|
b1 = getempty(nil);
|
|
|
|
|
n = b->nobj/2;
|
|
|
|
|
b->nobj -= n;
|
|
|
|
|
b1->nobj = n;
|
|
|
|
|
runtime·memmove(b1->obj, b->obj+b->nobj, n*sizeof b1->obj[0]);
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->gcstats.nhandoff++;
|
|
|
|
|
g->m->gcstats.nhandoffcnt += n;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
|
|
|
|
|
// Put b on full list - let first half of b get stolen.
|
2012-05-24 00:55:50 -06:00
|
|
|
|
runtime·lfstackpush(&work.full, &b->node);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
return b1;
|
2009-01-26 18:37:05 -07:00
|
|
|
|
}
|
|
|
|
|
|
2014-04-02 14:49:27 -06:00
|
|
|
|
BitVector
|
runtime: grow stack by copying
On stack overflow, if all frames on the stack are
copyable, we copy the frames to a new stack twice
as large as the old one. During GC, if a G is using
less than 1/4 of its stack, copy the stack to a stack
half its size.
TODO
- Do something about C frames. When a C frame is in the
stack segment, it isn't copyable. We allocate a new segment
in this case.
- For idempotent C code, we can abort it, copy the stack,
then retry. I'm working on a separate CL for this.
- For other C code, we can raise the stackguard
to the lowest Go frame so the next call that Go frame
makes triggers a copy, which will then succeed.
- Pick a starting stack size?
The plan is that eventually we reach a point where the
stack contains only copyable frames.
LGTM=rsc
R=dvyukov, rsc
CC=golang-codereviews
https://golang.org/cl/54650044
2014-02-27 00:28:44 -07:00
|
|
|
|
runtime·stackmapdata(StackMap *stackmap, int32 n)
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
{
|
2014-04-02 14:49:27 -06:00
|
|
|
|
if(n < 0 || n >= stackmap->n)
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
runtime·throw("stackmapdata: index out of range");
|
2014-04-02 14:49:27 -06:00
|
|
|
|
return (BitVector){stackmap->nbit, stackmap->data + n*((stackmap->nbit+31)/32)};
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Scan a stack frame: local variables and function arguments/results.
|
runtime: grow stack by copying
On stack overflow, if all frames on the stack are
copyable, we copy the frames to a new stack twice
as large as the old one. During GC, if a G is using
less than 1/4 of its stack, copy the stack to a stack
half its size.
TODO
- Do something about C frames. When a C frame is in the
stack segment, it isn't copyable. We allocate a new segment
in this case.
- For idempotent C code, we can abort it, copy the stack,
then retry. I'm working on a separate CL for this.
- For other C code, we can raise the stackguard
to the lowest Go frame so the next call that Go frame
makes triggers a copy, which will then succeed.
- Pick a starting stack size?
The plan is that eventually we reach a point where the
stack contains only copyable frames.
LGTM=rsc
R=dvyukov, rsc
CC=golang-codereviews
https://golang.org/cl/54650044
2014-02-27 00:28:44 -07:00
|
|
|
|
static bool
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanframe(Stkframe *frame, void *unused)
|
2013-03-28 15:36:23 -06:00
|
|
|
|
{
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
Func *f;
|
|
|
|
|
StackMap *stackmap;
|
2014-04-02 14:49:27 -06:00
|
|
|
|
BitVector bv;
|
2013-08-07 13:47:01 -06:00
|
|
|
|
uintptr size;
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
uintptr targetpc;
|
|
|
|
|
int32 pcdata;
|
2013-03-28 15:36:23 -06:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
USED(unused);
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
f = frame->fn;
|
2014-05-31 08:10:12 -06:00
|
|
|
|
targetpc = frame->continpc;
|
|
|
|
|
if(targetpc == 0) {
|
|
|
|
|
// Frame is dead.
|
|
|
|
|
return true;
|
|
|
|
|
}
|
2014-07-30 10:01:52 -06:00
|
|
|
|
if(Debug > 1)
|
|
|
|
|
runtime·printf("scanframe %s\n", runtime·funcname(f));
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
if(targetpc != f->entry)
|
|
|
|
|
targetpc--;
|
|
|
|
|
pcdata = runtime·pcdatavalue(f, PCDATA_StackMapIndex, targetpc);
|
|
|
|
|
if(pcdata == -1) {
|
|
|
|
|
// We do not have a valid pcdata value but there might be a
|
|
|
|
|
// stackmap for this function. It is likely that we are looking
|
|
|
|
|
// at the function prologue, assume so and hope for the best.
|
|
|
|
|
pcdata = 0;
|
|
|
|
|
}
|
|
|
|
|
|
2013-06-12 06:49:38 -06:00
|
|
|
|
// Scan local variables if stack frame has been allocated.
|
2013-08-07 13:47:01 -06:00
|
|
|
|
// Use pointer information if known.
|
2014-06-19 23:04:10 -06:00
|
|
|
|
stackmap = runtime·funcdata(f, FUNCDATA_LocalsPointerMaps);
|
|
|
|
|
if(stackmap == nil) {
|
|
|
|
|
// No locals information, scan everything.
|
2014-09-01 08:05:16 -06:00
|
|
|
|
size = frame->varp - frame->sp;
|
2014-06-19 23:04:10 -06:00
|
|
|
|
if(Debug > 2)
|
2014-09-01 08:05:16 -06:00
|
|
|
|
runtime·printf("frame %s unsized locals %p+%p\n", runtime·funcname(f), (byte*)(frame->varp-size), size);
|
|
|
|
|
scanblock((byte*)(frame->varp - size), size, ScanConservatively);
|
2014-06-19 23:04:10 -06:00
|
|
|
|
} else if(stackmap->n < 0) {
|
|
|
|
|
// Locals size information, scan just the locals.
|
|
|
|
|
size = -stackmap->n;
|
|
|
|
|
if(Debug > 2)
|
2014-09-01 08:05:16 -06:00
|
|
|
|
runtime·printf("frame %s conservative locals %p+%p\n", runtime·funcname(f), (byte*)(frame->varp-size), size);
|
|
|
|
|
scanblock((byte*)(frame->varp - size), size, ScanConservatively);
|
2014-06-19 23:04:10 -06:00
|
|
|
|
} else if(stackmap->n > 0) {
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// Locals bitmap information, scan just the pointers in locals.
|
2014-06-19 23:04:10 -06:00
|
|
|
|
if(pcdata < 0 || pcdata >= stackmap->n) {
|
|
|
|
|
// don't know where we are
|
|
|
|
|
runtime·printf("pcdata is %d and %d stack map entries for %s (targetpc=%p)\n",
|
|
|
|
|
pcdata, stackmap->n, runtime·funcname(f), targetpc);
|
|
|
|
|
runtime·throw("scanframe: bad symbol table");
|
2013-08-07 13:47:01 -06:00
|
|
|
|
}
|
2014-06-19 23:04:10 -06:00
|
|
|
|
bv = runtime·stackmapdata(stackmap, pcdata);
|
|
|
|
|
size = (bv.n * PtrSize) / BitsPerPointer;
|
2014-09-01 08:05:16 -06:00
|
|
|
|
scanblock((byte*)(frame->varp - size), bv.n/BitsPerPointer*PtrSize, (byte*)bv.data);
|
2013-07-19 14:04:09 -06:00
|
|
|
|
}
|
2013-06-12 06:49:38 -06:00
|
|
|
|
|
|
|
|
|
// Scan arguments.
|
|
|
|
|
// Use pointer information if known.
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
stackmap = runtime·funcdata(f, FUNCDATA_ArgsPointerMaps);
|
|
|
|
|
if(stackmap != nil) {
|
runtime: grow stack by copying
On stack overflow, if all frames on the stack are
copyable, we copy the frames to a new stack twice
as large as the old one. During GC, if a G is using
less than 1/4 of its stack, copy the stack to a stack
half its size.
TODO
- Do something about C frames. When a C frame is in the
stack segment, it isn't copyable. We allocate a new segment
in this case.
- For idempotent C code, we can abort it, copy the stack,
then retry. I'm working on a separate CL for this.
- For other C code, we can raise the stackguard
to the lowest Go frame so the next call that Go frame
makes triggers a copy, which will then succeed.
- Pick a starting stack size?
The plan is that eventually we reach a point where the
stack contains only copyable frames.
LGTM=rsc
R=dvyukov, rsc
CC=golang-codereviews
https://golang.org/cl/54650044
2014-02-27 00:28:44 -07:00
|
|
|
|
bv = runtime·stackmapdata(stackmap, pcdata);
|
2014-09-01 08:05:16 -06:00
|
|
|
|
scanblock((byte*)frame->argp, bv.n/BitsPerPointer*PtrSize, (byte*)bv.data);
|
reflect, runtime: fix crash in GC due to reflect.call + precise GC
Given
type Outer struct {
*Inner
...
}
the compiler generates the implementation of (*Outer).M dispatching to
the embedded Inner. The implementation is logically:
func (p *Outer) M() {
(p.Inner).M()
}
but since the only change here is the replacement of one pointer
receiver with another, the actual generated code overwrites the
original receiver with the p.Inner pointer and then jumps to the M
method expecting the *Inner receiver.
During reflect.Value.Call, we create an argument frame and the
associated data structures to describe it to the garbage collector,
populate the frame, call reflect.call to run a function call using
that frame, and then copy the results back out of the frame. The
reflect.call function does a memmove of the frame structure onto the
stack (to set up the inputs), runs the call, and the memmoves the
stack back to the frame structure (to preserve the outputs).
Originally reflect.call did not distinguish inputs from outputs: both
memmoves were for the full stack frame. However, in the case where the
called function was one of these wrappers, the rewritten receiver is
almost certainly a different type than the original receiver. This is
not a problem on the stack, where we use the program counter to
determine the type information and understand that during (*Outer).M
the receiver is an *Outer while during (*Inner).M the receiver in the
same memory word is now an *Inner. But in the statically typed
argument frame created by reflect, the receiver is always an *Outer.
Copying the modified receiver pointer off the stack into the frame
will store an *Inner there, and then if a garbage collection happens
to scan that argument frame before it is discarded, it will scan the
*Inner memory as if it were an *Outer. If the two have different
memory layouts, the collection will intepret the memory incorrectly.
Fix by only copying back the results.
Fixes #7725.
LGTM=khr
R=khr
CC=dave, golang-codereviews
https://golang.org/cl/85180043
2014-04-08 09:11:35 -06:00
|
|
|
|
} else {
|
|
|
|
|
if(Debug > 2)
|
|
|
|
|
runtime·printf("frame %s conservative args %p+%p\n", runtime·funcname(f), frame->argp, (uintptr)frame->arglen);
|
2014-09-01 08:05:16 -06:00
|
|
|
|
scanblock((byte*)frame->argp, frame->arglen, ScanConservatively);
|
reflect, runtime: fix crash in GC due to reflect.call + precise GC
Given
type Outer struct {
*Inner
...
}
the compiler generates the implementation of (*Outer).M dispatching to
the embedded Inner. The implementation is logically:
func (p *Outer) M() {
(p.Inner).M()
}
but since the only change here is the replacement of one pointer
receiver with another, the actual generated code overwrites the
original receiver with the p.Inner pointer and then jumps to the M
method expecting the *Inner receiver.
During reflect.Value.Call, we create an argument frame and the
associated data structures to describe it to the garbage collector,
populate the frame, call reflect.call to run a function call using
that frame, and then copy the results back out of the frame. The
reflect.call function does a memmove of the frame structure onto the
stack (to set up the inputs), runs the call, and the memmoves the
stack back to the frame structure (to preserve the outputs).
Originally reflect.call did not distinguish inputs from outputs: both
memmoves were for the full stack frame. However, in the case where the
called function was one of these wrappers, the rewritten receiver is
almost certainly a different type than the original receiver. This is
not a problem on the stack, where we use the program counter to
determine the type information and understand that during (*Outer).M
the receiver is an *Outer while during (*Inner).M the receiver in the
same memory word is now an *Inner. But in the statically typed
argument frame created by reflect, the receiver is always an *Outer.
Copying the modified receiver pointer off the stack into the frame
will store an *Inner there, and then if a garbage collection happens
to scan that argument frame before it is discarded, it will scan the
*Inner memory as if it were an *Outer. If the two have different
memory layouts, the collection will intepret the memory incorrectly.
Fix by only copying back the results.
Fixes #7725.
LGTM=khr
R=khr
CC=dave, golang-codereviews
https://golang.org/cl/85180043
2014-04-08 09:11:35 -06:00
|
|
|
|
}
|
runtime: grow stack by copying
On stack overflow, if all frames on the stack are
copyable, we copy the frames to a new stack twice
as large as the old one. During GC, if a G is using
less than 1/4 of its stack, copy the stack to a stack
half its size.
TODO
- Do something about C frames. When a C frame is in the
stack segment, it isn't copyable. We allocate a new segment
in this case.
- For idempotent C code, we can abort it, copy the stack,
then retry. I'm working on a separate CL for this.
- For other C code, we can raise the stackguard
to the lowest Go frame so the next call that Go frame
makes triggers a copy, which will then succeed.
- Pick a starting stack size?
The plan is that eventually we reach a point where the
stack contains only copyable frames.
LGTM=rsc
R=dvyukov, rsc
CC=golang-codereviews
https://golang.org/cl/54650044
2014-02-27 00:28:44 -07:00
|
|
|
|
return true;
|
2009-01-26 18:37:05 -07:00
|
|
|
|
}
|
|
|
|
|
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
static void
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanstack(G *gp)
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
{
|
|
|
|
|
M *mp;
|
runtime: convert traceback*.c to Go
The two converted files were nearly identical.
Instead of continuing that duplication, I merged them
into a single traceback.go.
Tested on arm, amd64, amd64p32, and 386.
LGTM=r
R=golang-codereviews, remyoudompheng, dave, r
CC=dvyukov, golang-codereviews, iant, khr
https://golang.org/cl/134200044
2014-09-02 13:12:53 -06:00
|
|
|
|
bool (*fn)(Stkframe*, void*);
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
|
2014-09-03 10:06:36 -06:00
|
|
|
|
if(runtime·readgstatus(gp)&Gscan == 0) {
|
|
|
|
|
runtime·printf("runtime: gp=%p, goid=%D, gp->atomicstatus=%d\n", gp, gp->goid, runtime·readgstatus(gp));
|
|
|
|
|
runtime·throw("mark - bad status");
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
switch(runtime·readgstatus(gp)&~Gscan) {
|
2014-01-21 02:06:57 -07:00
|
|
|
|
default:
|
2014-08-27 09:15:47 -06:00
|
|
|
|
runtime·printf("runtime: gp=%p, goid=%D, gp->atomicstatus=%d\n", gp, gp->goid, runtime·readgstatus(gp));
|
2014-01-21 02:06:57 -07:00
|
|
|
|
runtime·throw("mark - bad status");
|
|
|
|
|
case Gdead:
|
|
|
|
|
return;
|
|
|
|
|
case Grunning:
|
2014-08-27 09:15:47 -06:00
|
|
|
|
runtime·printf("runtime: gp=%p, goid=%D, gp->atomicstatus=%d\n", gp, gp->goid, runtime·readgstatus(gp));
|
2014-01-21 02:06:57 -07:00
|
|
|
|
runtime·throw("mark - world not stopped");
|
|
|
|
|
case Grunnable:
|
|
|
|
|
case Gsyscall:
|
|
|
|
|
case Gwaiting:
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
if(gp == g)
|
|
|
|
|
runtime·throw("can't scan our own stack");
|
|
|
|
|
if((mp = gp->m) != nil && mp->helpgc)
|
|
|
|
|
runtime·throw("can't scan gchelper stack");
|
runtime: grow stack by copying
On stack overflow, if all frames on the stack are
copyable, we copy the frames to a new stack twice
as large as the old one. During GC, if a G is using
less than 1/4 of its stack, copy the stack to a stack
half its size.
TODO
- Do something about C frames. When a C frame is in the
stack segment, it isn't copyable. We allocate a new segment
in this case.
- For idempotent C code, we can abort it, copy the stack,
then retry. I'm working on a separate CL for this.
- For other C code, we can raise the stackguard
to the lowest Go frame so the next call that Go frame
makes triggers a copy, which will then succeed.
- Pick a starting stack size?
The plan is that eventually we reach a point where the
stack contains only copyable frames.
LGTM=rsc
R=dvyukov, rsc
CC=golang-codereviews
https://golang.org/cl/54650044
2014-02-27 00:28:44 -07:00
|
|
|
|
|
runtime: assume precisestack, copystack, StackCopyAlways, ScanStackByFrames
Commit to stack copying for stack growth.
We're carrying around a surprising amount of cruft from older schemes.
I am confident that precise stack scans and stack copying are here to stay.
Delete fallback code for when precise stack info is disabled.
Delete fallback code for when copying stacks is disabled.
Delete fallback code for when StackCopyAlways is disabled.
Delete Stktop chain - there is only one stack segment now.
Delete M.moreargp, M.moreargsize, M.moreframesize, M.cret.
Delete G.writenbuf (unrelated, just dead).
Delete runtime.lessstack, runtime.oldstack.
Delete many amd64 morestack variants.
Delete initialization of morestack frame/arg sizes (shortens split prologue!).
Replace G's stackguard/stackbase/stack0/stacksize/
syscallstack/syscallguard/forkstackguard with simple stack
bounds (lo, hi).
Update liblink, runtime/cgo for adjustments to G.
LGTM=khr
R=khr, bradfitz
CC=golang-codereviews, iant, r
https://golang.org/cl/137410043
2014-09-09 11:39:57 -06:00
|
|
|
|
fn = scanframe;
|
|
|
|
|
runtime·gentraceback(~(uintptr)0, ~(uintptr)0, 0, gp, 0, nil, 0x7fffffff, &fn, nil, false);
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
}
|
|
|
|
|
|
2014-09-03 10:06:36 -06:00
|
|
|
|
// The gp has been moved to a gc safepoint. If there is gcphase specific
|
|
|
|
|
// work it is done here.
|
|
|
|
|
void
|
|
|
|
|
runtime·gcphasework(G *gp)
|
|
|
|
|
{
|
|
|
|
|
switch(runtime·gcphase) {
|
|
|
|
|
default:
|
|
|
|
|
runtime·throw("gcphasework in bad gcphase");
|
|
|
|
|
case GCoff:
|
|
|
|
|
case GCquiesce:
|
|
|
|
|
case GCstw:
|
|
|
|
|
case GCsweep:
|
|
|
|
|
// No work for now.
|
|
|
|
|
break;
|
|
|
|
|
case GCmark:
|
|
|
|
|
// Disabled until concurrent GC is implemented
|
|
|
|
|
// but indicate the scan has been done.
|
|
|
|
|
// scanstack(gp);
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
gp->gcworkdone = true;
|
|
|
|
|
}
|
|
|
|
|
|
2014-01-07 14:45:50 -07:00
|
|
|
|
void
|
|
|
|
|
runtime·queuefinalizer(byte *p, FuncVal *fn, uintptr nret, Type *fint, PtrType *ot)
|
2011-10-06 09:42:51 -06:00
|
|
|
|
{
|
|
|
|
|
FinBlock *block;
|
|
|
|
|
Finalizer *f;
|
2012-05-15 09:10:16 -06:00
|
|
|
|
|
2014-08-28 14:23:10 -06:00
|
|
|
|
runtime·lock(&runtime·finlock);
|
|
|
|
|
if(runtime·finq == nil || runtime·finq->cnt == runtime·finq->cap) {
|
|
|
|
|
if(runtime·finc == nil) {
|
|
|
|
|
runtime·finc = runtime·persistentalloc(FinBlockSize, 0, &mstats.gc_sys);
|
|
|
|
|
runtime·finc->cap = (FinBlockSize - sizeof(FinBlock)) / sizeof(Finalizer) + 1;
|
|
|
|
|
runtime·finc->alllink = allfin;
|
|
|
|
|
allfin = runtime·finc;
|
2011-10-06 09:42:51 -06:00
|
|
|
|
}
|
2014-08-28 14:23:10 -06:00
|
|
|
|
block = runtime·finc;
|
|
|
|
|
runtime·finc = block->next;
|
|
|
|
|
block->next = runtime·finq;
|
|
|
|
|
runtime·finq = block;
|
2011-10-06 09:42:51 -06:00
|
|
|
|
}
|
2014-08-28 14:23:10 -06:00
|
|
|
|
f = &runtime·finq->fin[runtime·finq->cnt];
|
|
|
|
|
runtime·finq->cnt++;
|
2011-10-06 09:42:51 -06:00
|
|
|
|
f->fn = fn;
|
|
|
|
|
f->nret = nret;
|
2013-08-14 12:54:31 -06:00
|
|
|
|
f->fint = fint;
|
2013-07-29 09:43:08 -06:00
|
|
|
|
f->ot = ot;
|
2011-10-06 09:42:51 -06:00
|
|
|
|
f->arg = p;
|
2014-03-26 05:11:36 -06:00
|
|
|
|
runtime·fingwake = true;
|
2014-08-28 14:23:10 -06:00
|
|
|
|
runtime·unlock(&runtime·finlock);
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
}
|
|
|
|
|
|
2014-03-25 16:09:49 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·iterate_finq(void (*callback)(FuncVal*, byte*, uintptr, Type*, PtrType*))
|
|
|
|
|
{
|
|
|
|
|
FinBlock *fb;
|
|
|
|
|
Finalizer *f;
|
|
|
|
|
uintptr i;
|
|
|
|
|
|
|
|
|
|
for(fb = allfin; fb; fb = fb->alllink) {
|
|
|
|
|
for(i = 0; i < fb->cnt; i++) {
|
|
|
|
|
f = &fb->fin[i];
|
|
|
|
|
callback(f->fn, f->arg, f->nret, f->fint, f->ot);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
void
|
|
|
|
|
runtime·MSpan_EnsureSwept(MSpan *s)
|
|
|
|
|
{
|
|
|
|
|
uint32 sg;
|
|
|
|
|
|
2014-02-24 09:53:20 -07:00
|
|
|
|
// Caller must disable preemption.
|
|
|
|
|
// Otherwise when this function returns the span can become unswept again
|
|
|
|
|
// (if GC is triggered on another goroutine).
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
if(g->m->locks == 0 && g->m->mallocing == 0 && g != g->m->g0)
|
2014-02-24 09:53:20 -07:00
|
|
|
|
runtime·throw("MSpan_EnsureSwept: m is not locked");
|
|
|
|
|
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
sg = runtime·mheap.sweepgen;
|
|
|
|
|
if(runtime·atomicload(&s->sweepgen) == sg)
|
|
|
|
|
return;
|
|
|
|
|
if(runtime·cas(&s->sweepgen, sg-2, sg-1)) {
|
2014-08-18 06:52:31 -06:00
|
|
|
|
runtime·MSpan_Sweep(s, false);
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
// unfortunate condition, and we don't have efficient means to wait
|
|
|
|
|
while(runtime·atomicload(&s->sweepgen) != sg)
|
2014-08-27 09:15:47 -06:00
|
|
|
|
runtime·osyield();
|
2011-10-06 09:42:51 -06:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Sweep frees or collects finalizers for blocks not marked in the mark phase.
|
2011-02-02 21:03:47 -07:00
|
|
|
|
// It clears the mark bits in preparation for the next GC round.
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
// Returns true if the span was returned to heap.
|
2014-08-18 06:52:31 -06:00
|
|
|
|
// If preserve=true, don't return it to heap nor relink in MCentral lists;
|
|
|
|
|
// caller takes care of it.
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
bool
|
2014-08-18 06:52:31 -06:00
|
|
|
|
runtime·MSpan_Sweep(MSpan *s, bool preserve)
|
2012-04-05 11:02:20 -06:00
|
|
|
|
{
|
2014-02-13 08:36:45 -07:00
|
|
|
|
int32 cl, n, npages, nfree;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
uintptr size, off, step;
|
2014-02-13 08:36:45 -07:00
|
|
|
|
uint32 sweepgen;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
byte *p, *bitp, shift, xbits, bits;
|
2012-04-05 11:02:20 -06:00
|
|
|
|
MCache *c;
|
|
|
|
|
byte *arena_start;
|
2014-08-13 10:42:55 -06:00
|
|
|
|
MLink head, *end, *link;
|
2014-01-07 14:45:50 -07:00
|
|
|
|
Special *special, **specialp, *y;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
bool res, sweepgenset;
|
2010-02-10 15:59:39 -07:00
|
|
|
|
|
2014-02-13 08:36:45 -07:00
|
|
|
|
// It's critical that we enter this function with preemption disabled,
|
|
|
|
|
// GC must not start while we are in the middle of this function.
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
if(g->m->locks == 0 && g->m->mallocing == 0 && g != g->m->g0)
|
2014-02-13 08:36:45 -07:00
|
|
|
|
runtime·throw("MSpan_Sweep: m is not locked");
|
|
|
|
|
sweepgen = runtime·mheap.sweepgen;
|
|
|
|
|
if(s->state != MSpanInUse || s->sweepgen != sweepgen-1) {
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
runtime·printf("MSpan_Sweep: state=%d sweepgen=%d mheap.sweepgen=%d\n",
|
2014-02-13 08:36:45 -07:00
|
|
|
|
s->state, s->sweepgen, sweepgen);
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
runtime·throw("MSpan_Sweep: bad span state");
|
|
|
|
|
}
|
2013-05-28 12:14:47 -06:00
|
|
|
|
arena_start = runtime·mheap.arena_start;
|
2012-04-05 11:02:20 -06:00
|
|
|
|
cl = s->sizeclass;
|
2012-09-24 18:08:05 -06:00
|
|
|
|
size = s->elemsize;
|
2012-04-05 11:02:20 -06:00
|
|
|
|
if(cl == 0) {
|
|
|
|
|
n = 1;
|
|
|
|
|
} else {
|
|
|
|
|
// Chunk full of small blocks.
|
|
|
|
|
npages = runtime·class_to_allocnpages[cl];
|
|
|
|
|
n = (npages << PageShift) / size;
|
|
|
|
|
}
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
res = false;
|
2012-04-12 02:01:24 -06:00
|
|
|
|
nfree = 0;
|
2012-09-24 18:08:05 -06:00
|
|
|
|
end = &head;
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
c = g->m->mcache;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
sweepgenset = false;
|
2013-12-18 18:13:59 -07:00
|
|
|
|
|
2014-08-13 10:42:55 -06:00
|
|
|
|
// Mark any free objects in this span so we don't collect them.
|
|
|
|
|
for(link = s->freelist; link != nil; link = link->next) {
|
|
|
|
|
off = (uintptr*)link - (uintptr*)arena_start;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
bitp = arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = (off % wordsPerBitmapByte) * gcBits;
|
2014-08-13 10:42:55 -06:00
|
|
|
|
*bitp |= bitMarked<<shift;
|
|
|
|
|
}
|
|
|
|
|
|
2014-01-07 14:45:50 -07:00
|
|
|
|
// Unlink & free special records for any objects we're about to free.
|
|
|
|
|
specialp = &s->specials;
|
|
|
|
|
special = *specialp;
|
|
|
|
|
while(special != nil) {
|
2014-01-24 11:35:11 -07:00
|
|
|
|
// A finalizer can be set for an inner byte of an object, find object beginning.
|
|
|
|
|
p = (byte*)(s->start << PageShift) + special->offset/size*size;
|
2014-01-07 14:45:50 -07:00
|
|
|
|
off = (uintptr*)p - (uintptr*)arena_start;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
bitp = arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = (off % wordsPerBitmapByte) * gcBits;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
bits = (*bitp>>shift) & bitMask;
|
2014-08-13 10:42:55 -06:00
|
|
|
|
if((bits&bitMarked) == 0) {
|
2014-01-24 11:35:11 -07:00
|
|
|
|
// Find the exact byte for which the special was setup
|
|
|
|
|
// (as opposed to object beginning).
|
|
|
|
|
p = (byte*)(s->start << PageShift) + special->offset;
|
2014-01-07 14:45:50 -07:00
|
|
|
|
// about to free object: splice out special record
|
|
|
|
|
y = special;
|
|
|
|
|
special = special->next;
|
|
|
|
|
*specialp = special;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
if(!runtime·freespecial(y, p, size, false)) {
|
2014-01-07 14:45:50 -07:00
|
|
|
|
// stop freeing of object if it has a finalizer
|
|
|
|
|
*bitp |= bitMarked << shift;
|
|
|
|
|
}
|
|
|
|
|
} else {
|
|
|
|
|
// object is still live: keep special record
|
|
|
|
|
specialp = &special->next;
|
|
|
|
|
special = *specialp;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2012-04-05 11:02:20 -06:00
|
|
|
|
// Sweep through n objects of given size starting at p.
|
|
|
|
|
// This thread owns the span now, so it can manipulate
|
|
|
|
|
// the block bitmap without atomic operations.
|
2014-01-07 14:45:50 -07:00
|
|
|
|
p = (byte*)(s->start << PageShift);
|
2014-08-19 07:38:00 -06:00
|
|
|
|
// Find bits for the beginning of the span.
|
|
|
|
|
off = (uintptr*)p - (uintptr*)arena_start;
|
|
|
|
|
bitp = arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = 0;
|
|
|
|
|
step = size/(PtrSize*wordsPerBitmapByte);
|
|
|
|
|
// Rewind to the previous quadruple as we move to the next
|
|
|
|
|
// in the beginning of the loop.
|
|
|
|
|
bitp += step;
|
|
|
|
|
if(step == 0) {
|
|
|
|
|
// 8-byte objects.
|
|
|
|
|
bitp++;
|
|
|
|
|
shift = gcBits;
|
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
for(; n > 0; n--, p += size) {
|
2014-08-19 07:38:00 -06:00
|
|
|
|
bitp -= step;
|
|
|
|
|
if(step == 0) {
|
|
|
|
|
if(shift != 0)
|
|
|
|
|
bitp--;
|
|
|
|
|
shift = gcBits - shift;
|
|
|
|
|
}
|
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
xbits = *bitp;
|
|
|
|
|
bits = (xbits>>shift) & bitMask;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// Allocated and marked object, reset bits to allocated.
|
2014-08-13 10:42:55 -06:00
|
|
|
|
if((bits&bitMarked) != 0) {
|
|
|
|
|
*bitp &= ~(bitMarked<<shift);
|
2012-04-05 11:02:20 -06:00
|
|
|
|
continue;
|
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// At this point we know that we are looking at garbage object
|
|
|
|
|
// that needs to be collected.
|
2014-04-01 11:30:10 -06:00
|
|
|
|
if(runtime·debug.allocfreetrace)
|
|
|
|
|
runtime·tracefree(p, size);
|
2014-08-13 10:42:55 -06:00
|
|
|
|
// Reset to allocated+noscan.
|
|
|
|
|
*bitp = (xbits & ~((bitMarked|(BitsMask<<2))<<shift)) | ((uintptr)BitsDead<<(shift+2));
|
2012-09-24 18:08:05 -06:00
|
|
|
|
if(cl == 0) {
|
2012-04-05 11:02:20 -06:00
|
|
|
|
// Free large span.
|
2014-08-18 06:52:31 -06:00
|
|
|
|
if(preserve)
|
|
|
|
|
runtime·throw("can't preserve large span");
|
2014-07-29 01:01:02 -06:00
|
|
|
|
runtime·unmarkspan(p, s->npages<<PageShift);
|
2014-02-13 09:10:31 -07:00
|
|
|
|
s->needzero = 1;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
// important to set sweepgen before returning it to heap
|
2014-02-13 08:36:45 -07:00
|
|
|
|
runtime·atomicstore(&s->sweepgen, sweepgen);
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
sweepgenset = true;
|
2014-07-31 02:55:40 -06:00
|
|
|
|
// NOTE(rsc,dvyukov): The original implementation of efence
|
|
|
|
|
// in CL 22060046 used SysFree instead of SysFault, so that
|
|
|
|
|
// the operating system would eventually give the memory
|
|
|
|
|
// back to us again, so that an efence program could run
|
|
|
|
|
// longer without running out of memory. Unfortunately,
|
|
|
|
|
// calling SysFree here without any kind of adjustment of the
|
|
|
|
|
// heap data structures means that when the memory does
|
|
|
|
|
// come back to us, we have the wrong metadata for it, either in
|
|
|
|
|
// the MSpan structures or in the garbage collection bitmap.
|
|
|
|
|
// Using SysFault here means that the program will run out of
|
|
|
|
|
// memory fairly quickly in efence mode, but at least it won't
|
|
|
|
|
// have mysterious crashes due to confused memory reuse.
|
|
|
|
|
// It should be possible to switch back to SysFree if we also
|
|
|
|
|
// implement and then call some kind of MHeap_DeleteSpan.
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(runtime·debug.efence) {
|
|
|
|
|
s->limit = nil; // prevent mlookup from finding this span
|
2014-03-06 16:34:29 -07:00
|
|
|
|
runtime·SysFault(p, size);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
} else
|
2013-12-06 15:40:45 -07:00
|
|
|
|
runtime·MHeap_Free(&runtime·mheap, s, 1);
|
2013-06-06 04:56:50 -06:00
|
|
|
|
c->local_nlargefree++;
|
|
|
|
|
c->local_largefree += size;
|
2014-07-30 10:01:52 -06:00
|
|
|
|
runtime·xadd64(&mstats.next_gc, -(uint64)(size * (runtime·gcpercent + 100)/100));
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
res = true;
|
2012-04-05 11:02:20 -06:00
|
|
|
|
} else {
|
|
|
|
|
// Free small object.
|
2014-01-20 23:53:51 -07:00
|
|
|
|
if(size > 2*sizeof(uintptr))
|
2013-04-04 15:18:52 -06:00
|
|
|
|
((uintptr*)p)[1] = (uintptr)0xdeaddeaddeaddeadll; // mark as "needs to be zeroed"
|
2014-01-20 23:53:51 -07:00
|
|
|
|
else if(size > sizeof(uintptr))
|
|
|
|
|
((uintptr*)p)[1] = 0;
|
|
|
|
|
|
2012-09-24 18:08:05 -06:00
|
|
|
|
end->next = (MLink*)p;
|
2012-04-12 02:01:24 -06:00
|
|
|
|
end = (MLink*)p;
|
|
|
|
|
nfree++;
|
2009-01-26 18:37:05 -07:00
|
|
|
|
}
|
2012-04-12 02:01:24 -06:00
|
|
|
|
}
|
|
|
|
|
|
2014-02-26 16:52:58 -07:00
|
|
|
|
// We need to set s->sweepgen = h->sweepgen only when all blocks are swept,
|
|
|
|
|
// because of the potential for a concurrent free/SetFinalizer.
|
|
|
|
|
// But we need to set it before we make the span available for allocation
|
|
|
|
|
// (return it to heap or mcentral), because allocation code assumes that a
|
|
|
|
|
// span is already swept if available for allocation.
|
|
|
|
|
|
|
|
|
|
if(!sweepgenset && nfree == 0) {
|
2014-02-13 08:36:45 -07:00
|
|
|
|
// The span must be in our exclusive ownership until we update sweepgen,
|
|
|
|
|
// check for potential races.
|
|
|
|
|
if(s->state != MSpanInUse || s->sweepgen != sweepgen-1) {
|
|
|
|
|
runtime·printf("MSpan_Sweep: state=%d sweepgen=%d mheap.sweepgen=%d\n",
|
|
|
|
|
s->state, s->sweepgen, sweepgen);
|
|
|
|
|
runtime·throw("MSpan_Sweep: bad span state after sweep");
|
|
|
|
|
}
|
|
|
|
|
runtime·atomicstore(&s->sweepgen, sweepgen);
|
|
|
|
|
}
|
2014-02-26 16:52:58 -07:00
|
|
|
|
if(nfree > 0) {
|
2013-06-06 04:56:50 -06:00
|
|
|
|
c->local_nsmallfree[cl] += nfree;
|
2012-04-12 02:01:24 -06:00
|
|
|
|
c->local_cachealloc -= nfree * size;
|
2014-07-30 10:01:52 -06:00
|
|
|
|
runtime·xadd64(&mstats.next_gc, -(uint64)(nfree * size * (runtime·gcpercent + 100)/100));
|
2014-08-18 06:52:31 -06:00
|
|
|
|
res = runtime·MCentral_FreeSpan(&runtime·mheap.central[cl].mcentral, s, nfree, head.next, end, preserve);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// MCentral_FreeSpan updates sweepgen
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
}
|
|
|
|
|
return res;
|
|
|
|
|
}
|
|
|
|
|
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
// State of background runtime·sweep.
|
|
|
|
|
// Pretected by runtime·gclock.
|
|
|
|
|
// Must match mgc0.go.
|
|
|
|
|
struct
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
{
|
|
|
|
|
G* g;
|
|
|
|
|
bool parked;
|
|
|
|
|
|
2014-08-24 02:05:07 -06:00
|
|
|
|
uint32 spanidx; // background sweeper position
|
2014-07-29 01:01:02 -06:00
|
|
|
|
|
|
|
|
|
uint32 nbgsweep;
|
|
|
|
|
uint32 npausesweep;
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
} runtime·sweep;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
|
|
|
|
|
// sweeps one span
|
|
|
|
|
// returns number of pages returned to heap, or -1 if there is nothing to sweep
|
|
|
|
|
uintptr
|
|
|
|
|
runtime·sweepone(void)
|
|
|
|
|
{
|
|
|
|
|
MSpan *s;
|
|
|
|
|
uint32 idx, sg;
|
|
|
|
|
uintptr npages;
|
|
|
|
|
|
|
|
|
|
// increment locks to ensure that the goroutine is not preempted
|
|
|
|
|
// in the middle of sweep thus leaving the span in an inconsistent state for next GC
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->locks++;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
sg = runtime·mheap.sweepgen;
|
|
|
|
|
for(;;) {
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
idx = runtime·xadd(&runtime·sweep.spanidx, 1) - 1;
|
2014-08-24 02:05:07 -06:00
|
|
|
|
if(idx >= work.nspan) {
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
runtime·mheap.sweepdone = true;
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->locks--;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
return -1;
|
|
|
|
|
}
|
2014-08-24 02:05:07 -06:00
|
|
|
|
s = work.spans[idx];
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
if(s->state != MSpanInUse) {
|
|
|
|
|
s->sweepgen = sg;
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
if(s->sweepgen != sg-2 || !runtime·cas(&s->sweepgen, sg-2, sg-1))
|
|
|
|
|
continue;
|
|
|
|
|
npages = s->npages;
|
2014-08-18 06:52:31 -06:00
|
|
|
|
if(!runtime·MSpan_Sweep(s, false))
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
npages = 0;
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->locks--;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
return npages;
|
2009-01-26 18:37:05 -07:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
static void
|
|
|
|
|
sweepone_m(void)
|
|
|
|
|
{
|
|
|
|
|
g->m->scalararg[0] = runtime·sweepone();
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#pragma textflag NOSPLIT
|
|
|
|
|
uintptr
|
|
|
|
|
runtime·gosweepone(void)
|
|
|
|
|
{
|
|
|
|
|
void (*fn)(void);
|
|
|
|
|
|
|
|
|
|
fn = sweepone_m;
|
|
|
|
|
runtime·onM(&fn);
|
|
|
|
|
return g->m->scalararg[0];
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#pragma textflag NOSPLIT
|
|
|
|
|
bool
|
|
|
|
|
runtime·gosweepdone(void)
|
|
|
|
|
{
|
|
|
|
|
return runtime·mheap.sweepdone;
|
|
|
|
|
}
|
|
|
|
|
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·gchelper(void)
|
|
|
|
|
{
|
2014-03-25 06:18:08 -06:00
|
|
|
|
uint32 nproc;
|
2014-01-15 08:38:08 -07:00
|
|
|
|
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->traceback = 2;
|
2013-03-21 02:48:02 -06:00
|
|
|
|
gchelperstart();
|
|
|
|
|
|
2012-05-24 00:55:50 -06:00
|
|
|
|
// parallel mark for over gc roots
|
|
|
|
|
runtime·parfordo(work.markfor);
|
2012-12-16 17:32:12 -07:00
|
|
|
|
|
2012-05-24 00:55:50 -06:00
|
|
|
|
// help other threads scan secondary blocks
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanblock(nil, 0, nil);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
|
2014-01-15 08:38:08 -07:00
|
|
|
|
nproc = work.nproc; // work.nproc can change right after we increment work.ndone
|
|
|
|
|
if(runtime·xadd(&work.ndone, +1) == nproc-1)
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
runtime·notewakeup(&work.alldone);
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->traceback = 0;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
}
|
|
|
|
|
|
2010-09-07 07:57:22 -06:00
|
|
|
|
static void
|
2013-06-06 04:56:50 -06:00
|
|
|
|
cachestats(void)
|
|
|
|
|
{
|
|
|
|
|
MCache *c;
|
|
|
|
|
P *p, **pp;
|
|
|
|
|
|
|
|
|
|
for(pp=runtime·allp; p=*pp; pp++) {
|
|
|
|
|
c = p->mcache;
|
|
|
|
|
if(c==nil)
|
|
|
|
|
continue;
|
|
|
|
|
runtime·purgecachedstats(c);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2014-01-21 02:06:57 -07:00
|
|
|
|
static void
|
|
|
|
|
flushallmcaches(void)
|
|
|
|
|
{
|
|
|
|
|
P *p, **pp;
|
|
|
|
|
MCache *c;
|
|
|
|
|
|
|
|
|
|
// Flush MCache's to MCentral.
|
|
|
|
|
for(pp=runtime·allp; p=*pp; pp++) {
|
|
|
|
|
c = p->mcache;
|
|
|
|
|
if(c==nil)
|
|
|
|
|
continue;
|
|
|
|
|
runtime·MCache_ReleaseAll(c);
|
undo CL 101570044 / 2c57aaea79c4
redo stack allocation. This is mostly the same as
the original CL with a few bug fixes.
1. add racemalloc() for stack allocations
2. fix poolalloc/poolfree to terminate free lists correctly.
3. adjust span ref count correctly.
4. don't use cache for sizes >= StackCacheSize.
Should fix bugs and memory leaks in original changelist.
««« original CL description
undo CL 104200047 / 318b04f28372
Breaks windows and race detector.
TBR=rsc
««« original CL description
runtime: stack allocator, separate from mallocgc
In order to move malloc to Go, we need to have a
separate stack allocator. If we run out of stack
during malloc, malloc will not be available
to allocate a new stack.
Stacks are the last remaining FlagNoGC objects in the
GC heap. Once they are out, we can get rid of the
distinction between the allocated/blockboundary bits.
(This will be in a separate change.)
Fixes #7468
Fixes #7424
LGTM=rsc, dvyukov
R=golang-codereviews, dvyukov, khr, dave, rsc
CC=golang-codereviews
https://golang.org/cl/104200047
»»»
TBR=rsc
CC=golang-codereviews
https://golang.org/cl/101570044
»»»
LGTM=dvyukov
R=dvyukov, dave, khr, alex.brainman
CC=golang-codereviews
https://golang.org/cl/112240044
2014-07-17 15:41:46 -06:00
|
|
|
|
runtime·stackcache_clear(c);
|
2014-01-21 02:06:57 -07:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
undo CL 101570044 / 2c57aaea79c4
redo stack allocation. This is mostly the same as
the original CL with a few bug fixes.
1. add racemalloc() for stack allocations
2. fix poolalloc/poolfree to terminate free lists correctly.
3. adjust span ref count correctly.
4. don't use cache for sizes >= StackCacheSize.
Should fix bugs and memory leaks in original changelist.
««« original CL description
undo CL 104200047 / 318b04f28372
Breaks windows and race detector.
TBR=rsc
««« original CL description
runtime: stack allocator, separate from mallocgc
In order to move malloc to Go, we need to have a
separate stack allocator. If we run out of stack
during malloc, malloc will not be available
to allocate a new stack.
Stacks are the last remaining FlagNoGC objects in the
GC heap. Once they are out, we can get rid of the
distinction between the allocated/blockboundary bits.
(This will be in a separate change.)
Fixes #7468
Fixes #7424
LGTM=rsc, dvyukov
R=golang-codereviews, dvyukov, khr, dave, rsc
CC=golang-codereviews
https://golang.org/cl/104200047
»»»
TBR=rsc
CC=golang-codereviews
https://golang.org/cl/101570044
»»»
LGTM=dvyukov
R=dvyukov, dave, khr, alex.brainman
CC=golang-codereviews
https://golang.org/cl/112240044
2014-07-17 15:41:46 -06:00
|
|
|
|
static void
|
|
|
|
|
flushallmcaches_m(G *gp)
|
|
|
|
|
{
|
|
|
|
|
flushallmcaches();
|
|
|
|
|
runtime·gogo(&gp->sched);
|
|
|
|
|
}
|
|
|
|
|
|
2014-03-25 16:09:49 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·updatememstats(GCStats *stats)
|
2010-09-07 07:57:22 -06:00
|
|
|
|
{
|
2012-12-18 09:30:29 -07:00
|
|
|
|
M *mp;
|
2013-06-06 04:56:50 -06:00
|
|
|
|
MSpan *s;
|
2011-07-18 12:52:57 -06:00
|
|
|
|
int32 i;
|
undo CL 101570044 / 2c57aaea79c4
redo stack allocation. This is mostly the same as
the original CL with a few bug fixes.
1. add racemalloc() for stack allocations
2. fix poolalloc/poolfree to terminate free lists correctly.
3. adjust span ref count correctly.
4. don't use cache for sizes >= StackCacheSize.
Should fix bugs and memory leaks in original changelist.
««« original CL description
undo CL 104200047 / 318b04f28372
Breaks windows and race detector.
TBR=rsc
««« original CL description
runtime: stack allocator, separate from mallocgc
In order to move malloc to Go, we need to have a
separate stack allocator. If we run out of stack
during malloc, malloc will not be available
to allocate a new stack.
Stacks are the last remaining FlagNoGC objects in the
GC heap. Once they are out, we can get rid of the
distinction between the allocated/blockboundary bits.
(This will be in a separate change.)
Fixes #7468
Fixes #7424
LGTM=rsc, dvyukov
R=golang-codereviews, dvyukov, khr, dave, rsc
CC=golang-codereviews
https://golang.org/cl/104200047
»»»
TBR=rsc
CC=golang-codereviews
https://golang.org/cl/101570044
»»»
LGTM=dvyukov
R=dvyukov, dave, khr, alex.brainman
CC=golang-codereviews
https://golang.org/cl/112240044
2014-07-17 15:41:46 -06:00
|
|
|
|
uint64 smallfree;
|
2012-04-05 10:48:28 -06:00
|
|
|
|
uint64 *src, *dst;
|
2014-09-03 09:35:22 -06:00
|
|
|
|
void (*fn)(G*);
|
2010-09-07 07:57:22 -06:00
|
|
|
|
|
2012-04-05 10:48:28 -06:00
|
|
|
|
if(stats)
|
|
|
|
|
runtime·memclr((byte*)stats, sizeof(*stats));
|
2012-12-18 09:30:29 -07:00
|
|
|
|
for(mp=runtime·allm; mp; mp=mp->alllink) {
|
2012-04-05 10:48:28 -06:00
|
|
|
|
if(stats) {
|
2012-12-18 09:30:29 -07:00
|
|
|
|
src = (uint64*)&mp->gcstats;
|
2012-04-05 10:48:28 -06:00
|
|
|
|
dst = (uint64*)stats;
|
|
|
|
|
for(i=0; i<sizeof(*stats)/sizeof(uint64); i++)
|
|
|
|
|
dst[i] += src[i];
|
2012-12-18 09:30:29 -07:00
|
|
|
|
runtime·memclr((byte*)&mp->gcstats, sizeof(mp->gcstats));
|
2012-04-05 10:48:28 -06:00
|
|
|
|
}
|
2013-03-01 04:49:16 -07:00
|
|
|
|
}
|
runtime: account for all sys memory in MemStats
Currently lots of sys allocations are not accounted in any of XxxSys,
including GC bitmap, spans table, GC roots blocks, GC finalizer blocks,
iface table, netpoll descriptors and more. Up to ~20% can unaccounted.
This change introduces 2 new stats: GCSys and OtherSys for GC metadata
and all other misc allocations, respectively.
Also ensures that all XxxSys indeed sum up to Sys. All sys memory allocation
functions require the stat for accounting, so that it's impossible to miss something.
Also fix updating of mcache_sys/inuse, they were not updated after deallocation.
test/bench/garbage/parser before:
Sys 670064344
HeapSys 610271232
StackSys 65536
MSpanSys 14204928
MCacheSys 16384
BuckHashSys 1439992
after:
Sys 670064344
HeapSys 610271232
StackSys 65536
MSpanSys 14188544
MCacheSys 16384
BuckHashSys 3194304
GCSys 39198688
OtherSys 3129656
Fixes #5799.
R=rsc, dave, alex.brainman
CC=golang-dev
https://golang.org/cl/12946043
2013-09-06 14:55:40 -06:00
|
|
|
|
mstats.mcache_inuse = runtime·mheap.cachealloc.inuse;
|
|
|
|
|
mstats.mspan_inuse = runtime·mheap.spanalloc.inuse;
|
|
|
|
|
mstats.sys = mstats.heap_sys + mstats.stacks_sys + mstats.mspan_sys +
|
|
|
|
|
mstats.mcache_sys + mstats.buckhash_sys + mstats.gc_sys + mstats.other_sys;
|
|
|
|
|
|
2013-06-06 04:56:50 -06:00
|
|
|
|
// Calculate memory allocator stats.
|
|
|
|
|
// During program execution we only count number of frees and amount of freed memory.
|
|
|
|
|
// Current number of alive object in the heap and amount of alive heap memory
|
|
|
|
|
// are calculated by scanning all spans.
|
|
|
|
|
// Total number of mallocs is calculated as number of frees plus number of alive objects.
|
|
|
|
|
// Similarly, total amount of allocated memory is calculated as amount of freed memory
|
|
|
|
|
// plus amount of alive heap memory.
|
|
|
|
|
mstats.alloc = 0;
|
|
|
|
|
mstats.total_alloc = 0;
|
|
|
|
|
mstats.nmalloc = 0;
|
|
|
|
|
mstats.nfree = 0;
|
|
|
|
|
for(i = 0; i < nelem(mstats.by_size); i++) {
|
|
|
|
|
mstats.by_size[i].nmalloc = 0;
|
|
|
|
|
mstats.by_size[i].nfree = 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Flush MCache's to MCentral.
|
2014-07-18 03:05:21 -06:00
|
|
|
|
if(g == g->m->g0)
|
|
|
|
|
flushallmcaches();
|
2014-09-03 09:35:22 -06:00
|
|
|
|
else {
|
|
|
|
|
fn = flushallmcaches_m;
|
|
|
|
|
runtime·mcall(&fn);
|
|
|
|
|
}
|
2013-06-06 04:56:50 -06:00
|
|
|
|
|
|
|
|
|
// Aggregate local stats.
|
|
|
|
|
cachestats();
|
|
|
|
|
|
|
|
|
|
// Scan all spans and count number of alive objects.
|
2014-08-24 02:05:07 -06:00
|
|
|
|
runtime·lock(&runtime·mheap.lock);
|
2013-06-06 04:56:50 -06:00
|
|
|
|
for(i = 0; i < runtime·mheap.nspan; i++) {
|
|
|
|
|
s = runtime·mheap.allspans[i];
|
|
|
|
|
if(s->state != MSpanInUse)
|
|
|
|
|
continue;
|
|
|
|
|
if(s->sizeclass == 0) {
|
|
|
|
|
mstats.nmalloc++;
|
|
|
|
|
mstats.alloc += s->elemsize;
|
|
|
|
|
} else {
|
|
|
|
|
mstats.nmalloc += s->ref;
|
|
|
|
|
mstats.by_size[s->sizeclass].nmalloc += s->ref;
|
|
|
|
|
mstats.alloc += s->ref*s->elemsize;
|
|
|
|
|
}
|
|
|
|
|
}
|
2014-08-24 02:05:07 -06:00
|
|
|
|
runtime·unlock(&runtime·mheap.lock);
|
2013-06-06 04:56:50 -06:00
|
|
|
|
|
|
|
|
|
// Aggregate by size class.
|
|
|
|
|
smallfree = 0;
|
|
|
|
|
mstats.nfree = runtime·mheap.nlargefree;
|
|
|
|
|
for(i = 0; i < nelem(mstats.by_size); i++) {
|
|
|
|
|
mstats.nfree += runtime·mheap.nsmallfree[i];
|
|
|
|
|
mstats.by_size[i].nfree = runtime·mheap.nsmallfree[i];
|
|
|
|
|
mstats.by_size[i].nmalloc += runtime·mheap.nsmallfree[i];
|
|
|
|
|
smallfree += runtime·mheap.nsmallfree[i] * runtime·class_to_size[i];
|
|
|
|
|
}
|
|
|
|
|
mstats.nmalloc += mstats.nfree;
|
|
|
|
|
|
|
|
|
|
// Calculate derived stats.
|
|
|
|
|
mstats.total_alloc = mstats.alloc + runtime·mheap.largefree + smallfree;
|
|
|
|
|
mstats.heap_alloc = mstats.alloc;
|
|
|
|
|
mstats.heap_objects = mstats.nmalloc - mstats.nfree;
|
2010-09-07 07:57:22 -06:00
|
|
|
|
}
|
|
|
|
|
|
2012-11-27 11:04:59 -07:00
|
|
|
|
// Structure of arguments passed to function gc().
|
2013-05-31 21:43:33 -06:00
|
|
|
|
// This allows the arguments to be passed via runtime·mcall.
|
2012-11-27 11:04:59 -07:00
|
|
|
|
struct gc_args
|
|
|
|
|
{
|
2013-05-31 21:43:33 -06:00
|
|
|
|
int64 start_time; // start time of GC in ns (just before stoptheworld)
|
2014-05-19 02:06:30 -06:00
|
|
|
|
bool eagersweep;
|
2012-11-27 11:04:59 -07:00
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
static void gc(struct gc_args *args);
|
2013-05-31 21:43:33 -06:00
|
|
|
|
static void mgc(G *gp);
|
2012-11-27 11:04:59 -07:00
|
|
|
|
|
2014-07-30 10:01:52 -06:00
|
|
|
|
int32
|
|
|
|
|
runtime·readgogc(void)
|
2013-02-03 22:00:55 -07:00
|
|
|
|
{
|
|
|
|
|
byte *p;
|
|
|
|
|
|
|
|
|
|
p = runtime·getenv("GOGC");
|
|
|
|
|
if(p == nil || p[0] == '\0')
|
|
|
|
|
return 100;
|
|
|
|
|
if(runtime·strcmp(p, (byte*)"off") == 0)
|
|
|
|
|
return -1;
|
|
|
|
|
return runtime·atoi(p);
|
|
|
|
|
}
|
|
|
|
|
|
2014-08-18 06:42:24 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·gcinit(void)
|
|
|
|
|
{
|
|
|
|
|
if(sizeof(Workbuf) != WorkbufSize)
|
|
|
|
|
runtime·throw("runtime: size of Workbuf is suboptimal");
|
|
|
|
|
|
|
|
|
|
work.markfor = runtime·parforalloc(MaxGcproc);
|
|
|
|
|
runtime·gcpercent = runtime·readgogc();
|
2014-08-27 18:15:05 -06:00
|
|
|
|
runtime·gcdatamask = unrollglobgcprog(runtime·gcdata, runtime·edata - runtime·data);
|
|
|
|
|
runtime·gcbssmask = unrollglobgcprog(runtime·gcbss, runtime·ebss - runtime·bss);
|
2014-08-18 06:42:24 -06:00
|
|
|
|
}
|
|
|
|
|
|
2014-07-30 10:01:52 -06:00
|
|
|
|
void
|
2014-08-06 15:33:57 -06:00
|
|
|
|
runtime·gc_m(void)
|
2014-07-30 10:01:52 -06:00
|
|
|
|
{
|
|
|
|
|
struct gc_args a;
|
|
|
|
|
G *gp;
|
|
|
|
|
|
|
|
|
|
gp = g->m->curg;
|
2014-08-27 09:15:47 -06:00
|
|
|
|
runtime·casgstatus(gp, Grunning, Gwaiting);
|
2014-08-21 10:41:09 -06:00
|
|
|
|
gp->waitreason = runtime·gostringnocopy((byte*)"garbage collection");
|
2014-07-30 10:01:52 -06:00
|
|
|
|
|
2014-08-19 01:53:20 -06:00
|
|
|
|
a.start_time = (uint64)(g->m->scalararg[0]) | ((uint64)(g->m->scalararg[1]) << 32);
|
|
|
|
|
a.eagersweep = g->m->scalararg[2];
|
2014-07-30 10:01:52 -06:00
|
|
|
|
gc(&a);
|
|
|
|
|
|
2014-08-27 09:15:47 -06:00
|
|
|
|
runtime·casgstatus(gp, Gwaiting, Grunning);
|
2014-07-30 10:01:52 -06:00
|
|
|
|
}
|
|
|
|
|
|
2012-11-27 11:04:59 -07:00
|
|
|
|
static void
|
|
|
|
|
gc(struct gc_args *args)
|
|
|
|
|
{
|
2013-01-22 02:44:49 -07:00
|
|
|
|
int64 t0, t1, t2, t3, t4;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
uint64 heap0, heap1, obj;
|
2012-11-27 11:04:59 -07:00
|
|
|
|
GCStats stats;
|
|
|
|
|
|
2014-04-01 11:30:10 -06:00
|
|
|
|
if(runtime·debug.allocfreetrace)
|
|
|
|
|
runtime·tracegc();
|
|
|
|
|
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->traceback = 2;
|
2013-05-31 21:43:33 -06:00
|
|
|
|
t0 = args->start_time;
|
2014-01-16 01:54:46 -07:00
|
|
|
|
work.tstart = args->start_time;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
|
2014-04-09 08:38:12 -06:00
|
|
|
|
t1 = 0;
|
|
|
|
|
if(runtime·debug.gctrace)
|
|
|
|
|
t1 = runtime·nanotime();
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
|
|
|
|
|
// Sweep what is not sweeped by bgsweep.
|
|
|
|
|
while(runtime·sweepone() != -1)
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
runtime·sweep.npausesweep++;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
|
2014-08-24 02:05:07 -06:00
|
|
|
|
// Cache runtime.mheap.allspans in work.spans to avoid conflicts with
|
|
|
|
|
// resizing/freeing allspans.
|
|
|
|
|
// New spans can be created while GC progresses, but they are not garbage for
|
|
|
|
|
// this round:
|
|
|
|
|
// - new stack spans can be created even while the world is stopped.
|
|
|
|
|
// - new malloc spans can be created during the concurrent sweep
|
|
|
|
|
|
|
|
|
|
// Even if this is stop-the-world, a concurrent exitsyscall can allocate a stack from heap.
|
|
|
|
|
runtime·lock(&runtime·mheap.lock);
|
|
|
|
|
// Free the old cached sweep array if necessary.
|
|
|
|
|
if(work.spans != nil && work.spans != runtime·mheap.allspans)
|
|
|
|
|
runtime·SysFree(work.spans, work.nspan*sizeof(work.spans[0]), &mstats.other_sys);
|
|
|
|
|
// Cache the current array for marking.
|
|
|
|
|
runtime·mheap.gcspans = runtime·mheap.allspans;
|
|
|
|
|
work.spans = runtime·mheap.allspans;
|
|
|
|
|
work.nspan = runtime·mheap.nspan;
|
|
|
|
|
runtime·unlock(&runtime·mheap.lock);
|
|
|
|
|
|
2012-05-22 11:35:52 -06:00
|
|
|
|
work.nwait = 0;
|
|
|
|
|
work.ndone = 0;
|
2012-05-15 09:10:16 -06:00
|
|
|
|
work.nproc = runtime·gcprocs();
|
2014-01-21 02:06:57 -07:00
|
|
|
|
runtime·parforsetup(work.markfor, work.nproc, RootCount + runtime·allglen, nil, false, markroot);
|
2012-05-15 09:10:16 -06:00
|
|
|
|
if(work.nproc > 1) {
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
runtime·noteclear(&work.alldone);
|
2012-05-15 09:10:16 -06:00
|
|
|
|
runtime·helpgc(work.nproc);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
}
|
|
|
|
|
|
2014-04-09 08:38:12 -06:00
|
|
|
|
t2 = 0;
|
|
|
|
|
if(runtime·debug.gctrace)
|
|
|
|
|
t2 = runtime·nanotime();
|
2013-01-22 02:44:49 -07:00
|
|
|
|
|
2013-03-21 02:48:02 -06:00
|
|
|
|
gchelperstart();
|
2012-05-24 00:55:50 -06:00
|
|
|
|
runtime·parfordo(work.markfor);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanblock(nil, 0, nil);
|
2012-05-24 00:55:50 -06:00
|
|
|
|
|
2014-04-09 08:38:12 -06:00
|
|
|
|
t3 = 0;
|
|
|
|
|
if(runtime·debug.gctrace)
|
|
|
|
|
t3 = runtime·nanotime();
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
|
2012-05-22 11:35:52 -06:00
|
|
|
|
if(work.nproc > 1)
|
|
|
|
|
runtime·notesleep(&work.alldone);
|
|
|
|
|
|
2013-06-06 04:56:50 -06:00
|
|
|
|
cachestats();
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
// next_gc calculation is tricky with concurrent sweep since we don't know size of live heap
|
|
|
|
|
// estimate what was live heap size after previous GC (for tracing only)
|
2014-07-30 10:01:52 -06:00
|
|
|
|
heap0 = mstats.next_gc*100/(runtime·gcpercent+100);
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
// conservatively set next_gc to high value assuming that everything is live
|
|
|
|
|
// concurrent/lazy sweep will reduce this number while discovering new garbage
|
2014-07-30 10:01:52 -06:00
|
|
|
|
mstats.next_gc = mstats.heap_alloc+mstats.heap_alloc*runtime·gcpercent/100;
|
2010-02-10 01:00:12 -07:00
|
|
|
|
|
2013-01-22 02:44:49 -07:00
|
|
|
|
t4 = runtime·nanotime();
|
2014-08-29 01:08:10 -06:00
|
|
|
|
runtime·atomicstore64(&mstats.last_gc, runtime·unixnanotime()); // must be Unix time to make sense to user
|
2013-01-22 02:44:49 -07:00
|
|
|
|
mstats.pause_ns[mstats.numgc%nelem(mstats.pause_ns)] = t4 - t0;
|
|
|
|
|
mstats.pause_total_ns += t4 - t0;
|
2010-02-08 15:32:22 -07:00
|
|
|
|
mstats.numgc++;
|
|
|
|
|
if(mstats.debuggc)
|
2013-01-22 02:44:49 -07:00
|
|
|
|
runtime·printf("pause %D\n", t4-t0);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
|
2013-06-28 08:37:06 -06:00
|
|
|
|
if(runtime·debug.gctrace) {
|
2013-06-06 04:56:50 -06:00
|
|
|
|
heap1 = mstats.heap_alloc;
|
2014-03-25 16:09:49 -06:00
|
|
|
|
runtime·updatememstats(&stats);
|
2014-03-06 10:33:00 -07:00
|
|
|
|
if(heap1 != mstats.heap_alloc) {
|
2014-03-06 12:56:22 -07:00
|
|
|
|
runtime·printf("runtime: mstats skew: heap=%D/%D\n", heap1, mstats.heap_alloc);
|
2014-03-06 10:33:00 -07:00
|
|
|
|
runtime·throw("mstats skew");
|
|
|
|
|
}
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
obj = mstats.nmalloc - mstats.nfree;
|
2013-06-06 04:56:50 -06:00
|
|
|
|
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
stats.nprocyield += work.markfor->nprocyield;
|
|
|
|
|
stats.nosyield += work.markfor->nosyield;
|
|
|
|
|
stats.nsleep += work.markfor->nsleep;
|
2013-06-06 04:56:50 -06:00
|
|
|
|
|
2014-04-10 12:34:48 -06:00
|
|
|
|
runtime·printf("gc%d(%d): %D+%D+%D+%D us, %D -> %D MB, %D (%D-%D) objects,"
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
" %d/%d/%d sweeps,"
|
2012-05-22 11:35:52 -06:00
|
|
|
|
" %D(%D) handoff, %D(%D) steal, %D/%D/%D yields\n",
|
2014-04-10 12:34:48 -06:00
|
|
|
|
mstats.numgc, work.nproc, (t1-t0)/1000, (t2-t1)/1000, (t3-t2)/1000, (t4-t3)/1000,
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
heap0>>20, heap1>>20, obj,
|
2011-02-02 21:03:47 -07:00
|
|
|
|
mstats.nmalloc, mstats.nfree,
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
work.nspan, runtime·sweep.nbgsweep, runtime·sweep.npausesweep,
|
2012-04-05 10:48:28 -06:00
|
|
|
|
stats.nhandoff, stats.nhandoffcnt,
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
work.markfor->nsteal, work.markfor->nstealcnt,
|
2012-04-05 10:48:28 -06:00
|
|
|
|
stats.nprocyield, stats.nosyield, stats.nsleep);
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
runtime·sweep.nbgsweep = runtime·sweep.npausesweep = 0;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
}
|
2012-05-22 11:35:52 -06:00
|
|
|
|
|
2014-08-24 02:05:07 -06:00
|
|
|
|
// See the comment in the beginning of this function as to why we need the following.
|
|
|
|
|
// Even if this is still stop-the-world, a concurrent exitsyscall can allocate a stack from heap.
|
|
|
|
|
runtime·lock(&runtime·mheap.lock);
|
|
|
|
|
// Free the old cached mark array if necessary.
|
|
|
|
|
if(work.spans != nil && work.spans != runtime·mheap.allspans)
|
|
|
|
|
runtime·SysFree(work.spans, work.nspan*sizeof(work.spans[0]), &mstats.other_sys);
|
|
|
|
|
// Cache the current array for sweeping.
|
|
|
|
|
runtime·mheap.gcspans = runtime·mheap.allspans;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
runtime·mheap.sweepgen += 2;
|
|
|
|
|
runtime·mheap.sweepdone = false;
|
2014-08-24 02:05:07 -06:00
|
|
|
|
work.spans = runtime·mheap.allspans;
|
|
|
|
|
work.nspan = runtime·mheap.nspan;
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
runtime·sweep.spanidx = 0;
|
2014-08-24 02:05:07 -06:00
|
|
|
|
runtime·unlock(&runtime·mheap.lock);
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
|
2014-02-12 13:03:27 -07:00
|
|
|
|
// Temporary disable concurrent sweep, because we see failures on builders.
|
2014-05-19 02:06:30 -06:00
|
|
|
|
if(ConcurrentSweep && !args->eagersweep) {
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
runtime·lock(&runtime·gclock);
|
|
|
|
|
if(runtime·sweep.g == nil)
|
|
|
|
|
runtime·sweep.g = runtime·newproc1(&bgsweepv, nil, 0, 0, gc);
|
|
|
|
|
else if(runtime·sweep.parked) {
|
|
|
|
|
runtime·sweep.parked = false;
|
|
|
|
|
runtime·ready(runtime·sweep.g);
|
2014-02-12 13:03:27 -07:00
|
|
|
|
}
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
runtime·unlock(&runtime·gclock);
|
2014-02-12 13:03:27 -07:00
|
|
|
|
} else {
|
|
|
|
|
// Sweep all spans eagerly.
|
|
|
|
|
while(runtime·sweepone() != -1)
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
runtime·sweep.npausesweep++;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
}
|
|
|
|
|
|
2014-09-01 16:51:12 -06:00
|
|
|
|
runtime·mProf_GC();
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->traceback = 0;
|
2010-03-26 15:15:30 -06:00
|
|
|
|
}
|
|
|
|
|
|
2014-01-30 02:28:19 -07:00
|
|
|
|
extern uintptr runtime·sizeof_C_MStats;
|
|
|
|
|
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
static void readmemstats_m(void);
|
|
|
|
|
|
|
|
|
|
#pragma textflag NOSPLIT
|
2011-07-21 22:55:01 -06:00
|
|
|
|
void
|
2012-02-06 11:16:26 -07:00
|
|
|
|
runtime·ReadMemStats(MStats *stats)
|
2011-07-21 22:55:01 -06:00
|
|
|
|
{
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
void (*fn)(void);
|
|
|
|
|
|
|
|
|
|
g->m->ptrarg[0] = stats;
|
|
|
|
|
fn = readmemstats_m;
|
|
|
|
|
runtime·onM(&fn);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
readmemstats_m(void)
|
|
|
|
|
{
|
|
|
|
|
MStats *stats;
|
|
|
|
|
|
|
|
|
|
stats = g->m->ptrarg[0];
|
|
|
|
|
g->m->ptrarg[0] = nil;
|
|
|
|
|
|
2012-02-22 19:45:01 -07:00
|
|
|
|
// Have to acquire worldsema to stop the world,
|
2011-07-21 22:55:01 -06:00
|
|
|
|
// because stoptheworld can only be used by
|
|
|
|
|
// one goroutine at a time, and there might be
|
|
|
|
|
// a pending garbage collection already calling it.
|
net: add special netFD mutex
The mutex, fdMutex, handles locking and lifetime of sysfd,
and serializes Read and Write methods.
This allows to strip 2 sync.Mutex.Lock calls,
2 sync.Mutex.Unlock calls, 1 defer and some amount
of misc overhead from every network operation.
On linux/amd64, Intel E5-2690:
benchmark old ns/op new ns/op delta
BenchmarkTCP4Persistent 9595 9454 -1.47%
BenchmarkTCP4Persistent-2 8978 8772 -2.29%
BenchmarkTCP4ConcurrentReadWrite 4900 4625 -5.61%
BenchmarkTCP4ConcurrentReadWrite-2 2603 2500 -3.96%
In general it strips 70-500 ns from every network operation depending
on processor model. On my relatively new E5-2690 it accounts to ~5%
of network op cost.
Fixes #6074.
R=golang-dev, bradfitz, alex.brainman, iant, mikioh.mikioh
CC=golang-dev
https://golang.org/cl/12418043
2013-08-09 11:43:00 -06:00
|
|
|
|
runtime·semacquire(&runtime·worldsema, false);
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->gcing = 1;
|
2011-07-21 22:55:01 -06:00
|
|
|
|
runtime·stoptheworld();
|
2014-03-25 16:09:49 -06:00
|
|
|
|
runtime·updatememstats(nil);
|
2014-01-30 02:28:19 -07:00
|
|
|
|
// Size of the trailing by_size array differs between Go and C,
|
|
|
|
|
// NumSizeClasses was changed, but we can not change Go struct because of backward compatibility.
|
2014-08-04 21:40:44 -06:00
|
|
|
|
runtime·memmove(stats, &mstats, runtime·sizeof_C_MStats);
|
undo CL 101570044 / 2c57aaea79c4
redo stack allocation. This is mostly the same as
the original CL with a few bug fixes.
1. add racemalloc() for stack allocations
2. fix poolalloc/poolfree to terminate free lists correctly.
3. adjust span ref count correctly.
4. don't use cache for sizes >= StackCacheSize.
Should fix bugs and memory leaks in original changelist.
««« original CL description
undo CL 104200047 / 318b04f28372
Breaks windows and race detector.
TBR=rsc
««« original CL description
runtime: stack allocator, separate from mallocgc
In order to move malloc to Go, we need to have a
separate stack allocator. If we run out of stack
during malloc, malloc will not be available
to allocate a new stack.
Stacks are the last remaining FlagNoGC objects in the
GC heap. Once they are out, we can get rid of the
distinction between the allocated/blockboundary bits.
(This will be in a separate change.)
Fixes #7468
Fixes #7424
LGTM=rsc, dvyukov
R=golang-codereviews, dvyukov, khr, dave, rsc
CC=golang-codereviews
https://golang.org/cl/104200047
»»»
TBR=rsc
CC=golang-codereviews
https://golang.org/cl/101570044
»»»
LGTM=dvyukov
R=dvyukov, dave, khr, alex.brainman
CC=golang-codereviews
https://golang.org/cl/112240044
2014-07-17 15:41:46 -06:00
|
|
|
|
|
|
|
|
|
// Stack numbers are part of the heap numbers, separate those out for user consumption
|
|
|
|
|
stats->stacks_sys = stats->stacks_inuse;
|
|
|
|
|
stats->heap_inuse -= stats->stacks_inuse;
|
|
|
|
|
stats->heap_sys -= stats->stacks_inuse;
|
|
|
|
|
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->gcing = 0;
|
|
|
|
|
g->m->locks++;
|
2012-02-22 19:45:01 -07:00
|
|
|
|
runtime·semrelease(&runtime·worldsema);
|
2012-05-15 09:10:16 -06:00
|
|
|
|
runtime·starttheworld();
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->locks--;
|
2011-07-21 22:55:01 -06:00
|
|
|
|
}
|
|
|
|
|
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
static void readgcstats_m(void);
|
|
|
|
|
|
|
|
|
|
#pragma textflag NOSPLIT
|
2013-02-03 22:00:55 -07:00
|
|
|
|
void
|
|
|
|
|
runtime∕debug·readGCStats(Slice *pauses)
|
|
|
|
|
{
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
void (*fn)(void);
|
|
|
|
|
|
|
|
|
|
g->m->ptrarg[0] = pauses;
|
|
|
|
|
fn = readgcstats_m;
|
|
|
|
|
runtime·onM(&fn);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
readgcstats_m(void)
|
|
|
|
|
{
|
|
|
|
|
Slice *pauses;
|
2013-02-03 22:00:55 -07:00
|
|
|
|
uint64 *p;
|
|
|
|
|
uint32 i, n;
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
|
|
|
|
|
pauses = g->m->ptrarg[0];
|
|
|
|
|
g->m->ptrarg[0] = nil;
|
2013-02-03 22:00:55 -07:00
|
|
|
|
|
|
|
|
|
// Calling code in runtime/debug should make the slice large enough.
|
|
|
|
|
if(pauses->cap < nelem(mstats.pause_ns)+3)
|
|
|
|
|
runtime·throw("runtime: short slice passed to readGCStats");
|
|
|
|
|
|
|
|
|
|
// Pass back: pauses, last gc (absolute time), number of gc, total pause ns.
|
|
|
|
|
p = (uint64*)pauses->array;
|
2014-08-07 07:00:02 -06:00
|
|
|
|
runtime·lock(&runtime·mheap.lock);
|
2013-02-03 22:00:55 -07:00
|
|
|
|
n = mstats.numgc;
|
|
|
|
|
if(n > nelem(mstats.pause_ns))
|
|
|
|
|
n = nelem(mstats.pause_ns);
|
|
|
|
|
|
|
|
|
|
// The pause buffer is circular. The most recent pause is at
|
|
|
|
|
// pause_ns[(numgc-1)%nelem(pause_ns)], and then backward
|
|
|
|
|
// from there to go back farther in time. We deliver the times
|
|
|
|
|
// most recent first (in p[0]).
|
|
|
|
|
for(i=0; i<n; i++)
|
|
|
|
|
p[i] = mstats.pause_ns[(mstats.numgc-1-i)%nelem(mstats.pause_ns)];
|
|
|
|
|
|
|
|
|
|
p[n] = mstats.last_gc;
|
|
|
|
|
p[n+1] = mstats.numgc;
|
|
|
|
|
p[n+2] = mstats.pause_total_ns;
|
2014-08-07 07:00:02 -06:00
|
|
|
|
runtime·unlock(&runtime·mheap.lock);
|
2013-02-03 22:00:55 -07:00
|
|
|
|
pauses->len = n+3;
|
|
|
|
|
}
|
|
|
|
|
|
2014-08-24 21:27:00 -06:00
|
|
|
|
void
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
runtime·setgcpercent_m(void)
|
|
|
|
|
{
|
2014-08-24 21:27:00 -06:00
|
|
|
|
int32 in;
|
runtime: use goc2c as much as possible
Package runtime's C functions written to be called from Go
started out written in C using carefully constructed argument
lists and the FLUSH macro to write a result back to memory.
For some functions, the appropriate parameter list ended up
being architecture-dependent due to differences in alignment,
so we added 'goc2c', which takes a .goc file containing Go func
declarations but C bodies, rewrites the Go func declaration to
equivalent C declarations for the target architecture, adds the
needed FLUSH statements, and writes out an equivalent C file.
That C file is compiled as part of package runtime.
Native Client's x86-64 support introduces the most complex
alignment rules yet, breaking many functions that could until
now be portably written in C. Using goc2c for those avoids the
breakage.
Separately, Keith's work on emitting stack information from
the C compiler would require the hand-written functions
to add #pragmas specifying how many arguments are result
parameters. Using goc2c for those avoids maintaining #pragmas.
For both reasons, use goc2c for as many Go-called C functions
as possible.
This CL is a replay of the bulk of CL 15400047 and CL 15790043,
both of which were reviewed as part of the NaCl port and are
checked in to the NaCl branch. This CL is part of bringing the
NaCl code into the main tree.
No new code here, just reformatting and occasional movement
into .h files.
LGTM=r
R=dave, alex.brainman, r
CC=golang-codereviews
https://golang.org/cl/65220044
2014-02-20 13:58:47 -07:00
|
|
|
|
int32 out;
|
|
|
|
|
|
2014-08-24 21:27:00 -06:00
|
|
|
|
in = (int32)(intptr)g->m->scalararg[0];
|
|
|
|
|
|
2014-08-07 07:00:02 -06:00
|
|
|
|
runtime·lock(&runtime·mheap.lock);
|
2014-07-30 10:01:52 -06:00
|
|
|
|
out = runtime·gcpercent;
|
2013-02-03 22:00:55 -07:00
|
|
|
|
if(in < 0)
|
|
|
|
|
in = -1;
|
2014-07-30 10:01:52 -06:00
|
|
|
|
runtime·gcpercent = in;
|
2014-08-07 07:00:02 -06:00
|
|
|
|
runtime·unlock(&runtime·mheap.lock);
|
2014-08-24 21:27:00 -06:00
|
|
|
|
|
|
|
|
|
g->m->scalararg[0] = (uintptr)(intptr)out;
|
2013-02-03 22:00:55 -07:00
|
|
|
|
}
|
|
|
|
|
|
2013-03-21 02:48:02 -06:00
|
|
|
|
static void
|
|
|
|
|
gchelperstart(void)
|
|
|
|
|
{
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
if(g->m->helpgc < 0 || g->m->helpgc >= MaxGcproc)
|
2013-03-21 02:48:02 -06:00
|
|
|
|
runtime·throw("gchelperstart: bad m->helpgc");
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
if(g != g->m->g0)
|
2013-05-31 21:43:33 -06:00
|
|
|
|
runtime·throw("gchelper not running on g0 stack");
|
2013-03-21 02:48:02 -06:00
|
|
|
|
}
|
|
|
|
|
|
2014-03-26 05:11:36 -06:00
|
|
|
|
G*
|
|
|
|
|
runtime·wakefing(void)
|
|
|
|
|
{
|
|
|
|
|
G *res;
|
|
|
|
|
|
|
|
|
|
res = nil;
|
2014-08-28 14:23:10 -06:00
|
|
|
|
runtime·lock(&runtime·finlock);
|
2014-03-26 05:11:36 -06:00
|
|
|
|
if(runtime·fingwait && runtime·fingwake) {
|
|
|
|
|
runtime·fingwait = false;
|
|
|
|
|
runtime·fingwake = false;
|
2014-07-30 10:01:52 -06:00
|
|
|
|
res = runtime·fing;
|
2014-03-26 05:11:36 -06:00
|
|
|
|
}
|
2014-08-28 14:23:10 -06:00
|
|
|
|
runtime·unlock(&runtime·finlock);
|
2014-03-26 05:11:36 -06:00
|
|
|
|
return res;
|
|
|
|
|
}
|
|
|
|
|
|
2014-08-07 03:34:30 -06:00
|
|
|
|
// Recursively unrolls GC program in prog.
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// mask is where to store the result.
|
|
|
|
|
// ppos is a pointer to position in mask, in bits.
|
|
|
|
|
// sparse says to generate 4-bits per word mask for heap (2-bits for data/bss otherwise).
|
|
|
|
|
static byte*
|
|
|
|
|
unrollgcprog1(byte *mask, byte *prog, uintptr *ppos, bool inplace, bool sparse)
|
2011-02-02 21:03:47 -07:00
|
|
|
|
{
|
2014-08-19 07:38:00 -06:00
|
|
|
|
uintptr pos, siz, i, off;
|
|
|
|
|
byte *arena_start, *prog1, v, *bitp, shift;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
arena_start = runtime·mheap.arena_start;
|
|
|
|
|
pos = *ppos;
|
|
|
|
|
for(;;) {
|
|
|
|
|
switch(prog[0]) {
|
|
|
|
|
case insData:
|
|
|
|
|
prog++;
|
|
|
|
|
siz = prog[0];
|
|
|
|
|
prog++;
|
|
|
|
|
for(i = 0; i < siz; i++) {
|
|
|
|
|
v = prog[i/PointersPerByte];
|
|
|
|
|
v >>= (i%PointersPerByte)*BitsPerPointer;
|
|
|
|
|
v &= BitsMask;
|
|
|
|
|
if(inplace) {
|
|
|
|
|
// Store directly into GC bitmap.
|
|
|
|
|
off = (uintptr*)(mask+pos) - (uintptr*)arena_start;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
bitp = arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = (off % wordsPerBitmapByte) * gcBits;
|
|
|
|
|
if(shift==0)
|
|
|
|
|
*bitp = 0;
|
|
|
|
|
*bitp |= v<<(shift+2);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
pos += PtrSize;
|
|
|
|
|
} else if(sparse) {
|
|
|
|
|
// 4-bits per word
|
|
|
|
|
v <<= (pos%8)+2;
|
|
|
|
|
mask[pos/8] |= v;
|
|
|
|
|
pos += gcBits;
|
|
|
|
|
} else {
|
|
|
|
|
// 2-bits per word
|
|
|
|
|
v <<= pos%8;
|
|
|
|
|
mask[pos/8] |= v;
|
|
|
|
|
pos += BitsPerPointer;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
prog += ROUND(siz*BitsPerPointer, 8)/8;
|
|
|
|
|
break;
|
|
|
|
|
case insArray:
|
|
|
|
|
prog++;
|
|
|
|
|
siz = 0;
|
|
|
|
|
for(i = 0; i < PtrSize; i++)
|
|
|
|
|
siz = (siz<<8) + prog[PtrSize-i-1];
|
|
|
|
|
prog += PtrSize;
|
|
|
|
|
prog1 = nil;
|
|
|
|
|
for(i = 0; i < siz; i++)
|
|
|
|
|
prog1 = unrollgcprog1(mask, prog, &pos, inplace, sparse);
|
|
|
|
|
if(prog1[0] != insArrayEnd)
|
|
|
|
|
runtime·throw("unrollgcprog: array does not end with insArrayEnd");
|
|
|
|
|
prog = prog1+1;
|
|
|
|
|
break;
|
|
|
|
|
case insArrayEnd:
|
|
|
|
|
case insEnd:
|
|
|
|
|
*ppos = pos;
|
|
|
|
|
return prog;
|
|
|
|
|
default:
|
|
|
|
|
runtime·throw("unrollgcprog: unknown instruction");
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Unrolls GC program prog for data/bss, returns dense GC mask.
|
2014-08-25 12:38:19 -06:00
|
|
|
|
static BitVector
|
2014-07-29 01:01:02 -06:00
|
|
|
|
unrollglobgcprog(byte *prog, uintptr size)
|
|
|
|
|
{
|
|
|
|
|
byte *mask;
|
|
|
|
|
uintptr pos, masksize;
|
|
|
|
|
|
|
|
|
|
masksize = ROUND(ROUND(size, PtrSize)/PtrSize*BitsPerPointer, 8)/8;
|
|
|
|
|
mask = runtime·persistentalloc(masksize+1, 0, &mstats.gc_sys);
|
|
|
|
|
mask[masksize] = 0xa1;
|
|
|
|
|
pos = 0;
|
|
|
|
|
prog = unrollgcprog1(mask, prog, &pos, false, false);
|
|
|
|
|
if(pos != size/PtrSize*BitsPerPointer) {
|
|
|
|
|
runtime·printf("unrollglobgcprog: bad program size, got %D, expect %D\n",
|
|
|
|
|
(uint64)pos, (uint64)size/PtrSize*BitsPerPointer);
|
|
|
|
|
runtime·throw("unrollglobgcprog: bad program size");
|
|
|
|
|
}
|
|
|
|
|
if(prog[0] != insEnd)
|
|
|
|
|
runtime·throw("unrollglobgcprog: program does not end with insEnd");
|
|
|
|
|
if(mask[masksize] != 0xa1)
|
|
|
|
|
runtime·throw("unrollglobgcprog: overflow");
|
2014-08-25 12:38:19 -06:00
|
|
|
|
return (BitVector){masksize*8, (uint32*)mask};
|
2014-07-29 01:01:02 -06:00
|
|
|
|
}
|
|
|
|
|
|
2014-08-07 03:34:30 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·unrollgcproginplace_m(void)
|
2014-07-29 01:01:02 -06:00
|
|
|
|
{
|
2014-08-19 07:38:00 -06:00
|
|
|
|
uintptr size, size0, pos, off;
|
|
|
|
|
byte *arena_start, *prog, *bitp, shift;
|
2014-08-07 03:34:30 -06:00
|
|
|
|
Type *typ;
|
|
|
|
|
void *v;
|
|
|
|
|
|
|
|
|
|
v = g->m->ptrarg[0];
|
|
|
|
|
typ = g->m->ptrarg[1];
|
|
|
|
|
size = g->m->scalararg[0];
|
|
|
|
|
size0 = g->m->scalararg[1];
|
|
|
|
|
g->m->ptrarg[0] = nil;
|
|
|
|
|
g->m->ptrarg[1] = nil;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
|
|
|
|
|
pos = 0;
|
|
|
|
|
prog = (byte*)typ->gc[1];
|
|
|
|
|
while(pos != size0)
|
|
|
|
|
unrollgcprog1(v, prog, &pos, true, true);
|
|
|
|
|
// Mark first word as bitAllocated.
|
|
|
|
|
arena_start = runtime·mheap.arena_start;
|
|
|
|
|
off = (uintptr*)v - (uintptr*)arena_start;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
bitp = arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = (off % wordsPerBitmapByte) * gcBits;
|
|
|
|
|
*bitp |= bitBoundary<<shift;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// Mark word after last as BitsDead.
|
|
|
|
|
if(size0 < size) {
|
|
|
|
|
off = (uintptr*)((byte*)v + size0) - (uintptr*)arena_start;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
bitp = arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = (off % wordsPerBitmapByte) * gcBits;
|
|
|
|
|
*bitp &= ~(bitPtrMask<<shift) | ((uintptr)BitsDead<<(shift+2));
|
2014-07-29 01:01:02 -06:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Unrolls GC program in typ->gc[1] into typ->gc[0]
|
2014-08-07 03:34:30 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·unrollgcprog_m(void)
|
2014-07-29 01:01:02 -06:00
|
|
|
|
{
|
2014-08-27 21:32:49 -06:00
|
|
|
|
static Mutex lock;
|
2014-08-07 03:34:30 -06:00
|
|
|
|
Type *typ;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
byte *mask, *prog;
|
|
|
|
|
uintptr pos;
|
|
|
|
|
uint32 x;
|
|
|
|
|
|
2014-08-07 03:34:30 -06:00
|
|
|
|
typ = g->m->ptrarg[0];
|
|
|
|
|
g->m->ptrarg[0] = nil;
|
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
runtime·lock(&lock);
|
|
|
|
|
mask = (byte*)typ->gc[0];
|
|
|
|
|
if(mask[0] == 0) {
|
|
|
|
|
pos = 8; // skip the unroll flag
|
|
|
|
|
prog = (byte*)typ->gc[1];
|
|
|
|
|
prog = unrollgcprog1(mask, prog, &pos, false, true);
|
|
|
|
|
if(prog[0] != insEnd)
|
|
|
|
|
runtime·throw("unrollgcprog: program does not end with insEnd");
|
|
|
|
|
if(((typ->size/PtrSize)%2) != 0) {
|
|
|
|
|
// repeat the program twice
|
|
|
|
|
prog = (byte*)typ->gc[1];
|
|
|
|
|
unrollgcprog1(mask, prog, &pos, false, true);
|
|
|
|
|
}
|
|
|
|
|
// atomic way to say mask[0] = 1
|
|
|
|
|
x = ((uint32*)mask)[0];
|
|
|
|
|
runtime·atomicstore((uint32*)mask, x|1);
|
|
|
|
|
}
|
|
|
|
|
runtime·unlock(&lock);
|
2013-12-18 18:13:59 -07:00
|
|
|
|
}
|
|
|
|
|
|
2011-02-02 21:03:47 -07:00
|
|
|
|
// mark the span of memory at v as having n blocks of the given size.
|
|
|
|
|
// if leftover is true, there is left over space at the end of the span.
|
|
|
|
|
void
|
|
|
|
|
runtime·markspan(void *v, uintptr size, uintptr n, bool leftover)
|
|
|
|
|
{
|
2014-08-19 07:38:00 -06:00
|
|
|
|
uintptr i, off, step;
|
|
|
|
|
byte *b;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
|
2013-05-28 12:14:47 -06:00
|
|
|
|
if((byte*)v+size*n > (byte*)runtime·mheap.arena_used || (byte*)v < runtime·mheap.arena_start)
|
2011-02-02 21:03:47 -07:00
|
|
|
|
runtime·throw("markspan: bad pointer");
|
|
|
|
|
|
2014-08-19 07:38:00 -06:00
|
|
|
|
// Find bits of the beginning of the span.
|
|
|
|
|
off = (uintptr*)v - (uintptr*)runtime·mheap.arena_start; // word offset
|
|
|
|
|
b = runtime·mheap.arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
if((off%wordsPerBitmapByte) != 0)
|
|
|
|
|
runtime·throw("markspan: unaligned length");
|
|
|
|
|
|
|
|
|
|
// Okay to use non-atomic ops here, because we control
|
|
|
|
|
// the entire span, and each bitmap byte has bits for only
|
|
|
|
|
// one span, so no other goroutines are changing these bitmap words.
|
|
|
|
|
|
|
|
|
|
if(size == PtrSize) {
|
|
|
|
|
// Possible only on 64-bits (minimal size class is 8 bytes).
|
|
|
|
|
// Poor man's memset(0x11).
|
|
|
|
|
if(0x11 != ((bitBoundary+BitsDead)<<gcBits) + (bitBoundary+BitsDead))
|
|
|
|
|
runtime·throw("markspan: bad bits");
|
|
|
|
|
if((n%(wordsPerBitmapByte*PtrSize)) != 0)
|
|
|
|
|
runtime·throw("markspan: unaligned length");
|
|
|
|
|
b = b - n/wordsPerBitmapByte + 1; // find first byte
|
|
|
|
|
if(((uintptr)b%PtrSize) != 0)
|
|
|
|
|
runtime·throw("markspan: unaligned pointer");
|
|
|
|
|
for(i = 0; i != n; i += wordsPerBitmapByte*PtrSize, b += PtrSize)
|
|
|
|
|
*(uintptr*)b = (uintptr)0x1111111111111111ULL; // bitBoundary+BitsDead
|
|
|
|
|
return;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
}
|
2014-08-19 07:38:00 -06:00
|
|
|
|
|
|
|
|
|
if(leftover)
|
|
|
|
|
n++; // mark a boundary just past end of last block too
|
|
|
|
|
step = size/(PtrSize*wordsPerBitmapByte);
|
|
|
|
|
for(i = 0; i != n; i++, b -= step)
|
|
|
|
|
*b = bitBoundary|(BitsDead<<2);
|
2011-02-02 21:03:47 -07:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// unmark the span of memory at v of length n bytes.
|
|
|
|
|
void
|
|
|
|
|
runtime·unmarkspan(void *v, uintptr n)
|
|
|
|
|
{
|
2014-08-19 07:38:00 -06:00
|
|
|
|
uintptr off;
|
|
|
|
|
byte *b;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
|
2013-05-28 12:14:47 -06:00
|
|
|
|
if((byte*)v+n > (byte*)runtime·mheap.arena_used || (byte*)v < runtime·mheap.arena_start)
|
2011-02-02 21:03:47 -07:00
|
|
|
|
runtime·throw("markspan: bad pointer");
|
|
|
|
|
|
2014-08-19 07:38:00 -06:00
|
|
|
|
off = (uintptr*)v - (uintptr*)runtime·mheap.arena_start; // word offset
|
|
|
|
|
if((off % (PtrSize*wordsPerBitmapByte)) != 0)
|
2011-02-02 21:03:47 -07:00
|
|
|
|
runtime·throw("markspan: unaligned pointer");
|
2014-08-19 07:38:00 -06:00
|
|
|
|
b = runtime·mheap.arena_start - off/wordsPerBitmapByte - 1;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
n /= PtrSize;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
if(n%(PtrSize*wordsPerBitmapByte) != 0)
|
2011-02-02 21:03:47 -07:00
|
|
|
|
runtime·throw("unmarkspan: unaligned length");
|
2011-02-16 11:21:20 -07:00
|
|
|
|
// Okay to use non-atomic ops here, because we control
|
|
|
|
|
// the entire span, and each bitmap word has bits for only
|
|
|
|
|
// one span, so no other goroutines are changing these
|
|
|
|
|
// bitmap words.
|
2014-08-19 07:38:00 -06:00
|
|
|
|
n /= wordsPerBitmapByte;
|
|
|
|
|
runtime·memclr(b - n + 1, n);
|
2011-02-02 21:03:47 -07:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
|
|
|
|
runtime·MHeap_MapBits(MHeap *h)
|
|
|
|
|
{
|
|
|
|
|
// Caller has added extra mappings to the arena.
|
|
|
|
|
// Add extra mappings of bitmap words as needed.
|
|
|
|
|
// We allocate extra bitmap pieces in chunks of bitmapChunk.
|
|
|
|
|
enum {
|
|
|
|
|
bitmapChunk = 8192
|
|
|
|
|
};
|
|
|
|
|
uintptr n;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
|
2014-08-19 07:38:00 -06:00
|
|
|
|
n = (h->arena_used - h->arena_start) / (PtrSize*wordsPerBitmapByte);
|
2013-05-28 12:04:34 -06:00
|
|
|
|
n = ROUND(n, bitmapChunk);
|
all: merge NaCl branch (part 1)
See golang.org/s/go13nacl for design overview.
This CL is the mostly mechanical changes from rsc's Go 1.2 based NaCl branch, specifically 39cb35750369 to 500771b477cf from https://code.google.com/r/rsc-go13nacl. This CL does not include working NaCl support, there are probably two or three more large merges to come.
CL 15750044 is not included as it involves more invasive changes to the linker which will need to be merged separately.
The exact change lists included are
15050047: syscall: support for Native Client
15360044: syscall: unzip implementation for Native Client
15370044: syscall: Native Client SRPC implementation
15400047: cmd/dist, cmd/go, go/build, test: support for Native Client
15410048: runtime: support for Native Client
15410049: syscall: file descriptor table for Native Client
15410050: syscall: in-memory file system for Native Client
15440048: all: update +build lines for Native Client port
15540045: cmd/6g, cmd/8g, cmd/gc: support for Native Client
15570045: os: support for Native Client
15680044: crypto/..., hash/crc32, reflect, sync/atomic: support for amd64p32
15690044: net: support for Native Client
15690048: runtime: support for fake time like on Go Playground
15690051: build: disable various tests on Native Client
LGTM=rsc
R=rsc
CC=golang-codereviews
https://golang.org/cl/68150047
2014-02-25 07:47:42 -07:00
|
|
|
|
n = ROUND(n, PhysPageSize);
|
2011-02-02 21:03:47 -07:00
|
|
|
|
if(h->bitmap_mapped >= n)
|
|
|
|
|
return;
|
|
|
|
|
|
2014-03-25 14:22:19 -06:00
|
|
|
|
runtime·SysMap(h->arena_start - n, n - h->bitmap_mapped, h->arena_reserved, &mstats.gc_sys);
|
2011-02-02 21:03:47 -07:00
|
|
|
|
h->bitmap_mapped = n;
|
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
|
|
|
|
|
static bool
|
|
|
|
|
getgcmaskcb(Stkframe *frame, void *ctxt)
|
|
|
|
|
{
|
|
|
|
|
Stkframe *frame0;
|
|
|
|
|
|
|
|
|
|
frame0 = ctxt;
|
2014-09-05 15:59:31 -06:00
|
|
|
|
if(frame->sp <= frame0->sp && frame0->sp < frame->varp) {
|
2014-07-29 01:01:02 -06:00
|
|
|
|
*frame0 = *frame;
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Returns GC type info for object p for testing.
|
|
|
|
|
void
|
|
|
|
|
runtime·getgcmask(byte *p, Type *t, byte **mask, uintptr *len)
|
|
|
|
|
{
|
|
|
|
|
Stkframe frame;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
uintptr i, n, off;
|
|
|
|
|
byte *base, bits, shift, *b;
|
runtime: convert traceback*.c to Go
The two converted files were nearly identical.
Instead of continuing that duplication, I merged them
into a single traceback.go.
Tested on arm, amd64, amd64p32, and 386.
LGTM=r
R=golang-codereviews, remyoudompheng, dave, r
CC=dvyukov, golang-codereviews, iant, khr
https://golang.org/cl/134200044
2014-09-02 13:12:53 -06:00
|
|
|
|
bool (*cb)(Stkframe*, void*);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
|
|
|
|
|
*mask = nil;
|
|
|
|
|
*len = 0;
|
|
|
|
|
|
|
|
|
|
// data
|
2014-08-27 18:15:05 -06:00
|
|
|
|
if(p >= runtime·data && p < runtime·edata) {
|
2014-07-29 01:01:02 -06:00
|
|
|
|
n = ((PtrType*)t)->elem->size;
|
|
|
|
|
*len = n/PtrSize;
|
2014-09-08 23:08:34 -06:00
|
|
|
|
*mask = runtime·mallocgc(*len, nil, FlagNoScan);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
for(i = 0; i < n; i += PtrSize) {
|
2014-08-27 18:15:05 -06:00
|
|
|
|
off = (p+i-runtime·data)/PtrSize;
|
2014-08-25 12:38:19 -06:00
|
|
|
|
bits = (((byte*)runtime·gcdatamask.data)[off/PointersPerByte] >> ((off%PointersPerByte)*BitsPerPointer))&BitsMask;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
(*mask)[i/PtrSize] = bits;
|
|
|
|
|
}
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
// bss
|
2014-08-27 18:15:05 -06:00
|
|
|
|
if(p >= runtime·bss && p < runtime·ebss) {
|
2014-07-29 01:01:02 -06:00
|
|
|
|
n = ((PtrType*)t)->elem->size;
|
|
|
|
|
*len = n/PtrSize;
|
2014-09-08 23:08:34 -06:00
|
|
|
|
*mask = runtime·mallocgc(*len, nil, FlagNoScan);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
for(i = 0; i < n; i += PtrSize) {
|
2014-08-27 18:15:05 -06:00
|
|
|
|
off = (p+i-runtime·bss)/PtrSize;
|
2014-08-25 12:38:19 -06:00
|
|
|
|
bits = (((byte*)runtime·gcbssmask.data)[off/PointersPerByte] >> ((off%PointersPerByte)*BitsPerPointer))&BitsMask;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
(*mask)[i/PtrSize] = bits;
|
|
|
|
|
}
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
// heap
|
|
|
|
|
if(runtime·mlookup(p, &base, &n, nil)) {
|
|
|
|
|
*len = n/PtrSize;
|
2014-09-08 23:08:34 -06:00
|
|
|
|
*mask = runtime·mallocgc(*len, nil, FlagNoScan);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
for(i = 0; i < n; i += PtrSize) {
|
|
|
|
|
off = (uintptr*)(base+i) - (uintptr*)runtime·mheap.arena_start;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
b = runtime·mheap.arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = (off % wordsPerBitmapByte) * gcBits;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
bits = (*b >> (shift+2))&BitsMask;
|
|
|
|
|
(*mask)[i/PtrSize] = bits;
|
|
|
|
|
}
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
// stack
|
|
|
|
|
frame.fn = nil;
|
|
|
|
|
frame.sp = (uintptr)p;
|
runtime: convert traceback*.c to Go
The two converted files were nearly identical.
Instead of continuing that duplication, I merged them
into a single traceback.go.
Tested on arm, amd64, amd64p32, and 386.
LGTM=r
R=golang-codereviews, remyoudompheng, dave, r
CC=dvyukov, golang-codereviews, iant, khr
https://golang.org/cl/134200044
2014-09-02 13:12:53 -06:00
|
|
|
|
cb = getgcmaskcb;
|
2014-09-05 15:59:31 -06:00
|
|
|
|
runtime·gentraceback(g->m->curg->sched.pc, g->m->curg->sched.sp, 0, g->m->curg, 0, nil, 1000, &cb, &frame, false);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(frame.fn != nil) {
|
|
|
|
|
Func *f;
|
|
|
|
|
StackMap *stackmap;
|
|
|
|
|
BitVector bv;
|
|
|
|
|
uintptr size;
|
|
|
|
|
uintptr targetpc;
|
|
|
|
|
int32 pcdata;
|
|
|
|
|
|
|
|
|
|
f = frame.fn;
|
|
|
|
|
targetpc = frame.continpc;
|
|
|
|
|
if(targetpc == 0)
|
|
|
|
|
return;
|
|
|
|
|
if(targetpc != f->entry)
|
|
|
|
|
targetpc--;
|
|
|
|
|
pcdata = runtime·pcdatavalue(f, PCDATA_StackMapIndex, targetpc);
|
|
|
|
|
if(pcdata == -1)
|
|
|
|
|
return;
|
|
|
|
|
stackmap = runtime·funcdata(f, FUNCDATA_LocalsPointerMaps);
|
|
|
|
|
if(stackmap == nil || stackmap->n <= 0)
|
|
|
|
|
return;
|
|
|
|
|
bv = runtime·stackmapdata(stackmap, pcdata);
|
|
|
|
|
size = bv.n/BitsPerPointer*PtrSize;
|
|
|
|
|
n = ((PtrType*)t)->elem->size;
|
|
|
|
|
*len = n/PtrSize;
|
2014-09-08 23:08:34 -06:00
|
|
|
|
*mask = runtime·mallocgc(*len, nil, FlagNoScan);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
for(i = 0; i < n; i += PtrSize) {
|
2014-09-01 08:05:16 -06:00
|
|
|
|
off = (p+i-(byte*)frame.varp+size)/PtrSize;
|
2014-08-15 12:36:12 -06:00
|
|
|
|
bits = (bv.data[off*BitsPerPointer/32] >> ((off*BitsPerPointer)%32))&BitsMask;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
(*mask)[i/PtrSize] = bits;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
2014-08-25 10:25:22 -06:00
|
|
|
|
|
|
|
|
|
void runtime·gc_unixnanotime(int64 *now);
|
|
|
|
|
|
liblink, runtime: diagnose and fix C code running on Go stack
This CL contains compiler+runtime changes that detect C code
running on Go (not g0, not gsignal) stacks, and it contains
corrections for what it detected.
The detection works by changing the C prologue to use a different
stack guard word in the G than Go prologue does. On the g0 and
gsignal stacks, that stack guard word is set to the usual
stack guard value. But on ordinary Go stacks, that stack
guard word is set to ^0, which will make any stack split
check fail. The C prologue then calls morestackc instead
of morestack, and morestackc aborts the program with
a message about running C code on a Go stack.
This check catches all C code running on the Go stack
except NOSPLIT code. The NOSPLIT code is allowed,
so the check is complete. Since it is a dynamic check,
the code must execute to be caught. But unlike the static
checks we've been using in cmd/ld, the dynamic check
works with function pointers and other indirect calls.
For example it caught sigpanic being pushed onto Go
stacks in the signal handlers.
Fixes #8667.
LGTM=khr, iant
R=golang-codereviews, khr, iant
CC=golang-codereviews, r
https://golang.org/cl/133700043
2014-09-08 12:05:23 -06:00
|
|
|
|
int64
|
|
|
|
|
runtime·unixnanotime(void)
|
2014-08-25 10:25:22 -06:00
|
|
|
|
{
|
|
|
|
|
int64 now;
|
|
|
|
|
|
|
|
|
|
runtime·gc_unixnanotime(&now);
|
|
|
|
|
return now;
|
|
|
|
|
}
|