2009-01-26 18:37:05 -07:00
|
|
|
|
// Copyright 2009 The Go Authors. All rights reserved.
|
|
|
|
|
// Use of this source code is governed by a BSD-style
|
|
|
|
|
// license that can be found in the LICENSE file.
|
|
|
|
|
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
// Garbage collector (GC).
|
|
|
|
|
//
|
|
|
|
|
// GC is:
|
|
|
|
|
// - mark&sweep
|
|
|
|
|
// - mostly precise (with the exception of some C-allocated objects, assembly frames/arguments, etc)
|
|
|
|
|
// - parallel (up to MaxGcproc threads)
|
|
|
|
|
// - partially concurrent (mark is stop-the-world, while sweep is concurrent)
|
|
|
|
|
// - non-moving/non-compacting
|
|
|
|
|
// - full (non-partial)
|
|
|
|
|
//
|
|
|
|
|
// GC rate.
|
|
|
|
|
// Next GC is after we've allocated an extra amount of memory proportional to
|
|
|
|
|
// the amount already in use. The proportion is controlled by GOGC environment variable
|
|
|
|
|
// (100 by default). If GOGC=100 and we're using 4M, we'll GC again when we get to 8M
|
|
|
|
|
// (this mark is tracked in next_gc variable). This keeps the GC cost in linear
|
|
|
|
|
// proportion to the allocation cost. Adjusting GOGC just changes the linear constant
|
|
|
|
|
// (and also the amount of extra memory used).
|
|
|
|
|
//
|
|
|
|
|
// Concurrent sweep.
|
|
|
|
|
// The sweep phase proceeds concurrently with normal program execution.
|
|
|
|
|
// The heap is swept span-by-span both lazily (when a goroutine needs another span)
|
|
|
|
|
// and concurrently in a background goroutine (this helps programs that are not CPU bound).
|
|
|
|
|
// However, at the end of the stop-the-world GC phase we don't know the size of the live heap,
|
|
|
|
|
// and so next_gc calculation is tricky and happens as follows.
|
|
|
|
|
// At the end of the stop-the-world phase next_gc is conservatively set based on total
|
|
|
|
|
// heap size; all spans are marked as "needs sweeping".
|
|
|
|
|
// Whenever a span is swept, next_gc is decremented by GOGC*newly_freed_memory.
|
|
|
|
|
// The background sweeper goroutine simply sweeps spans one-by-one bringing next_gc
|
|
|
|
|
// closer to the target value. However, this is not enough to avoid over-allocating memory.
|
|
|
|
|
// Consider that a goroutine wants to allocate a new span for a large object and
|
|
|
|
|
// there are no free swept spans, but there are small-object unswept spans.
|
|
|
|
|
// If the goroutine naively allocates a new span, it can surpass the yet-unknown
|
|
|
|
|
// target next_gc value. In order to prevent such cases (1) when a goroutine needs
|
|
|
|
|
// to allocate a new small-object span, it sweeps small-object spans for the same
|
|
|
|
|
// object size until it frees at least one object; (2) when a goroutine needs to
|
|
|
|
|
// allocate large-object span from heap, it sweeps spans until it frees at least
|
|
|
|
|
// that many pages into heap. Together these two measures ensure that we don't surpass
|
|
|
|
|
// target next_gc value by a large margin. There is an exception: if a goroutine sweeps
|
|
|
|
|
// and frees two nonadjacent one-page spans to the heap, it will allocate a new two-page span,
|
|
|
|
|
// but there can still be other one-page unswept spans which could be combined into a two-page span.
|
|
|
|
|
// It's critical to ensure that no operations proceed on unswept spans (that would corrupt
|
|
|
|
|
// mark bits in GC bitmap). During GC all mcaches are flushed into the central cache,
|
|
|
|
|
// so they are empty. When a goroutine grabs a new span into mcache, it sweeps it.
|
|
|
|
|
// When a goroutine explicitly frees an object or sets a finalizer, it ensures that
|
|
|
|
|
// the span is swept (either by sweeping it, or by waiting for the concurrent sweep to finish).
|
|
|
|
|
// The finalizer goroutine is kicked off only when all spans are swept.
|
|
|
|
|
// When the next GC starts, it sweeps all not-yet-swept spans (if any).
|
2009-01-26 18:37:05 -07:00
|
|
|
|
|
|
|
|
|
#include "runtime.h"
|
2011-12-16 13:33:58 -07:00
|
|
|
|
#include "arch_GOARCH.h"
|
2009-01-26 18:37:05 -07:00
|
|
|
|
#include "malloc.h"
|
runtime: stack split + garbage collection bug
The g->sched.sp saved stack pointer and the
g->stackbase and g->stackguard stack bounds
can change even while "the world is stopped",
because a goroutine has to call functions (and
therefore might split its stack) when exiting a
system call to check whether the world is stopped
(and if so, wait until the world continues).
That means the garbage collector cannot access
those values safely (without a race) for goroutines
executing system calls. Instead, save a consistent
triple in g->gcsp, g->gcstack, g->gcguard during
entersyscall and have the garbage collector refer
to those.
The old code was occasionally seeing (because of
the race) an sp and stk that did not correspond to
each other, so that stk - sp was not the number of
stack bytes following sp. In that case, if sp < stk
then the call scanblock(sp, stk - sp) scanned too
many bytes (anything between the two pointers,
which pointed into different allocation blocks).
If sp > stk then stk - sp wrapped around.
On 32-bit, stk - sp is a uintptr (uint32) converted
to int64 in the call to scanblock, so a large (~4G)
but positive number. Scanblock would try to scan
that many bytes and eventually fault accessing
unmapped memory. On 64-bit, stk - sp is a uintptr (uint64)
promoted to int64 in the call to scanblock, so a negative
number. Scanblock would not scan anything, possibly
causing in-use blocks to be freed.
In short, 32-bit platforms would have seen either
ineffective garbage collection or crashes during garbage
collection, while 64-bit platforms would have seen
either ineffective or incorrect garbage collection.
You can see the invalid arguments to scanblock in the
stack traces in issue 1620.
Fixes #1620.
Fixes #1746.
R=iant, r
CC=golang-dev
https://golang.org/cl/4437075
2011-04-27 21:21:12 -06:00
|
|
|
|
#include "stack.h"
|
2012-12-16 17:32:12 -07:00
|
|
|
|
#include "mgc0.h"
|
reflect, runtime: fix crash in GC due to reflect.call + precise GC
Given
type Outer struct {
*Inner
...
}
the compiler generates the implementation of (*Outer).M dispatching to
the embedded Inner. The implementation is logically:
func (p *Outer) M() {
(p.Inner).M()
}
but since the only change here is the replacement of one pointer
receiver with another, the actual generated code overwrites the
original receiver with the p.Inner pointer and then jumps to the M
method expecting the *Inner receiver.
During reflect.Value.Call, we create an argument frame and the
associated data structures to describe it to the garbage collector,
populate the frame, call reflect.call to run a function call using
that frame, and then copy the results back out of the frame. The
reflect.call function does a memmove of the frame structure onto the
stack (to set up the inputs), runs the call, and the memmoves the
stack back to the frame structure (to preserve the outputs).
Originally reflect.call did not distinguish inputs from outputs: both
memmoves were for the full stack frame. However, in the case where the
called function was one of these wrappers, the rewritten receiver is
almost certainly a different type than the original receiver. This is
not a problem on the stack, where we use the program counter to
determine the type information and understand that during (*Outer).M
the receiver is an *Outer while during (*Inner).M the receiver in the
same memory word is now an *Inner. But in the statically typed
argument frame created by reflect, the receiver is always an *Outer.
Copying the modified receiver pointer off the stack into the frame
will store an *Inner there, and then if a garbage collection happens
to scan that argument frame before it is discarded, it will scan the
*Inner memory as if it were an *Outer. If the two have different
memory layouts, the collection will intepret the memory incorrectly.
Fix by only copying back the results.
Fixes #7725.
LGTM=khr
R=khr
CC=dave, golang-codereviews
https://golang.org/cl/85180043
2014-04-08 09:11:35 -06:00
|
|
|
|
#include "chan.h"
|
2012-10-07 12:05:32 -06:00
|
|
|
|
#include "race.h"
|
2013-01-10 13:45:46 -07:00
|
|
|
|
#include "type.h"
|
|
|
|
|
#include "typekind.h"
|
2013-07-19 14:04:09 -06:00
|
|
|
|
#include "funcdata.h"
|
2013-08-29 13:36:59 -06:00
|
|
|
|
#include "../../cmd/ld/textflag.h"
|
2009-01-26 18:37:05 -07:00
|
|
|
|
|
|
|
|
|
enum {
|
2014-07-29 01:01:02 -06:00
|
|
|
|
Debug = 0,
|
|
|
|
|
ConcurrentSweep = 1,
|
|
|
|
|
PreciseScan = 1,
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
WorkbufSize = 4*1024,
|
2014-01-30 02:28:19 -07:00
|
|
|
|
FinBlockSize = 4*1024,
|
2014-01-21 02:06:57 -07:00
|
|
|
|
RootData = 0,
|
|
|
|
|
RootBss = 1,
|
|
|
|
|
RootFinalizers = 2,
|
2014-07-29 01:01:02 -06:00
|
|
|
|
RootSpans = 3,
|
2014-01-21 02:06:57 -07:00
|
|
|
|
RootFlushCaches = 4,
|
|
|
|
|
RootCount = 5,
|
2009-01-26 18:37:05 -07:00
|
|
|
|
};
|
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
#define ScanConservatively ((byte*)1)
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
|
|
|
|
|
// Initialized from $GOGC. GOGC=off means no gc.
|
2014-08-18 06:42:24 -06:00
|
|
|
|
extern int32 runtime·gcpercent;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
|
2014-04-14 11:13:32 -06:00
|
|
|
|
static FuncVal* poolcleanup;
|
2013-12-18 12:08:34 -07:00
|
|
|
|
|
|
|
|
|
void
|
2014-04-14 11:13:32 -06:00
|
|
|
|
sync·runtime_registerPoolCleanup(FuncVal *f)
|
2013-12-18 12:08:34 -07:00
|
|
|
|
{
|
2014-04-14 11:13:32 -06:00
|
|
|
|
poolcleanup = f;
|
2013-12-18 12:08:34 -07:00
|
|
|
|
}
|
|
|
|
|
|
2014-07-30 10:01:52 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·clearpools(void)
|
2013-12-18 12:08:34 -07:00
|
|
|
|
{
|
2014-01-21 00:20:23 -07:00
|
|
|
|
P *p, **pp;
|
2014-01-27 13:26:56 -07:00
|
|
|
|
MCache *c;
|
2014-01-21 00:20:23 -07:00
|
|
|
|
int32 i;
|
2013-12-18 12:08:34 -07:00
|
|
|
|
|
2014-01-21 00:20:23 -07:00
|
|
|
|
// clear sync.Pool's
|
2014-04-14 11:13:32 -06:00
|
|
|
|
if(poolcleanup != nil)
|
|
|
|
|
reflect·call(poolcleanup, nil, 0, 0);
|
2014-01-21 00:20:23 -07:00
|
|
|
|
|
|
|
|
|
for(pp=runtime·allp; p=*pp; pp++) {
|
2014-01-24 11:35:11 -07:00
|
|
|
|
// clear tinyalloc pool
|
2014-01-27 13:26:56 -07:00
|
|
|
|
c = p->mcache;
|
|
|
|
|
if(c != nil) {
|
|
|
|
|
c->tiny = nil;
|
|
|
|
|
c->tinysize = 0;
|
2014-08-21 10:41:09 -06:00
|
|
|
|
c->sudogcache = nil;
|
2014-01-27 13:26:56 -07:00
|
|
|
|
}
|
2014-01-24 11:35:11 -07:00
|
|
|
|
// clear defer pools
|
2014-01-21 00:20:23 -07:00
|
|
|
|
for(i=0; i<nelem(p->deferpool); i++)
|
|
|
|
|
p->deferpool[i] = nil;
|
|
|
|
|
}
|
2013-12-18 12:08:34 -07:00
|
|
|
|
}
|
|
|
|
|
|
2012-02-22 19:45:01 -07:00
|
|
|
|
// Holding worldsema grants an M the right to try to stop the world.
|
|
|
|
|
// The procedure is:
|
|
|
|
|
//
|
|
|
|
|
// runtime·semacquire(&runtime·worldsema);
|
|
|
|
|
// m->gcing = 1;
|
|
|
|
|
// runtime·stoptheworld();
|
|
|
|
|
//
|
|
|
|
|
// ... do stuff ...
|
|
|
|
|
//
|
|
|
|
|
// m->gcing = 0;
|
|
|
|
|
// runtime·semrelease(&runtime·worldsema);
|
|
|
|
|
// runtime·starttheworld();
|
|
|
|
|
//
|
|
|
|
|
uint32 runtime·worldsema = 1;
|
|
|
|
|
|
2011-02-02 21:03:47 -07:00
|
|
|
|
typedef struct Workbuf Workbuf;
|
|
|
|
|
struct Workbuf
|
2010-09-07 07:57:22 -06:00
|
|
|
|
{
|
2014-07-29 01:01:02 -06:00
|
|
|
|
LFNode node; // must be first
|
|
|
|
|
uintptr nobj;
|
|
|
|
|
byte* obj[(WorkbufSize-sizeof(LFNode)-sizeof(uintptr))/PtrSize];
|
2010-09-07 07:57:22 -06:00
|
|
|
|
};
|
|
|
|
|
|
2011-10-06 09:42:51 -06:00
|
|
|
|
typedef struct Finalizer Finalizer;
|
|
|
|
|
struct Finalizer
|
|
|
|
|
{
|
2013-02-21 15:01:13 -07:00
|
|
|
|
FuncVal *fn;
|
2011-10-06 09:42:51 -06:00
|
|
|
|
void *arg;
|
2012-09-24 12:58:34 -06:00
|
|
|
|
uintptr nret;
|
2013-08-14 12:54:31 -06:00
|
|
|
|
Type *fint;
|
2013-07-29 09:43:08 -06:00
|
|
|
|
PtrType *ot;
|
2011-10-06 09:42:51 -06:00
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
typedef struct FinBlock FinBlock;
|
|
|
|
|
struct FinBlock
|
|
|
|
|
{
|
|
|
|
|
FinBlock *alllink;
|
|
|
|
|
FinBlock *next;
|
|
|
|
|
int32 cnt;
|
|
|
|
|
int32 cap;
|
|
|
|
|
Finalizer fin[1];
|
|
|
|
|
};
|
|
|
|
|
|
2009-08-20 17:09:38 -06:00
|
|
|
|
extern byte data[];
|
2012-12-16 17:32:12 -07:00
|
|
|
|
extern byte edata[];
|
|
|
|
|
extern byte bss[];
|
2012-02-21 20:08:42 -07:00
|
|
|
|
extern byte ebss[];
|
2009-01-26 18:37:05 -07:00
|
|
|
|
|
2012-12-16 17:32:12 -07:00
|
|
|
|
extern byte gcdata[];
|
|
|
|
|
extern byte gcbss[];
|
|
|
|
|
|
2014-03-26 05:11:36 -06:00
|
|
|
|
static Lock finlock; // protects the following variables
|
|
|
|
|
static FinBlock *finq; // list of finalizers that are to be executed
|
|
|
|
|
static FinBlock *finc; // cache of free blocks
|
|
|
|
|
static FinBlock *allfin; // list of all blocks
|
|
|
|
|
bool runtime·fingwait;
|
|
|
|
|
bool runtime·fingwake;
|
2014-08-18 06:42:24 -06:00
|
|
|
|
byte* runtime·gcdatamask;
|
|
|
|
|
byte* runtime·gcbssmask;
|
2014-03-26 05:11:36 -06:00
|
|
|
|
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
static Lock gclock;
|
2010-04-07 21:38:02 -06:00
|
|
|
|
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
static void runfinq(void);
|
|
|
|
|
static void bgsweep(void);
|
2011-02-02 21:03:47 -07:00
|
|
|
|
static Workbuf* getempty(Workbuf*);
|
|
|
|
|
static Workbuf* getfull(Workbuf*);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
static void putempty(Workbuf*);
|
|
|
|
|
static Workbuf* handoff(Workbuf*);
|
2013-03-21 02:48:02 -06:00
|
|
|
|
static void gchelperstart(void);
|
2014-01-21 02:06:57 -07:00
|
|
|
|
static void flushallmcaches(void);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
static bool scanframe(Stkframe *frame, void *unused);
|
|
|
|
|
static void scanstack(G *gp);
|
|
|
|
|
static byte* unrollglobgcprog(byte *prog, uintptr size);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
static FuncVal runfinqv = {runfinq};
|
|
|
|
|
static FuncVal bgsweepv = {bgsweep};
|
|
|
|
|
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
static struct {
|
2012-05-24 00:55:50 -06:00
|
|
|
|
uint64 full; // lock-free list of full blocks
|
|
|
|
|
uint64 empty; // lock-free list of empty blocks
|
|
|
|
|
byte pad0[CacheLineSize]; // prevents false-sharing between full/empty and nproc/nwait
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
uint32 nproc;
|
2014-01-16 01:54:46 -07:00
|
|
|
|
int64 tstart;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
volatile uint32 nwait;
|
|
|
|
|
volatile uint32 ndone;
|
|
|
|
|
Note alldone;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
ParFor* markfor;
|
2014-08-24 02:05:07 -06:00
|
|
|
|
|
|
|
|
|
// Copy of mheap.allspans for marker or sweeper.
|
|
|
|
|
MSpan** spans;
|
|
|
|
|
uint32 nspan;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
} work;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
|
2012-12-16 17:32:12 -07:00
|
|
|
|
// scanblock scans a block of n bytes starting at pointer b for references
|
|
|
|
|
// to other objects, scanning any it finds recursively until there are no
|
|
|
|
|
// unscanned objects left. Instead of using an explicit recursion, it keeps
|
|
|
|
|
// a work list in the Workbuf* structures and loops in the main function
|
|
|
|
|
// body. Keeping an explicit work list is easier on the stack allocator and
|
|
|
|
|
// more efficient.
|
|
|
|
|
static void
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanblock(byte *b, uintptr n, byte *ptrmask)
|
2012-12-16 17:32:12 -07:00
|
|
|
|
{
|
2014-08-19 07:38:00 -06:00
|
|
|
|
byte *obj, *p, *arena_start, *arena_used, **wp, *scanbuf[8], *ptrbitp, *bitp, bits, xbits, shift, cached;
|
|
|
|
|
uintptr i, nobj, size, idx, x, off, scanbufpos;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
intptr ncached;
|
|
|
|
|
Workbuf *wbuf;
|
|
|
|
|
String *str;
|
|
|
|
|
Slice *slice;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
Iface *iface;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
Eface *eface;
|
|
|
|
|
Type *typ;
|
|
|
|
|
MSpan *s;
|
|
|
|
|
PageID k;
|
|
|
|
|
bool keepworking;
|
2012-12-16 17:32:12 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// Cache memory arena parameters in local vars.
|
2013-05-28 12:14:47 -06:00
|
|
|
|
arena_start = runtime·mheap.arena_start;
|
|
|
|
|
arena_used = runtime·mheap.arena_used;
|
2012-12-16 17:32:12 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
wbuf = getempty(nil);
|
|
|
|
|
nobj = wbuf->nobj;
|
|
|
|
|
wp = &wbuf->obj[nobj];
|
|
|
|
|
keepworking = b == nil;
|
|
|
|
|
scanbufpos = 0;
|
|
|
|
|
for(i = 0; i < nelem(scanbuf); i++)
|
|
|
|
|
scanbuf[i] = nil;
|
|
|
|
|
|
2014-08-19 07:38:00 -06:00
|
|
|
|
ptrbitp = nil;
|
|
|
|
|
cached = 0;
|
|
|
|
|
ncached = 0;
|
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// ptrmask can have 3 possible values:
|
|
|
|
|
// 1. nil - obtain pointer mask from GC bitmap.
|
|
|
|
|
// 2. ScanConservatively - don't use any mask, scan conservatively.
|
|
|
|
|
// 3. pointer to a compact mask (for stacks and data).
|
|
|
|
|
if(b != nil)
|
|
|
|
|
goto scanobj;
|
2012-12-16 17:32:12 -07:00
|
|
|
|
for(;;) {
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(nobj == 0) {
|
|
|
|
|
// Out of work in workbuf.
|
|
|
|
|
// First, see is there is any work in scanbuf.
|
|
|
|
|
for(i = 0; i < nelem(scanbuf); i++) {
|
|
|
|
|
b = scanbuf[scanbufpos];
|
|
|
|
|
scanbuf[scanbufpos++] = nil;
|
|
|
|
|
if(scanbufpos == nelem(scanbuf))
|
|
|
|
|
scanbufpos = 0;
|
|
|
|
|
if(b != nil) {
|
|
|
|
|
n = arena_used - b; // scan until bitBoundary or BitsDead
|
|
|
|
|
ptrmask = nil; // use GC bitmap for pointer info
|
|
|
|
|
goto scanobj;
|
2013-04-08 14:36:35 -06:00
|
|
|
|
}
|
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(!keepworking) {
|
|
|
|
|
putempty(wbuf);
|
|
|
|
|
return;
|
2013-01-18 14:56:17 -07:00
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// Refill workbuf from global queue.
|
|
|
|
|
wbuf = getfull(wbuf);
|
|
|
|
|
if(wbuf == nil)
|
|
|
|
|
return;
|
|
|
|
|
nobj = wbuf->nobj;
|
|
|
|
|
wp = &wbuf->obj[nobj];
|
2013-01-10 13:45:46 -07:00
|
|
|
|
}
|
2012-12-16 17:32:12 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// If another proc wants a pointer, give it some.
|
|
|
|
|
if(work.nwait > 0 && nobj > 4 && work.full == 0) {
|
|
|
|
|
wbuf->nobj = nobj;
|
|
|
|
|
wbuf = handoff(wbuf);
|
|
|
|
|
nobj = wbuf->nobj;
|
|
|
|
|
wp = &wbuf->obj[nobj];
|
|
|
|
|
}
|
2013-03-15 14:07:52 -06:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
wp--;
|
|
|
|
|
nobj--;
|
|
|
|
|
b = *wp;
|
|
|
|
|
n = arena_used - b; // scan until next bitBoundary or BitsDead
|
|
|
|
|
ptrmask = nil; // use GC bitmap for pointer info
|
|
|
|
|
|
|
|
|
|
scanobj:
|
|
|
|
|
if(!PreciseScan) {
|
|
|
|
|
if(ptrmask == nil) {
|
|
|
|
|
// Heap obj, obtain real size.
|
|
|
|
|
if(!runtime·mlookup(b, &p, &n, nil))
|
|
|
|
|
continue; // not an allocated obj
|
|
|
|
|
if(b != p)
|
|
|
|
|
runtime·throw("bad heap object");
|
2013-03-15 14:07:52 -06:00
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
ptrmask = ScanConservatively;
|
|
|
|
|
}
|
2014-08-19 07:38:00 -06:00
|
|
|
|
// Find bits of the beginning of the object.
|
|
|
|
|
if(ptrmask == nil) {
|
|
|
|
|
off = (uintptr*)b - (uintptr*)arena_start;
|
|
|
|
|
ptrbitp = arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = (off % wordsPerBitmapByte) * gcBits;
|
|
|
|
|
cached = *ptrbitp >> shift;
|
|
|
|
|
cached &= ~bitBoundary;
|
|
|
|
|
ncached = (8 - shift)/gcBits;
|
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
for(i = 0; i < n; i += PtrSize) {
|
|
|
|
|
obj = nil;
|
|
|
|
|
// Find bits for this word.
|
|
|
|
|
if(ptrmask == nil) {
|
|
|
|
|
// Check is we have reached end of span.
|
|
|
|
|
if((((uintptr)b+i)%PageSize) == 0 &&
|
|
|
|
|
runtime·mheap.spans[(b-arena_start)>>PageShift] != runtime·mheap.spans[(b+i-arena_start)>>PageShift])
|
|
|
|
|
break;
|
|
|
|
|
// Consult GC bitmap.
|
|
|
|
|
if(ncached <= 0) {
|
|
|
|
|
// Refill cache.
|
2014-08-19 07:38:00 -06:00
|
|
|
|
cached = *--ptrbitp;
|
|
|
|
|
ncached = 2;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
bits = cached;
|
|
|
|
|
cached >>= gcBits;
|
|
|
|
|
ncached--;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
if((bits&bitBoundary) != 0)
|
2014-07-29 01:01:02 -06:00
|
|
|
|
break; // reached beginning of the next object
|
|
|
|
|
bits = (bits>>2)&BitsMask;
|
|
|
|
|
if(bits == BitsDead)
|
|
|
|
|
break; // reached no-scan part of the object
|
|
|
|
|
} else if(ptrmask != ScanConservatively) // dense mask (stack or data)
|
|
|
|
|
bits = (ptrmask[(i/PtrSize)/4]>>(((i/PtrSize)%4)*BitsPerPointer))&BitsMask;
|
|
|
|
|
else
|
|
|
|
|
bits = BitsPointer;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(bits == BitsScalar || bits == BitsDead)
|
2013-03-15 14:07:52 -06:00
|
|
|
|
continue;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(bits == BitsPointer) {
|
|
|
|
|
obj = *(byte**)(b+i);
|
|
|
|
|
goto markobj;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// Find the next pair of bits.
|
|
|
|
|
if(ptrmask == nil) {
|
|
|
|
|
if(ncached <= 0) {
|
2014-08-19 07:38:00 -06:00
|
|
|
|
// Refill cache.
|
|
|
|
|
cached = *--ptrbitp;
|
|
|
|
|
ncached = 2;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
bits = (cached>>2)&BitsMask;
|
|
|
|
|
} else
|
|
|
|
|
bits = (ptrmask[((i+PtrSize)/PtrSize)/4]>>((((i+PtrSize)/PtrSize)%4)*BitsPerPointer))&BitsMask;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
switch(bits) {
|
|
|
|
|
case BitsString:
|
|
|
|
|
str = (String*)(b+i);
|
|
|
|
|
if(str->len > 0)
|
|
|
|
|
obj = str->str;
|
|
|
|
|
break;
|
|
|
|
|
case BitsSlice:
|
|
|
|
|
slice = (Slice*)(b+i);
|
|
|
|
|
if(Debug && slice->cap < slice->len) {
|
|
|
|
|
g->m->traceback = 2;
|
|
|
|
|
runtime·printf("bad slice in object %p: %p/%p/%p\n",
|
|
|
|
|
b, slice->array, slice->len, slice->cap);
|
|
|
|
|
runtime·throw("bad slice in heap object");
|
2013-01-10 13:45:46 -07:00
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(slice->cap > 0)
|
|
|
|
|
obj = slice->array;
|
|
|
|
|
break;
|
|
|
|
|
case BitsIface:
|
|
|
|
|
iface = (Iface*)(b+i);
|
|
|
|
|
if(iface->tab != nil) {
|
|
|
|
|
typ = iface->tab->type;
|
2014-08-18 19:13:11 -06:00
|
|
|
|
if(!(typ->kind&KindDirectIface) || !(typ->kind&KindNoPointers))
|
2014-07-29 01:01:02 -06:00
|
|
|
|
obj = iface->data;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
break;
|
|
|
|
|
case BitsEface:
|
|
|
|
|
eface = (Eface*)(b+i);
|
|
|
|
|
typ = eface->type;
|
|
|
|
|
if(typ != nil) {
|
2014-08-18 19:13:11 -06:00
|
|
|
|
if(!(typ->kind&KindDirectIface) || !(typ->kind&KindNoPointers))
|
2014-07-29 01:01:02 -06:00
|
|
|
|
obj = eface->data;
|
2012-12-16 17:32:12 -07:00
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
break;
|
2012-12-16 17:32:12 -07:00
|
|
|
|
}
|
2013-01-10 13:45:46 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(bits == BitsSlice) {
|
|
|
|
|
i += 2*PtrSize;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
if(ncached == 2)
|
|
|
|
|
ncached = 0;
|
|
|
|
|
else if(ptrmask == nil) {
|
|
|
|
|
// Refill cache and consume one quadruple.
|
|
|
|
|
cached = *--ptrbitp;
|
|
|
|
|
cached >>= gcBits;
|
|
|
|
|
ncached = 1;
|
|
|
|
|
}
|
2013-01-10 13:45:46 -07:00
|
|
|
|
} else {
|
2014-07-29 01:01:02 -06:00
|
|
|
|
i += PtrSize;
|
|
|
|
|
cached >>= gcBits;
|
|
|
|
|
ncached--;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
}
|
2012-12-16 17:32:12 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
markobj:
|
|
|
|
|
// At this point we have extracted the next potential pointer.
|
|
|
|
|
// Check if it points into heap.
|
|
|
|
|
if(obj == nil || obj < arena_start || obj >= arena_used)
|
2013-03-19 12:51:03 -06:00
|
|
|
|
continue;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// Mark the object.
|
|
|
|
|
off = (uintptr*)obj - (uintptr*)arena_start;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
bitp = arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = (off % wordsPerBitmapByte) * gcBits;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
xbits = *bitp;
|
|
|
|
|
bits = (xbits >> shift) & bitMask;
|
2014-08-13 10:42:55 -06:00
|
|
|
|
if((bits&bitBoundary) == 0) {
|
2014-08-19 07:38:00 -06:00
|
|
|
|
// Not a beginning of a block, consult span table to find the block beginning.
|
2014-07-29 01:01:02 -06:00
|
|
|
|
k = (uintptr)obj>>PageShift;
|
|
|
|
|
x = k;
|
|
|
|
|
x -= (uintptr)arena_start>>PageShift;
|
|
|
|
|
s = runtime·mheap.spans[x];
|
|
|
|
|
if(s == nil || k < s->start || obj >= s->limit || s->state != MSpanInUse)
|
2013-03-19 12:51:03 -06:00
|
|
|
|
continue;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
p = (byte*)((uintptr)s->start<<PageShift);
|
|
|
|
|
if(s->sizeclass != 0) {
|
|
|
|
|
size = s->elemsize;
|
|
|
|
|
idx = ((byte*)obj - p)/size;
|
|
|
|
|
p = p+idx*size;
|
2013-03-19 12:51:03 -06:00
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(p == obj) {
|
2014-08-13 10:42:55 -06:00
|
|
|
|
runtime·printf("runtime: failed to find block beginning for %p s=%p s->limit=%p\n",
|
|
|
|
|
p, s->start*PageSize, s->limit);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
runtime·throw("failed to find block beginning");
|
|
|
|
|
}
|
|
|
|
|
obj = p;
|
|
|
|
|
goto markobj;
|
2013-03-19 12:51:03 -06:00
|
|
|
|
}
|
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// Now we have bits, bitp, and shift correct for
|
|
|
|
|
// obj pointing at the base of the object.
|
2014-08-13 10:42:55 -06:00
|
|
|
|
// Only care about not marked objects.
|
|
|
|
|
if((bits&bitMarked) != 0)
|
2014-07-29 01:01:02 -06:00
|
|
|
|
continue;
|
2014-08-14 11:38:24 -06:00
|
|
|
|
// If obj size is greater than 8, then each byte of GC bitmap
|
|
|
|
|
// contains info for at most one object. In such case we use
|
|
|
|
|
// non-atomic byte store to mark the object. This can lead
|
|
|
|
|
// to double enqueue of the object for scanning, but scanning
|
|
|
|
|
// is an idempotent operation, so it is OK. This cannot lead
|
|
|
|
|
// to bitmap corruption because the single marked bit is the
|
|
|
|
|
// only thing that can change in the byte.
|
|
|
|
|
// For 8-byte objects we use non-atomic store, if the other
|
|
|
|
|
// quadruple is already marked. Otherwise we resort to CAS
|
|
|
|
|
// loop for marking.
|
2014-08-19 07:38:00 -06:00
|
|
|
|
if((xbits&(bitMask|(bitMask<<gcBits))) != (bitBoundary|(bitBoundary<<gcBits)) ||
|
2014-08-15 23:07:55 -06:00
|
|
|
|
work.nproc == 1)
|
2014-08-19 07:38:00 -06:00
|
|
|
|
*bitp = xbits | (bitMarked<<shift);
|
|
|
|
|
else
|
|
|
|
|
runtime·atomicor8(bitp, bitMarked<<shift);
|
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(((xbits>>(shift+2))&BitsMask) == BitsDead)
|
|
|
|
|
continue; // noscan object
|
|
|
|
|
|
|
|
|
|
// Queue the obj for scanning.
|
|
|
|
|
PREFETCH(obj);
|
|
|
|
|
obj = (byte*)((uintptr)obj & ~(PtrSize-1));
|
|
|
|
|
p = scanbuf[scanbufpos];
|
|
|
|
|
scanbuf[scanbufpos++] = obj;
|
|
|
|
|
if(scanbufpos == nelem(scanbuf))
|
|
|
|
|
scanbufpos = 0;
|
|
|
|
|
if(p == nil)
|
|
|
|
|
continue;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// If workbuf is full, obtain an empty one.
|
|
|
|
|
if(nobj >= nelem(wbuf->obj)) {
|
|
|
|
|
wbuf->nobj = nobj;
|
|
|
|
|
wbuf = getempty(wbuf);
|
|
|
|
|
nobj = wbuf->nobj;
|
|
|
|
|
wp = &wbuf->obj[nobj];
|
|
|
|
|
}
|
|
|
|
|
*wp = p;
|
|
|
|
|
wp++;
|
|
|
|
|
nobj++;
|
2013-01-10 13:45:46 -07:00
|
|
|
|
}
|
2012-12-16 17:32:12 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(Debug && ptrmask == nil) {
|
|
|
|
|
// For heap objects ensure that we did not overscan.
|
|
|
|
|
n = 0;
|
|
|
|
|
p = nil;
|
|
|
|
|
if(!runtime·mlookup(b, &p, &n, nil) || b != p || i > n) {
|
|
|
|
|
runtime·printf("runtime: scanned (%p,%p), heap object (%p,%p)\n", b, i, p, n);
|
|
|
|
|
runtime·throw("scanblock: scanned invalid object");
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
}
|
2009-01-26 18:37:05 -07:00
|
|
|
|
}
|
2011-02-02 21:03:47 -07:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2012-05-24 00:55:50 -06:00
|
|
|
|
static void
|
|
|
|
|
markroot(ParFor *desc, uint32 i)
|
|
|
|
|
{
|
2014-01-21 02:06:57 -07:00
|
|
|
|
FinBlock *fb;
|
2014-08-24 02:05:07 -06:00
|
|
|
|
MSpan *s;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
uint32 spanidx, sg;
|
2014-01-21 02:06:57 -07:00
|
|
|
|
G *gp;
|
2014-01-24 11:35:11 -07:00
|
|
|
|
void *p;
|
2012-12-16 17:32:12 -07:00
|
|
|
|
|
2012-05-24 00:55:50 -06:00
|
|
|
|
USED(&desc);
|
2014-03-25 16:09:49 -06:00
|
|
|
|
// Note: if you add a case here, please also update heapdump.c:dumproots.
|
2014-01-21 02:06:57 -07:00
|
|
|
|
switch(i) {
|
|
|
|
|
case RootData:
|
2014-08-18 06:42:24 -06:00
|
|
|
|
scanblock(data, edata - data, runtime·gcdatamask);
|
2014-01-21 02:06:57 -07:00
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
case RootBss:
|
2014-08-18 06:42:24 -06:00
|
|
|
|
scanblock(bss, ebss - bss, runtime·gcbssmask);
|
2014-01-21 02:06:57 -07:00
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
case RootFinalizers:
|
|
|
|
|
for(fb=allfin; fb; fb=fb->alllink)
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanblock((byte*)fb->fin, fb->cnt*sizeof(fb->fin[0]), ScanConservatively);
|
2014-01-21 02:06:57 -07:00
|
|
|
|
break;
|
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
case RootSpans:
|
|
|
|
|
// mark MSpan.specials
|
2014-08-24 02:05:07 -06:00
|
|
|
|
sg = runtime·mheap.sweepgen;
|
|
|
|
|
for(spanidx=0; spanidx<work.nspan; spanidx++) {
|
2014-01-21 02:06:57 -07:00
|
|
|
|
Special *sp;
|
|
|
|
|
SpecialFinalizer *spf;
|
|
|
|
|
|
2014-08-24 02:05:07 -06:00
|
|
|
|
s = work.spans[spanidx];
|
undo CL 101570044 / 2c57aaea79c4
redo stack allocation. This is mostly the same as
the original CL with a few bug fixes.
1. add racemalloc() for stack allocations
2. fix poolalloc/poolfree to terminate free lists correctly.
3. adjust span ref count correctly.
4. don't use cache for sizes >= StackCacheSize.
Should fix bugs and memory leaks in original changelist.
««« original CL description
undo CL 104200047 / 318b04f28372
Breaks windows and race detector.
TBR=rsc
««« original CL description
runtime: stack allocator, separate from mallocgc
In order to move malloc to Go, we need to have a
separate stack allocator. If we run out of stack
during malloc, malloc will not be available
to allocate a new stack.
Stacks are the last remaining FlagNoGC objects in the
GC heap. Once they are out, we can get rid of the
distinction between the allocated/blockboundary bits.
(This will be in a separate change.)
Fixes #7468
Fixes #7424
LGTM=rsc, dvyukov
R=golang-codereviews, dvyukov, khr, dave, rsc
CC=golang-codereviews
https://golang.org/cl/104200047
»»»
TBR=rsc
CC=golang-codereviews
https://golang.org/cl/101570044
»»»
LGTM=dvyukov
R=dvyukov, dave, khr, alex.brainman
CC=golang-codereviews
https://golang.org/cl/112240044
2014-07-17 15:41:46 -06:00
|
|
|
|
if(s->state != MSpanInUse)
|
|
|
|
|
continue;
|
2014-02-13 09:10:31 -07:00
|
|
|
|
if(s->sweepgen != sg) {
|
|
|
|
|
runtime·printf("sweep %d %d\n", s->sweepgen, sg);
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
runtime·throw("gc: unswept span");
|
2014-02-13 09:10:31 -07:00
|
|
|
|
}
|
2014-01-21 02:06:57 -07:00
|
|
|
|
for(sp = s->specials; sp != nil; sp = sp->next) {
|
|
|
|
|
if(sp->kind != KindSpecialFinalizer)
|
|
|
|
|
continue;
|
|
|
|
|
// don't mark finalized object, but scan it so we
|
|
|
|
|
// retain everything it points to.
|
|
|
|
|
spf = (SpecialFinalizer*)sp;
|
2014-01-24 11:35:11 -07:00
|
|
|
|
// A finalizer can be set for an inner byte of an object, find object beginning.
|
2014-08-07 07:00:02 -06:00
|
|
|
|
p = (void*)((s->start << PageShift) + spf->special.offset/s->elemsize*s->elemsize);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanblock(p, s->elemsize, nil);
|
|
|
|
|
scanblock((void*)&spf->fn, PtrSize, ScanConservatively);
|
2014-01-21 02:06:57 -07:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
case RootFlushCaches:
|
|
|
|
|
flushallmcaches();
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
default:
|
|
|
|
|
// the rest is scanning goroutine stacks
|
|
|
|
|
if(i - RootCount >= runtime·allglen)
|
|
|
|
|
runtime·throw("markroot: bad index");
|
|
|
|
|
gp = runtime·allg[i - RootCount];
|
|
|
|
|
// remember when we've first observed the G blocked
|
|
|
|
|
// needed only to output in traceback
|
|
|
|
|
if((gp->status == Gwaiting || gp->status == Gsyscall) && gp->waitsince == 0)
|
|
|
|
|
gp->waitsince = work.tstart;
|
2014-08-07 02:55:28 -06:00
|
|
|
|
// Shrink a stack if not much of it is being used.
|
|
|
|
|
runtime·shrinkstack(gp);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanstack(gp);
|
2014-01-21 02:06:57 -07:00
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
}
|
2012-05-24 00:55:50 -06:00
|
|
|
|
}
|
|
|
|
|
|
2011-02-02 21:03:47 -07:00
|
|
|
|
// Get an empty work buffer off the work.empty list,
|
|
|
|
|
// allocating new buffers as needed.
|
|
|
|
|
static Workbuf*
|
|
|
|
|
getempty(Workbuf *b)
|
|
|
|
|
{
|
2014-08-05 15:50:37 -06:00
|
|
|
|
MCache *c;
|
|
|
|
|
|
2012-05-24 00:55:50 -06:00
|
|
|
|
if(b != nil)
|
|
|
|
|
runtime·lfstackpush(&work.full, &b->node);
|
2014-08-05 15:50:37 -06:00
|
|
|
|
b = nil;
|
|
|
|
|
c = g->m->mcache;
|
|
|
|
|
if(c->gcworkbuf != nil) {
|
|
|
|
|
b = c->gcworkbuf;
|
|
|
|
|
c->gcworkbuf = nil;
|
|
|
|
|
}
|
|
|
|
|
if(b == nil)
|
|
|
|
|
b = (Workbuf*)runtime·lfstackpop(&work.empty);
|
2014-07-21 15:56:01 -06:00
|
|
|
|
if(b == nil)
|
|
|
|
|
b = runtime·persistentalloc(sizeof(*b), CacheLineSize, &mstats.gc_sys);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
b->nobj = 0;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
return b;
|
|
|
|
|
}
|
|
|
|
|
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
static void
|
|
|
|
|
putempty(Workbuf *b)
|
|
|
|
|
{
|
2014-08-05 15:50:37 -06:00
|
|
|
|
MCache *c;
|
|
|
|
|
|
|
|
|
|
c = g->m->mcache;
|
|
|
|
|
if(c->gcworkbuf == nil) {
|
|
|
|
|
c->gcworkbuf = b;
|
|
|
|
|
return;
|
|
|
|
|
}
|
2012-05-24 00:55:50 -06:00
|
|
|
|
runtime·lfstackpush(&work.empty, &b->node);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
}
|
|
|
|
|
|
2014-08-05 15:50:37 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·gcworkbuffree(void *b)
|
|
|
|
|
{
|
|
|
|
|
if(b != nil)
|
|
|
|
|
putempty(b);
|
|
|
|
|
}
|
|
|
|
|
|
2011-02-02 21:03:47 -07:00
|
|
|
|
// Get a full work buffer off the work.full list, or return nil.
|
|
|
|
|
static Workbuf*
|
|
|
|
|
getfull(Workbuf *b)
|
|
|
|
|
{
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
int32 i;
|
|
|
|
|
|
2012-05-24 00:55:50 -06:00
|
|
|
|
if(b != nil)
|
|
|
|
|
runtime·lfstackpush(&work.empty, &b->node);
|
|
|
|
|
b = (Workbuf*)runtime·lfstackpop(&work.full);
|
|
|
|
|
if(b != nil || work.nproc == 1)
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
return b;
|
|
|
|
|
|
|
|
|
|
runtime·xadd(&work.nwait, +1);
|
|
|
|
|
for(i=0;; i++) {
|
2012-05-24 00:55:50 -06:00
|
|
|
|
if(work.full != 0) {
|
|
|
|
|
runtime·xadd(&work.nwait, -1);
|
|
|
|
|
b = (Workbuf*)runtime·lfstackpop(&work.full);
|
|
|
|
|
if(b != nil)
|
|
|
|
|
return b;
|
|
|
|
|
runtime·xadd(&work.nwait, +1);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
}
|
|
|
|
|
if(work.nwait == work.nproc)
|
|
|
|
|
return nil;
|
2012-04-05 10:48:28 -06:00
|
|
|
|
if(i < 10) {
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->gcstats.nprocyield++;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
runtime·procyield(20);
|
2012-04-05 10:48:28 -06:00
|
|
|
|
} else if(i < 20) {
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->gcstats.nosyield++;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
runtime·osyield();
|
2012-04-05 10:48:28 -06:00
|
|
|
|
} else {
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->gcstats.nsleep++;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
runtime·usleep(100);
|
2012-04-05 10:48:28 -06:00
|
|
|
|
}
|
2009-01-26 18:37:05 -07:00
|
|
|
|
}
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static Workbuf*
|
|
|
|
|
handoff(Workbuf *b)
|
|
|
|
|
{
|
|
|
|
|
int32 n;
|
|
|
|
|
Workbuf *b1;
|
|
|
|
|
|
|
|
|
|
// Make new buffer with half of b's pointers.
|
|
|
|
|
b1 = getempty(nil);
|
|
|
|
|
n = b->nobj/2;
|
|
|
|
|
b->nobj -= n;
|
|
|
|
|
b1->nobj = n;
|
|
|
|
|
runtime·memmove(b1->obj, b->obj+b->nobj, n*sizeof b1->obj[0]);
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->gcstats.nhandoff++;
|
|
|
|
|
g->m->gcstats.nhandoffcnt += n;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
|
|
|
|
|
// Put b on full list - let first half of b get stolen.
|
2012-05-24 00:55:50 -06:00
|
|
|
|
runtime·lfstackpush(&work.full, &b->node);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
return b1;
|
2009-01-26 18:37:05 -07:00
|
|
|
|
}
|
|
|
|
|
|
2014-04-02 14:49:27 -06:00
|
|
|
|
BitVector
|
runtime: grow stack by copying
On stack overflow, if all frames on the stack are
copyable, we copy the frames to a new stack twice
as large as the old one. During GC, if a G is using
less than 1/4 of its stack, copy the stack to a stack
half its size.
TODO
- Do something about C frames. When a C frame is in the
stack segment, it isn't copyable. We allocate a new segment
in this case.
- For idempotent C code, we can abort it, copy the stack,
then retry. I'm working on a separate CL for this.
- For other C code, we can raise the stackguard
to the lowest Go frame so the next call that Go frame
makes triggers a copy, which will then succeed.
- Pick a starting stack size?
The plan is that eventually we reach a point where the
stack contains only copyable frames.
LGTM=rsc
R=dvyukov, rsc
CC=golang-codereviews
https://golang.org/cl/54650044
2014-02-27 00:28:44 -07:00
|
|
|
|
runtime·stackmapdata(StackMap *stackmap, int32 n)
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
{
|
2014-04-02 14:49:27 -06:00
|
|
|
|
if(n < 0 || n >= stackmap->n)
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
runtime·throw("stackmapdata: index out of range");
|
2014-04-02 14:49:27 -06:00
|
|
|
|
return (BitVector){stackmap->nbit, stackmap->data + n*((stackmap->nbit+31)/32)};
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Scan a stack frame: local variables and function arguments/results.
|
runtime: grow stack by copying
On stack overflow, if all frames on the stack are
copyable, we copy the frames to a new stack twice
as large as the old one. During GC, if a G is using
less than 1/4 of its stack, copy the stack to a stack
half its size.
TODO
- Do something about C frames. When a C frame is in the
stack segment, it isn't copyable. We allocate a new segment
in this case.
- For idempotent C code, we can abort it, copy the stack,
then retry. I'm working on a separate CL for this.
- For other C code, we can raise the stackguard
to the lowest Go frame so the next call that Go frame
makes triggers a copy, which will then succeed.
- Pick a starting stack size?
The plan is that eventually we reach a point where the
stack contains only copyable frames.
LGTM=rsc
R=dvyukov, rsc
CC=golang-codereviews
https://golang.org/cl/54650044
2014-02-27 00:28:44 -07:00
|
|
|
|
static bool
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanframe(Stkframe *frame, void *unused)
|
2013-03-28 15:36:23 -06:00
|
|
|
|
{
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
Func *f;
|
|
|
|
|
StackMap *stackmap;
|
2014-04-02 14:49:27 -06:00
|
|
|
|
BitVector bv;
|
2013-08-07 13:47:01 -06:00
|
|
|
|
uintptr size;
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
uintptr targetpc;
|
|
|
|
|
int32 pcdata;
|
2013-03-28 15:36:23 -06:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
USED(unused);
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
f = frame->fn;
|
2014-05-31 08:10:12 -06:00
|
|
|
|
targetpc = frame->continpc;
|
|
|
|
|
if(targetpc == 0) {
|
|
|
|
|
// Frame is dead.
|
|
|
|
|
return true;
|
|
|
|
|
}
|
2014-07-30 10:01:52 -06:00
|
|
|
|
if(Debug > 1)
|
|
|
|
|
runtime·printf("scanframe %s\n", runtime·funcname(f));
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
if(targetpc != f->entry)
|
|
|
|
|
targetpc--;
|
|
|
|
|
pcdata = runtime·pcdatavalue(f, PCDATA_StackMapIndex, targetpc);
|
|
|
|
|
if(pcdata == -1) {
|
|
|
|
|
// We do not have a valid pcdata value but there might be a
|
|
|
|
|
// stackmap for this function. It is likely that we are looking
|
|
|
|
|
// at the function prologue, assume so and hope for the best.
|
|
|
|
|
pcdata = 0;
|
|
|
|
|
}
|
|
|
|
|
|
2013-06-12 06:49:38 -06:00
|
|
|
|
// Scan local variables if stack frame has been allocated.
|
2013-08-07 13:47:01 -06:00
|
|
|
|
// Use pointer information if known.
|
2014-06-19 23:04:10 -06:00
|
|
|
|
stackmap = runtime·funcdata(f, FUNCDATA_LocalsPointerMaps);
|
|
|
|
|
if(stackmap == nil) {
|
|
|
|
|
// No locals information, scan everything.
|
|
|
|
|
size = frame->varp - (byte*)frame->sp;
|
|
|
|
|
if(Debug > 2)
|
|
|
|
|
runtime·printf("frame %s unsized locals %p+%p\n", runtime·funcname(f), frame->varp-size, size);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanblock(frame->varp - size, size, ScanConservatively);
|
2014-06-19 23:04:10 -06:00
|
|
|
|
} else if(stackmap->n < 0) {
|
|
|
|
|
// Locals size information, scan just the locals.
|
|
|
|
|
size = -stackmap->n;
|
|
|
|
|
if(Debug > 2)
|
|
|
|
|
runtime·printf("frame %s conservative locals %p+%p\n", runtime·funcname(f), frame->varp-size, size);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanblock(frame->varp - size, size, ScanConservatively);
|
2014-06-19 23:04:10 -06:00
|
|
|
|
} else if(stackmap->n > 0) {
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// Locals bitmap information, scan just the pointers in locals.
|
2014-06-19 23:04:10 -06:00
|
|
|
|
if(pcdata < 0 || pcdata >= stackmap->n) {
|
|
|
|
|
// don't know where we are
|
|
|
|
|
runtime·printf("pcdata is %d and %d stack map entries for %s (targetpc=%p)\n",
|
|
|
|
|
pcdata, stackmap->n, runtime·funcname(f), targetpc);
|
|
|
|
|
runtime·throw("scanframe: bad symbol table");
|
2013-08-07 13:47:01 -06:00
|
|
|
|
}
|
2014-06-19 23:04:10 -06:00
|
|
|
|
bv = runtime·stackmapdata(stackmap, pcdata);
|
|
|
|
|
size = (bv.n * PtrSize) / BitsPerPointer;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanblock(frame->varp - size, bv.n/BitsPerPointer*PtrSize, (byte*)bv.data);
|
2013-07-19 14:04:09 -06:00
|
|
|
|
}
|
2013-06-12 06:49:38 -06:00
|
|
|
|
|
|
|
|
|
// Scan arguments.
|
|
|
|
|
// Use pointer information if known.
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
stackmap = runtime·funcdata(f, FUNCDATA_ArgsPointerMaps);
|
|
|
|
|
if(stackmap != nil) {
|
runtime: grow stack by copying
On stack overflow, if all frames on the stack are
copyable, we copy the frames to a new stack twice
as large as the old one. During GC, if a G is using
less than 1/4 of its stack, copy the stack to a stack
half its size.
TODO
- Do something about C frames. When a C frame is in the
stack segment, it isn't copyable. We allocate a new segment
in this case.
- For idempotent C code, we can abort it, copy the stack,
then retry. I'm working on a separate CL for this.
- For other C code, we can raise the stackguard
to the lowest Go frame so the next call that Go frame
makes triggers a copy, which will then succeed.
- Pick a starting stack size?
The plan is that eventually we reach a point where the
stack contains only copyable frames.
LGTM=rsc
R=dvyukov, rsc
CC=golang-codereviews
https://golang.org/cl/54650044
2014-02-27 00:28:44 -07:00
|
|
|
|
bv = runtime·stackmapdata(stackmap, pcdata);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanblock(frame->argp, bv.n/BitsPerPointer*PtrSize, (byte*)bv.data);
|
reflect, runtime: fix crash in GC due to reflect.call + precise GC
Given
type Outer struct {
*Inner
...
}
the compiler generates the implementation of (*Outer).M dispatching to
the embedded Inner. The implementation is logically:
func (p *Outer) M() {
(p.Inner).M()
}
but since the only change here is the replacement of one pointer
receiver with another, the actual generated code overwrites the
original receiver with the p.Inner pointer and then jumps to the M
method expecting the *Inner receiver.
During reflect.Value.Call, we create an argument frame and the
associated data structures to describe it to the garbage collector,
populate the frame, call reflect.call to run a function call using
that frame, and then copy the results back out of the frame. The
reflect.call function does a memmove of the frame structure onto the
stack (to set up the inputs), runs the call, and the memmoves the
stack back to the frame structure (to preserve the outputs).
Originally reflect.call did not distinguish inputs from outputs: both
memmoves were for the full stack frame. However, in the case where the
called function was one of these wrappers, the rewritten receiver is
almost certainly a different type than the original receiver. This is
not a problem on the stack, where we use the program counter to
determine the type information and understand that during (*Outer).M
the receiver is an *Outer while during (*Inner).M the receiver in the
same memory word is now an *Inner. But in the statically typed
argument frame created by reflect, the receiver is always an *Outer.
Copying the modified receiver pointer off the stack into the frame
will store an *Inner there, and then if a garbage collection happens
to scan that argument frame before it is discarded, it will scan the
*Inner memory as if it were an *Outer. If the two have different
memory layouts, the collection will intepret the memory incorrectly.
Fix by only copying back the results.
Fixes #7725.
LGTM=khr
R=khr
CC=dave, golang-codereviews
https://golang.org/cl/85180043
2014-04-08 09:11:35 -06:00
|
|
|
|
} else {
|
|
|
|
|
if(Debug > 2)
|
|
|
|
|
runtime·printf("frame %s conservative args %p+%p\n", runtime·funcname(f), frame->argp, (uintptr)frame->arglen);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanblock(frame->argp, frame->arglen, ScanConservatively);
|
reflect, runtime: fix crash in GC due to reflect.call + precise GC
Given
type Outer struct {
*Inner
...
}
the compiler generates the implementation of (*Outer).M dispatching to
the embedded Inner. The implementation is logically:
func (p *Outer) M() {
(p.Inner).M()
}
but since the only change here is the replacement of one pointer
receiver with another, the actual generated code overwrites the
original receiver with the p.Inner pointer and then jumps to the M
method expecting the *Inner receiver.
During reflect.Value.Call, we create an argument frame and the
associated data structures to describe it to the garbage collector,
populate the frame, call reflect.call to run a function call using
that frame, and then copy the results back out of the frame. The
reflect.call function does a memmove of the frame structure onto the
stack (to set up the inputs), runs the call, and the memmoves the
stack back to the frame structure (to preserve the outputs).
Originally reflect.call did not distinguish inputs from outputs: both
memmoves were for the full stack frame. However, in the case where the
called function was one of these wrappers, the rewritten receiver is
almost certainly a different type than the original receiver. This is
not a problem on the stack, where we use the program counter to
determine the type information and understand that during (*Outer).M
the receiver is an *Outer while during (*Inner).M the receiver in the
same memory word is now an *Inner. But in the statically typed
argument frame created by reflect, the receiver is always an *Outer.
Copying the modified receiver pointer off the stack into the frame
will store an *Inner there, and then if a garbage collection happens
to scan that argument frame before it is discarded, it will scan the
*Inner memory as if it were an *Outer. If the two have different
memory layouts, the collection will intepret the memory incorrectly.
Fix by only copying back the results.
Fixes #7725.
LGTM=khr
R=khr
CC=dave, golang-codereviews
https://golang.org/cl/85180043
2014-04-08 09:11:35 -06:00
|
|
|
|
}
|
runtime: grow stack by copying
On stack overflow, if all frames on the stack are
copyable, we copy the frames to a new stack twice
as large as the old one. During GC, if a G is using
less than 1/4 of its stack, copy the stack to a stack
half its size.
TODO
- Do something about C frames. When a C frame is in the
stack segment, it isn't copyable. We allocate a new segment
in this case.
- For idempotent C code, we can abort it, copy the stack,
then retry. I'm working on a separate CL for this.
- For other C code, we can raise the stackguard
to the lowest Go frame so the next call that Go frame
makes triggers a copy, which will then succeed.
- Pick a starting stack size?
The plan is that eventually we reach a point where the
stack contains only copyable frames.
LGTM=rsc
R=dvyukov, rsc
CC=golang-codereviews
https://golang.org/cl/54650044
2014-02-27 00:28:44 -07:00
|
|
|
|
return true;
|
2009-01-26 18:37:05 -07:00
|
|
|
|
}
|
|
|
|
|
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
static void
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanstack(G *gp)
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
{
|
|
|
|
|
M *mp;
|
|
|
|
|
int32 n;
|
|
|
|
|
Stktop *stk;
|
|
|
|
|
uintptr sp, guard;
|
|
|
|
|
|
2014-01-21 02:06:57 -07:00
|
|
|
|
switch(gp->status){
|
|
|
|
|
default:
|
|
|
|
|
runtime·printf("unexpected G.status %d (goroutine %p %D)\n", gp->status, gp, gp->goid);
|
|
|
|
|
runtime·throw("mark - bad status");
|
|
|
|
|
case Gdead:
|
|
|
|
|
return;
|
|
|
|
|
case Grunning:
|
|
|
|
|
runtime·throw("mark - world not stopped");
|
|
|
|
|
case Grunnable:
|
|
|
|
|
case Gsyscall:
|
|
|
|
|
case Gwaiting:
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
if(gp == g)
|
|
|
|
|
runtime·throw("can't scan our own stack");
|
|
|
|
|
if((mp = gp->m) != nil && mp->helpgc)
|
|
|
|
|
runtime·throw("can't scan gchelper stack");
|
runtime: grow stack by copying
On stack overflow, if all frames on the stack are
copyable, we copy the frames to a new stack twice
as large as the old one. During GC, if a G is using
less than 1/4 of its stack, copy the stack to a stack
half its size.
TODO
- Do something about C frames. When a C frame is in the
stack segment, it isn't copyable. We allocate a new segment
in this case.
- For idempotent C code, we can abort it, copy the stack,
then retry. I'm working on a separate CL for this.
- For other C code, we can raise the stackguard
to the lowest Go frame so the next call that Go frame
makes triggers a copy, which will then succeed.
- Pick a starting stack size?
The plan is that eventually we reach a point where the
stack contains only copyable frames.
LGTM=rsc
R=dvyukov, rsc
CC=golang-codereviews
https://golang.org/cl/54650044
2014-02-27 00:28:44 -07:00
|
|
|
|
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
if(gp->syscallstack != (uintptr)nil) {
|
|
|
|
|
// Scanning another goroutine that is about to enter or might
|
|
|
|
|
// have just exited a system call. It may be executing code such
|
|
|
|
|
// as schedlock and may have needed to start a new stack segment.
|
|
|
|
|
// Use the stack segment and stack pointer at the time of
|
|
|
|
|
// the system call instead, since that won't change underfoot.
|
|
|
|
|
sp = gp->syscallsp;
|
|
|
|
|
stk = (Stktop*)gp->syscallstack;
|
|
|
|
|
guard = gp->syscallguard;
|
|
|
|
|
} else {
|
|
|
|
|
// Scanning another goroutine's stack.
|
|
|
|
|
// The goroutine is usually asleep (the world is stopped).
|
|
|
|
|
sp = gp->sched.sp;
|
|
|
|
|
stk = (Stktop*)gp->stackbase;
|
|
|
|
|
guard = gp->stackguard;
|
|
|
|
|
}
|
|
|
|
|
if(ScanStackByFrames) {
|
|
|
|
|
USED(sp);
|
|
|
|
|
USED(stk);
|
|
|
|
|
USED(guard);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
runtime·gentraceback(~(uintptr)0, ~(uintptr)0, 0, gp, 0, nil, 0x7fffffff, scanframe, nil, false);
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
} else {
|
|
|
|
|
n = 0;
|
|
|
|
|
while(stk) {
|
|
|
|
|
if(sp < guard-StackGuard || (uintptr)stk < sp) {
|
|
|
|
|
runtime·printf("scanstack inconsistent: g%D#%d sp=%p not in [%p,%p]\n", gp->goid, n, sp, guard-StackGuard, stk);
|
|
|
|
|
runtime·throw("scanstack");
|
|
|
|
|
}
|
reflect, runtime: fix crash in GC due to reflect.call + precise GC
Given
type Outer struct {
*Inner
...
}
the compiler generates the implementation of (*Outer).M dispatching to
the embedded Inner. The implementation is logically:
func (p *Outer) M() {
(p.Inner).M()
}
but since the only change here is the replacement of one pointer
receiver with another, the actual generated code overwrites the
original receiver with the p.Inner pointer and then jumps to the M
method expecting the *Inner receiver.
During reflect.Value.Call, we create an argument frame and the
associated data structures to describe it to the garbage collector,
populate the frame, call reflect.call to run a function call using
that frame, and then copy the results back out of the frame. The
reflect.call function does a memmove of the frame structure onto the
stack (to set up the inputs), runs the call, and the memmoves the
stack back to the frame structure (to preserve the outputs).
Originally reflect.call did not distinguish inputs from outputs: both
memmoves were for the full stack frame. However, in the case where the
called function was one of these wrappers, the rewritten receiver is
almost certainly a different type than the original receiver. This is
not a problem on the stack, where we use the program counter to
determine the type information and understand that during (*Outer).M
the receiver is an *Outer while during (*Inner).M the receiver in the
same memory word is now an *Inner. But in the statically typed
argument frame created by reflect, the receiver is always an *Outer.
Copying the modified receiver pointer off the stack into the frame
will store an *Inner there, and then if a garbage collection happens
to scan that argument frame before it is discarded, it will scan the
*Inner memory as if it were an *Outer. If the two have different
memory layouts, the collection will intepret the memory incorrectly.
Fix by only copying back the results.
Fixes #7725.
LGTM=khr
R=khr
CC=dave, golang-codereviews
https://golang.org/cl/85180043
2014-04-08 09:11:35 -06:00
|
|
|
|
if(Debug > 2)
|
|
|
|
|
runtime·printf("conservative stack %p+%p\n", (byte*)sp, (uintptr)stk-sp);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanblock((byte*)sp, (uintptr)stk - sp, ScanConservatively);
|
cmd/5g, cmd/5l, cmd/6g, cmd/6l, cmd/8g, cmd/8l, cmd/gc, runtime: generate pointer maps by liveness analysis
This change allows the garbage collector to examine stack
slots that are determined as live and containing a pointer
value by the garbage collector. This results in a mean
reduction of 65% in the number of stack slots scanned during
an invocation of "GOGC=1 all.bash".
Unfortunately, this does not yet allow garbage collection to
be precise for the stack slots computed as live. Pointers
confound the determination of what definitions reach a given
instruction. In general, this problem is not solvable without
runtime cost but some advanced cooperation from the compiler
might mitigate common cases.
R=golang-dev, rsc, cshapiro
CC=golang-dev
https://golang.org/cl/14430048
2013-12-05 18:35:22 -07:00
|
|
|
|
sp = stk->gobuf.sp;
|
|
|
|
|
guard = stk->stackguard;
|
|
|
|
|
stk = (Stktop*)stk->stackbase;
|
|
|
|
|
n++;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2014-01-07 14:45:50 -07:00
|
|
|
|
void
|
|
|
|
|
runtime·queuefinalizer(byte *p, FuncVal *fn, uintptr nret, Type *fint, PtrType *ot)
|
2011-10-06 09:42:51 -06:00
|
|
|
|
{
|
|
|
|
|
FinBlock *block;
|
|
|
|
|
Finalizer *f;
|
2012-05-15 09:10:16 -06:00
|
|
|
|
|
2014-03-26 05:11:36 -06:00
|
|
|
|
runtime·lock(&finlock);
|
2011-10-06 09:42:51 -06:00
|
|
|
|
if(finq == nil || finq->cnt == finq->cap) {
|
|
|
|
|
if(finc == nil) {
|
2014-01-30 02:28:19 -07:00
|
|
|
|
finc = runtime·persistentalloc(FinBlockSize, 0, &mstats.gc_sys);
|
|
|
|
|
finc->cap = (FinBlockSize - sizeof(FinBlock)) / sizeof(Finalizer) + 1;
|
2011-10-06 09:42:51 -06:00
|
|
|
|
finc->alllink = allfin;
|
|
|
|
|
allfin = finc;
|
|
|
|
|
}
|
|
|
|
|
block = finc;
|
|
|
|
|
finc = block->next;
|
|
|
|
|
block->next = finq;
|
|
|
|
|
finq = block;
|
|
|
|
|
}
|
|
|
|
|
f = &finq->fin[finq->cnt];
|
|
|
|
|
finq->cnt++;
|
|
|
|
|
f->fn = fn;
|
|
|
|
|
f->nret = nret;
|
2013-08-14 12:54:31 -06:00
|
|
|
|
f->fint = fint;
|
2013-07-29 09:43:08 -06:00
|
|
|
|
f->ot = ot;
|
2011-10-06 09:42:51 -06:00
|
|
|
|
f->arg = p;
|
2014-03-26 05:11:36 -06:00
|
|
|
|
runtime·fingwake = true;
|
|
|
|
|
runtime·unlock(&finlock);
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
}
|
|
|
|
|
|
2014-03-25 16:09:49 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·iterate_finq(void (*callback)(FuncVal*, byte*, uintptr, Type*, PtrType*))
|
|
|
|
|
{
|
|
|
|
|
FinBlock *fb;
|
|
|
|
|
Finalizer *f;
|
|
|
|
|
uintptr i;
|
|
|
|
|
|
|
|
|
|
for(fb = allfin; fb; fb = fb->alllink) {
|
|
|
|
|
for(i = 0; i < fb->cnt; i++) {
|
|
|
|
|
f = &fb->fin[i];
|
|
|
|
|
callback(f->fn, f->arg, f->nret, f->fint, f->ot);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
void
|
|
|
|
|
runtime·MSpan_EnsureSwept(MSpan *s)
|
|
|
|
|
{
|
|
|
|
|
uint32 sg;
|
|
|
|
|
|
2014-02-24 09:53:20 -07:00
|
|
|
|
// Caller must disable preemption.
|
|
|
|
|
// Otherwise when this function returns the span can become unswept again
|
|
|
|
|
// (if GC is triggered on another goroutine).
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
if(g->m->locks == 0 && g->m->mallocing == 0 && g != g->m->g0)
|
2014-02-24 09:53:20 -07:00
|
|
|
|
runtime·throw("MSpan_EnsureSwept: m is not locked");
|
|
|
|
|
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
sg = runtime·mheap.sweepgen;
|
|
|
|
|
if(runtime·atomicload(&s->sweepgen) == sg)
|
|
|
|
|
return;
|
|
|
|
|
if(runtime·cas(&s->sweepgen, sg-2, sg-1)) {
|
2014-08-18 06:52:31 -06:00
|
|
|
|
runtime·MSpan_Sweep(s, false);
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
// unfortunate condition, and we don't have efficient means to wait
|
|
|
|
|
while(runtime·atomicload(&s->sweepgen) != sg)
|
|
|
|
|
runtime·osyield();
|
2011-10-06 09:42:51 -06:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Sweep frees or collects finalizers for blocks not marked in the mark phase.
|
2011-02-02 21:03:47 -07:00
|
|
|
|
// It clears the mark bits in preparation for the next GC round.
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
// Returns true if the span was returned to heap.
|
2014-08-18 06:52:31 -06:00
|
|
|
|
// If preserve=true, don't return it to heap nor relink in MCentral lists;
|
|
|
|
|
// caller takes care of it.
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
bool
|
2014-08-18 06:52:31 -06:00
|
|
|
|
runtime·MSpan_Sweep(MSpan *s, bool preserve)
|
2012-04-05 11:02:20 -06:00
|
|
|
|
{
|
2014-02-13 08:36:45 -07:00
|
|
|
|
int32 cl, n, npages, nfree;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
uintptr size, off, step;
|
2014-02-13 08:36:45 -07:00
|
|
|
|
uint32 sweepgen;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
byte *p, *bitp, shift, xbits, bits;
|
2012-04-05 11:02:20 -06:00
|
|
|
|
MCache *c;
|
|
|
|
|
byte *arena_start;
|
2014-08-13 10:42:55 -06:00
|
|
|
|
MLink head, *end, *link;
|
2014-01-07 14:45:50 -07:00
|
|
|
|
Special *special, **specialp, *y;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
bool res, sweepgenset;
|
2010-02-10 15:59:39 -07:00
|
|
|
|
|
2014-02-13 08:36:45 -07:00
|
|
|
|
// It's critical that we enter this function with preemption disabled,
|
|
|
|
|
// GC must not start while we are in the middle of this function.
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
if(g->m->locks == 0 && g->m->mallocing == 0 && g != g->m->g0)
|
2014-02-13 08:36:45 -07:00
|
|
|
|
runtime·throw("MSpan_Sweep: m is not locked");
|
|
|
|
|
sweepgen = runtime·mheap.sweepgen;
|
|
|
|
|
if(s->state != MSpanInUse || s->sweepgen != sweepgen-1) {
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
runtime·printf("MSpan_Sweep: state=%d sweepgen=%d mheap.sweepgen=%d\n",
|
2014-02-13 08:36:45 -07:00
|
|
|
|
s->state, s->sweepgen, sweepgen);
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
runtime·throw("MSpan_Sweep: bad span state");
|
|
|
|
|
}
|
2013-05-28 12:14:47 -06:00
|
|
|
|
arena_start = runtime·mheap.arena_start;
|
2012-04-05 11:02:20 -06:00
|
|
|
|
cl = s->sizeclass;
|
2012-09-24 18:08:05 -06:00
|
|
|
|
size = s->elemsize;
|
2012-04-05 11:02:20 -06:00
|
|
|
|
if(cl == 0) {
|
|
|
|
|
n = 1;
|
|
|
|
|
} else {
|
|
|
|
|
// Chunk full of small blocks.
|
|
|
|
|
npages = runtime·class_to_allocnpages[cl];
|
|
|
|
|
n = (npages << PageShift) / size;
|
|
|
|
|
}
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
res = false;
|
2012-04-12 02:01:24 -06:00
|
|
|
|
nfree = 0;
|
2012-09-24 18:08:05 -06:00
|
|
|
|
end = &head;
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
c = g->m->mcache;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
sweepgenset = false;
|
2013-12-18 18:13:59 -07:00
|
|
|
|
|
2014-08-13 10:42:55 -06:00
|
|
|
|
// Mark any free objects in this span so we don't collect them.
|
|
|
|
|
for(link = s->freelist; link != nil; link = link->next) {
|
|
|
|
|
off = (uintptr*)link - (uintptr*)arena_start;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
bitp = arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = (off % wordsPerBitmapByte) * gcBits;
|
2014-08-13 10:42:55 -06:00
|
|
|
|
*bitp |= bitMarked<<shift;
|
|
|
|
|
}
|
|
|
|
|
|
2014-01-07 14:45:50 -07:00
|
|
|
|
// Unlink & free special records for any objects we're about to free.
|
|
|
|
|
specialp = &s->specials;
|
|
|
|
|
special = *specialp;
|
|
|
|
|
while(special != nil) {
|
2014-01-24 11:35:11 -07:00
|
|
|
|
// A finalizer can be set for an inner byte of an object, find object beginning.
|
|
|
|
|
p = (byte*)(s->start << PageShift) + special->offset/size*size;
|
2014-01-07 14:45:50 -07:00
|
|
|
|
off = (uintptr*)p - (uintptr*)arena_start;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
bitp = arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = (off % wordsPerBitmapByte) * gcBits;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
bits = (*bitp>>shift) & bitMask;
|
2014-08-13 10:42:55 -06:00
|
|
|
|
if((bits&bitMarked) == 0) {
|
2014-01-24 11:35:11 -07:00
|
|
|
|
// Find the exact byte for which the special was setup
|
|
|
|
|
// (as opposed to object beginning).
|
|
|
|
|
p = (byte*)(s->start << PageShift) + special->offset;
|
2014-01-07 14:45:50 -07:00
|
|
|
|
// about to free object: splice out special record
|
|
|
|
|
y = special;
|
|
|
|
|
special = special->next;
|
|
|
|
|
*specialp = special;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
if(!runtime·freespecial(y, p, size, false)) {
|
2014-01-07 14:45:50 -07:00
|
|
|
|
// stop freeing of object if it has a finalizer
|
|
|
|
|
*bitp |= bitMarked << shift;
|
|
|
|
|
}
|
|
|
|
|
} else {
|
|
|
|
|
// object is still live: keep special record
|
|
|
|
|
specialp = &special->next;
|
|
|
|
|
special = *specialp;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2012-04-05 11:02:20 -06:00
|
|
|
|
// Sweep through n objects of given size starting at p.
|
|
|
|
|
// This thread owns the span now, so it can manipulate
|
|
|
|
|
// the block bitmap without atomic operations.
|
2014-01-07 14:45:50 -07:00
|
|
|
|
p = (byte*)(s->start << PageShift);
|
2014-08-19 07:38:00 -06:00
|
|
|
|
// Find bits for the beginning of the span.
|
|
|
|
|
off = (uintptr*)p - (uintptr*)arena_start;
|
|
|
|
|
bitp = arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = 0;
|
|
|
|
|
step = size/(PtrSize*wordsPerBitmapByte);
|
|
|
|
|
// Rewind to the previous quadruple as we move to the next
|
|
|
|
|
// in the beginning of the loop.
|
|
|
|
|
bitp += step;
|
|
|
|
|
if(step == 0) {
|
|
|
|
|
// 8-byte objects.
|
|
|
|
|
bitp++;
|
|
|
|
|
shift = gcBits;
|
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
for(; n > 0; n--, p += size) {
|
2014-08-19 07:38:00 -06:00
|
|
|
|
bitp -= step;
|
|
|
|
|
if(step == 0) {
|
|
|
|
|
if(shift != 0)
|
|
|
|
|
bitp--;
|
|
|
|
|
shift = gcBits - shift;
|
|
|
|
|
}
|
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
xbits = *bitp;
|
|
|
|
|
bits = (xbits>>shift) & bitMask;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// Allocated and marked object, reset bits to allocated.
|
2014-08-13 10:42:55 -06:00
|
|
|
|
if((bits&bitMarked) != 0) {
|
|
|
|
|
*bitp &= ~(bitMarked<<shift);
|
2012-04-05 11:02:20 -06:00
|
|
|
|
continue;
|
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// At this point we know that we are looking at garbage object
|
|
|
|
|
// that needs to be collected.
|
2014-04-01 11:30:10 -06:00
|
|
|
|
if(runtime·debug.allocfreetrace)
|
|
|
|
|
runtime·tracefree(p, size);
|
2014-08-13 10:42:55 -06:00
|
|
|
|
// Reset to allocated+noscan.
|
|
|
|
|
*bitp = (xbits & ~((bitMarked|(BitsMask<<2))<<shift)) | ((uintptr)BitsDead<<(shift+2));
|
2012-09-24 18:08:05 -06:00
|
|
|
|
if(cl == 0) {
|
2012-04-05 11:02:20 -06:00
|
|
|
|
// Free large span.
|
2014-08-18 06:52:31 -06:00
|
|
|
|
if(preserve)
|
|
|
|
|
runtime·throw("can't preserve large span");
|
2014-07-29 01:01:02 -06:00
|
|
|
|
runtime·unmarkspan(p, s->npages<<PageShift);
|
2014-02-13 09:10:31 -07:00
|
|
|
|
s->needzero = 1;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
// important to set sweepgen before returning it to heap
|
2014-02-13 08:36:45 -07:00
|
|
|
|
runtime·atomicstore(&s->sweepgen, sweepgen);
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
sweepgenset = true;
|
2014-07-31 02:55:40 -06:00
|
|
|
|
// NOTE(rsc,dvyukov): The original implementation of efence
|
|
|
|
|
// in CL 22060046 used SysFree instead of SysFault, so that
|
|
|
|
|
// the operating system would eventually give the memory
|
|
|
|
|
// back to us again, so that an efence program could run
|
|
|
|
|
// longer without running out of memory. Unfortunately,
|
|
|
|
|
// calling SysFree here without any kind of adjustment of the
|
|
|
|
|
// heap data structures means that when the memory does
|
|
|
|
|
// come back to us, we have the wrong metadata for it, either in
|
|
|
|
|
// the MSpan structures or in the garbage collection bitmap.
|
|
|
|
|
// Using SysFault here means that the program will run out of
|
|
|
|
|
// memory fairly quickly in efence mode, but at least it won't
|
|
|
|
|
// have mysterious crashes due to confused memory reuse.
|
|
|
|
|
// It should be possible to switch back to SysFree if we also
|
|
|
|
|
// implement and then call some kind of MHeap_DeleteSpan.
|
2014-07-29 01:01:02 -06:00
|
|
|
|
if(runtime·debug.efence) {
|
|
|
|
|
s->limit = nil; // prevent mlookup from finding this span
|
2014-03-06 16:34:29 -07:00
|
|
|
|
runtime·SysFault(p, size);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
} else
|
2013-12-06 15:40:45 -07:00
|
|
|
|
runtime·MHeap_Free(&runtime·mheap, s, 1);
|
2013-06-06 04:56:50 -06:00
|
|
|
|
c->local_nlargefree++;
|
|
|
|
|
c->local_largefree += size;
|
2014-07-30 10:01:52 -06:00
|
|
|
|
runtime·xadd64(&mstats.next_gc, -(uint64)(size * (runtime·gcpercent + 100)/100));
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
res = true;
|
2012-04-05 11:02:20 -06:00
|
|
|
|
} else {
|
|
|
|
|
// Free small object.
|
2014-01-20 23:53:51 -07:00
|
|
|
|
if(size > 2*sizeof(uintptr))
|
2013-04-04 15:18:52 -06:00
|
|
|
|
((uintptr*)p)[1] = (uintptr)0xdeaddeaddeaddeadll; // mark as "needs to be zeroed"
|
2014-01-20 23:53:51 -07:00
|
|
|
|
else if(size > sizeof(uintptr))
|
|
|
|
|
((uintptr*)p)[1] = 0;
|
|
|
|
|
|
2012-09-24 18:08:05 -06:00
|
|
|
|
end->next = (MLink*)p;
|
2012-04-12 02:01:24 -06:00
|
|
|
|
end = (MLink*)p;
|
|
|
|
|
nfree++;
|
2009-01-26 18:37:05 -07:00
|
|
|
|
}
|
2012-04-12 02:01:24 -06:00
|
|
|
|
}
|
|
|
|
|
|
2014-02-26 16:52:58 -07:00
|
|
|
|
// We need to set s->sweepgen = h->sweepgen only when all blocks are swept,
|
|
|
|
|
// because of the potential for a concurrent free/SetFinalizer.
|
|
|
|
|
// But we need to set it before we make the span available for allocation
|
|
|
|
|
// (return it to heap or mcentral), because allocation code assumes that a
|
|
|
|
|
// span is already swept if available for allocation.
|
|
|
|
|
|
|
|
|
|
if(!sweepgenset && nfree == 0) {
|
2014-02-13 08:36:45 -07:00
|
|
|
|
// The span must be in our exclusive ownership until we update sweepgen,
|
|
|
|
|
// check for potential races.
|
|
|
|
|
if(s->state != MSpanInUse || s->sweepgen != sweepgen-1) {
|
|
|
|
|
runtime·printf("MSpan_Sweep: state=%d sweepgen=%d mheap.sweepgen=%d\n",
|
|
|
|
|
s->state, s->sweepgen, sweepgen);
|
|
|
|
|
runtime·throw("MSpan_Sweep: bad span state after sweep");
|
|
|
|
|
}
|
|
|
|
|
runtime·atomicstore(&s->sweepgen, sweepgen);
|
|
|
|
|
}
|
2014-02-26 16:52:58 -07:00
|
|
|
|
if(nfree > 0) {
|
2013-06-06 04:56:50 -06:00
|
|
|
|
c->local_nsmallfree[cl] += nfree;
|
2012-04-12 02:01:24 -06:00
|
|
|
|
c->local_cachealloc -= nfree * size;
|
2014-07-30 10:01:52 -06:00
|
|
|
|
runtime·xadd64(&mstats.next_gc, -(uint64)(nfree * size * (runtime·gcpercent + 100)/100));
|
2014-08-18 06:52:31 -06:00
|
|
|
|
res = runtime·MCentral_FreeSpan(&runtime·mheap.central[cl].mcentral, s, nfree, head.next, end, preserve);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// MCentral_FreeSpan updates sweepgen
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
}
|
|
|
|
|
return res;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// State of background sweep.
|
|
|
|
|
// Pretected by gclock.
|
|
|
|
|
static struct
|
|
|
|
|
{
|
|
|
|
|
G* g;
|
|
|
|
|
bool parked;
|
|
|
|
|
|
2014-08-24 02:05:07 -06:00
|
|
|
|
uint32 spanidx; // background sweeper position
|
2014-07-29 01:01:02 -06:00
|
|
|
|
|
|
|
|
|
uint32 nbgsweep;
|
|
|
|
|
uint32 npausesweep;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
} sweep;
|
|
|
|
|
|
|
|
|
|
// background sweeping goroutine
|
|
|
|
|
static void
|
|
|
|
|
bgsweep(void)
|
|
|
|
|
{
|
|
|
|
|
g->issystem = 1;
|
|
|
|
|
for(;;) {
|
|
|
|
|
while(runtime·sweepone() != -1) {
|
2014-07-29 01:01:02 -06:00
|
|
|
|
sweep.nbgsweep++;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
runtime·gosched();
|
|
|
|
|
}
|
|
|
|
|
runtime·lock(&gclock);
|
2014-03-14 11:21:44 -06:00
|
|
|
|
if(!runtime·mheap.sweepdone) {
|
|
|
|
|
// It's possible if GC has happened between sweepone has
|
|
|
|
|
// returned -1 and gclock lock.
|
|
|
|
|
runtime·unlock(&gclock);
|
|
|
|
|
continue;
|
|
|
|
|
}
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
sweep.parked = true;
|
2014-04-15 09:48:17 -06:00
|
|
|
|
g->isbackground = true;
|
2014-08-21 10:41:09 -06:00
|
|
|
|
runtime·parkunlock(&gclock, runtime·gostringnocopy((byte*)"GC sweep wait"));
|
2014-04-15 09:48:17 -06:00
|
|
|
|
g->isbackground = false;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// sweeps one span
|
|
|
|
|
// returns number of pages returned to heap, or -1 if there is nothing to sweep
|
|
|
|
|
uintptr
|
|
|
|
|
runtime·sweepone(void)
|
|
|
|
|
{
|
|
|
|
|
MSpan *s;
|
|
|
|
|
uint32 idx, sg;
|
|
|
|
|
uintptr npages;
|
|
|
|
|
|
|
|
|
|
// increment locks to ensure that the goroutine is not preempted
|
|
|
|
|
// in the middle of sweep thus leaving the span in an inconsistent state for next GC
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->locks++;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
sg = runtime·mheap.sweepgen;
|
|
|
|
|
for(;;) {
|
|
|
|
|
idx = runtime·xadd(&sweep.spanidx, 1) - 1;
|
2014-08-24 02:05:07 -06:00
|
|
|
|
if(idx >= work.nspan) {
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
runtime·mheap.sweepdone = true;
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->locks--;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
return -1;
|
|
|
|
|
}
|
2014-08-24 02:05:07 -06:00
|
|
|
|
s = work.spans[idx];
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
if(s->state != MSpanInUse) {
|
|
|
|
|
s->sweepgen = sg;
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
if(s->sweepgen != sg-2 || !runtime·cas(&s->sweepgen, sg-2, sg-1))
|
|
|
|
|
continue;
|
|
|
|
|
npages = s->npages;
|
2014-08-18 06:52:31 -06:00
|
|
|
|
if(!runtime·MSpan_Sweep(s, false))
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
npages = 0;
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->locks--;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
return npages;
|
2009-01-26 18:37:05 -07:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·gchelper(void)
|
|
|
|
|
{
|
2014-03-25 06:18:08 -06:00
|
|
|
|
uint32 nproc;
|
2014-01-15 08:38:08 -07:00
|
|
|
|
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->traceback = 2;
|
2013-03-21 02:48:02 -06:00
|
|
|
|
gchelperstart();
|
|
|
|
|
|
2012-05-24 00:55:50 -06:00
|
|
|
|
// parallel mark for over gc roots
|
|
|
|
|
runtime·parfordo(work.markfor);
|
2012-12-16 17:32:12 -07:00
|
|
|
|
|
2012-05-24 00:55:50 -06:00
|
|
|
|
// help other threads scan secondary blocks
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanblock(nil, 0, nil);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
|
2014-01-15 08:38:08 -07:00
|
|
|
|
nproc = work.nproc; // work.nproc can change right after we increment work.ndone
|
|
|
|
|
if(runtime·xadd(&work.ndone, +1) == nproc-1)
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
runtime·notewakeup(&work.alldone);
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->traceback = 0;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
}
|
|
|
|
|
|
2010-09-07 07:57:22 -06:00
|
|
|
|
static void
|
2013-06-06 04:56:50 -06:00
|
|
|
|
cachestats(void)
|
|
|
|
|
{
|
|
|
|
|
MCache *c;
|
|
|
|
|
P *p, **pp;
|
|
|
|
|
|
|
|
|
|
for(pp=runtime·allp; p=*pp; pp++) {
|
|
|
|
|
c = p->mcache;
|
|
|
|
|
if(c==nil)
|
|
|
|
|
continue;
|
|
|
|
|
runtime·purgecachedstats(c);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2014-01-21 02:06:57 -07:00
|
|
|
|
static void
|
|
|
|
|
flushallmcaches(void)
|
|
|
|
|
{
|
|
|
|
|
P *p, **pp;
|
|
|
|
|
MCache *c;
|
|
|
|
|
|
|
|
|
|
// Flush MCache's to MCentral.
|
|
|
|
|
for(pp=runtime·allp; p=*pp; pp++) {
|
|
|
|
|
c = p->mcache;
|
|
|
|
|
if(c==nil)
|
|
|
|
|
continue;
|
|
|
|
|
runtime·MCache_ReleaseAll(c);
|
undo CL 101570044 / 2c57aaea79c4
redo stack allocation. This is mostly the same as
the original CL with a few bug fixes.
1. add racemalloc() for stack allocations
2. fix poolalloc/poolfree to terminate free lists correctly.
3. adjust span ref count correctly.
4. don't use cache for sizes >= StackCacheSize.
Should fix bugs and memory leaks in original changelist.
««« original CL description
undo CL 104200047 / 318b04f28372
Breaks windows and race detector.
TBR=rsc
««« original CL description
runtime: stack allocator, separate from mallocgc
In order to move malloc to Go, we need to have a
separate stack allocator. If we run out of stack
during malloc, malloc will not be available
to allocate a new stack.
Stacks are the last remaining FlagNoGC objects in the
GC heap. Once they are out, we can get rid of the
distinction between the allocated/blockboundary bits.
(This will be in a separate change.)
Fixes #7468
Fixes #7424
LGTM=rsc, dvyukov
R=golang-codereviews, dvyukov, khr, dave, rsc
CC=golang-codereviews
https://golang.org/cl/104200047
»»»
TBR=rsc
CC=golang-codereviews
https://golang.org/cl/101570044
»»»
LGTM=dvyukov
R=dvyukov, dave, khr, alex.brainman
CC=golang-codereviews
https://golang.org/cl/112240044
2014-07-17 15:41:46 -06:00
|
|
|
|
runtime·stackcache_clear(c);
|
2014-01-21 02:06:57 -07:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
undo CL 101570044 / 2c57aaea79c4
redo stack allocation. This is mostly the same as
the original CL with a few bug fixes.
1. add racemalloc() for stack allocations
2. fix poolalloc/poolfree to terminate free lists correctly.
3. adjust span ref count correctly.
4. don't use cache for sizes >= StackCacheSize.
Should fix bugs and memory leaks in original changelist.
««« original CL description
undo CL 104200047 / 318b04f28372
Breaks windows and race detector.
TBR=rsc
««« original CL description
runtime: stack allocator, separate from mallocgc
In order to move malloc to Go, we need to have a
separate stack allocator. If we run out of stack
during malloc, malloc will not be available
to allocate a new stack.
Stacks are the last remaining FlagNoGC objects in the
GC heap. Once they are out, we can get rid of the
distinction between the allocated/blockboundary bits.
(This will be in a separate change.)
Fixes #7468
Fixes #7424
LGTM=rsc, dvyukov
R=golang-codereviews, dvyukov, khr, dave, rsc
CC=golang-codereviews
https://golang.org/cl/104200047
»»»
TBR=rsc
CC=golang-codereviews
https://golang.org/cl/101570044
»»»
LGTM=dvyukov
R=dvyukov, dave, khr, alex.brainman
CC=golang-codereviews
https://golang.org/cl/112240044
2014-07-17 15:41:46 -06:00
|
|
|
|
static void
|
|
|
|
|
flushallmcaches_m(G *gp)
|
|
|
|
|
{
|
|
|
|
|
flushallmcaches();
|
|
|
|
|
runtime·gogo(&gp->sched);
|
|
|
|
|
}
|
|
|
|
|
|
2014-03-25 16:09:49 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·updatememstats(GCStats *stats)
|
2010-09-07 07:57:22 -06:00
|
|
|
|
{
|
2012-12-18 09:30:29 -07:00
|
|
|
|
M *mp;
|
2013-06-06 04:56:50 -06:00
|
|
|
|
MSpan *s;
|
2011-07-18 12:52:57 -06:00
|
|
|
|
int32 i;
|
undo CL 101570044 / 2c57aaea79c4
redo stack allocation. This is mostly the same as
the original CL with a few bug fixes.
1. add racemalloc() for stack allocations
2. fix poolalloc/poolfree to terminate free lists correctly.
3. adjust span ref count correctly.
4. don't use cache for sizes >= StackCacheSize.
Should fix bugs and memory leaks in original changelist.
««« original CL description
undo CL 104200047 / 318b04f28372
Breaks windows and race detector.
TBR=rsc
««« original CL description
runtime: stack allocator, separate from mallocgc
In order to move malloc to Go, we need to have a
separate stack allocator. If we run out of stack
during malloc, malloc will not be available
to allocate a new stack.
Stacks are the last remaining FlagNoGC objects in the
GC heap. Once they are out, we can get rid of the
distinction between the allocated/blockboundary bits.
(This will be in a separate change.)
Fixes #7468
Fixes #7424
LGTM=rsc, dvyukov
R=golang-codereviews, dvyukov, khr, dave, rsc
CC=golang-codereviews
https://golang.org/cl/104200047
»»»
TBR=rsc
CC=golang-codereviews
https://golang.org/cl/101570044
»»»
LGTM=dvyukov
R=dvyukov, dave, khr, alex.brainman
CC=golang-codereviews
https://golang.org/cl/112240044
2014-07-17 15:41:46 -06:00
|
|
|
|
uint64 smallfree;
|
2012-04-05 10:48:28 -06:00
|
|
|
|
uint64 *src, *dst;
|
2010-09-07 07:57:22 -06:00
|
|
|
|
|
2012-04-05 10:48:28 -06:00
|
|
|
|
if(stats)
|
|
|
|
|
runtime·memclr((byte*)stats, sizeof(*stats));
|
2012-12-18 09:30:29 -07:00
|
|
|
|
for(mp=runtime·allm; mp; mp=mp->alllink) {
|
2012-04-05 10:48:28 -06:00
|
|
|
|
if(stats) {
|
2012-12-18 09:30:29 -07:00
|
|
|
|
src = (uint64*)&mp->gcstats;
|
2012-04-05 10:48:28 -06:00
|
|
|
|
dst = (uint64*)stats;
|
|
|
|
|
for(i=0; i<sizeof(*stats)/sizeof(uint64); i++)
|
|
|
|
|
dst[i] += src[i];
|
2012-12-18 09:30:29 -07:00
|
|
|
|
runtime·memclr((byte*)&mp->gcstats, sizeof(mp->gcstats));
|
2012-04-05 10:48:28 -06:00
|
|
|
|
}
|
2013-03-01 04:49:16 -07:00
|
|
|
|
}
|
runtime: account for all sys memory in MemStats
Currently lots of sys allocations are not accounted in any of XxxSys,
including GC bitmap, spans table, GC roots blocks, GC finalizer blocks,
iface table, netpoll descriptors and more. Up to ~20% can unaccounted.
This change introduces 2 new stats: GCSys and OtherSys for GC metadata
and all other misc allocations, respectively.
Also ensures that all XxxSys indeed sum up to Sys. All sys memory allocation
functions require the stat for accounting, so that it's impossible to miss something.
Also fix updating of mcache_sys/inuse, they were not updated after deallocation.
test/bench/garbage/parser before:
Sys 670064344
HeapSys 610271232
StackSys 65536
MSpanSys 14204928
MCacheSys 16384
BuckHashSys 1439992
after:
Sys 670064344
HeapSys 610271232
StackSys 65536
MSpanSys 14188544
MCacheSys 16384
BuckHashSys 3194304
GCSys 39198688
OtherSys 3129656
Fixes #5799.
R=rsc, dave, alex.brainman
CC=golang-dev
https://golang.org/cl/12946043
2013-09-06 14:55:40 -06:00
|
|
|
|
mstats.mcache_inuse = runtime·mheap.cachealloc.inuse;
|
|
|
|
|
mstats.mspan_inuse = runtime·mheap.spanalloc.inuse;
|
|
|
|
|
mstats.sys = mstats.heap_sys + mstats.stacks_sys + mstats.mspan_sys +
|
|
|
|
|
mstats.mcache_sys + mstats.buckhash_sys + mstats.gc_sys + mstats.other_sys;
|
|
|
|
|
|
2013-06-06 04:56:50 -06:00
|
|
|
|
// Calculate memory allocator stats.
|
|
|
|
|
// During program execution we only count number of frees and amount of freed memory.
|
|
|
|
|
// Current number of alive object in the heap and amount of alive heap memory
|
|
|
|
|
// are calculated by scanning all spans.
|
|
|
|
|
// Total number of mallocs is calculated as number of frees plus number of alive objects.
|
|
|
|
|
// Similarly, total amount of allocated memory is calculated as amount of freed memory
|
|
|
|
|
// plus amount of alive heap memory.
|
|
|
|
|
mstats.alloc = 0;
|
|
|
|
|
mstats.total_alloc = 0;
|
|
|
|
|
mstats.nmalloc = 0;
|
|
|
|
|
mstats.nfree = 0;
|
|
|
|
|
for(i = 0; i < nelem(mstats.by_size); i++) {
|
|
|
|
|
mstats.by_size[i].nmalloc = 0;
|
|
|
|
|
mstats.by_size[i].nfree = 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Flush MCache's to MCentral.
|
2014-07-18 03:05:21 -06:00
|
|
|
|
if(g == g->m->g0)
|
|
|
|
|
flushallmcaches();
|
|
|
|
|
else
|
|
|
|
|
runtime·mcall(flushallmcaches_m);
|
2013-06-06 04:56:50 -06:00
|
|
|
|
|
|
|
|
|
// Aggregate local stats.
|
|
|
|
|
cachestats();
|
|
|
|
|
|
|
|
|
|
// Scan all spans and count number of alive objects.
|
2014-08-24 02:05:07 -06:00
|
|
|
|
runtime·lock(&runtime·mheap.lock);
|
2013-06-06 04:56:50 -06:00
|
|
|
|
for(i = 0; i < runtime·mheap.nspan; i++) {
|
|
|
|
|
s = runtime·mheap.allspans[i];
|
|
|
|
|
if(s->state != MSpanInUse)
|
|
|
|
|
continue;
|
|
|
|
|
if(s->sizeclass == 0) {
|
|
|
|
|
mstats.nmalloc++;
|
|
|
|
|
mstats.alloc += s->elemsize;
|
|
|
|
|
} else {
|
|
|
|
|
mstats.nmalloc += s->ref;
|
|
|
|
|
mstats.by_size[s->sizeclass].nmalloc += s->ref;
|
|
|
|
|
mstats.alloc += s->ref*s->elemsize;
|
|
|
|
|
}
|
|
|
|
|
}
|
2014-08-24 02:05:07 -06:00
|
|
|
|
runtime·unlock(&runtime·mheap.lock);
|
2013-06-06 04:56:50 -06:00
|
|
|
|
|
|
|
|
|
// Aggregate by size class.
|
|
|
|
|
smallfree = 0;
|
|
|
|
|
mstats.nfree = runtime·mheap.nlargefree;
|
|
|
|
|
for(i = 0; i < nelem(mstats.by_size); i++) {
|
|
|
|
|
mstats.nfree += runtime·mheap.nsmallfree[i];
|
|
|
|
|
mstats.by_size[i].nfree = runtime·mheap.nsmallfree[i];
|
|
|
|
|
mstats.by_size[i].nmalloc += runtime·mheap.nsmallfree[i];
|
|
|
|
|
smallfree += runtime·mheap.nsmallfree[i] * runtime·class_to_size[i];
|
|
|
|
|
}
|
|
|
|
|
mstats.nmalloc += mstats.nfree;
|
|
|
|
|
|
|
|
|
|
// Calculate derived stats.
|
|
|
|
|
mstats.total_alloc = mstats.alloc + runtime·mheap.largefree + smallfree;
|
|
|
|
|
mstats.heap_alloc = mstats.alloc;
|
|
|
|
|
mstats.heap_objects = mstats.nmalloc - mstats.nfree;
|
2010-09-07 07:57:22 -06:00
|
|
|
|
}
|
|
|
|
|
|
2012-11-27 11:04:59 -07:00
|
|
|
|
// Structure of arguments passed to function gc().
|
2013-05-31 21:43:33 -06:00
|
|
|
|
// This allows the arguments to be passed via runtime·mcall.
|
2012-11-27 11:04:59 -07:00
|
|
|
|
struct gc_args
|
|
|
|
|
{
|
2013-05-31 21:43:33 -06:00
|
|
|
|
int64 start_time; // start time of GC in ns (just before stoptheworld)
|
2014-05-19 02:06:30 -06:00
|
|
|
|
bool eagersweep;
|
2012-11-27 11:04:59 -07:00
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
static void gc(struct gc_args *args);
|
2013-05-31 21:43:33 -06:00
|
|
|
|
static void mgc(G *gp);
|
2012-11-27 11:04:59 -07:00
|
|
|
|
|
2014-07-30 10:01:52 -06:00
|
|
|
|
int32
|
|
|
|
|
runtime·readgogc(void)
|
2013-02-03 22:00:55 -07:00
|
|
|
|
{
|
|
|
|
|
byte *p;
|
|
|
|
|
|
|
|
|
|
p = runtime·getenv("GOGC");
|
|
|
|
|
if(p == nil || p[0] == '\0')
|
|
|
|
|
return 100;
|
|
|
|
|
if(runtime·strcmp(p, (byte*)"off") == 0)
|
|
|
|
|
return -1;
|
|
|
|
|
return runtime·atoi(p);
|
|
|
|
|
}
|
|
|
|
|
|
2014-08-18 06:42:24 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·gcinit(void)
|
|
|
|
|
{
|
|
|
|
|
if(sizeof(Workbuf) != WorkbufSize)
|
|
|
|
|
runtime·throw("runtime: size of Workbuf is suboptimal");
|
|
|
|
|
|
|
|
|
|
work.markfor = runtime·parforalloc(MaxGcproc);
|
|
|
|
|
runtime·gcpercent = runtime·readgogc();
|
|
|
|
|
runtime·gcdatamask = unrollglobgcprog(gcdata, edata - data);
|
|
|
|
|
runtime·gcbssmask = unrollglobgcprog(gcbss, ebss - bss);
|
|
|
|
|
}
|
|
|
|
|
|
2014-05-19 02:06:30 -06:00
|
|
|
|
// force = 1 - do GC regardless of current heap usage
|
|
|
|
|
// force = 2 - go GC and eager sweep
|
2009-01-26 18:37:05 -07:00
|
|
|
|
void
|
runtime: ,s/[a-zA-Z0-9_]+/runtime·&/g, almost
Prefix all external symbols in runtime by runtime·,
to avoid conflicts with possible symbols of the same
name in linked-in C libraries. The obvious conflicts
are printf, malloc, and free, but hide everything to
avoid future pain.
The symbols left alone are:
** known to cgo **
_cgo_free
_cgo_malloc
libcgo_thread_start
initcgo
ncgocall
** known to linker **
_rt0_$GOARCH
_rt0_$GOARCH_$GOOS
text
etext
data
end
pclntab
epclntab
symtab
esymtab
** known to C compiler **
_divv
_modv
_div64by32
etc (arch specific)
Tested on darwin/386, darwin/amd64, linux/386, linux/amd64.
Built (but not tested) for freebsd/386, freebsd/amd64, linux/arm, windows/386.
R=r, PeterGo
CC=golang-dev
https://golang.org/cl/2899041
2010-11-04 12:00:19 -06:00
|
|
|
|
runtime·gc(int32 force)
|
2009-01-26 18:37:05 -07:00
|
|
|
|
{
|
2013-05-31 21:43:33 -06:00
|
|
|
|
struct gc_args a;
|
|
|
|
|
int32 i;
|
2009-01-26 18:37:05 -07:00
|
|
|
|
|
|
|
|
|
// The gc is turned off (via enablegc) until
|
|
|
|
|
// the bootstrap has completed.
|
|
|
|
|
// Also, malloc gets called in the guts
|
|
|
|
|
// of a number of libraries that might be
|
|
|
|
|
// holding locks. To avoid priority inversion
|
|
|
|
|
// problems, don't bother trying to run gc
|
|
|
|
|
// while holding a lock. The next mallocgc
|
|
|
|
|
// without a lock will do the gc instead.
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
if(!mstats.enablegc || g == g->m->g0 || g->m->locks > 0 || runtime·panicking)
|
2009-01-26 18:37:05 -07:00
|
|
|
|
return;
|
|
|
|
|
|
2014-07-30 10:01:52 -06:00
|
|
|
|
if(runtime·gcpercent < 0)
|
2009-01-26 18:37:05 -07:00
|
|
|
|
return;
|
|
|
|
|
|
net: add special netFD mutex
The mutex, fdMutex, handles locking and lifetime of sysfd,
and serializes Read and Write methods.
This allows to strip 2 sync.Mutex.Lock calls,
2 sync.Mutex.Unlock calls, 1 defer and some amount
of misc overhead from every network operation.
On linux/amd64, Intel E5-2690:
benchmark old ns/op new ns/op delta
BenchmarkTCP4Persistent 9595 9454 -1.47%
BenchmarkTCP4Persistent-2 8978 8772 -2.29%
BenchmarkTCP4ConcurrentReadWrite 4900 4625 -5.61%
BenchmarkTCP4ConcurrentReadWrite-2 2603 2500 -3.96%
In general it strips 70-500 ns from every network operation depending
on processor model. On my relatively new E5-2690 it accounts to ~5%
of network op cost.
Fixes #6074.
R=golang-dev, bradfitz, alex.brainman, iant, mikioh.mikioh
CC=golang-dev
https://golang.org/cl/12418043
2013-08-09 11:43:00 -06:00
|
|
|
|
runtime·semacquire(&runtime·worldsema, false);
|
2014-05-19 02:06:30 -06:00
|
|
|
|
if(force==0 && mstats.heap_alloc < mstats.next_gc) {
|
2013-05-31 21:43:33 -06:00
|
|
|
|
// typically threads which lost the race to grab
|
|
|
|
|
// worldsema exit here when gc is done.
|
|
|
|
|
runtime·semrelease(&runtime·worldsema);
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Ok, we're doing it! Stop everybody else
|
|
|
|
|
a.start_time = runtime·nanotime();
|
2014-05-19 02:06:30 -06:00
|
|
|
|
a.eagersweep = force >= 2;
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->gcing = 1;
|
2013-05-31 21:43:33 -06:00
|
|
|
|
runtime·stoptheworld();
|
2014-01-14 08:39:50 -07:00
|
|
|
|
|
2014-07-30 10:01:52 -06:00
|
|
|
|
runtime·clearpools();
|
2013-12-18 12:08:34 -07:00
|
|
|
|
|
2013-05-31 21:43:33 -06:00
|
|
|
|
// Run gc on the g0 stack. We do this so that the g stack
|
|
|
|
|
// we're currently running on will no longer change. Cuts
|
|
|
|
|
// the root set down a bit (g0 stacks are not scanned, and
|
|
|
|
|
// we don't need to scan gc's internal state). Also an
|
|
|
|
|
// enabler for copyable stacks.
|
2013-06-28 08:37:06 -06:00
|
|
|
|
for(i = 0; i < (runtime·debug.gctrace > 1 ? 2 : 1); i++) {
|
2014-04-09 08:38:12 -06:00
|
|
|
|
if(i > 0)
|
|
|
|
|
a.start_time = runtime·nanotime();
|
2013-08-21 16:17:45 -06:00
|
|
|
|
// switch to g0, call gc(&a), then switch back
|
|
|
|
|
g->param = &a;
|
|
|
|
|
g->status = Gwaiting;
|
2014-08-21 10:41:09 -06:00
|
|
|
|
g->waitreason = runtime·gostringnocopy((byte*)"garbage collection");
|
2013-08-21 16:17:45 -06:00
|
|
|
|
runtime·mcall(mgc);
|
2013-05-31 21:43:33 -06:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// all done
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->gcing = 0;
|
|
|
|
|
g->m->locks++;
|
2013-05-31 21:43:33 -06:00
|
|
|
|
runtime·semrelease(&runtime·worldsema);
|
|
|
|
|
runtime·starttheworld();
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->locks--;
|
2014-02-12 13:54:21 -07:00
|
|
|
|
|
|
|
|
|
// now that gc is done, kick off finalizer thread if needed
|
|
|
|
|
if(!ConcurrentSweep) {
|
|
|
|
|
// give the queued finalizers, if any, a chance to run
|
|
|
|
|
runtime·gosched();
|
|
|
|
|
}
|
2012-11-27 11:04:59 -07:00
|
|
|
|
}
|
|
|
|
|
|
2013-05-31 21:43:33 -06:00
|
|
|
|
static void
|
|
|
|
|
mgc(G *gp)
|
|
|
|
|
{
|
|
|
|
|
gc(gp->param);
|
|
|
|
|
gp->param = nil;
|
runtime: record proper goroutine state during stack split
Until now, the goroutine state has been scattered during the
execution of newstack and oldstack. It's all there, and those routines
know how to get back to a working goroutine, but other pieces of
the system, like stack traces, do not. If something does interrupt
the newstack or oldstack execution, the rest of the system can't
understand the goroutine. For example, if newstack decides there
is an overflow and calls throw, the stack tracer wouldn't dump the
goroutine correctly.
For newstack to save a useful state snapshot, it needs to be able
to rewind the PC in the function that triggered the split back to
the beginning of the function. (The PC is a few instructions in, just
after the call to morestack.) To make that possible, we change the
prologues to insert a jmp back to the beginning of the function
after the call to morestack. That is, the prologue used to be roughly:
TEXT myfunc
check for split
jmpcond nosplit
call morestack
nosplit:
sub $xxx, sp
Now an extra instruction is inserted after the call:
TEXT myfunc
start:
check for split
jmpcond nosplit
call morestack
jmp start
nosplit:
sub $xxx, sp
The jmp is not executed directly. It is decoded and simulated by
runtime.rewindmorestack to discover the beginning of the function,
and then the call to morestack returns directly to the start label
instead of to the jump instruction. So logically the jmp is still
executed, just not by the cpu.
The prologue thus repeats in the case of a function that needs a
stack split, but against the cost of the split itself, the extra few
instructions are noise. The repeated prologue has the nice effect of
making a stack split double-check that the new stack is big enough:
if morestack happens to return on a too-small stack, we'll now notice
before corruption happens.
The ability for newstack to rewind to the beginning of the function
should help preemption too. If newstack decides that it was called
for preemption instead of a stack split, it now has the goroutine state
correctly paused if rescheduling is needed, and when the goroutine
can run again, it can return to the start label on its original stack
and re-execute the split check.
Here is an example of a split stack overflow showing the full
trace, without any special cases in the stack printer.
(This one was triggered by making the split check incorrect.)
runtime: newstack framesize=0x0 argsize=0x18 sp=0x6aebd0 stack=[0x6b0000, 0x6b0fa0]
morebuf={pc:0x69f5b sp:0x6aebd8 lr:0x0}
sched={pc:0x68880 sp:0x6aebd0 lr:0x0 ctxt:0x34e700}
runtime: split stack overflow: 0x6aebd0 < 0x6b0000
fatal error: runtime: split stack overflow
goroutine 1 [stack split]:
runtime.mallocgc(0x290, 0x100000000, 0x1)
/Users/rsc/g/go/src/pkg/runtime/zmalloc_darwin_amd64.c:21 fp=0x6aebd8
runtime.new()
/Users/rsc/g/go/src/pkg/runtime/zmalloc_darwin_amd64.c:682 +0x5b fp=0x6aec08
go/build.(*Context).Import(0x5ae340, 0xc210030c71, 0xa, 0xc2100b4380, 0x1b, ...)
/Users/rsc/g/go/src/pkg/go/build/build.go:424 +0x3a fp=0x6b00a0
main.loadImport(0xc210030c71, 0xa, 0xc2100b4380, 0x1b, 0xc2100b42c0, ...)
/Users/rsc/g/go/src/cmd/go/pkg.go:249 +0x371 fp=0x6b01a8
main.(*Package).load(0xc21017c800, 0xc2100b42c0, 0xc2101828c0, 0x0, 0x0, ...)
/Users/rsc/g/go/src/cmd/go/pkg.go:431 +0x2801 fp=0x6b0c98
main.loadPackage(0x369040, 0x7, 0xc2100b42c0, 0x0)
/Users/rsc/g/go/src/cmd/go/pkg.go:709 +0x857 fp=0x6b0f80
----- stack segment boundary -----
main.(*builder).action(0xc2100902a0, 0x0, 0x0, 0xc2100e6c00, 0xc2100e5750, ...)
/Users/rsc/g/go/src/cmd/go/build.go:539 +0x437 fp=0x6b14a0
main.(*builder).action(0xc2100902a0, 0x0, 0x0, 0xc21015b400, 0x2, ...)
/Users/rsc/g/go/src/cmd/go/build.go:528 +0x1d2 fp=0x6b1658
main.(*builder).test(0xc2100902a0, 0xc210092000, 0x0, 0x0, 0xc21008ff60, ...)
/Users/rsc/g/go/src/cmd/go/test.go:622 +0x1b53 fp=0x6b1f68
----- stack segment boundary -----
main.runTest(0x5a6b20, 0xc21000a020, 0x2, 0x2)
/Users/rsc/g/go/src/cmd/go/test.go:366 +0xd09 fp=0x6a5cf0
main.main()
/Users/rsc/g/go/src/cmd/go/main.go:161 +0x4f9 fp=0x6a5f78
runtime.main()
/Users/rsc/g/go/src/pkg/runtime/proc.c:183 +0x92 fp=0x6a5fa0
runtime.goexit()
/Users/rsc/g/go/src/pkg/runtime/proc.c:1266 fp=0x6a5fa8
And here is a seg fault during oldstack:
SIGSEGV: segmentation violation
PC=0x1b2a6
runtime.oldstack()
/Users/rsc/g/go/src/pkg/runtime/stack.c:159 +0x76
runtime.lessstack()
/Users/rsc/g/go/src/pkg/runtime/asm_amd64.s:270 +0x22
goroutine 1 [stack unsplit]:
fmt.(*pp).printArg(0x2102e64e0, 0xe5c80, 0x2102c9220, 0x73, 0x0, ...)
/Users/rsc/g/go/src/pkg/fmt/print.go:818 +0x3d3 fp=0x221031e6f8
fmt.(*pp).doPrintf(0x2102e64e0, 0x12fb20, 0x2, 0x221031eb98, 0x1, ...)
/Users/rsc/g/go/src/pkg/fmt/print.go:1183 +0x15cb fp=0x221031eaf0
fmt.Sprintf(0x12fb20, 0x2, 0x221031eb98, 0x1, 0x1, ...)
/Users/rsc/g/go/src/pkg/fmt/print.go:234 +0x67 fp=0x221031eb40
flag.(*stringValue).String(0x2102c9210, 0x1, 0x0)
/Users/rsc/g/go/src/pkg/flag/flag.go:180 +0xb3 fp=0x221031ebb0
flag.(*FlagSet).Var(0x2102f6000, 0x293d38, 0x2102c9210, 0x143490, 0xa, ...)
/Users/rsc/g/go/src/pkg/flag/flag.go:633 +0x40 fp=0x221031eca0
flag.(*FlagSet).StringVar(0x2102f6000, 0x2102c9210, 0x143490, 0xa, 0x12fa60, ...)
/Users/rsc/g/go/src/pkg/flag/flag.go:550 +0x91 fp=0x221031ece8
flag.(*FlagSet).String(0x2102f6000, 0x143490, 0xa, 0x12fa60, 0x0, ...)
/Users/rsc/g/go/src/pkg/flag/flag.go:563 +0x87 fp=0x221031ed38
flag.String(0x143490, 0xa, 0x12fa60, 0x0, 0x161950, ...)
/Users/rsc/g/go/src/pkg/flag/flag.go:570 +0x6b fp=0x221031ed80
testing.init()
/Users/rsc/g/go/src/pkg/testing/testing.go:-531 +0xbb fp=0x221031edc0
strings_test.init()
/Users/rsc/g/go/src/pkg/strings/strings_test.go:1115 +0x62 fp=0x221031ef70
main.init()
strings/_test/_testmain.go:90 +0x3d fp=0x221031ef78
runtime.main()
/Users/rsc/g/go/src/pkg/runtime/proc.c:180 +0x8a fp=0x221031efa0
runtime.goexit()
/Users/rsc/g/go/src/pkg/runtime/proc.c:1269 fp=0x221031efa8
goroutine 2 [runnable]:
runtime.MHeap_Scavenger()
/Users/rsc/g/go/src/pkg/runtime/mheap.c:438
runtime.goexit()
/Users/rsc/g/go/src/pkg/runtime/proc.c:1269
created by runtime.main
/Users/rsc/g/go/src/pkg/runtime/proc.c:166
rax 0x23ccc0
rbx 0x23ccc0
rcx 0x0
rdx 0x38
rdi 0x2102c0170
rsi 0x221032cfe0
rbp 0x221032cfa0
rsp 0x7fff5fbff5b0
r8 0x2102c0120
r9 0x221032cfa0
r10 0x221032c000
r11 0x104ce8
r12 0xe5c80
r13 0x1be82baac718
r14 0x13091135f7d69200
r15 0x0
rip 0x1b2a6
rflags 0x10246
cs 0x2b
fs 0x0
gs 0x0
Fixes #5723.
R=r, dvyukov, go.peter.90, dave, iant
CC=golang-dev
https://golang.org/cl/10360048
2013-06-27 09:32:01 -06:00
|
|
|
|
gp->status = Grunning;
|
2013-06-12 13:22:26 -06:00
|
|
|
|
runtime·gogo(&gp->sched);
|
2013-05-31 21:43:33 -06:00
|
|
|
|
}
|
2013-02-21 15:01:13 -07:00
|
|
|
|
|
2014-07-30 10:01:52 -06:00
|
|
|
|
void
|
2014-08-06 15:33:57 -06:00
|
|
|
|
runtime·gc_m(void)
|
2014-07-30 10:01:52 -06:00
|
|
|
|
{
|
|
|
|
|
struct gc_args a;
|
|
|
|
|
G *gp;
|
|
|
|
|
|
|
|
|
|
gp = g->m->curg;
|
|
|
|
|
gp->status = Gwaiting;
|
2014-08-21 10:41:09 -06:00
|
|
|
|
gp->waitreason = runtime·gostringnocopy((byte*)"garbage collection");
|
2014-07-30 10:01:52 -06:00
|
|
|
|
|
2014-08-19 01:53:20 -06:00
|
|
|
|
a.start_time = (uint64)(g->m->scalararg[0]) | ((uint64)(g->m->scalararg[1]) << 32);
|
|
|
|
|
a.eagersweep = g->m->scalararg[2];
|
2014-07-30 10:01:52 -06:00
|
|
|
|
gc(&a);
|
|
|
|
|
|
|
|
|
|
gp->status = Grunning;
|
|
|
|
|
}
|
|
|
|
|
|
2012-11-27 11:04:59 -07:00
|
|
|
|
static void
|
|
|
|
|
gc(struct gc_args *args)
|
|
|
|
|
{
|
2013-01-22 02:44:49 -07:00
|
|
|
|
int64 t0, t1, t2, t3, t4;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
uint64 heap0, heap1, obj;
|
2012-11-27 11:04:59 -07:00
|
|
|
|
GCStats stats;
|
|
|
|
|
|
2014-04-01 11:30:10 -06:00
|
|
|
|
if(runtime·debug.allocfreetrace)
|
|
|
|
|
runtime·tracegc();
|
|
|
|
|
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->traceback = 2;
|
2013-05-31 21:43:33 -06:00
|
|
|
|
t0 = args->start_time;
|
2014-01-16 01:54:46 -07:00
|
|
|
|
work.tstart = args->start_time;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
|
2014-04-09 08:38:12 -06:00
|
|
|
|
t1 = 0;
|
|
|
|
|
if(runtime·debug.gctrace)
|
|
|
|
|
t1 = runtime·nanotime();
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
|
|
|
|
|
// Sweep what is not sweeped by bgsweep.
|
|
|
|
|
while(runtime·sweepone() != -1)
|
2014-07-29 01:01:02 -06:00
|
|
|
|
sweep.npausesweep++;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
|
2014-08-24 02:05:07 -06:00
|
|
|
|
// Cache runtime.mheap.allspans in work.spans to avoid conflicts with
|
|
|
|
|
// resizing/freeing allspans.
|
|
|
|
|
// New spans can be created while GC progresses, but they are not garbage for
|
|
|
|
|
// this round:
|
|
|
|
|
// - new stack spans can be created even while the world is stopped.
|
|
|
|
|
// - new malloc spans can be created during the concurrent sweep
|
|
|
|
|
|
|
|
|
|
// Even if this is stop-the-world, a concurrent exitsyscall can allocate a stack from heap.
|
|
|
|
|
runtime·lock(&runtime·mheap.lock);
|
|
|
|
|
// Free the old cached sweep array if necessary.
|
|
|
|
|
if(work.spans != nil && work.spans != runtime·mheap.allspans)
|
|
|
|
|
runtime·SysFree(work.spans, work.nspan*sizeof(work.spans[0]), &mstats.other_sys);
|
|
|
|
|
// Cache the current array for marking.
|
|
|
|
|
runtime·mheap.gcspans = runtime·mheap.allspans;
|
|
|
|
|
work.spans = runtime·mheap.allspans;
|
|
|
|
|
work.nspan = runtime·mheap.nspan;
|
|
|
|
|
runtime·unlock(&runtime·mheap.lock);
|
|
|
|
|
|
2012-05-22 11:35:52 -06:00
|
|
|
|
work.nwait = 0;
|
|
|
|
|
work.ndone = 0;
|
2012-05-15 09:10:16 -06:00
|
|
|
|
work.nproc = runtime·gcprocs();
|
2014-01-21 02:06:57 -07:00
|
|
|
|
runtime·parforsetup(work.markfor, work.nproc, RootCount + runtime·allglen, nil, false, markroot);
|
2012-05-15 09:10:16 -06:00
|
|
|
|
if(work.nproc > 1) {
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
runtime·noteclear(&work.alldone);
|
2012-05-15 09:10:16 -06:00
|
|
|
|
runtime·helpgc(work.nproc);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
}
|
|
|
|
|
|
2014-04-09 08:38:12 -06:00
|
|
|
|
t2 = 0;
|
|
|
|
|
if(runtime·debug.gctrace)
|
|
|
|
|
t2 = runtime·nanotime();
|
2013-01-22 02:44:49 -07:00
|
|
|
|
|
2013-03-21 02:48:02 -06:00
|
|
|
|
gchelperstart();
|
2012-05-24 00:55:50 -06:00
|
|
|
|
runtime·parfordo(work.markfor);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
scanblock(nil, 0, nil);
|
2012-05-24 00:55:50 -06:00
|
|
|
|
|
2014-04-09 08:38:12 -06:00
|
|
|
|
t3 = 0;
|
|
|
|
|
if(runtime·debug.gctrace)
|
|
|
|
|
t3 = runtime·nanotime();
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
|
2012-05-22 11:35:52 -06:00
|
|
|
|
if(work.nproc > 1)
|
|
|
|
|
runtime·notesleep(&work.alldone);
|
|
|
|
|
|
2013-06-06 04:56:50 -06:00
|
|
|
|
cachestats();
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
// next_gc calculation is tricky with concurrent sweep since we don't know size of live heap
|
|
|
|
|
// estimate what was live heap size after previous GC (for tracing only)
|
2014-07-30 10:01:52 -06:00
|
|
|
|
heap0 = mstats.next_gc*100/(runtime·gcpercent+100);
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
// conservatively set next_gc to high value assuming that everything is live
|
|
|
|
|
// concurrent/lazy sweep will reduce this number while discovering new garbage
|
2014-07-30 10:01:52 -06:00
|
|
|
|
mstats.next_gc = mstats.heap_alloc+mstats.heap_alloc*runtime·gcpercent/100;
|
2010-02-10 01:00:12 -07:00
|
|
|
|
|
2013-01-22 02:44:49 -07:00
|
|
|
|
t4 = runtime·nanotime();
|
2014-05-12 23:53:03 -06:00
|
|
|
|
mstats.last_gc = runtime·unixnanotime(); // must be Unix time to make sense to user
|
2013-01-22 02:44:49 -07:00
|
|
|
|
mstats.pause_ns[mstats.numgc%nelem(mstats.pause_ns)] = t4 - t0;
|
|
|
|
|
mstats.pause_total_ns += t4 - t0;
|
2010-02-08 15:32:22 -07:00
|
|
|
|
mstats.numgc++;
|
|
|
|
|
if(mstats.debuggc)
|
2013-01-22 02:44:49 -07:00
|
|
|
|
runtime·printf("pause %D\n", t4-t0);
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
|
2013-06-28 08:37:06 -06:00
|
|
|
|
if(runtime·debug.gctrace) {
|
2013-06-06 04:56:50 -06:00
|
|
|
|
heap1 = mstats.heap_alloc;
|
2014-03-25 16:09:49 -06:00
|
|
|
|
runtime·updatememstats(&stats);
|
2014-03-06 10:33:00 -07:00
|
|
|
|
if(heap1 != mstats.heap_alloc) {
|
2014-03-06 12:56:22 -07:00
|
|
|
|
runtime·printf("runtime: mstats skew: heap=%D/%D\n", heap1, mstats.heap_alloc);
|
2014-03-06 10:33:00 -07:00
|
|
|
|
runtime·throw("mstats skew");
|
|
|
|
|
}
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
obj = mstats.nmalloc - mstats.nfree;
|
2013-06-06 04:56:50 -06:00
|
|
|
|
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
stats.nprocyield += work.markfor->nprocyield;
|
|
|
|
|
stats.nosyield += work.markfor->nosyield;
|
|
|
|
|
stats.nsleep += work.markfor->nsleep;
|
2013-06-06 04:56:50 -06:00
|
|
|
|
|
2014-04-10 12:34:48 -06:00
|
|
|
|
runtime·printf("gc%d(%d): %D+%D+%D+%D us, %D -> %D MB, %D (%D-%D) objects,"
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
" %d/%d/%d sweeps,"
|
2012-05-22 11:35:52 -06:00
|
|
|
|
" %D(%D) handoff, %D(%D) steal, %D/%D/%D yields\n",
|
2014-04-10 12:34:48 -06:00
|
|
|
|
mstats.numgc, work.nproc, (t1-t0)/1000, (t2-t1)/1000, (t3-t2)/1000, (t4-t3)/1000,
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
heap0>>20, heap1>>20, obj,
|
2011-02-02 21:03:47 -07:00
|
|
|
|
mstats.nmalloc, mstats.nfree,
|
2014-08-24 02:05:07 -06:00
|
|
|
|
work.nspan, sweep.nbgsweep, sweep.npausesweep,
|
2012-04-05 10:48:28 -06:00
|
|
|
|
stats.nhandoff, stats.nhandoffcnt,
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
work.markfor->nsteal, work.markfor->nstealcnt,
|
2012-04-05 10:48:28 -06:00
|
|
|
|
stats.nprocyield, stats.nosyield, stats.nsleep);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
sweep.nbgsweep = sweep.npausesweep = 0;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
}
|
2012-05-22 11:35:52 -06:00
|
|
|
|
|
2014-08-24 02:05:07 -06:00
|
|
|
|
// See the comment in the beginning of this function as to why we need the following.
|
|
|
|
|
// Even if this is still stop-the-world, a concurrent exitsyscall can allocate a stack from heap.
|
|
|
|
|
runtime·lock(&runtime·mheap.lock);
|
|
|
|
|
// Free the old cached mark array if necessary.
|
|
|
|
|
if(work.spans != nil && work.spans != runtime·mheap.allspans)
|
|
|
|
|
runtime·SysFree(work.spans, work.nspan*sizeof(work.spans[0]), &mstats.other_sys);
|
|
|
|
|
// Cache the current array for sweeping.
|
|
|
|
|
runtime·mheap.gcspans = runtime·mheap.allspans;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
runtime·mheap.sweepgen += 2;
|
|
|
|
|
runtime·mheap.sweepdone = false;
|
2014-08-24 02:05:07 -06:00
|
|
|
|
work.spans = runtime·mheap.allspans;
|
|
|
|
|
work.nspan = runtime·mheap.nspan;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
sweep.spanidx = 0;
|
2014-08-24 02:05:07 -06:00
|
|
|
|
runtime·unlock(&runtime·mheap.lock);
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
|
2014-02-12 13:03:27 -07:00
|
|
|
|
// Temporary disable concurrent sweep, because we see failures on builders.
|
2014-05-19 02:06:30 -06:00
|
|
|
|
if(ConcurrentSweep && !args->eagersweep) {
|
2014-02-12 13:03:27 -07:00
|
|
|
|
runtime·lock(&gclock);
|
|
|
|
|
if(sweep.g == nil)
|
|
|
|
|
sweep.g = runtime·newproc1(&bgsweepv, nil, 0, 0, runtime·gc);
|
|
|
|
|
else if(sweep.parked) {
|
|
|
|
|
sweep.parked = false;
|
|
|
|
|
runtime·ready(sweep.g);
|
|
|
|
|
}
|
|
|
|
|
runtime·unlock(&gclock);
|
|
|
|
|
} else {
|
|
|
|
|
// Sweep all spans eagerly.
|
|
|
|
|
while(runtime·sweepone() != -1)
|
2014-07-29 01:01:02 -06:00
|
|
|
|
sweep.npausesweep++;
|
runtime: concurrent GC sweep
Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.
It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.
At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.
For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.
Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.
http-1
allocated 7036 7033 -0.04%
allocs 60 60 +0.00%
cputime 51050 46700 -8.52%
gc-pause-one 34060569 1777993 -94.78%
gc-pause-total 2554 133 -94.79%
latency-50 178448 170926 -4.22%
latency-95 284350 198294 -30.26%
latency-99 345191 220652 -36.08%
rss 101564416 101007360 -0.55%
sys-gc 6606832 6541296 -0.99%
sys-heap 88801280 87752704 -1.18%
sys-other 7334208 7405928 +0.98%
sys-stack 524288 524288 +0.00%
sys-total 103266608 102224216 -1.01%
time 50339 46533 -7.56%
virtual-mem 292990976 293728256 +0.25%
garbage-1
allocated 2983818 2990889 +0.24%
allocs 62880 62902 +0.03%
cputime 16480000 16190000 -1.76%
gc-pause-one 828462467 487875135 -41.11%
gc-pause-total 4142312 2439375 -41.11%
rss 1151709184 1153712128 +0.17%
sys-gc 66068352 66068352 +0.00%
sys-heap 1039728640 1039728640 +0.00%
sys-other 37776064 40770176 +7.93%
sys-stack 8781824 8781824 +0.00%
sys-total 1152354880 1155348992 +0.26%
time 16496998 16199876 -1.80%
virtual-mem 1409564672 1402281984 -0.52%
LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043
2014-02-12 11:16:42 -07:00
|
|
|
|
}
|
|
|
|
|
|
2012-02-22 19:45:01 -07:00
|
|
|
|
runtime·MProf_GC();
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->traceback = 0;
|
2010-03-26 15:15:30 -06:00
|
|
|
|
}
|
|
|
|
|
|
2014-01-30 02:28:19 -07:00
|
|
|
|
extern uintptr runtime·sizeof_C_MStats;
|
|
|
|
|
|
2011-07-21 22:55:01 -06:00
|
|
|
|
void
|
2012-02-06 11:16:26 -07:00
|
|
|
|
runtime·ReadMemStats(MStats *stats)
|
2011-07-21 22:55:01 -06:00
|
|
|
|
{
|
2012-02-22 19:45:01 -07:00
|
|
|
|
// Have to acquire worldsema to stop the world,
|
2011-07-21 22:55:01 -06:00
|
|
|
|
// because stoptheworld can only be used by
|
|
|
|
|
// one goroutine at a time, and there might be
|
|
|
|
|
// a pending garbage collection already calling it.
|
net: add special netFD mutex
The mutex, fdMutex, handles locking and lifetime of sysfd,
and serializes Read and Write methods.
This allows to strip 2 sync.Mutex.Lock calls,
2 sync.Mutex.Unlock calls, 1 defer and some amount
of misc overhead from every network operation.
On linux/amd64, Intel E5-2690:
benchmark old ns/op new ns/op delta
BenchmarkTCP4Persistent 9595 9454 -1.47%
BenchmarkTCP4Persistent-2 8978 8772 -2.29%
BenchmarkTCP4ConcurrentReadWrite 4900 4625 -5.61%
BenchmarkTCP4ConcurrentReadWrite-2 2603 2500 -3.96%
In general it strips 70-500 ns from every network operation depending
on processor model. On my relatively new E5-2690 it accounts to ~5%
of network op cost.
Fixes #6074.
R=golang-dev, bradfitz, alex.brainman, iant, mikioh.mikioh
CC=golang-dev
https://golang.org/cl/12418043
2013-08-09 11:43:00 -06:00
|
|
|
|
runtime·semacquire(&runtime·worldsema, false);
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->gcing = 1;
|
2011-07-21 22:55:01 -06:00
|
|
|
|
runtime·stoptheworld();
|
2014-03-25 16:09:49 -06:00
|
|
|
|
runtime·updatememstats(nil);
|
2014-01-30 02:28:19 -07:00
|
|
|
|
// Size of the trailing by_size array differs between Go and C,
|
|
|
|
|
// NumSizeClasses was changed, but we can not change Go struct because of backward compatibility.
|
2014-08-04 21:40:44 -06:00
|
|
|
|
runtime·memmove(stats, &mstats, runtime·sizeof_C_MStats);
|
undo CL 101570044 / 2c57aaea79c4
redo stack allocation. This is mostly the same as
the original CL with a few bug fixes.
1. add racemalloc() for stack allocations
2. fix poolalloc/poolfree to terminate free lists correctly.
3. adjust span ref count correctly.
4. don't use cache for sizes >= StackCacheSize.
Should fix bugs and memory leaks in original changelist.
««« original CL description
undo CL 104200047 / 318b04f28372
Breaks windows and race detector.
TBR=rsc
««« original CL description
runtime: stack allocator, separate from mallocgc
In order to move malloc to Go, we need to have a
separate stack allocator. If we run out of stack
during malloc, malloc will not be available
to allocate a new stack.
Stacks are the last remaining FlagNoGC objects in the
GC heap. Once they are out, we can get rid of the
distinction between the allocated/blockboundary bits.
(This will be in a separate change.)
Fixes #7468
Fixes #7424
LGTM=rsc, dvyukov
R=golang-codereviews, dvyukov, khr, dave, rsc
CC=golang-codereviews
https://golang.org/cl/104200047
»»»
TBR=rsc
CC=golang-codereviews
https://golang.org/cl/101570044
»»»
LGTM=dvyukov
R=dvyukov, dave, khr, alex.brainman
CC=golang-codereviews
https://golang.org/cl/112240044
2014-07-17 15:41:46 -06:00
|
|
|
|
|
|
|
|
|
// Stack numbers are part of the heap numbers, separate those out for user consumption
|
|
|
|
|
stats->stacks_sys = stats->stacks_inuse;
|
|
|
|
|
stats->heap_inuse -= stats->stacks_inuse;
|
|
|
|
|
stats->heap_sys -= stats->stacks_inuse;
|
|
|
|
|
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->gcing = 0;
|
|
|
|
|
g->m->locks++;
|
2012-02-22 19:45:01 -07:00
|
|
|
|
runtime·semrelease(&runtime·worldsema);
|
2012-05-15 09:10:16 -06:00
|
|
|
|
runtime·starttheworld();
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
g->m->locks--;
|
2011-07-21 22:55:01 -06:00
|
|
|
|
}
|
|
|
|
|
|
2013-02-03 22:00:55 -07:00
|
|
|
|
void
|
|
|
|
|
runtime∕debug·readGCStats(Slice *pauses)
|
|
|
|
|
{
|
|
|
|
|
uint64 *p;
|
|
|
|
|
uint32 i, n;
|
|
|
|
|
|
|
|
|
|
// Calling code in runtime/debug should make the slice large enough.
|
|
|
|
|
if(pauses->cap < nelem(mstats.pause_ns)+3)
|
|
|
|
|
runtime·throw("runtime: short slice passed to readGCStats");
|
|
|
|
|
|
|
|
|
|
// Pass back: pauses, last gc (absolute time), number of gc, total pause ns.
|
|
|
|
|
p = (uint64*)pauses->array;
|
2014-08-07 07:00:02 -06:00
|
|
|
|
runtime·lock(&runtime·mheap.lock);
|
2013-02-03 22:00:55 -07:00
|
|
|
|
n = mstats.numgc;
|
|
|
|
|
if(n > nelem(mstats.pause_ns))
|
|
|
|
|
n = nelem(mstats.pause_ns);
|
|
|
|
|
|
|
|
|
|
// The pause buffer is circular. The most recent pause is at
|
|
|
|
|
// pause_ns[(numgc-1)%nelem(pause_ns)], and then backward
|
|
|
|
|
// from there to go back farther in time. We deliver the times
|
|
|
|
|
// most recent first (in p[0]).
|
|
|
|
|
for(i=0; i<n; i++)
|
|
|
|
|
p[i] = mstats.pause_ns[(mstats.numgc-1-i)%nelem(mstats.pause_ns)];
|
|
|
|
|
|
|
|
|
|
p[n] = mstats.last_gc;
|
|
|
|
|
p[n+1] = mstats.numgc;
|
|
|
|
|
p[n+2] = mstats.pause_total_ns;
|
2014-08-07 07:00:02 -06:00
|
|
|
|
runtime·unlock(&runtime·mheap.lock);
|
2013-02-03 22:00:55 -07:00
|
|
|
|
pauses->len = n+3;
|
|
|
|
|
}
|
|
|
|
|
|
2014-08-24 21:27:00 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·setgcpercent_m(void) {
|
|
|
|
|
int32 in;
|
runtime: use goc2c as much as possible
Package runtime's C functions written to be called from Go
started out written in C using carefully constructed argument
lists and the FLUSH macro to write a result back to memory.
For some functions, the appropriate parameter list ended up
being architecture-dependent due to differences in alignment,
so we added 'goc2c', which takes a .goc file containing Go func
declarations but C bodies, rewrites the Go func declaration to
equivalent C declarations for the target architecture, adds the
needed FLUSH statements, and writes out an equivalent C file.
That C file is compiled as part of package runtime.
Native Client's x86-64 support introduces the most complex
alignment rules yet, breaking many functions that could until
now be portably written in C. Using goc2c for those avoids the
breakage.
Separately, Keith's work on emitting stack information from
the C compiler would require the hand-written functions
to add #pragmas specifying how many arguments are result
parameters. Using goc2c for those avoids maintaining #pragmas.
For both reasons, use goc2c for as many Go-called C functions
as possible.
This CL is a replay of the bulk of CL 15400047 and CL 15790043,
both of which were reviewed as part of the NaCl port and are
checked in to the NaCl branch. This CL is part of bringing the
NaCl code into the main tree.
No new code here, just reformatting and occasional movement
into .h files.
LGTM=r
R=dave, alex.brainman, r
CC=golang-codereviews
https://golang.org/cl/65220044
2014-02-20 13:58:47 -07:00
|
|
|
|
int32 out;
|
|
|
|
|
|
2014-08-24 21:27:00 -06:00
|
|
|
|
in = (int32)(intptr)g->m->scalararg[0];
|
|
|
|
|
|
2014-08-07 07:00:02 -06:00
|
|
|
|
runtime·lock(&runtime·mheap.lock);
|
2014-07-30 10:01:52 -06:00
|
|
|
|
out = runtime·gcpercent;
|
2013-02-03 22:00:55 -07:00
|
|
|
|
if(in < 0)
|
|
|
|
|
in = -1;
|
2014-07-30 10:01:52 -06:00
|
|
|
|
runtime·gcpercent = in;
|
2014-08-07 07:00:02 -06:00
|
|
|
|
runtime·unlock(&runtime·mheap.lock);
|
2014-08-24 21:27:00 -06:00
|
|
|
|
|
|
|
|
|
g->m->scalararg[0] = (uintptr)(intptr)out;
|
2013-02-03 22:00:55 -07:00
|
|
|
|
}
|
|
|
|
|
|
2013-03-21 02:48:02 -06:00
|
|
|
|
static void
|
|
|
|
|
gchelperstart(void)
|
|
|
|
|
{
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
if(g->m->helpgc < 0 || g->m->helpgc >= MaxGcproc)
|
2013-03-21 02:48:02 -06:00
|
|
|
|
runtime·throw("gchelperstart: bad m->helpgc");
|
all: remove 'extern register M *m' from runtime
The runtime has historically held two dedicated values g (current goroutine)
and m (current thread) in 'extern register' slots (TLS on x86, real registers
backed by TLS on ARM).
This CL removes the extern register m; code now uses g->m.
On ARM, this frees up the register that formerly held m (R9).
This is important for NaCl, because NaCl ARM code cannot use R9 at all.
The Go 1 macrobenchmarks (those with per-op times >= 10 µs) are unaffected:
BenchmarkBinaryTree17 5491374955 5471024381 -0.37%
BenchmarkFannkuch11 4357101311 4275174828 -1.88%
BenchmarkGobDecode 11029957 11364184 +3.03%
BenchmarkGobEncode 6852205 6784822 -0.98%
BenchmarkGzip 650795967 650152275 -0.10%
BenchmarkGunzip 140962363 141041670 +0.06%
BenchmarkHTTPClientServer 71581 73081 +2.10%
BenchmarkJSONEncode 31928079 31913356 -0.05%
BenchmarkJSONDecode 117470065 113689916 -3.22%
BenchmarkMandelbrot200 6008923 5998712 -0.17%
BenchmarkGoParse 6310917 6327487 +0.26%
BenchmarkRegexpMatchMedium_1K 114568 114763 +0.17%
BenchmarkRegexpMatchHard_1K 168977 169244 +0.16%
BenchmarkRevcomp 935294971 914060918 -2.27%
BenchmarkTemplate 145917123 148186096 +1.55%
Minux previous reported larger variations, but these were caused by
run-to-run noise, not repeatable slowdowns.
Actual code changes by Minux.
I only did the docs and the benchmarking.
LGTM=dvyukov, iant, minux
R=minux, josharian, iant, dave, bradfitz, dvyukov
CC=golang-codereviews
https://golang.org/cl/109050043
2014-06-26 09:54:39 -06:00
|
|
|
|
if(g != g->m->g0)
|
2013-05-31 21:43:33 -06:00
|
|
|
|
runtime·throw("gchelper not running on g0 stack");
|
2013-03-21 02:48:02 -06:00
|
|
|
|
}
|
|
|
|
|
|
2010-03-26 15:15:30 -06:00
|
|
|
|
static void
|
|
|
|
|
runfinq(void)
|
|
|
|
|
{
|
2011-10-06 09:42:51 -06:00
|
|
|
|
Finalizer *f;
|
|
|
|
|
FinBlock *fb, *next;
|
2010-03-26 15:15:30 -06:00
|
|
|
|
byte *frame;
|
2011-10-06 09:42:51 -06:00
|
|
|
|
uint32 framesz, framecap, i;
|
2013-08-14 12:54:31 -06:00
|
|
|
|
Eface *ef, ef1;
|
2010-03-26 15:15:30 -06:00
|
|
|
|
|
2014-03-07 09:27:01 -07:00
|
|
|
|
// This function blocks for long periods of time, and because it is written in C
|
|
|
|
|
// we have no liveness information. Zero everything so that uninitialized pointers
|
|
|
|
|
// do not cause memory leaks.
|
|
|
|
|
f = nil;
|
|
|
|
|
fb = nil;
|
|
|
|
|
next = nil;
|
2011-10-06 09:42:51 -06:00
|
|
|
|
frame = nil;
|
|
|
|
|
framecap = 0;
|
2014-03-07 09:27:01 -07:00
|
|
|
|
framesz = 0;
|
|
|
|
|
i = 0;
|
|
|
|
|
ef = nil;
|
|
|
|
|
ef1.type = nil;
|
|
|
|
|
ef1.data = nil;
|
|
|
|
|
|
|
|
|
|
// force flush to memory
|
|
|
|
|
USED(&f);
|
|
|
|
|
USED(&fb);
|
|
|
|
|
USED(&next);
|
|
|
|
|
USED(&framesz);
|
|
|
|
|
USED(&i);
|
|
|
|
|
USED(&ef);
|
|
|
|
|
USED(&ef1);
|
|
|
|
|
|
2010-03-26 15:15:30 -06:00
|
|
|
|
for(;;) {
|
2014-03-26 05:11:36 -06:00
|
|
|
|
runtime·lock(&finlock);
|
2011-10-06 09:42:51 -06:00
|
|
|
|
fb = finq;
|
2010-03-26 15:15:30 -06:00
|
|
|
|
finq = nil;
|
2011-10-06 09:42:51 -06:00
|
|
|
|
if(fb == nil) {
|
2014-03-26 05:11:36 -06:00
|
|
|
|
runtime·fingwait = true;
|
2014-04-15 09:48:17 -06:00
|
|
|
|
g->isbackground = true;
|
2014-08-21 10:41:09 -06:00
|
|
|
|
runtime·parkunlock(&finlock, runtime·gostringnocopy((byte*)"finalizer wait"));
|
2014-04-15 09:48:17 -06:00
|
|
|
|
g->isbackground = false;
|
2010-03-26 15:15:30 -06:00
|
|
|
|
continue;
|
|
|
|
|
}
|
2014-03-26 05:11:36 -06:00
|
|
|
|
runtime·unlock(&finlock);
|
2012-11-14 05:58:10 -07:00
|
|
|
|
if(raceenabled)
|
|
|
|
|
runtime·racefingo();
|
2011-10-06 09:42:51 -06:00
|
|
|
|
for(; fb; fb=next) {
|
|
|
|
|
next = fb->next;
|
|
|
|
|
for(i=0; i<fb->cnt; i++) {
|
|
|
|
|
f = &fb->fin[i];
|
2013-07-29 09:43:08 -06:00
|
|
|
|
framesz = sizeof(Eface) + f->nret;
|
2011-10-06 09:42:51 -06:00
|
|
|
|
if(framecap < framesz) {
|
2013-07-19 08:01:33 -06:00
|
|
|
|
// The frame does not contain pointers interesting for GC,
|
2014-02-24 09:53:50 -07:00
|
|
|
|
// all not yet finalized objects are stored in finq.
|
2013-08-23 18:28:47 -06:00
|
|
|
|
// If we do not mark it as FlagNoScan,
|
2013-07-19 08:01:33 -06:00
|
|
|
|
// the last finalized object is not collected.
|
2014-08-07 03:04:04 -06:00
|
|
|
|
frame = runtime·mallocgc(framesz, 0, FlagNoScan);
|
2011-10-06 09:42:51 -06:00
|
|
|
|
framecap = framesz;
|
|
|
|
|
}
|
2013-08-14 12:54:31 -06:00
|
|
|
|
if(f->fint == nil)
|
|
|
|
|
runtime·throw("missing type in runfinq");
|
2014-08-18 19:13:11 -06:00
|
|
|
|
if((f->fint->kind&KindMask) == KindPtr) {
|
2013-08-14 12:54:31 -06:00
|
|
|
|
// direct use of pointer
|
2013-07-29 09:43:08 -06:00
|
|
|
|
*(void**)frame = f->arg;
|
2013-08-14 12:54:31 -06:00
|
|
|
|
} else if(((InterfaceType*)f->fint)->mhdr.len == 0) {
|
|
|
|
|
// convert to empty interface
|
2013-07-29 09:43:08 -06:00
|
|
|
|
ef = (Eface*)frame;
|
2014-08-07 07:00:02 -06:00
|
|
|
|
ef->type = &f->ot->typ;
|
2013-07-29 09:43:08 -06:00
|
|
|
|
ef->data = f->arg;
|
2013-08-14 12:54:31 -06:00
|
|
|
|
} else {
|
|
|
|
|
// convert to interface with methods, via empty interface.
|
2014-08-07 07:00:02 -06:00
|
|
|
|
ef1.type = &f->ot->typ;
|
2013-08-14 12:54:31 -06:00
|
|
|
|
ef1.data = f->arg;
|
|
|
|
|
if(!runtime·ifaceE2I2((InterfaceType*)f->fint, ef1, (Iface*)frame))
|
|
|
|
|
runtime·throw("invalid type conversion in runfinq");
|
2013-07-29 09:43:08 -06:00
|
|
|
|
}
|
reflect, runtime: fix crash in GC due to reflect.call + precise GC
Given
type Outer struct {
*Inner
...
}
the compiler generates the implementation of (*Outer).M dispatching to
the embedded Inner. The implementation is logically:
func (p *Outer) M() {
(p.Inner).M()
}
but since the only change here is the replacement of one pointer
receiver with another, the actual generated code overwrites the
original receiver with the p.Inner pointer and then jumps to the M
method expecting the *Inner receiver.
During reflect.Value.Call, we create an argument frame and the
associated data structures to describe it to the garbage collector,
populate the frame, call reflect.call to run a function call using
that frame, and then copy the results back out of the frame. The
reflect.call function does a memmove of the frame structure onto the
stack (to set up the inputs), runs the call, and the memmoves the
stack back to the frame structure (to preserve the outputs).
Originally reflect.call did not distinguish inputs from outputs: both
memmoves were for the full stack frame. However, in the case where the
called function was one of these wrappers, the rewritten receiver is
almost certainly a different type than the original receiver. This is
not a problem on the stack, where we use the program counter to
determine the type information and understand that during (*Outer).M
the receiver is an *Outer while during (*Inner).M the receiver in the
same memory word is now an *Inner. But in the statically typed
argument frame created by reflect, the receiver is always an *Outer.
Copying the modified receiver pointer off the stack into the frame
will store an *Inner there, and then if a garbage collection happens
to scan that argument frame before it is discarded, it will scan the
*Inner memory as if it were an *Outer. If the two have different
memory layouts, the collection will intepret the memory incorrectly.
Fix by only copying back the results.
Fixes #7725.
LGTM=khr
R=khr
CC=dave, golang-codereviews
https://golang.org/cl/85180043
2014-04-08 09:11:35 -06:00
|
|
|
|
reflect·call(f->fn, frame, framesz, framesz);
|
2011-10-06 09:42:51 -06:00
|
|
|
|
f->fn = nil;
|
|
|
|
|
f->arg = nil;
|
2013-07-29 09:43:08 -06:00
|
|
|
|
f->ot = nil;
|
2011-10-06 09:42:51 -06:00
|
|
|
|
}
|
|
|
|
|
fb->cnt = 0;
|
2014-03-26 05:11:36 -06:00
|
|
|
|
runtime·lock(&finlock);
|
2011-10-06 09:42:51 -06:00
|
|
|
|
fb->next = finc;
|
|
|
|
|
finc = fb;
|
2014-03-26 05:11:36 -06:00
|
|
|
|
runtime·unlock(&finlock);
|
2010-03-26 15:15:30 -06:00
|
|
|
|
}
|
2014-03-07 09:27:01 -07:00
|
|
|
|
|
|
|
|
|
// Zero everything that's dead, to avoid memory leaks.
|
|
|
|
|
// See comment at top of function.
|
|
|
|
|
f = nil;
|
|
|
|
|
fb = nil;
|
|
|
|
|
next = nil;
|
|
|
|
|
i = 0;
|
|
|
|
|
ef = nil;
|
|
|
|
|
ef1.type = nil;
|
|
|
|
|
ef1.data = nil;
|
runtime: ,s/[a-zA-Z0-9_]+/runtime·&/g, almost
Prefix all external symbols in runtime by runtime·,
to avoid conflicts with possible symbols of the same
name in linked-in C libraries. The obvious conflicts
are printf, malloc, and free, but hide everything to
avoid future pain.
The symbols left alone are:
** known to cgo **
_cgo_free
_cgo_malloc
libcgo_thread_start
initcgo
ncgocall
** known to linker **
_rt0_$GOARCH
_rt0_$GOARCH_$GOOS
text
etext
data
end
pclntab
epclntab
symtab
esymtab
** known to C compiler **
_divv
_modv
_div64by32
etc (arch specific)
Tested on darwin/386, darwin/amd64, linux/386, linux/amd64.
Built (but not tested) for freebsd/386, freebsd/amd64, linux/arm, windows/386.
R=r, PeterGo
CC=golang-dev
https://golang.org/cl/2899041
2010-11-04 12:00:19 -06:00
|
|
|
|
runtime·gc(1); // trigger another gc to clean up the finalized objects, if possible
|
2010-03-26 15:15:30 -06:00
|
|
|
|
}
|
2009-01-26 18:37:05 -07:00
|
|
|
|
}
|
2011-02-02 21:03:47 -07:00
|
|
|
|
|
2014-03-26 05:11:36 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·createfing(void)
|
2014-03-14 13:32:12 -06:00
|
|
|
|
{
|
2014-07-30 10:01:52 -06:00
|
|
|
|
if(runtime·fing != nil)
|
2014-03-14 13:32:12 -06:00
|
|
|
|
return;
|
2014-03-26 05:11:36 -06:00
|
|
|
|
// Here we use gclock instead of finlock,
|
|
|
|
|
// because newproc1 can allocate, which can cause on-demand span sweep,
|
|
|
|
|
// which can queue finalizers, which would deadlock.
|
2014-03-14 13:32:12 -06:00
|
|
|
|
runtime·lock(&gclock);
|
2014-07-30 10:01:52 -06:00
|
|
|
|
if(runtime·fing == nil)
|
|
|
|
|
runtime·fing = runtime·newproc1(&runfinqv, nil, 0, 0, runtime·gc);
|
2014-03-14 13:32:12 -06:00
|
|
|
|
runtime·unlock(&gclock);
|
|
|
|
|
}
|
|
|
|
|
|
2014-07-30 10:01:52 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·createfingM(G *gp)
|
|
|
|
|
{
|
|
|
|
|
runtime·createfing();
|
|
|
|
|
runtime·gogo(&gp->sched);
|
|
|
|
|
}
|
|
|
|
|
|
2014-03-26 05:11:36 -06:00
|
|
|
|
G*
|
|
|
|
|
runtime·wakefing(void)
|
|
|
|
|
{
|
|
|
|
|
G *res;
|
|
|
|
|
|
|
|
|
|
res = nil;
|
|
|
|
|
runtime·lock(&finlock);
|
|
|
|
|
if(runtime·fingwait && runtime·fingwake) {
|
|
|
|
|
runtime·fingwait = false;
|
|
|
|
|
runtime·fingwake = false;
|
2014-07-30 10:01:52 -06:00
|
|
|
|
res = runtime·fing;
|
2014-03-26 05:11:36 -06:00
|
|
|
|
}
|
|
|
|
|
runtime·unlock(&finlock);
|
|
|
|
|
return res;
|
|
|
|
|
}
|
|
|
|
|
|
2014-08-07 03:34:30 -06:00
|
|
|
|
// Recursively unrolls GC program in prog.
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// mask is where to store the result.
|
|
|
|
|
// ppos is a pointer to position in mask, in bits.
|
|
|
|
|
// sparse says to generate 4-bits per word mask for heap (2-bits for data/bss otherwise).
|
|
|
|
|
static byte*
|
|
|
|
|
unrollgcprog1(byte *mask, byte *prog, uintptr *ppos, bool inplace, bool sparse)
|
2011-02-02 21:03:47 -07:00
|
|
|
|
{
|
2014-08-19 07:38:00 -06:00
|
|
|
|
uintptr pos, siz, i, off;
|
|
|
|
|
byte *arena_start, *prog1, v, *bitp, shift;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
arena_start = runtime·mheap.arena_start;
|
|
|
|
|
pos = *ppos;
|
|
|
|
|
for(;;) {
|
|
|
|
|
switch(prog[0]) {
|
|
|
|
|
case insData:
|
|
|
|
|
prog++;
|
|
|
|
|
siz = prog[0];
|
|
|
|
|
prog++;
|
|
|
|
|
for(i = 0; i < siz; i++) {
|
|
|
|
|
v = prog[i/PointersPerByte];
|
|
|
|
|
v >>= (i%PointersPerByte)*BitsPerPointer;
|
|
|
|
|
v &= BitsMask;
|
|
|
|
|
if(inplace) {
|
|
|
|
|
// Store directly into GC bitmap.
|
|
|
|
|
off = (uintptr*)(mask+pos) - (uintptr*)arena_start;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
bitp = arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = (off % wordsPerBitmapByte) * gcBits;
|
|
|
|
|
if(shift==0)
|
|
|
|
|
*bitp = 0;
|
|
|
|
|
*bitp |= v<<(shift+2);
|
2014-07-29 01:01:02 -06:00
|
|
|
|
pos += PtrSize;
|
|
|
|
|
} else if(sparse) {
|
|
|
|
|
// 4-bits per word
|
|
|
|
|
v <<= (pos%8)+2;
|
|
|
|
|
mask[pos/8] |= v;
|
|
|
|
|
pos += gcBits;
|
|
|
|
|
} else {
|
|
|
|
|
// 2-bits per word
|
|
|
|
|
v <<= pos%8;
|
|
|
|
|
mask[pos/8] |= v;
|
|
|
|
|
pos += BitsPerPointer;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
prog += ROUND(siz*BitsPerPointer, 8)/8;
|
|
|
|
|
break;
|
|
|
|
|
case insArray:
|
|
|
|
|
prog++;
|
|
|
|
|
siz = 0;
|
|
|
|
|
for(i = 0; i < PtrSize; i++)
|
|
|
|
|
siz = (siz<<8) + prog[PtrSize-i-1];
|
|
|
|
|
prog += PtrSize;
|
|
|
|
|
prog1 = nil;
|
|
|
|
|
for(i = 0; i < siz; i++)
|
|
|
|
|
prog1 = unrollgcprog1(mask, prog, &pos, inplace, sparse);
|
|
|
|
|
if(prog1[0] != insArrayEnd)
|
|
|
|
|
runtime·throw("unrollgcprog: array does not end with insArrayEnd");
|
|
|
|
|
prog = prog1+1;
|
|
|
|
|
break;
|
|
|
|
|
case insArrayEnd:
|
|
|
|
|
case insEnd:
|
|
|
|
|
*ppos = pos;
|
|
|
|
|
return prog;
|
|
|
|
|
default:
|
|
|
|
|
runtime·throw("unrollgcprog: unknown instruction");
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Unrolls GC program prog for data/bss, returns dense GC mask.
|
|
|
|
|
static byte*
|
|
|
|
|
unrollglobgcprog(byte *prog, uintptr size)
|
|
|
|
|
{
|
|
|
|
|
byte *mask;
|
|
|
|
|
uintptr pos, masksize;
|
|
|
|
|
|
|
|
|
|
masksize = ROUND(ROUND(size, PtrSize)/PtrSize*BitsPerPointer, 8)/8;
|
|
|
|
|
mask = runtime·persistentalloc(masksize+1, 0, &mstats.gc_sys);
|
|
|
|
|
mask[masksize] = 0xa1;
|
|
|
|
|
pos = 0;
|
|
|
|
|
prog = unrollgcprog1(mask, prog, &pos, false, false);
|
|
|
|
|
if(pos != size/PtrSize*BitsPerPointer) {
|
|
|
|
|
runtime·printf("unrollglobgcprog: bad program size, got %D, expect %D\n",
|
|
|
|
|
(uint64)pos, (uint64)size/PtrSize*BitsPerPointer);
|
|
|
|
|
runtime·throw("unrollglobgcprog: bad program size");
|
|
|
|
|
}
|
|
|
|
|
if(prog[0] != insEnd)
|
|
|
|
|
runtime·throw("unrollglobgcprog: program does not end with insEnd");
|
|
|
|
|
if(mask[masksize] != 0xa1)
|
|
|
|
|
runtime·throw("unrollglobgcprog: overflow");
|
|
|
|
|
return mask;
|
|
|
|
|
}
|
|
|
|
|
|
2014-08-07 03:34:30 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·unrollgcproginplace_m(void)
|
2014-07-29 01:01:02 -06:00
|
|
|
|
{
|
2014-08-19 07:38:00 -06:00
|
|
|
|
uintptr size, size0, pos, off;
|
|
|
|
|
byte *arena_start, *prog, *bitp, shift;
|
2014-08-07 03:34:30 -06:00
|
|
|
|
Type *typ;
|
|
|
|
|
void *v;
|
|
|
|
|
|
|
|
|
|
v = g->m->ptrarg[0];
|
|
|
|
|
typ = g->m->ptrarg[1];
|
|
|
|
|
size = g->m->scalararg[0];
|
|
|
|
|
size0 = g->m->scalararg[1];
|
|
|
|
|
g->m->ptrarg[0] = nil;
|
|
|
|
|
g->m->ptrarg[1] = nil;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
|
|
|
|
|
pos = 0;
|
|
|
|
|
prog = (byte*)typ->gc[1];
|
|
|
|
|
while(pos != size0)
|
|
|
|
|
unrollgcprog1(v, prog, &pos, true, true);
|
|
|
|
|
// Mark first word as bitAllocated.
|
|
|
|
|
arena_start = runtime·mheap.arena_start;
|
|
|
|
|
off = (uintptr*)v - (uintptr*)arena_start;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
bitp = arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = (off % wordsPerBitmapByte) * gcBits;
|
|
|
|
|
*bitp |= bitBoundary<<shift;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
// Mark word after last as BitsDead.
|
|
|
|
|
if(size0 < size) {
|
|
|
|
|
off = (uintptr*)((byte*)v + size0) - (uintptr*)arena_start;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
bitp = arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = (off % wordsPerBitmapByte) * gcBits;
|
|
|
|
|
*bitp &= ~(bitPtrMask<<shift) | ((uintptr)BitsDead<<(shift+2));
|
2014-07-29 01:01:02 -06:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Unrolls GC program in typ->gc[1] into typ->gc[0]
|
2014-08-07 03:34:30 -06:00
|
|
|
|
void
|
|
|
|
|
runtime·unrollgcprog_m(void)
|
2014-07-29 01:01:02 -06:00
|
|
|
|
{
|
|
|
|
|
static Lock lock;
|
2014-08-07 03:34:30 -06:00
|
|
|
|
Type *typ;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
byte *mask, *prog;
|
|
|
|
|
uintptr pos;
|
|
|
|
|
uint32 x;
|
|
|
|
|
|
2014-08-07 03:34:30 -06:00
|
|
|
|
typ = g->m->ptrarg[0];
|
|
|
|
|
g->m->ptrarg[0] = nil;
|
|
|
|
|
|
2014-07-29 01:01:02 -06:00
|
|
|
|
runtime·lock(&lock);
|
|
|
|
|
mask = (byte*)typ->gc[0];
|
|
|
|
|
if(mask[0] == 0) {
|
|
|
|
|
pos = 8; // skip the unroll flag
|
|
|
|
|
prog = (byte*)typ->gc[1];
|
|
|
|
|
prog = unrollgcprog1(mask, prog, &pos, false, true);
|
|
|
|
|
if(prog[0] != insEnd)
|
|
|
|
|
runtime·throw("unrollgcprog: program does not end with insEnd");
|
|
|
|
|
if(((typ->size/PtrSize)%2) != 0) {
|
|
|
|
|
// repeat the program twice
|
|
|
|
|
prog = (byte*)typ->gc[1];
|
|
|
|
|
unrollgcprog1(mask, prog, &pos, false, true);
|
|
|
|
|
}
|
|
|
|
|
// atomic way to say mask[0] = 1
|
|
|
|
|
x = ((uint32*)mask)[0];
|
|
|
|
|
runtime·atomicstore((uint32*)mask, x|1);
|
|
|
|
|
}
|
|
|
|
|
runtime·unlock(&lock);
|
2013-12-18 18:13:59 -07:00
|
|
|
|
}
|
|
|
|
|
|
2011-02-02 21:03:47 -07:00
|
|
|
|
// mark the span of memory at v as having n blocks of the given size.
|
|
|
|
|
// if leftover is true, there is left over space at the end of the span.
|
|
|
|
|
void
|
|
|
|
|
runtime·markspan(void *v, uintptr size, uintptr n, bool leftover)
|
|
|
|
|
{
|
2014-08-19 07:38:00 -06:00
|
|
|
|
uintptr i, off, step;
|
|
|
|
|
byte *b;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
|
2013-05-28 12:14:47 -06:00
|
|
|
|
if((byte*)v+size*n > (byte*)runtime·mheap.arena_used || (byte*)v < runtime·mheap.arena_start)
|
2011-02-02 21:03:47 -07:00
|
|
|
|
runtime·throw("markspan: bad pointer");
|
|
|
|
|
|
2014-08-19 07:38:00 -06:00
|
|
|
|
// Find bits of the beginning of the span.
|
|
|
|
|
off = (uintptr*)v - (uintptr*)runtime·mheap.arena_start; // word offset
|
|
|
|
|
b = runtime·mheap.arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
if((off%wordsPerBitmapByte) != 0)
|
|
|
|
|
runtime·throw("markspan: unaligned length");
|
|
|
|
|
|
|
|
|
|
// Okay to use non-atomic ops here, because we control
|
|
|
|
|
// the entire span, and each bitmap byte has bits for only
|
|
|
|
|
// one span, so no other goroutines are changing these bitmap words.
|
|
|
|
|
|
|
|
|
|
if(size == PtrSize) {
|
|
|
|
|
// Possible only on 64-bits (minimal size class is 8 bytes).
|
|
|
|
|
// Poor man's memset(0x11).
|
|
|
|
|
if(0x11 != ((bitBoundary+BitsDead)<<gcBits) + (bitBoundary+BitsDead))
|
|
|
|
|
runtime·throw("markspan: bad bits");
|
|
|
|
|
if((n%(wordsPerBitmapByte*PtrSize)) != 0)
|
|
|
|
|
runtime·throw("markspan: unaligned length");
|
|
|
|
|
b = b - n/wordsPerBitmapByte + 1; // find first byte
|
|
|
|
|
if(((uintptr)b%PtrSize) != 0)
|
|
|
|
|
runtime·throw("markspan: unaligned pointer");
|
|
|
|
|
for(i = 0; i != n; i += wordsPerBitmapByte*PtrSize, b += PtrSize)
|
|
|
|
|
*(uintptr*)b = (uintptr)0x1111111111111111ULL; // bitBoundary+BitsDead
|
|
|
|
|
return;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
}
|
2014-08-19 07:38:00 -06:00
|
|
|
|
|
|
|
|
|
if(leftover)
|
|
|
|
|
n++; // mark a boundary just past end of last block too
|
|
|
|
|
step = size/(PtrSize*wordsPerBitmapByte);
|
|
|
|
|
for(i = 0; i != n; i++, b -= step)
|
|
|
|
|
*b = bitBoundary|(BitsDead<<2);
|
2011-02-02 21:03:47 -07:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// unmark the span of memory at v of length n bytes.
|
|
|
|
|
void
|
|
|
|
|
runtime·unmarkspan(void *v, uintptr n)
|
|
|
|
|
{
|
2014-08-19 07:38:00 -06:00
|
|
|
|
uintptr off;
|
|
|
|
|
byte *b;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
|
2013-05-28 12:14:47 -06:00
|
|
|
|
if((byte*)v+n > (byte*)runtime·mheap.arena_used || (byte*)v < runtime·mheap.arena_start)
|
2011-02-02 21:03:47 -07:00
|
|
|
|
runtime·throw("markspan: bad pointer");
|
|
|
|
|
|
2014-08-19 07:38:00 -06:00
|
|
|
|
off = (uintptr*)v - (uintptr*)runtime·mheap.arena_start; // word offset
|
|
|
|
|
if((off % (PtrSize*wordsPerBitmapByte)) != 0)
|
2011-02-02 21:03:47 -07:00
|
|
|
|
runtime·throw("markspan: unaligned pointer");
|
2014-08-19 07:38:00 -06:00
|
|
|
|
b = runtime·mheap.arena_start - off/wordsPerBitmapByte - 1;
|
2011-02-02 21:03:47 -07:00
|
|
|
|
n /= PtrSize;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
if(n%(PtrSize*wordsPerBitmapByte) != 0)
|
2011-02-02 21:03:47 -07:00
|
|
|
|
runtime·throw("unmarkspan: unaligned length");
|
2011-02-16 11:21:20 -07:00
|
|
|
|
// Okay to use non-atomic ops here, because we control
|
|
|
|
|
// the entire span, and each bitmap word has bits for only
|
|
|
|
|
// one span, so no other goroutines are changing these
|
|
|
|
|
// bitmap words.
|
2014-08-19 07:38:00 -06:00
|
|
|
|
n /= wordsPerBitmapByte;
|
|
|
|
|
runtime·memclr(b - n + 1, n);
|
2011-02-02 21:03:47 -07:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
|
|
|
|
runtime·MHeap_MapBits(MHeap *h)
|
|
|
|
|
{
|
|
|
|
|
// Caller has added extra mappings to the arena.
|
|
|
|
|
// Add extra mappings of bitmap words as needed.
|
|
|
|
|
// We allocate extra bitmap pieces in chunks of bitmapChunk.
|
|
|
|
|
enum {
|
|
|
|
|
bitmapChunk = 8192
|
|
|
|
|
};
|
|
|
|
|
uintptr n;
|
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
R=dvyukov
CC=golang-dev
https://golang.org/cl/4641082
2011-09-30 07:40:01 -06:00
|
|
|
|
|
2014-08-19 07:38:00 -06:00
|
|
|
|
n = (h->arena_used - h->arena_start) / (PtrSize*wordsPerBitmapByte);
|
2013-05-28 12:04:34 -06:00
|
|
|
|
n = ROUND(n, bitmapChunk);
|
all: merge NaCl branch (part 1)
See golang.org/s/go13nacl for design overview.
This CL is the mostly mechanical changes from rsc's Go 1.2 based NaCl branch, specifically 39cb35750369 to 500771b477cf from https://code.google.com/r/rsc-go13nacl. This CL does not include working NaCl support, there are probably two or three more large merges to come.
CL 15750044 is not included as it involves more invasive changes to the linker which will need to be merged separately.
The exact change lists included are
15050047: syscall: support for Native Client
15360044: syscall: unzip implementation for Native Client
15370044: syscall: Native Client SRPC implementation
15400047: cmd/dist, cmd/go, go/build, test: support for Native Client
15410048: runtime: support for Native Client
15410049: syscall: file descriptor table for Native Client
15410050: syscall: in-memory file system for Native Client
15440048: all: update +build lines for Native Client port
15540045: cmd/6g, cmd/8g, cmd/gc: support for Native Client
15570045: os: support for Native Client
15680044: crypto/..., hash/crc32, reflect, sync/atomic: support for amd64p32
15690044: net: support for Native Client
15690048: runtime: support for fake time like on Go Playground
15690051: build: disable various tests on Native Client
LGTM=rsc
R=rsc
CC=golang-codereviews
https://golang.org/cl/68150047
2014-02-25 07:47:42 -07:00
|
|
|
|
n = ROUND(n, PhysPageSize);
|
2011-02-02 21:03:47 -07:00
|
|
|
|
if(h->bitmap_mapped >= n)
|
|
|
|
|
return;
|
|
|
|
|
|
2014-03-25 14:22:19 -06:00
|
|
|
|
runtime·SysMap(h->arena_start - n, n - h->bitmap_mapped, h->arena_reserved, &mstats.gc_sys);
|
2011-02-02 21:03:47 -07:00
|
|
|
|
h->bitmap_mapped = n;
|
|
|
|
|
}
|
2014-07-29 01:01:02 -06:00
|
|
|
|
|
|
|
|
|
static bool
|
|
|
|
|
getgcmaskcb(Stkframe *frame, void *ctxt)
|
|
|
|
|
{
|
|
|
|
|
Stkframe *frame0;
|
|
|
|
|
|
|
|
|
|
frame0 = ctxt;
|
|
|
|
|
if(frame0->sp >= (uintptr)frame->varp - frame->sp && frame0->sp < (uintptr)frame->varp) {
|
|
|
|
|
*frame0 = *frame;
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Returns GC type info for object p for testing.
|
|
|
|
|
void
|
|
|
|
|
runtime·getgcmask(byte *p, Type *t, byte **mask, uintptr *len)
|
|
|
|
|
{
|
|
|
|
|
Stkframe frame;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
uintptr i, n, off;
|
|
|
|
|
byte *base, bits, shift, *b;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
|
|
|
|
|
*mask = nil;
|
|
|
|
|
*len = 0;
|
|
|
|
|
|
|
|
|
|
// data
|
|
|
|
|
if(p >= data && p < edata) {
|
|
|
|
|
n = ((PtrType*)t)->elem->size;
|
|
|
|
|
*len = n/PtrSize;
|
|
|
|
|
*mask = runtime·mallocgc(*len, nil, 0);
|
|
|
|
|
for(i = 0; i < n; i += PtrSize) {
|
|
|
|
|
off = (p+i-data)/PtrSize;
|
2014-08-18 06:42:24 -06:00
|
|
|
|
bits = (runtime·gcdatamask[off/PointersPerByte] >> ((off%PointersPerByte)*BitsPerPointer))&BitsMask;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
(*mask)[i/PtrSize] = bits;
|
|
|
|
|
}
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
// bss
|
|
|
|
|
if(p >= bss && p < ebss) {
|
|
|
|
|
n = ((PtrType*)t)->elem->size;
|
|
|
|
|
*len = n/PtrSize;
|
|
|
|
|
*mask = runtime·mallocgc(*len, nil, 0);
|
|
|
|
|
for(i = 0; i < n; i += PtrSize) {
|
|
|
|
|
off = (p+i-bss)/PtrSize;
|
2014-08-18 06:42:24 -06:00
|
|
|
|
bits = (runtime·gcbssmask[off/PointersPerByte] >> ((off%PointersPerByte)*BitsPerPointer))&BitsMask;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
(*mask)[i/PtrSize] = bits;
|
|
|
|
|
}
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
// heap
|
|
|
|
|
if(runtime·mlookup(p, &base, &n, nil)) {
|
|
|
|
|
*len = n/PtrSize;
|
|
|
|
|
*mask = runtime·mallocgc(*len, nil, 0);
|
|
|
|
|
for(i = 0; i < n; i += PtrSize) {
|
|
|
|
|
off = (uintptr*)(base+i) - (uintptr*)runtime·mheap.arena_start;
|
2014-08-19 07:38:00 -06:00
|
|
|
|
b = runtime·mheap.arena_start - off/wordsPerBitmapByte - 1;
|
|
|
|
|
shift = (off % wordsPerBitmapByte) * gcBits;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
bits = (*b >> (shift+2))&BitsMask;
|
|
|
|
|
(*mask)[i/PtrSize] = bits;
|
|
|
|
|
}
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
// stack
|
|
|
|
|
frame.fn = nil;
|
|
|
|
|
frame.sp = (uintptr)p;
|
|
|
|
|
runtime·gentraceback((uintptr)runtime·getcallerpc(&p), (uintptr)runtime·getcallersp(&p), 0, g, 0, nil, 1000, getgcmaskcb, &frame, false);
|
|
|
|
|
if(frame.fn != nil) {
|
|
|
|
|
Func *f;
|
|
|
|
|
StackMap *stackmap;
|
|
|
|
|
BitVector bv;
|
|
|
|
|
uintptr size;
|
|
|
|
|
uintptr targetpc;
|
|
|
|
|
int32 pcdata;
|
|
|
|
|
|
|
|
|
|
f = frame.fn;
|
|
|
|
|
targetpc = frame.continpc;
|
|
|
|
|
if(targetpc == 0)
|
|
|
|
|
return;
|
|
|
|
|
if(targetpc != f->entry)
|
|
|
|
|
targetpc--;
|
|
|
|
|
pcdata = runtime·pcdatavalue(f, PCDATA_StackMapIndex, targetpc);
|
|
|
|
|
if(pcdata == -1)
|
|
|
|
|
return;
|
|
|
|
|
stackmap = runtime·funcdata(f, FUNCDATA_LocalsPointerMaps);
|
|
|
|
|
if(stackmap == nil || stackmap->n <= 0)
|
|
|
|
|
return;
|
|
|
|
|
bv = runtime·stackmapdata(stackmap, pcdata);
|
|
|
|
|
size = bv.n/BitsPerPointer*PtrSize;
|
|
|
|
|
n = ((PtrType*)t)->elem->size;
|
|
|
|
|
*len = n/PtrSize;
|
|
|
|
|
*mask = runtime·mallocgc(*len, nil, 0);
|
|
|
|
|
for(i = 0; i < n; i += PtrSize) {
|
|
|
|
|
off = (p+i-frame.varp+size)/PtrSize;
|
2014-08-15 12:36:12 -06:00
|
|
|
|
bits = (bv.data[off*BitsPerPointer/32] >> ((off*BitsPerPointer)%32))&BitsMask;
|
2014-07-29 01:01:02 -06:00
|
|
|
|
(*mask)[i/PtrSize] = bits;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
2014-08-25 10:25:22 -06:00
|
|
|
|
|
|
|
|
|
void runtime·gc_unixnanotime(int64 *now);
|
|
|
|
|
|
|
|
|
|
int64 runtime·unixnanotime(void)
|
|
|
|
|
{
|
|
|
|
|
int64 now;
|
|
|
|
|
|
|
|
|
|
runtime·gc_unixnanotime(&now);
|
|
|
|
|
return now;
|
|
|
|
|
}
|