runtime: fix over-aggressive proportional sweep

Currently, sweeping is performed before allocating a span by charging
for the entire size of the span requested, rather than the number of
bytes actually available for allocation from the returned span. That
is, if the returned span is 8K, but already has 6K in use, the mutator
is charged for 8K of heap allocation even though it can only allocate
2K more from the span. As a result, proportional sweep is
over-aggressive and tends to finish much earlier than it needs to.
This effect is more amplified by fragmented heaps.

Fix this by reimbursing the mutator for the used space in a span once
it has allocated that span. We still have to charge up-front for the
worst-case because we don't know which span the mutator will get, but
at least we can correct the over-charge once it has a span, which will
go toward later span allocations.

This has negligible effect on the throughput of the go1 benchmarks and
the garbage benchmark.

Fixes #12040.

Change-Id: I0e23e7a4ccf126cca000fed5067b20017028dd6b
Reviewed-on: https://go-review.googlesource.com/16515
Reviewed-by: Rick Hudson <rlh@golang.org>

src/runtime/mcentral.go

@@ -101,6 +101,10 @@ havespan:
 	if n == 0 {
 		throw("empty span")
 	}
+	usedBytes := uintptr(s.ref) * s.elemsize
+	if usedBytes > 0 {
+		reimburseSweepCredit(usedBytes)
+	}
 	if s.freelist.ptr() == nil {
 		throw("freelist empty")
 	}

src/runtime/mgcsweep.go

@@ -354,6 +354,11 @@ func mSpan_Sweep(s *mspan, preserve bool) bool {
 // also sweep pages (e.g., for a large allocation), it can pass a
 // non-zero callerSweepPages to leave that many pages unswept.
 //
+// deductSweepCredit makes a worst-case assumption that all spanBytes
+// bytes of the ultimately allocated span will be available for object
+// allocation. The caller should call reimburseSweepCredit if that
+// turns out not to be the case once the span is allocated.
+//
 // deductSweepCredit is the core of the "proportional sweep" system.
 // It uses statistics gathered by the garbage collector to perform
 // enough sweeping so that all pages are swept during the concurrent
@@ -379,6 +384,17 @@ func deductSweepCredit(spanBytes uintptr, callerSweepPages uintptr) {
 	}
 }
 
+// reimburseSweepCredit records that unusableBytes bytes of a
+// just-allocated span are not available for object allocation. This
+// offsets the worst-case charge performed by deductSweepCredit.
+func reimburseSweepCredit(unusableBytes uintptr) {
+	if mheap_.sweepPagesPerByte == 0 {
+		// Nobody cares about the credit. Avoid the atomic.
+		return
+	}
+	xadd64(&mheap_.spanBytesAlloc, -int64(unusableBytes))
+}
+
 func dumpFreeList(s *mspan) {
 	printlock()
 	print("runtime: free list of span ", s, ":\n")