runtime: force mutator to give work buffer to GC

The scheduler, work buffer's dispose, and write barriers
can conspire to hide the a pointer from the GC's concurent
mark phase. If this pointer is the only path to a large
amount of marking the STW mark termination phase may take
a lot of time.

Consider the following:
1) dispose places a work buffer on the partial queue
2) the GC is busy so it does not immediately remove and
   process the work buffer
3) the scheduler runs a mutator whose write barrier dequeues the
   work buffer from the partial queue so the GC won't see it
This repeats until the GC reaches the mark termination
phase where the GC finally discovers the pointer along
with a lot of work to do.

This CL fixes the problem by having the mutator
dispose of the buffer to the full queue instead of
the partial queue. Since the write buffer never asks for full
buffers the conspiracy described above is not possible.

Updates #11694.

Change-Id: I2ce832f9657a7570f800e8ce4459cd9e304ef43b
Reviewed-on: https://go-review.googlesource.com/12840
Reviewed-by: Austin Clements <austin@google.com>

src/runtime/mgc.go

@@ -754,8 +754,9 @@ func (s *bgMarkSignal) clear() {
 }
 
 var work struct {
-	full    uint64                // lock-free list of full blocks workbuf
-	empty   uint64                // lock-free list of empty blocks workbuf
+	full  uint64 // lock-free list of full blocks workbuf
+	empty uint64 // lock-free list of empty blocks workbuf
+	// TODO(rlh): partial no longer used, remove. (issue #11922)
 	partial uint64                // lock-free list of partially filled blocks workbuf
 	pad0    [_CacheLineSize]uint8 // prevents false-sharing between full/empty and nproc/nwait
 	nproc   uint32

src/runtime/mgcwork.go

@@ -153,10 +153,18 @@ func (ww *gcWork) get() uintptr {
 }
 
 // dispose returns any cached pointers to the global queue.
+// The buffers are being put on the full queue so that the
+// write barriers will not simply reacquire them before the
+// GC can inspect them. This helps reduce the mutator's
+// ability to hide pointers during the concurrent mark phase.
+//
 //go:nowritebarrier
 func (w *gcWork) dispose() {
 	if wbuf := w.wbuf; wbuf != 0 {
-		putpartial(wbuf.ptr(), 167)
+		if wbuf.ptr().nobj == 0 {
+			throw("dispose: workbuf is empty")
+		}
+		putfull(wbuf.ptr(), 166)
 		w.wbuf = 0
 	}
 	if w.bytesMarked != 0 {