qbit/go
1
0
Fork 0
mirror of https://github.com/golang/go synced 2024-11-11 18:21:40 -07:00
Code Releases Activity

cmd/internal/obj: make aix/ppc64 builds reproducible

Browse Source 
sort.Slice was being used to sort some newly added entries by name
to make the ctxt.Data slice reproducible, but some existing entries
have the same name, and the effect was to take the non-determinism
of the tail entries and scatter it into the earlier, deterministic section
when multiple entries had the same name.

The specific entries with the same name are type SDWARFVAR, which
all have an empty name but different relocations. If they are shuffled,
then the relocation symbols are visited in a different order, which
enters them into the string table in a different order, which results in
different object files, different object file hashes, and different build IDs
for the final executables.

Use sort.SliceStable to avoid reordering entries we don't mean to reorder.

Also add a simple test for scheduling-related non-determinism.
I debugged this originally using 'go install -race cmd/compile',
but that was slow and turned out not to be terribly reliable.
Using a few different GOMAXPROCS settings turns out to be a much more
effective (and faster) way to scramble scheduling decisions.

Fixes #60759.

Change-Id: Ia966b02b9fdaefa971f998a09319ca375bdf8604
Reviewed-on: https://go-review.googlesource.com/c/go/+/502755
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Auto-Submit: Russ Cox <rsc@golang.org>
TryBot-Bypass: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
This commit is contained in:
Russ Cox 2023-06-12 22:56:23 -04:00 committed by Gopher Robot
parent 8b5ed3cdae
commit ccf75b36ff
2 changed files with 31 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
src/cmd/go/testdata/script/repro_build.txt vendored Normal file
View File

@ -0,0 +1,22 @@
# Check that goroutine scheduling does not affect compiler output.
# If it does, reproducible builds will not work very well.
[short] skip
[GOOS:aix] env CGO_ENABLED=0 # go.dev/issue/56896
env GOMAXPROCS=16
go build -a -o http16.o net/http
env GOMAXPROCS=17
go build -a -o http17.o net/http
cmp -q http16.o http17.o
env GOMAXPROCS=18
go build -a -o http18.o net/http
cmp -q http16.o http18.o
# Check that goroutine scheduling does not affect linker output.
env GOMAXPROCS=16
go build -a -o gofmt16.exe cmd/gofmt
env GOMAXPROCS=17
go build -a -o gofmt17.exe cmd/gofmt
cmp -q gofmt16.exe gofmt17.exe
env GOMAXPROCS=18
go build -a -o gofmt18.exe cmd/gofmt
cmp -q gofmt16.exe gofmt18.exe

14
src/cmd/internal/obj/sym.go
View File

@ -189,11 +189,15 @@ func (ctxt *Link) GCLocalsSym(data []byte) *LSym {
// in which case all the symbols are non-package (for now).
func (ctxt *Link) NumberSyms() {
if ctxt.Headtype == objabi.Haix {
// Data must be sorted to keep a constant order in TOC symbols.
// As they are created during Progedit, two symbols can be switched between
// two different compilations. Therefore, BuildID will be different.
// TODO: find a better place and optimize to only sort TOC symbols
sort.Slice(ctxt.Data, func(i, j int) bool {
// Data must be in a reliable order for reproducible builds.
// The original entries are in a reliable order, but the TOC symbols
// that are added in Progedit are added by different goroutines
// that can be scheduled independently. We need to reorder those
// symbols reliably. Sort by name but use a stable sort, so that
// any original entries with the same name (all DWARFVAR symbols
// have empty names but different relocation sets) are not shuffled.
// TODO: Find a better place and optimize to only sort TOC symbols.
sort.SliceStable(ctxt.Data, func(i, j int) bool {
return ctxt.Data[i].Name < ctxt.Data[j].Name
})
}