1
0
mirror of https://github.com/golang/go synced 2024-11-17 13:44:43 -07:00
Commit Graph

44410 Commits

Author SHA1 Message Date
Alexander Rakoczy
ba9e108899 cmd: update golang.org/x/xerrors
This pulls in CL 247217.

Fixes #40573

Change-Id: I89eeebb5da9a4668adc6b5c5155651e5da421d59
Reviewed-on: https://go-review.googlesource.com/c/go/+/247186
Run-TryBot: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-08-06 19:17:13 +00:00
Katie Hockman
027d7241ce encoding/binary: read at most MaxVarintLen64 bytes in ReadUvarint
This CL ensures that ReadUvarint consumes only a limited
amount of input (instead of an unbounded amount).

On some inputs, ReadUvarint could read an arbitrary number
of bytes before deciding to return an overflow error.
After this CL, ReadUvarint returns that same overflow
error sooner, after reading at most MaxVarintLen64 bytes.

Fix authored by Robert Griesemer and Filippo Valsorda.

Thanks to Diederik Loerakker, Jonny Rhea, Raúl Kripalani,
and Preston Van Loon for reporting this.

Fixes #40618
Fixes CVE-2020-16845

Change-Id: Ie0cb15972f14c38b7cf7af84c45c4ce54909bb8f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/812099
Reviewed-by: Filippo Valsorda <valsorda@google.com>
Reviewed-on: https://go-review.googlesource.com/c/go/+/247120
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-08-06 17:24:10 +00:00
Michael Matloob
6f08e89ec3 cmd/go: fix error stacks when there are scanner errors
After golang.org/cl/228784 setLoadPackageDataError tries to decide whether an
error is caused by an imported package or an importing package by examining the
error itself to decide. Ideally, the errors themselves would belong to a
specific interface or some other property to make it unambiguous that they
were import errors. Since they don't, setLoadPackageDataError just checked
for nogoerrors and classified all other errors as import errors. But
it missed scanner errors which are also "caused" by the imported
package.

Fixes #40544

Change-Id: I39159bfdc286bee73697decd07b8aa9451f2db06
Reviewed-on: https://go-review.googlesource.com/c/go/+/246717
Run-TryBot: Michael Matloob <matloob@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2020-08-05 18:24:52 +00:00
Damien Neil
f235275097 net/http: fix cancelation of requests with a readTrackingBody wrapper
Use the original *Request in the reqCanceler map, not the transient
wrapper created to handle body rewinding.

Change the key of reqCanceler to a struct{*Request}, to make it more
difficult to accidentally use the wrong request as the key.

Fixes #40453.

Change-Id: I4e61ee9ff2c794fb4c920a3a66c9a0458693d757
Reviewed-on: https://go-review.googlesource.com/c/go/+/245357
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
2020-08-04 19:27:13 +00:00
Cherry Zhang
b85033d153 [dev.link] cmd/link: fix pclntab symbol handling on AIX
On AIX, container symbols are handled in a weird way (unlike
other platforms): the outer symbol needs to have size (but still
no data), and the inner symbols must not be in the symbol table
(otherwise it overlaps with the outer symbol, which the system
linker doesn't like).

As of CL 241598, pclntab becomes a container symbol. We need to
follow the rule above for AIX.

Fix AIX build.

Change-Id: Ie2515a4cabbd8cf3f6d3868643a28f64ca3365a2
Reviewed-on: https://go-review.googlesource.com/c/go/+/246479
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-08-03 23:45:46 +00:00
Cherry Zhang
5c9b540378 [dev.link] cmd/internal/goobj: delete old object file reader
Change-Id: Ieebab205e2cea2b4665c830b7424d543812787ff
Reviewed-on: https://go-review.googlesource.com/c/go/+/246441
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-08-03 21:13:48 +00:00
Cherry Zhang
db924fd99e [dev.link] cmd/compile, cmd/link: generate itablink at link time
Currently, at compile time, for each itab symbol, we create an
"itablink" symbol which holds solely the address of the itab
symbol. At link time, all the itablink symbols are grouped
together to form the itablinks slice.

This CL removes the itablink symbols, and directly generate the
itablinks slice in the linker. This removes a number of symbols,
which are dupOK and generally have long names. And also removes
a special handling of itablink symbols in the deadcode pass which
iterates through all symbols.

Change-Id: I475c3c8899e9fbeec9abc7647b1e4a69aa5c3c5a
Reviewed-on: https://go-review.googlesource.com/c/go/+/245901
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-08-03 21:13:25 +00:00
Cherry Zhang
f204ca3051 [dev.link] cmd/link: drop hash maps after loading
The hash maps are used to deduplicate hashed symbols. Once we
loaded all the symbols, we no longer need the hash maps. Drop
them.

Linking cmd/compile,

name         old live-B     new live-B     delta
Loadlib_GC      13.1M ± 0%     11.3M ± 0%   -13.62%  (p=0.008 n=5+5)

Change-Id: I4bb1f84e1111a56d9e777cd6a68f7d974b60e321
Reviewed-on: https://go-review.googlesource.com/c/go/+/245721
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-08-03 21:12:56 +00:00
Cherry Zhang
cecb7a1cf3 [dev.link] cmd/compile, cmd/link: make itab symbols content-addressable
Extend the content-addressable symbol mechanism to itab symbols.
Itab symbols require global uniqueness (as at run time we compare
pointers), so it needs to be reliably deduplicated. Currently the
content hash depends on symbol name expansion, so we can only do
this when all Go packages are built with know package paths. Fall
back to checking names if any Go package is built with unknown
package path.

Change-Id: Icf5e8873755050c20e5fc6549f6de1c883254c89
Reviewed-on: https://go-review.googlesource.com/c/go/+/245719
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-08-03 21:12:28 +00:00
Cherry Zhang
69748f0ce4 [dev.link] cmd/compile, cmd/link: reference type symbol of defined type by index
The type descriptor symbol of a defined (named) type (and pointer
to it) is defined only in the package that defines the type. It
is not dupOK, unlike other type descriptors. So it can be
referenced by index. Currently it is referenced by name for
cross-package references, because the index is not exported and
so not known to the referencing package.

This CL passes the index through the export data, so the symbol
can be referenced by index, and does not need to be looked up by
name. This also makes such symbol references consistent: it is
referenced by index within the defining package and also cross-
package, which makes it easier for content hashing (in later CLs).

One complication is that we need to set flags on referenced
symbols (specifically, the UsedInIface flag). Before, they are
non-package refs, which naturally carry flags in the object file.
For indexed refs, we currently don't put their flags in the
object file. Introduce a new block for this.

Change-Id: I8126f8e318ac4e6609eb2ac136201fd6c264c256
Reviewed-on: https://go-review.googlesource.com/c/go/+/245718
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-08-03 21:11:33 +00:00
Jeremy Faller
e4e1c6a7af [dev.link] add compilation unit index to func
Not used yet, but add the compilation unit for a function to func.

Change-Id: I7c43fa9f1da044ca63bab030062519771b9f4418
Reviewed-on: https://go-review.googlesource.com/c/go/+/244547
Run-TryBot: Jeremy Faller <jeremy@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-08-03 17:58:03 +00:00
Jeremy Faller
89cf569a45 [dev.link] move FuncID creation into the compiler/assembler
Leaving creation of the funcID till the linker requires the linker to
load the function and file names into memory. Moving these into the
compiler/assembler prevents this.

This work is a step towards moving all func metadata into the compiler.

Change-Id: Iebffdc5a909adbd03ac263fde3f4c3d492fb1eac
Reviewed-on: https://go-review.googlesource.com/c/go/+/244024
Run-TryBot: Jeremy Faller <jeremy@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Austin Clements <austin@google.com>
2020-08-03 17:56:50 +00:00
Keith Randall
f92337422e runtime/race: fix ppc64le build
The .syso test also fails for ppc64le. Not sure why. For now, just
disable the test for that architecture. The test really only needs to
run on a single builder of any arch.

Change-Id: I346cdc01ada09d43c4c504fbc30be806f59d5422
Reviewed-on: https://go-review.googlesource.com/c/go/+/246358
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-08-02 00:04:25 +00:00
Keith Randall
e49b2308a5 runtime/race: rebuild some .syso files to remove getauxval dependency
We can't depend on getauxval because it only exists in glibc >= 2.16.
Tsan has been updated to avoid that dependency
(https://reviews.llvm.org/D84859). This CL rebuilds the affected
.syso files, and adds a test to make sure we don't regress.

Fixes #37485

Change-Id: I891f54d28ec0d7da50a8df1adadc76dd6e7ab3e0
Reviewed-on: https://go-review.googlesource.com/c/go/+/246258
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
2020-08-01 06:32:57 +00:00
Toshihiro Shiino
10523c0efb doc/go1.15: fix a few trivial inconsistencies
For #37419

Change-Id: I8ede539df5d5344aeb44ba1a7e2383363d92157f
Reviewed-on: https://go-review.googlesource.com/c/go/+/245977
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-08-01 01:11:19 +00:00
Cherry Zhang
847b9be3f6 [dev.link] cmd/link: make symbol attribute setting more reliable
For dupOK symbols, their attributes should be OR'd. Most of the
attributes are expected to be set consistently across multiple
definitions, but UsedInIface must be OR'd, and for alignment we
need to pick the largest one. Currently the attributes are not
always OR'd, depending on addSym returning true or false. This
doesn't cause any real problem, but it would be a problem if we
make type descriptor symbols content-addressable.

This CL removes the second result of addSym, and lets preloadSyms
always set the attributes. Also removes the alignment handling on
addSym, handles it in preloadSyms only.

Change-Id: I06b3f0adb733f6681956ea9ef54736baa86ae7bc
Reviewed-on: https://go-review.googlesource.com/c/go/+/245720
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-07-31 21:02:26 +00:00
Cherry Zhang
c4ee16eda9 [dev.link] all: merge branch 'master' into dev.link
Clean merge.

Change-Id: If78d97b9ac69511e4de7aa9532257d4fabebdcbc
2020-07-31 13:21:24 -04:00
Matthew Dempsky
7388956b76 cmd/cgo: fix mangling of enum and union types
Consider this test package:

    package p

    // enum E { E0 };
    // union U { long x; };
    // void f(enum E e, union U* up) {}
    import "C"

    func f() {
    	C.f(C.enum_E(C.E0), (*C.union_U)(nil))
    }

In Go 1.14, cgo translated this to (omitting irrelevant details):

    type _Ctype_union_U [8]byte

    func f() {
    	_Cfunc_f(uint32(_Ciconst_E0), (*[8]byte)(nil))
    }

    func _Cfunc_f(p0 uint32, p1 *[8]byte) (r1 _Ctype_void) { ... }

Notably, _Ctype_union_U was declared as a defined type, but uses were
being rewritten into uses of the underlying type, which matched how
_Cfunc_f was declared.

After CL 230037, cgo started consistently rewriting "C.foo" type
expressions as "_Ctype_foo", which caused it to start emitting:

    type _Ctype_enum_E uint32
    type _Ctype_union_U [8]byte

    func f() {
    	_Cfunc_f(_Ctype_enum_E(_Ciconst_E0), (*_Ctype_union_U)(nil))
    }

    // _Cfunc_f unchanged

Of course, this fails to type-check because _Ctype_enum_E and
_Ctype_union_U are defined types.

This CL changes cgo to emit:

    type _Ctype_enum_E = uint32
    type _Ctype_union_U = [8]byte

    // f unchanged since CL 230037
    // _Cfunc_f still unchanged

It would probably be better to fix this in (*typeConv).loadType so
that cgo generated code uses the _Ctype_foo aliases too. But as it
wouldn't have any effect on actual compilation, it's not worth the
risk of touching it at this point in the release cycle.

Updates #39537.
Fixes #40494.

Change-Id: I88269660b40aeda80a9a9433777601a781b48ac0
Reviewed-on: https://go-review.googlesource.com/c/go/+/246057
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-07-31 16:35:33 +00:00
Michael Anthony Knyszek
b56791cdea runtime: validate candidate searchAddr in pageAlloc.find
Currently pageAlloc.find attempts to find a better estimate for the
first free page in the heap, even if the space its looking for isn't
necessarily going to be the first free page in the heap (e.g. if npages
>= 2). However, in doing so it has the potential to return a searchAddr
candidate that doesn't actually correspond to mapped memory, but this
candidate might still be adopted. As a result, pageAlloc.alloc's fast
path may look at unmapped summary memory and segfault. This case is rare
on most operating systems since the heap is kept fairly contiguous, so
the chance that the candidate searchAddr discovered is unmapped is
fairly low. Even so, this is totally possible and outside the user's
control when it happens (in fact, it's likely to happen consistently for
a given user on a given system).

Fix this problem by ensuring that our candidate always points to mapped
memory. We do this by looking at mheap's arenas structure first. If it
turns out our candidate doesn't correspond to mapped memory, then we
look at inUse to round up the searchAddr to the next mapped address.

While we're here, clean up some documentation related to searchAddr.

Fixes #40191.

Change-Id: I759efec78987e4a8fde466ae45aabbaa3d9d4214
Reviewed-on: https://go-review.googlesource.com/c/go/+/242680
Run-TryBot: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-07-31 14:35:39 +00:00
Jeremy Faller
128f4e51f2 [dev.link] ensure package path is set when TEXT symbols are created
We're reworking pclntab generation in the linker, and with that we're
moving FuncID generation in to the compiler. Determining the FuncID is
done by a lookup on the package.function name; therefore, we need the
package whenever we make the TEXT symbols.

Change-Id: I805445ffbf2f895f06ce3a91fb09126d012bf86e
Reviewed-on: https://go-review.googlesource.com/c/go/+/245318
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-07-31 13:55:19 +00:00
Jeremy Faller
6ac9914383 [dev.link] create runtime.funcnametab
Move the function names out of runtime.pclntab_old, creating
runtime.funcnametab.  There is an unfortunate artifact in this change in
that calculating the funcID still requires loading the name. Future work
will likely pull this out and put it into the object file Funcs.

ls -l cmd/compile (darwin):
  before: 18524016
  after:  18519952

The difference in size can be attributed to alignment in pclntab_old.

Change-Id: Ibcbb230d4632178f8fcd0667165f5335786381f8
Reviewed-on: https://go-review.googlesource.com/c/go/+/243223
Reviewed-by: Austin Clements <austin@google.com>
2020-07-31 13:55:07 +00:00
Jeremy Faller
3067a8dc02 [dev.link] cmd/link: use pclntabState and eliminate globals
Non functional change.

As runtime.pclntab breaks up, it'll be easier if we can just pass around
the pclntab state. Also, eliminate the globals in pclntab.

Change-Id: I2a5849e8f5f422a336a881e53a261e3997d11c44
Reviewed-on: https://go-review.googlesource.com/c/go/+/242599
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-07-31 13:54:51 +00:00
Cherry Zhang
365059e1d1 [dev.link] cmd/link: add back SUNDEFEXT case
The SUNDEFEXT case was lost during the refactoring. Add it back.

Fix ppc64le build.

Change-Id: I14594ee2c3e0a794c93839247fb3e6206c2e657a
Reviewed-on: https://go-review.googlesource.com/c/go/+/245919
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-07-30 20:03:52 +00:00
Jeremy Faller
ba9c639470 [dev.link] cmd/link: add runtime.pcheader
As of July 2020, a fair amount of the new linker's live memory, and
runtime is spent generating pclntab. In an effort to streamline that
code, this change starts breaking up the generation of runtime.pclntab
into smaller chunks that can run later in a link. These changes are
described in an (as yet not widely distributed) document that lays out
an improved format. Largely the work consists of breaking up
runtime.pclntab into smaller pieces, stopping much of the data
rewriting, and getting runtime.pclntab into a form where we can reason
about its size and look to shrink it. This change is the first part of
that work -- just pulling out the header, and demonstrating where a
majority of that work will be.

Change-Id: I65618d0d0c780f7e5977c9df4abdbd1696fedfcb
Reviewed-on: https://go-review.googlesource.com/c/go/+/241598
Run-TryBot: Jeremy Faller <jeremy@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Austin Clements <austin@google.com>
2020-07-30 19:36:06 +00:00
Katie Hockman
10374e2435 testing: fix quotation marks
Change-Id: I4b816e26718ef5521afba2b200a6333373b09c58
Reviewed-on: https://go-review.googlesource.com/c/go/+/245136
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-07-30 19:08:42 +00:00
Cherry Zhang
b3e3c339ff [dev.link] cmd/internal/obj: trim trailing zeros for content hashing
The symbol's data in the object file (sym.P) may already not
contain trailing zeros (e,g, for [10]int{1}), but sometimes it
does (e.g. for [10]int{1,0}). The linker can already handle this
case. We just always trim the trailing zeros for content hashing,
so it can deduplicate [10]int{1} and [10]int{1,0}.

Note: in theory we could just trim the zeros in the symbol data
as well. But currently the linker depends on reading symbol data
for certain symbols (e.g. type symbol decoding), and trimming
will complicates things in the linker.

Change-Id: I9e90e41e6ac808b36855b0713a85e61c33bf093a
Reviewed-on: https://go-review.googlesource.com/c/go/+/245717
Run-TryBot: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-07-30 18:37:08 +00:00
Cholerae Hu
7f86080476 cmd/compile: don't addLocalInductiveFacts if there is no direct edge from if block to phi block
Currently in addLocalInductiveFacts, we only check whether
direct edge from if block to phi block exists. If not, the
following logic will treat the phi block as the first successor,
which is wrong.

This patch makes prove pass more conservative, so we disable
some cases in test/prove.go. We will do some optimization in
the following CL and enable these cases then.

Fixes #40367.

Change-Id: I27cf0248f3a82312a6f7dabe11c79a1a34cf5412
Reviewed-on: https://go-review.googlesource.com/c/go/+/244579
Reviewed-by: Zach Jones <zachj1@gmail.com>
Reviewed-by: Keith Randall <khr@golang.org>
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-07-30 17:23:11 +00:00
Cherry Zhang
4a92371291 [dev.link] cmd/link: remove "2", another round
Rename Reloc2 to Reloc, At2 to At, Aux2 to Aux.

Change-Id: Ic98d83c080e8cd80fbe1837c8f0aa134033508ce
Reviewed-on: https://go-review.googlesource.com/c/go/+/245578
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-07-30 16:37:23 +00:00
Cherry Zhang
80b287fd28 [dev.link] cmd/link: remove loader.Reloc
We have Reloc and Reloc2. Reloc2 is the better approach and most
code uses Reloc2. There are still uses of Reloc. This CL migrates
them to Reloc2, and removes Reloc.

Change-Id: Id5f6a6019e1e044add682d05e70ebb1548ec58d9
Reviewed-on: https://go-review.googlesource.com/c/go/+/245577
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-07-30 16:37:04 +00:00
Cherry Zhang
317c1ca9f2 [dev.link] cmd/link: refactor ExtReloc data structures
We used to generate all external relocations in memory, then emit
the relocation records at a later pass. The data structures were
chosen so that it takes as little memory as possible. Now we just
stream out external relocations, and ExtReloc is just a local
variable. Change the data structure to avoid repeated read of
some fields. Also get rid of ExtRelocView, as it is no longer
necessary.

Change-Id: I40209bbe4387af231b29788125c3b4ebb0ff4a33
Reviewed-on: https://go-review.googlesource.com/c/go/+/245479
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-07-30 16:36:25 +00:00
Cherry Zhang
880f43c87f [dev.link] cmd/link: move arch-specific extreloc to common code
Change-Id: Ia833818862b277ac83266919f39e5c25faac895e
Reviewed-on: https://go-review.googlesource.com/c/go/+/245478
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-07-30 16:36:09 +00:00
Michael Munday
54e75e8f9d crypto/ed25519: remove s390x KDSA implementation
This reverts CL 202578 and CL 230677 which added an optimization
to use KDSA when available on s390x.

Inconsistencies have been found between the two implementations
in their handling of certain edge cases. Since the Go 1.15 release
is extremely soon it seems prudent to remove this optimization
for now and revisit it in a future release.

Fixes #40475.

Change-Id: Ifb2ed9b9e573784df57383671f1c29d8abae90d4
Reviewed-on: https://go-review.googlesource.com/c/go/+/245497
Run-TryBot: Michael Munday <mike.munday@ibm.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ruixin(Peter) Bao <ruixin.bao@ibm.com>
Reviewed-by: Filippo Valsorda <filippo@golang.org>
2020-07-30 16:00:05 +00:00
Michael Anthony Knyszek
6b4dcf19fa runtime: hold sched.lock over globrunqputbatch in runqputbatch
globrunqputbatch should never be called without sched.lock held.
runqputbatch's documentation even says it may acquire sched.lock in
order to call it.

Fixes #40457.

Change-Id: I5421b64f1da3a6087dfebbef7203db0c95d213a8
Reviewed-on: https://go-review.googlesource.com/c/go/+/245377
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-07-30 15:46:39 +00:00
Cherry Zhang
2369e01234 [dev.link] cmd/link: fix accidental escape in extreloc
We accidentally passed the address of a local to a function
pointer, where we should pass the address of a global.

Linking cmd/compile with external linking:

Asmb2_GC       32.5ms ± 5%    21.6ms ± 3%  -33.57%  (p=0.016 n=5+4)
Asmb2_GC       29.2MB ± 0%     6.4MB ± 0%  -78.20%  (p=0.008 n=5+5)
Asmb2_GC        1.43M ± 0%     0.00M ± 4%  -99.98%  (p=0.008 n=5+5)

Change-Id: I4754189bcc20f824627d95858ba35285d53c614d
Reviewed-on: https://go-review.googlesource.com/c/go/+/245337
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-07-28 19:47:05 +00:00
Michael Pratt
85afa2eb19 runtime: ensure startm new M is consistently visible to checkdead
If no M is available, startm first grabs an idle P, then drops
sched.lock and calls newm to start a new M to run than P.

Unfortunately, that leaves a window in which a G (e.g., returning from a
syscall) may find no idle P, add to the global runq, and then in stopm
discover that there are no running M's, a condition that should be
impossible with runnable G's.

To avoid this condition, we pre-allocate the new M ID in startm before
dropping sched.lock. This ensures that checkdead will see the M as
running, and since that new M must eventually run the scheduler, it will
handle any pending work as necessary.

Outside of startm, most other calls to newm/allocm don't have a P at
all. The only exception is startTheWorldWithSema, which always has an M
if there is 1 P (i.e., the currently running M), and if there is >1 P
the findrunnable spinning dance ensures the problem never occurs.

This has been tested with strategically placed sleeps in the runtime to
help induce the correct race ordering, but the timing on this is too
narrow for a test that can be checked in.

Fixes #40368

Change-Id: If5e0293a430cc85154b7ed55bc6dadf9b340abe2
Reviewed-on: https://go-review.googlesource.com/c/go/+/245018
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
2020-07-28 16:59:04 +00:00
Keith Randall
c4fed25553 cmd/compile: add floating point load+op operations to addressing modes pass
They were missed as part of the refactoring to use a separate
addressing modes pass.

Fixes #40426

Change-Id: Ie0418b2fac4ba1ffe720644ac918f6d728d5e420
Reviewed-on: https://go-review.googlesource.com/c/go/+/244859
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-07-27 18:24:32 +00:00
Cherry Zhang
19a932ceb8 cmd/link: don't mark shared library symbols reachable unconditionally
During the transitioning period, we mark symbols from Go shared
libraries reachable unconditionally. That might be useful when
there was still a large portion of the linker using sym.Symbols,
and only reachable symbols were converted to sym.Symbols. Marking
them reachable brings them to the dynamic symbol table, even if
they are not needed, increased the binary size unexpectedly.

That time has passed. Now we largely operate on loader symbols,
and it is not needed to mark them reachable anymore.

Fixes #40416.

Change-Id: I1e2bdb93a960ba7dc96575fabe15af93d8e95329
Reviewed-on: https://go-review.googlesource.com/c/go/+/244839
Run-TryBot: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-07-27 16:04:55 +00:00
Ian Lance Taylor
8696ae82c9 syscall: use correct file descriptor in dup2 fallback path
This fixes a mistake in CL 220422. This changes code that is only
executed on Linux kernel versions earlier than 2.6.27.

Change-Id: I01280184f4d7b75e06387c38f1891e8f0a81f793
Reviewed-on: https://go-review.googlesource.com/c/go/+/244630
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
2020-07-25 20:13:27 +00:00
Ian Lance Taylor
9591515f51 runtime, sync: add copyright headers to new files
For #38029

Change-Id: I71de2b66c1de617d32c46d4f2c1866f9ff1756ec
Reviewed-on: https://go-review.googlesource.com/c/go/+/244631
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Dan Scales <danscales@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-07-25 03:26:17 +00:00
Cherry Zhang
4b3cfcee58 [dev.link] cmd/link: remove non-streaming external relocation code
Now we support streaming external relocations everywhere.

Change-Id: I8d107c8239fe979bd5410e6a7f3fe471ac3e8b35
Reviewed-on: https://go-review.googlesource.com/c/go/+/244764
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-07-24 19:08:19 +00:00
Cherry Zhang
4c544ddaea [dev.link] all: merge branch 'master' into dev.link
Clean merge.

Change-Id: Ied3a0a9d098ab8dd28a38fdfb36c1f17b004fef7
2020-07-24 13:18:51 -04:00
Filippo Valsorda
074f2d800f doc/go1.15: surface the crypto/x509 CommonName deprecation note
Updates #39568
Updates #37419
Updates #24151

Change-Id: I44c940e09e26a039076396bbfecb2b1574197cf7
Reviewed-on: https://go-review.googlesource.com/c/go/+/243221
Reviewed-by: Kevin Burke <kev@inburke.com>
2020-07-24 17:09:32 +00:00
Austin Clements
78c20c81aa doc/go1.15: announce GO386=387 deprecation
For #40255.
Updates #37419.

Change-Id: If9210c855cc2eea079e7e469463d4203888748f7
Reviewed-on: https://go-review.googlesource.com/c/go/+/243137
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2020-07-23 18:07:28 +00:00
Cherry Zhang
9d22325681 [dev.link] cmd/link: stream external relocations on PPC64
Both ELF and Xcoff.

Now we support streaming on all platforms. Later CLs will clean
up the old code.

Change-Id: Ieeef7844a3e229429983a8bc108d7f3fabf618e9
Reviewed-on: https://go-review.googlesource.com/c/go/+/244358
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-07-23 16:44:27 +00:00
Cherry Zhang
e4a3e57f47 [dev.link] cmd/link: stream external relocations on ARM and on Windows
Do them in the same CL so ARM's archreloc doesn't need to support
both streaming and non-streaming.

TODO: we haven't switched to using mmap to emit external
relocations on Windows.

Change-Id: Ica2ee89c03fc74839efd6b9e26c80585fcdce45c
Reviewed-on: https://go-review.googlesource.com/c/go/+/244357
Run-TryBot: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-07-23 16:38:56 +00:00
Cherry Zhang
71e2133304 [dev.link] cmd/link: stream external relocations on S390X
All the bits are there. Just need to enable it.

Change-Id: I12ee25317c6385838493dadc900fb57c3b49a416
Reviewed-on: https://go-review.googlesource.com/c/go/+/244277
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-07-23 16:38:43 +00:00
Cherry Zhang
ed6b8af509 [dev.link] cmd/link: stream external relocations on MIPS (32/64)
Change-Id: I47fbeb3a49754395dceff51af371638fd43350ff
Reviewed-on: https://go-review.googlesource.com/c/go/+/244097
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-07-23 16:38:36 +00:00
Cherry Zhang
ea708dc94c [dev.link] cmd/link: avoid reading symbol Data in archreloc on ARM64
The previous CL changed it to drop the Data when writing out a
symbol. Don't read the data.

Fix ARM64 build.

Change-Id: I121e9b0ebef123dbbc4ddffc02bf1a42788532f4
Reviewed-on: https://go-review.googlesource.com/c/go/+/244038
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-07-23 16:38:27 +00:00
Ian Lance Taylor
71218dbc40 runtime: don't mlock on Ubuntu 5.4 systems
For #35777
For #37436
Fixes #40184

Change-Id: I68561497d9258e994d1c6c48d4fb41ac6130ee3a
Reviewed-on: https://go-review.googlesource.com/c/go/+/244059
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
2020-07-22 21:31:19 +00:00
Cherry Zhang
ee8541e5b8 [dev.link] cmd/link: remove OutData
OutData was used for a symbol to point to its data in the output
buffer, in order to apply relocations. Now we fold relocation
application to Asmb next to symbol data writing. We can just pass
the output data as a local variable.

Linking cmd/compile,

name         old time/op    new time/op    delta
Asmb_GC        19.0ms ±10%    16.6ms ± 9%  -12.50%  (p=0.032 n=5+5)

name         old alloc/op   new alloc/op   delta
Asmb_GC        3.78MB ± 0%    0.14MB ± 1%  -96.41%  (p=0.008 n=5+5)

name         old live-B     new live-B     delta
Asmb_GC         27.5M ± 0%     23.9M ± 0%  -13.24%  (p=0.008 n=5+5)

Change-Id: Id870a10dce2a0a7447a05029c6d0ab39b47d0a12
Reviewed-on: https://go-review.googlesource.com/c/go/+/244017
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2020-07-21 21:23:44 +00:00