Support streaming external relocations on ARM64. Support
architecture-specific relocations.
Also support streaming external relocations on Darwin. Do it in
the same CL so ARM64's archreloc doesn't need to support both
streaming and non-streaming.
Change-Id: Ia7fee9957892f98c065022c69a51f47402f4d6e2
Reviewed-on: https://go-review.googlesource.com/c/go/+/243644
Reviewed-by: Jeremy Faller <jeremy@golang.org>
For content-addressable symbols, we build its content hash based
on the symbol data and relocations. When the compiler builds the
symbol data, it may not always include the trailing zeros, e.g.
the data of [10]int64{1,2,3} is only the first 24 bytes.
Therefore, we may end up with symbols with the same contents
(thus same hash) but different sizes. This is not actually a hash
collision. In this case, we can deduplicate them and keep the one
with the larger size.
Change-Id: If6834542d7914cc00f917d7db151955e5aee6f30
Reviewed-on: https://go-review.googlesource.com/c/go/+/243718
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
The ICE reported as #33308 was fixed by a related CL; this change adds
a regression test with the crasher.
Fixes#33308
Change-Id: I3260075dbe3823b56b8825e6269e57a0fad185a6
Reviewed-on: https://go-review.googlesource.com/c/go/+/243458
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
For content-addressable symbols with relocations, we build a
content hash based on its content and relocations. Depending on
the category of the referenced symbol, we choose different hash
algorithms such that the hash is globally consistent.
For now, we only support content-addressable symbols with
relocations when the current package's import path is known, so
that the symbol names are fully expanded. Otherwise, if the
referenced symbol is a named symbol whose name is not fully
expanded, the hash won't be globally consistent, and can cause
erroneous collisions. This is fine for now, as the deduplication
is just an optimization, not a requirement for correctness (until
we get to type descriptors).
Change-Id: I639e4e03dd749b5d71f0a55c2525926575b1ac30
Reviewed-on: https://go-review.googlesource.com/c/go/+/243142
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
For now, we only do this for symbols without relocations.
Mark static temps "local", as they are not referenced across DSO
boundaries. And deduplicating a local symbol and a non-local
symbol can be problematic.
Change-Id: I0a3dc4138aaeea7fd4f326998f32ab6305da8e4b
Reviewed-on: https://go-review.googlesource.com/c/go/+/243141
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
The StdFormat flag was added as part of CL 231461, where the primary aim
was to fix the bug #37476. It's expected that the existing printer modes
only adjust spacing but do not change any of the code text itself. A new
printing flag served as a way for cmd/gofmt and go/format to delegate
a part of formatting work to the printer—where it's more more convenient
and efficient to perform—while maintaining current low-level printing
behavior of go/printer unmodified.
We already have cmd/gofmt and the go/format API that implement standard
formatting of Go source code, so there isn't a need to expose StdFormat
flag to the world, as it can only cause confusion.
Consider that to format source in canonical gofmt style completely it
may require tasks A, B, C to be done. In one version of Go, the printer
may do both A and B, while cmd/gofmt and go/format will do the remaining
task C. In another version, the printer may take on doing just A, while
cmd/gofmt and go/format will perform B and C. This makes it hard to add
a gofmt-like mode to the printer without compromising on above fluidity.
This change prefers to shift back some complexity to the implementation
of the standard library, allowing us to avoid creating the new exported
printing flag just for the internal needs of gofmt and go/format today.
We may still want to re-think the API and consider if something better
should be added, but unfortunately there isn't time for Go 1.15. We are
not adding new APIs now, so we can defer this decision until Go 1.16 or
later, when there is more time.
For #37476.
For #37453.
For #39489.
For #37419.
Change-Id: I0bb07156dca852b043487099dcf05c5350b29e20
Reviewed-on: https://go-review.googlesource.com/c/go/+/240683
Reviewed-by: Robert Griesemer <gri@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
While investigating #34121, fixed by CL 193605,
I discovered another case where Reset was not quite
resetting enough.
This specific case is not a problem in Reset itself but
rather that the Huffman bit writer in one code path
is using uninitialized memory left over from a previous
block, making the compression not choose the optimal
compression method.
Fixes#34121.
Change-Id: I29245b28214d924e382f91e2c56b4b8a9b7da13d
Reviewed-on: https://go-review.googlesource.com/c/go/+/243140
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Modify the overflow detection logic to shuffle the contents
of the table to a lower offset to avoid leaking the effects
of a previous use of compress.Writer past Reset calls.
Fixes#34121
Change-Id: I9963eadfa5482881e7b7adbad4c2cae146b669ab
GitHub-Last-Rev: 8b35798cdd
GitHub-Pull-Request: golang/go#34128
Reviewed-on: https://go-review.googlesource.com/c/go/+/193605
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Currently, when external linking, in relocsym (in asmb pass), we
convert Go relocations to an in-memory representation of external
relocations, and then in asmb2 pass we write them out to the
output file. This is not memory efficient.
This CL makes it not do the conversion but directly stream out
the external relocations based on Go relocations. Currently only
do this on AMD64 ELF systems.
This reduces memory usage, but makes the asmb2 pass a little
slower.
Linking cmd/compile with external linking:
name old time/op new time/op delta
Asmb_GC 83.8ms ± 7% 70.4ms ± 4% -16.03% (p=0.008 n=5+5)
Asmb2_GC 95.6ms ± 4% 118.2ms ± 5% +23.65% (p=0.008 n=5+5)
TotalTime_GC 1.59s ± 2% 1.62s ± 1% ~ (p=0.151 n=5+5)
name old alloc/op new alloc/op delta
Asmb_GC 26.0MB ± 0% 4.1MB ± 0% -84.15% (p=0.008 n=5+5)
Asmb2_GC 8.19MB ± 0% 8.18MB ± 0% ~ (p=0.222 n=5+5)
name old live-B new live-B delta
Asmb_GC 49.2M ± 0% 27.4M ± 0% -44.38% (p=0.008 n=5+5)
Asmb2_GC 51.5M ± 0% 29.7M ± 0% -42.33% (p=0.008 n=5+5)
TODO: figure out what is slow. Possible improvements:
- Remove redundant work in relocsym.
- Maybe there is a better representation for external relocations
now.
- Fine-grained parallelism in emitting external relocations.
- The old elfrelocsect only iterates over external relocations,
now we iterate over all relocations. Is it too many?
Change-Id: Ib0a8ee8c88d65864c62b89a8d634614f7f2c813e
Reviewed-on: https://go-review.googlesource.com/c/go/+/242603
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
For symbols of size 8 bytes or below, we can map them to 64-bit
hash values using the identity function. There is no need to use
longer and more expensive hash functions.
For them, we introduce another pseudo-package, PkgIdxHashed64. It
is like PkgIdxHashed except that the hash function is different.
Note that the hash value is not affected with trailing zeros,
e.g. "A" and "A\0\0\0" have the same hash value. This allows
deduplicating a few more symbols. When deduplicating them, we
need to keep the longer one.
Change-Id: Iad0c2e9e569b6a59ca6a121fb8c8f0c018c6da03
Reviewed-on: https://go-review.googlesource.com/c/go/+/242362
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Fill in the data at compile time, and get rid of the preprocess
function in the linker.
We need to be careful with symbol alignment: data symbols are
generally naturally aligned, except for string symbols which are
not aligned. When deduplicating two symbols with same content but
different alignments, we need to keep the biggest alignment.
Change-Id: I4bd96adfdc5f704b5bf3a0e723457c9bfe16a684
Reviewed-on: https://go-review.googlesource.com/c/go/+/242081
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
This CL introduces content-addressable symbols (a.k.a. hashed
symbols) to object files. Content-addressable symbols are
identified and referenced by their content hashes, instead of by
names.
In the object file, a new pseudo-package index PkgIdxHashed is
introduced, for content-addressable symbols, and a new block is
added to store their hashes. The hashes are used by the linker to
identify and deduplicate the symbols.
For now, we only support content-addressable symbols that are
always locally defined (i.e. no cross-package references).
As a proof of concept, make string constant symbols content-
addressable.
Change-Id: Iaf53efd74c0ffb54fa95f784628cc84e95844536
Reviewed-on: https://go-review.googlesource.com/c/go/+/242079
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
We don't ship the old linker in binary releases. Skip the test if
we cannot find the old linker.
Fixes#39509.
Change-Id: I1af5552bc56aff5314a384bcb5f3717b725d68e5
Reviewed-on: https://go-review.googlesource.com/c/go/+/242604
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
The expectContinueReader writes to the connection on the first
Request.Body read. Since a Handler might be doing a read in parallel or
before a write, expectContinueReader needs to synchronize with the
ResponseWriter, and abort if a response already went out.
The tests will land in a separate CL.
Fixes#34902
Fixes CVE-2020-15586
Change-Id: Icdd8dd539f45e8863762bd378194bb4741e875fc
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/793350
Reviewed-by: Filippo Valsorda <valsorda@google.com>
Reviewed-on: https://go-review.googlesource.com/c/go/+/242598
Run-TryBot: Katie Hockman <katie@golang.org>
Reviewed-by: Filippo Valsorda <filippo@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
When using the platform verifier on Windows (because Roots is nil) we
were always enforcing server auth EKUs if DNSName was set, and none
otherwise. If an application was setting KeyUsages, they were not being
respected.
Started correctly surfacing IncompatibleUsage errors from the system
verifier, as those are the ones applications will see if they are
affected by this change.
Also refactored verify_test.go to make it easier to add tests for this,
and replaced the EKULeaf chain with a new one that doesn't have a SHA-1
signature.
Thanks to Niall Newman for reporting this.
Fixes#39360
Fixes CVE-2020-14039
Change-Id: If5c00d615f2944f7d57007891aae1307f9571c32
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/774414
Reviewed-by: Katie Hockman <katiehockman@google.com>
Reviewed-on: https://go-review.googlesource.com/c/go/+/242597
Run-TryBot: Katie Hockman <katie@golang.org>
Reviewed-by: Filippo Valsorda <filippo@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Copy and adapt tests from text/template, to exercise more of html/template's copy.
Various differences in behavior are flagged with NOTE comments or t.Skip
and documented in #40075. Many of them are probably bugs.
One clarifying test case added to both text/template and html/template.
No changes to the package itself.
Change-Id: Ifefad83d647db846040d24c2741a0244b00ade82
Reviewed-on: https://go-review.googlesource.com/c/go/+/241084
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
If the linker panics, it usually helps dumping all symbols'
information for debugging. Do it under -v.
Change-Id: I66f9e32a0133e763a631f17a218dcdc805c5df2f
Reviewed-on: https://go-review.googlesource.com/c/go/+/242078
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
The special case is no longer needed, didn't actually work, and
we no longer even save this map anywhere (see CL 240621 for more
information).
Change-Id: I19bcf32cace22decf50fd6414d4519cc51cbb0be
Reviewed-on: https://go-review.googlesource.com/c/go/+/241982
Reviewed-by: Austin Clements <austin@google.com>
After Dial timeout, force close the TCP connection by writing "hangup"
to the control file. This unblocks the "connect" command if the
connection is taking too long to establish, and frees up the control
file FD.
Fixes#40118
Change-Id: I1cef8539cd9fe0793e32b49c9d0ef636b4b26e1d
Reviewed-on: https://go-review.googlesource.com/c/go/+/241638
Run-TryBot: David du Colombier <0intro@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: David du Colombier <0intro@gmail.com>
Set the AttrStatic flag on compiler-emitted TOC symbols for ppc64; these
symbols don't need to go into the final symbol table in Go binaries.
This fixes a buglet introduced by CL 240539 that was causing failures
on the aix builder.
Change-Id: If8b63bcf6d2791f1ec5a0c371d2d11e806202fd2
Reviewed-on: https://go-review.googlesource.com/c/go/+/241637
Run-TryBot: Than McIntosh <thanm@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Also add a test to lock in this policy.
Fixes#40065
Change-Id: Iedc4586f2f5598046d84132a8f3bba8f2e93ddc2
Reviewed-on: https://go-review.googlesource.com/c/go/+/241274
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
It is called by the signal handler before switching to gsignal
(sigtrampgo -> sigfwdgo -> dieFromSignal -> raise)
which means that it must not split the stack.
All other instances of raise are already marked nosplit.
Fixes#40076
Change-Id: I4794491331af48c46d0d8ebc82d34c6483f0e6cd
Reviewed-on: https://go-review.googlesource.com/c/go/+/241121
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: David Chase <drchase@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Use of a nil *File as an argument should not result in a panic,
but result in the ErrInvalid error being returned.
Fix the copy_file_range implementation to preserve this semantic.
Fixes#40115
Change-Id: Iad5ac39664a3efb7964cf55685be636940a8db13
Reviewed-on: https://go-review.googlesource.com/c/go/+/241417
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Don't emit symbol table entries for compiler-generated file-local
symbols (this category includes .stmp_* temporaries and *.stkobj
symbols). Note that user-written static symbols within assembler
sources will still be added to the symbol table. Apply the same test
when emitting DWARF for global variables.
Change-Id: I4db77a2750a0b575e051dfea895c4742cf6709a6
Reviewed-on: https://go-review.googlesource.com/c/go/+/240539
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Mark compiler-generated ".stmp_%d" and "<fn>.stkobj" symbols as
AttrStatic, so as to tell the linker that they do not need to be
inserted into its name lookup tables.
Change-Id: I59ffd11659b2c54c2d0ad41275d05c3f919e3b88
Reviewed-on: https://go-review.googlesource.com/c/go/+/240497
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
I'm from Hootsuite. We're a Canadian tech company who provides products
and services to businesses, organizations and individuals to really help
them succeed on social. We have leveraged Go in our stack for the past
4+ years. I am super happy to give back to Go on behalf of Hootsuite
through a small contribution to pkgsite (with a few more in the works).
We love this project and we love open source :)
Hopefully we can give back more in the future!
Kush
Change-Id: Id534a41d78e17e1fa48a8ddecd1ca110cf812388
GitHub-Last-Rev: 297b8b06e7
GitHub-Pull-Request: golang/go#40088
Reviewed-on: https://go-review.googlesource.com/c/go/+/241218
Reviewed-by: Julie Qiu <julie@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Since Plan 9 doesn't allow us to listen on 0.0.0.0, the Listener
address that's read in from /net is the IPv6 address ::. Convert
this address to 0.0.0.0 when the network is tcp4 or udp4.
Fixes#40045
Change-Id: Icfb69b823e5b80603742d23c3762a812996fe43f
Reviewed-on: https://go-review.googlesource.com/c/go/+/240918
Run-TryBot: David du Colombier <0intro@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: David du Colombier <0intro@gmail.com>
This doesn't change how ExtraNames are printed, so as not to cause
unnecessary churn of current outputs. Switched the ExtraNames check to a
nil check as we are checking for just-parsed values.
Fixes#39924Fixes#39873
Change-Id: Ifa07cfc1a057d73643710a774ef8a154222db187
Reviewed-on: https://go-review.googlesource.com/c/go/+/240543
Run-TryBot: Filippo Valsorda <filippo@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
TestDependencies defines the dependency policy
(what can depend on what) for the standard library.
The standard library has outgrown the idea of writing
the policy as a plain map literal. Also, the checker was
ignoring vendored packages, which makes it miss real
problems.
This commit adds a little language for describing
partial orders and rewrites the policy in that language.
It also changes the checker to look inside vendored
packages and adds those to the policy as well.
This turned up one important problem: net is depending
on fmt, unicode via golang.org/x/net/dns/dnsmessage,
filed as #40070.
This is a test-only change, so it should be appropriate
even for the release freeze, especially since it identified
a real bug.
Change-Id: I9b79f30761f167b8587204c959baa973583e39f2
Reviewed-on: https://go-review.googlesource.com/c/go/+/241078
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Pull in dns/dnsmessage fix to drop fmt dependency.
Also pulled in minor bug fix in x/crypto (dep of net).
Change-Id: I0f618e81a24ac24a5adae00af34b728b243dfdbb
Reviewed-on: https://go-review.googlesource.com/c/go/+/241257
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
The additions were generated using golang.org/x/build/cmd/relnote.
Updates #37419
Change-Id: I99e934377e4bb511c44908039ad6d3bfd3ee35ce
Reviewed-on: https://go-review.googlesource.com/c/go/+/241065
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Following CL 240399 and CL 240400, do the same for Mach-O.
Linking cmd/compile with external linking,
name old time/op new time/op delta
Asmb2_GC 32.7ms ± 2% 13.5ms ± 6% -58.56% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
Asmb2_GC 16.5MB ± 0% 6.4MB ± 0% -61.15% (p=0.008 n=5+5)
Change-Id: I0fd7019d8713d1940e5fbbce4ee8eebd926451a1
Reviewed-on: https://go-review.googlesource.com/c/go/+/241178
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
The loader writeBlock() function has code that tries to skip the
initial portion of the input symbols list depending on the address of
the section being written-- this code is dead (skipping is never
triggered) due to similar skipping in the callers; remove this
preamble.
Change-Id: I9769694a3194faf73ebebbbc10ceba4928c3087c
Reviewed-on: https://go-review.googlesource.com/c/go/+/241067
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
The errors on these lines are meant to be discarded.
Add a comment to make that extra clear.
Change-Id: I38f72af6dfbb0e86677087baf47780b3cc6e7d40
Reviewed-on: https://go-review.googlesource.com/c/go/+/241083
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Introduce a new loader method "SetCarrierSym", to be used when
establishing container/containee symbol relationships for symbol
bucketing in the symtab phase.
This new method is intended to be employed in situations where you
have a series of related symbols will be represented by a single
carrier symbol as a combined entity. The pattern here is that the
sub-symbols contain content but will be anonymous from a symbol table
perspective; the carrier symbol has no content itself but will appear
in the symbol table. Examples of carrier symbols that follow this
model are "runtime.itablink" and "runtime.typelink".
Change-Id: I1a3391a71062c7c740cb108b3fa210b7f69b81ed
Reviewed-on: https://go-review.googlesource.com/c/go/+/240509
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Introduce a new loader method "AddInteriorSym" to be used when
establishing container/containee symbol relationships for host object
sub-symbols and GOT/dynamic sub-symbols.
Interior symbols are employed in situations where you have a
"container" or "payload" symbol that has content, and then a series of
"interior" sub-symbols that point into a portion of the container
symbol's content. Each interior symbol will typically have a useful
name / size / value, but no content of its own. From a symbol table
perspective the container symbol is anonymous, but the interior
symbols are added to the output symbol table.
Change-Id: I919ed5dbbfe2ef2c9a76214f7ea9b384a1be6297
Reviewed-on: https://go-review.googlesource.com/c/go/+/240508
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
