Both ELF and Xcoff.
Now we support streaming on all platforms. Later CLs will clean
up the old code.
Change-Id: Ieeef7844a3e229429983a8bc108d7f3fabf618e9
Reviewed-on: https://go-review.googlesource.com/c/go/+/244358
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Do them in the same CL so ARM's archreloc doesn't need to support
both streaming and non-streaming.
TODO: we haven't switched to using mmap to emit external
relocations on Windows.
Change-Id: Ica2ee89c03fc74839efd6b9e26c80585fcdce45c
Reviewed-on: https://go-review.googlesource.com/c/go/+/244357
Run-TryBot: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
All the bits are there. Just need to enable it.
Change-Id: I12ee25317c6385838493dadc900fb57c3b49a416
Reviewed-on: https://go-review.googlesource.com/c/go/+/244277
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
The previous CL changed it to drop the Data when writing out a
symbol. Don't read the data.
Fix ARM64 build.
Change-Id: I121e9b0ebef123dbbc4ddffc02bf1a42788532f4
Reviewed-on: https://go-review.googlesource.com/c/go/+/244038
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
OutData was used for a symbol to point to its data in the output
buffer, in order to apply relocations. Now we fold relocation
application to Asmb next to symbol data writing. We can just pass
the output data as a local variable.
Linking cmd/compile,
name old time/op new time/op delta
Asmb_GC 19.0ms ±10% 16.6ms ± 9% -12.50% (p=0.032 n=5+5)
name old alloc/op new alloc/op delta
Asmb_GC 3.78MB ± 0% 0.14MB ± 1% -96.41% (p=0.008 n=5+5)
name old live-B new live-B delta
Asmb_GC 27.5M ± 0% 23.9M ± 0% -13.24% (p=0.008 n=5+5)
Change-Id: Id870a10dce2a0a7447a05029c6d0ab39b47d0a12
Reviewed-on: https://go-review.googlesource.com/c/go/+/244017
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Support streaming external relocations on ARM64. Support
architecture-specific relocations.
Also support streaming external relocations on Darwin. Do it in
the same CL so ARM64's archreloc doesn't need to support both
streaming and non-streaming.
Change-Id: Ia7fee9957892f98c065022c69a51f47402f4d6e2
Reviewed-on: https://go-review.googlesource.com/c/go/+/243644
Reviewed-by: Jeremy Faller <jeremy@golang.org>
For content-addressable symbols, we build its content hash based
on the symbol data and relocations. When the compiler builds the
symbol data, it may not always include the trailing zeros, e.g.
the data of [10]int64{1,2,3} is only the first 24 bytes.
Therefore, we may end up with symbols with the same contents
(thus same hash) but different sizes. This is not actually a hash
collision. In this case, we can deduplicate them and keep the one
with the larger size.
Change-Id: If6834542d7914cc00f917d7db151955e5aee6f30
Reviewed-on: https://go-review.googlesource.com/c/go/+/243718
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
The ICE reported as #33308 was fixed by a related CL; this change adds
a regression test with the crasher.
Fixes#33308
Change-Id: I3260075dbe3823b56b8825e6269e57a0fad185a6
Reviewed-on: https://go-review.googlesource.com/c/go/+/243458
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
For content-addressable symbols with relocations, we build a
content hash based on its content and relocations. Depending on
the category of the referenced symbol, we choose different hash
algorithms such that the hash is globally consistent.
For now, we only support content-addressable symbols with
relocations when the current package's import path is known, so
that the symbol names are fully expanded. Otherwise, if the
referenced symbol is a named symbol whose name is not fully
expanded, the hash won't be globally consistent, and can cause
erroneous collisions. This is fine for now, as the deduplication
is just an optimization, not a requirement for correctness (until
we get to type descriptors).
Change-Id: I639e4e03dd749b5d71f0a55c2525926575b1ac30
Reviewed-on: https://go-review.googlesource.com/c/go/+/243142
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
For now, we only do this for symbols without relocations.
Mark static temps "local", as they are not referenced across DSO
boundaries. And deduplicating a local symbol and a non-local
symbol can be problematic.
Change-Id: I0a3dc4138aaeea7fd4f326998f32ab6305da8e4b
Reviewed-on: https://go-review.googlesource.com/c/go/+/243141
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
The StdFormat flag was added as part of CL 231461, where the primary aim
was to fix the bug #37476. It's expected that the existing printer modes
only adjust spacing but do not change any of the code text itself. A new
printing flag served as a way for cmd/gofmt and go/format to delegate
a part of formatting work to the printer—where it's more more convenient
and efficient to perform—while maintaining current low-level printing
behavior of go/printer unmodified.
We already have cmd/gofmt and the go/format API that implement standard
formatting of Go source code, so there isn't a need to expose StdFormat
flag to the world, as it can only cause confusion.
Consider that to format source in canonical gofmt style completely it
may require tasks A, B, C to be done. In one version of Go, the printer
may do both A and B, while cmd/gofmt and go/format will do the remaining
task C. In another version, the printer may take on doing just A, while
cmd/gofmt and go/format will perform B and C. This makes it hard to add
a gofmt-like mode to the printer without compromising on above fluidity.
This change prefers to shift back some complexity to the implementation
of the standard library, allowing us to avoid creating the new exported
printing flag just for the internal needs of gofmt and go/format today.
We may still want to re-think the API and consider if something better
should be added, but unfortunately there isn't time for Go 1.15. We are
not adding new APIs now, so we can defer this decision until Go 1.16 or
later, when there is more time.
For #37476.
For #37453.
For #39489.
For #37419.
Change-Id: I0bb07156dca852b043487099dcf05c5350b29e20
Reviewed-on: https://go-review.googlesource.com/c/go/+/240683
Reviewed-by: Robert Griesemer <gri@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
While investigating #34121, fixed by CL 193605,
I discovered another case where Reset was not quite
resetting enough.
This specific case is not a problem in Reset itself but
rather that the Huffman bit writer in one code path
is using uninitialized memory left over from a previous
block, making the compression not choose the optimal
compression method.
Fixes#34121.
Change-Id: I29245b28214d924e382f91e2c56b4b8a9b7da13d
Reviewed-on: https://go-review.googlesource.com/c/go/+/243140
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Modify the overflow detection logic to shuffle the contents
of the table to a lower offset to avoid leaking the effects
of a previous use of compress.Writer past Reset calls.
Fixes#34121
Change-Id: I9963eadfa5482881e7b7adbad4c2cae146b669ab
GitHub-Last-Rev: 8b35798cdd
GitHub-Pull-Request: golang/go#34128
Reviewed-on: https://go-review.googlesource.com/c/go/+/193605
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Currently, when external linking, in relocsym (in asmb pass), we
convert Go relocations to an in-memory representation of external
relocations, and then in asmb2 pass we write them out to the
output file. This is not memory efficient.
This CL makes it not do the conversion but directly stream out
the external relocations based on Go relocations. Currently only
do this on AMD64 ELF systems.
This reduces memory usage, but makes the asmb2 pass a little
slower.
Linking cmd/compile with external linking:
name old time/op new time/op delta
Asmb_GC 83.8ms ± 7% 70.4ms ± 4% -16.03% (p=0.008 n=5+5)
Asmb2_GC 95.6ms ± 4% 118.2ms ± 5% +23.65% (p=0.008 n=5+5)
TotalTime_GC 1.59s ± 2% 1.62s ± 1% ~ (p=0.151 n=5+5)
name old alloc/op new alloc/op delta
Asmb_GC 26.0MB ± 0% 4.1MB ± 0% -84.15% (p=0.008 n=5+5)
Asmb2_GC 8.19MB ± 0% 8.18MB ± 0% ~ (p=0.222 n=5+5)
name old live-B new live-B delta
Asmb_GC 49.2M ± 0% 27.4M ± 0% -44.38% (p=0.008 n=5+5)
Asmb2_GC 51.5M ± 0% 29.7M ± 0% -42.33% (p=0.008 n=5+5)
TODO: figure out what is slow. Possible improvements:
- Remove redundant work in relocsym.
- Maybe there is a better representation for external relocations
now.
- Fine-grained parallelism in emitting external relocations.
- The old elfrelocsect only iterates over external relocations,
now we iterate over all relocations. Is it too many?
Change-Id: Ib0a8ee8c88d65864c62b89a8d634614f7f2c813e
Reviewed-on: https://go-review.googlesource.com/c/go/+/242603
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
For symbols of size 8 bytes or below, we can map them to 64-bit
hash values using the identity function. There is no need to use
longer and more expensive hash functions.
For them, we introduce another pseudo-package, PkgIdxHashed64. It
is like PkgIdxHashed except that the hash function is different.
Note that the hash value is not affected with trailing zeros,
e.g. "A" and "A\0\0\0" have the same hash value. This allows
deduplicating a few more symbols. When deduplicating them, we
need to keep the longer one.
Change-Id: Iad0c2e9e569b6a59ca6a121fb8c8f0c018c6da03
Reviewed-on: https://go-review.googlesource.com/c/go/+/242362
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Fill in the data at compile time, and get rid of the preprocess
function in the linker.
We need to be careful with symbol alignment: data symbols are
generally naturally aligned, except for string symbols which are
not aligned. When deduplicating two symbols with same content but
different alignments, we need to keep the biggest alignment.
Change-Id: I4bd96adfdc5f704b5bf3a0e723457c9bfe16a684
Reviewed-on: https://go-review.googlesource.com/c/go/+/242081
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
This CL introduces content-addressable symbols (a.k.a. hashed
symbols) to object files. Content-addressable symbols are
identified and referenced by their content hashes, instead of by
names.
In the object file, a new pseudo-package index PkgIdxHashed is
introduced, for content-addressable symbols, and a new block is
added to store their hashes. The hashes are used by the linker to
identify and deduplicate the symbols.
For now, we only support content-addressable symbols that are
always locally defined (i.e. no cross-package references).
As a proof of concept, make string constant symbols content-
addressable.
Change-Id: Iaf53efd74c0ffb54fa95f784628cc84e95844536
Reviewed-on: https://go-review.googlesource.com/c/go/+/242079
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
We don't ship the old linker in binary releases. Skip the test if
we cannot find the old linker.
Fixes#39509.
Change-Id: I1af5552bc56aff5314a384bcb5f3717b725d68e5
Reviewed-on: https://go-review.googlesource.com/c/go/+/242604
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
The expectContinueReader writes to the connection on the first
Request.Body read. Since a Handler might be doing a read in parallel or
before a write, expectContinueReader needs to synchronize with the
ResponseWriter, and abort if a response already went out.
The tests will land in a separate CL.
Fixes#34902
Fixes CVE-2020-15586
Change-Id: Icdd8dd539f45e8863762bd378194bb4741e875fc
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/793350
Reviewed-by: Filippo Valsorda <valsorda@google.com>
Reviewed-on: https://go-review.googlesource.com/c/go/+/242598
Run-TryBot: Katie Hockman <katie@golang.org>
Reviewed-by: Filippo Valsorda <filippo@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
When using the platform verifier on Windows (because Roots is nil) we
were always enforcing server auth EKUs if DNSName was set, and none
otherwise. If an application was setting KeyUsages, they were not being
respected.
Started correctly surfacing IncompatibleUsage errors from the system
verifier, as those are the ones applications will see if they are
affected by this change.
Also refactored verify_test.go to make it easier to add tests for this,
and replaced the EKULeaf chain with a new one that doesn't have a SHA-1
signature.
Thanks to Niall Newman for reporting this.
Fixes#39360
Fixes CVE-2020-14039
Change-Id: If5c00d615f2944f7d57007891aae1307f9571c32
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/774414
Reviewed-by: Katie Hockman <katiehockman@google.com>
Reviewed-on: https://go-review.googlesource.com/c/go/+/242597
Run-TryBot: Katie Hockman <katie@golang.org>
Reviewed-by: Filippo Valsorda <filippo@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Copy and adapt tests from text/template, to exercise more of html/template's copy.
Various differences in behavior are flagged with NOTE comments or t.Skip
and documented in #40075. Many of them are probably bugs.
One clarifying test case added to both text/template and html/template.
No changes to the package itself.
Change-Id: Ifefad83d647db846040d24c2741a0244b00ade82
Reviewed-on: https://go-review.googlesource.com/c/go/+/241084
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
If the linker panics, it usually helps dumping all symbols'
information for debugging. Do it under -v.
Change-Id: I66f9e32a0133e763a631f17a218dcdc805c5df2f
Reviewed-on: https://go-review.googlesource.com/c/go/+/242078
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
The special case is no longer needed, didn't actually work, and
we no longer even save this map anywhere (see CL 240621 for more
information).
Change-Id: I19bcf32cace22decf50fd6414d4519cc51cbb0be
Reviewed-on: https://go-review.googlesource.com/c/go/+/241982
Reviewed-by: Austin Clements <austin@google.com>
After Dial timeout, force close the TCP connection by writing "hangup"
to the control file. This unblocks the "connect" command if the
connection is taking too long to establish, and frees up the control
file FD.
Fixes#40118
Change-Id: I1cef8539cd9fe0793e32b49c9d0ef636b4b26e1d
Reviewed-on: https://go-review.googlesource.com/c/go/+/241638
Run-TryBot: David du Colombier <0intro@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: David du Colombier <0intro@gmail.com>
Set the AttrStatic flag on compiler-emitted TOC symbols for ppc64; these
symbols don't need to go into the final symbol table in Go binaries.
This fixes a buglet introduced by CL 240539 that was causing failures
on the aix builder.
Change-Id: If8b63bcf6d2791f1ec5a0c371d2d11e806202fd2
Reviewed-on: https://go-review.googlesource.com/c/go/+/241637
Run-TryBot: Than McIntosh <thanm@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Also add a test to lock in this policy.
Fixes#40065
Change-Id: Iedc4586f2f5598046d84132a8f3bba8f2e93ddc2
Reviewed-on: https://go-review.googlesource.com/c/go/+/241274
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
It is called by the signal handler before switching to gsignal
(sigtrampgo -> sigfwdgo -> dieFromSignal -> raise)
which means that it must not split the stack.
All other instances of raise are already marked nosplit.
Fixes#40076
Change-Id: I4794491331af48c46d0d8ebc82d34c6483f0e6cd
Reviewed-on: https://go-review.googlesource.com/c/go/+/241121
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: David Chase <drchase@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Use of a nil *File as an argument should not result in a panic,
but result in the ErrInvalid error being returned.
Fix the copy_file_range implementation to preserve this semantic.
Fixes#40115
Change-Id: Iad5ac39664a3efb7964cf55685be636940a8db13
Reviewed-on: https://go-review.googlesource.com/c/go/+/241417
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Don't emit symbol table entries for compiler-generated file-local
symbols (this category includes .stmp_* temporaries and *.stkobj
symbols). Note that user-written static symbols within assembler
sources will still be added to the symbol table. Apply the same test
when emitting DWARF for global variables.
Change-Id: I4db77a2750a0b575e051dfea895c4742cf6709a6
Reviewed-on: https://go-review.googlesource.com/c/go/+/240539
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Mark compiler-generated ".stmp_%d" and "<fn>.stkobj" symbols as
AttrStatic, so as to tell the linker that they do not need to be
inserted into its name lookup tables.
Change-Id: I59ffd11659b2c54c2d0ad41275d05c3f919e3b88
Reviewed-on: https://go-review.googlesource.com/c/go/+/240497
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
I'm from Hootsuite. We're a Canadian tech company who provides products
and services to businesses, organizations and individuals to really help
them succeed on social. We have leveraged Go in our stack for the past
4+ years. I am super happy to give back to Go on behalf of Hootsuite
through a small contribution to pkgsite (with a few more in the works).
We love this project and we love open source :)
Hopefully we can give back more in the future!
Kush
Change-Id: Id534a41d78e17e1fa48a8ddecd1ca110cf812388
GitHub-Last-Rev: 297b8b06e7
GitHub-Pull-Request: golang/go#40088
Reviewed-on: https://go-review.googlesource.com/c/go/+/241218
Reviewed-by: Julie Qiu <julie@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Since Plan 9 doesn't allow us to listen on 0.0.0.0, the Listener
address that's read in from /net is the IPv6 address ::. Convert
this address to 0.0.0.0 when the network is tcp4 or udp4.
Fixes#40045
Change-Id: Icfb69b823e5b80603742d23c3762a812996fe43f
Reviewed-on: https://go-review.googlesource.com/c/go/+/240918
Run-TryBot: David du Colombier <0intro@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: David du Colombier <0intro@gmail.com>
