1
0
mirror of https://github.com/golang/go synced 2024-11-08 03:36:12 -07:00
Commit Graph

346 Commits

Author SHA1 Message Date
Bryan C. Mills
c5cf662407 all: move internal/x to vendor/golang.org/x and revendor using 'go mod vendor'
This also updates the vendored-in versions of several packages: 'go
mod vendor' selects a consistent version of each module, but we had
previously vendored an ad-hoc selection of packages.

Notably, x/crypto/hkdf was previously vendored in at a much newer
commit than the rest of x/crypto. Bringing the rest of x/crypto up to
that commit introduced an import of golang.org/x/sys/cpu, which broke
the js/wasm build, requiring an upgrade of x/sys to pick up CL 165749.

Updates #30228
Updates #30241
Updates #25822

Change-Id: I5b3dbc232b7e6a048a158cbd8d36137af1efb711
Reviewed-on: https://go-review.googlesource.com/c/go/+/164623
Reviewed-by: Filippo Valsorda <filippo@golang.org>
2019-03-11 20:28:54 +00:00
Bryan C. Mills
0271d41ed6 internal/testenv: remove SetModVendor
It turns out not to be necessary. Russ expressed a preference for
avoiding module fetches over making 'go mod tidy' work within std and
cmd right away, so for now we will make the loader use the vendor
directory for the standard library even if '-mod=vendor' is not set
explicitly.

Updates #30228

Change-Id: Idf7208e63da8cb7bfe281b93ec21b61d40334947
Reviewed-on: https://go-review.googlesource.com/c/go/+/166357
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
2019-03-08 23:40:24 +00:00
Daniel Martí
49662bc6b0 all: simplify multiple for loops
If a for loop has a simple condition and begins with a simple
"if x { break; }"; we can simply add "!x" to the loop's condition.

While at it, simplify a few assignments to use the common pattern
"x := staticDefault; if cond { x = otherValue(); }".

Finally, simplify a couple of var declarations.

Change-Id: I413982c6abd32905adc85a9a666cb3819139c19f
Reviewed-on: https://go-review.googlesource.com/c/go/+/165342
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-03-08 14:29:19 +00:00
Wèi Cōngruì
40d8c3d3e8 internal/poll: fix deadlock in Write if len(buf) > maxRW
fd.l.Lock shouldn't be called in a loop.

Change-Id: I3afbc184aa06a60175c9a39319985b5810ecb144
Reviewed-on: https://go-review.googlesource.com/c/go/+/165598
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-03-06 18:25:59 +00:00
Wèi Cōngruì
a60b56adbe internal/poll: make FD.isFile mean whether it isn't socket on Windows
Before this change, if a directory was closed twice on Windows,
the returning error would be "use of closed network connection".

Some code assumes FD.isFile means whether the fd isn't a network
socket, which is true on Unix. But isFile reports whether
the fd is a normal file rather than directory or console on Windows.

With this change, isFile will have the same meaning on different
platforms. And the change adds a new field kind to replace isConsole
and isDir.

Change-Id: Ib12265f1e12fa3d0239ae925291128a84be59cc2
GitHub-Last-Rev: 3f031756de
GitHub-Pull-Request: golang/go#30589
Reviewed-on: https://go-review.googlesource.com/c/go/+/165257
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-03-06 18:04:33 +00:00
Tobias Klauser
029a5af6a1 internal/bytealg: use word-wise comparison for Compare on arm
Use word-wise comparison for aligned buffers, otherwise fall back to the
current byte-wise comparison.

name                           old time/op    new time/op    delta
BytesCompare/1-4                 41.3ns ± 0%    36.4ns ± 1%   -11.73%  (p=0.008 n=5+5)
BytesCompare/2-4                 39.5ns ± 0%    39.5ns ± 1%      ~     (p=0.960 n=5+5)
BytesCompare/4-4                 45.3ns ± 0%    41.0ns ± 1%    -9.40%  (p=0.008 n=5+5)
BytesCompare/8-4                 64.8ns ± 1%    44.7ns ± 0%   -31.12%  (p=0.008 n=5+5)
BytesCompare/16-4                86.3ns ± 0%    55.1ns ± 0%   -36.21%  (p=0.008 n=5+5)
BytesCompare/32-4                 135ns ± 0%      70ns ± 1%   -47.73%  (p=0.008 n=5+5)
BytesCompare/64-4                 231ns ± 1%      99ns ± 0%   -57.27%  (p=0.016 n=5+4)
BytesCompare/128-4                424ns ± 0%     147ns ± 0%   -65.31%  (p=0.000 n=4+5)
BytesCompare/256-4                810ns ± 0%     243ns ± 0%   -69.96%  (p=0.008 n=5+5)
BytesCompare/512-4               1.59µs ± 0%    0.44µs ± 0%   -72.43%  (p=0.008 n=5+5)
BytesCompare/1024-4              3.14µs ± 1%    0.83µs ± 1%   -73.56%  (p=0.008 n=5+5)
BytesCompare/2048-4              6.23µs ± 0%    1.61µs ± 1%   -74.21%  (p=0.008 n=5+5)
CompareBytesEqual-4              79.4ns ± 0%    52.2ns ± 0%   -34.23%  (p=0.008 n=5+5)
CompareBytesToNil-4              31.0ns ± 0%    30.3ns ± 0%    -2.32%  (p=0.008 n=5+5)
CompareBytesEmpty-4              25.7ns ± 0%    25.7ns ± 0%      ~     (p=0.556 n=4+5)
CompareBytesIdentical-4          25.7ns ± 0%    25.7ns ± 0%      ~     (p=1.000 n=5+5)
CompareBytesSameLength-4         49.1ns ± 0%    48.5ns ± 0%    -1.26%  (p=0.008 n=5+5)
CompareBytesDifferentLength-4    49.8ns ± 1%    49.3ns ± 0%    -1.08%  (p=0.008 n=5+5)
CompareBytesBigUnaligned-4       5.71ms ± 1%    5.68ms ± 1%      ~     (p=0.222 n=5+5)
CompareBytesBig-4                4.95ms ± 0%    2.28ms ± 1%   -53.81%  (p=0.008 n=5+5)
CompareBytesBigIdentical-4       27.2ns ± 1%    27.3ns ± 1%      ~     (p=0.310 n=5+5)

name                           old speed      new speed      delta
CompareBytesBigUnaligned-4      184MB/s ± 1%   185MB/s ± 1%      ~     (p=0.222 n=5+5)
CompareBytesBig-4               212MB/s ± 0%   459MB/s ± 1%  +116.51%  (p=0.008 n=5+5)
CompareBytesBigIdentical-4     38.5TB/s ± 0%  38.4TB/s ± 1%      ~     (p=0.421 n=5+5)

Also, this reduces time for TestCompareBytes by about 20 sec on a
linux-arm builder via gomote.

Updates #29001

Change-Id: I25f148739b9ccb7cb1fc97b3d8763549b0a66c16
Reviewed-on: https://go-review.googlesource.com/c/go/+/165338
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-03-06 16:09:38 +00:00
Bryan C. Mills
9670e81c2e all: add -mod=vendor to GOFLAGS in tests that execute 'go' commands within std or cmd
Updates #30228
Updates #30240
Updates #30241

Change-Id: Idc311ba77e99909318b5b86f8ef82d4878f73e47
Reviewed-on: https://go-review.googlesource.com/c/go/+/165378
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
2019-03-05 18:21:29 +00:00
Josh Bleecher Snyder
4fb900e9ca internal/poll: copy and use errnoErr to avoid allocations
Converting a syscall.Errno to an interface is
a significant source of allocations in os/exec.

Elsewhere in the tree, we have pre-allocated errors
for common errno values. Use the same trick here.

This CL makes yet another copy of this code.
The problem is that there isn't really a great place to share it.

The existing copies are in:

cmd/vendor/golang.org/x/sys/unix
cmd/vendor/golang.org/x/sys/windows
cmd/vendor/golang.org/x/sys/windows/registry
internal/syscall/windows
internal/syscall/windows/registry
syscall

internal/poll can't import from cmd/vendor, and cmd/vendor
can't import from internal/*, so we can ignore cmd/vendor.

We could put the unix version in internal/syscall/unix
and then have a platform-independent wrapper in internal/syscall.
But syscall couldn't use it; internal/syscall/* depends on syscall.
So that only allows code re-use with internal/syscall/windows/*.

We could create a new very low level internal package, internal/errno.
But syscall couldn't use it, because it has to import syscall
to get access to syscall.Errno.
So that only allows code re-use with internal/syscall/windows/*.

It's not clear that that any of these options pulls its weight.

The obvious and "correct" place for this is syscall.
But we can't export syscall's version, because package syscall is frozen.

So just copy the code. There's not much of it.

name            old alloc/op   new alloc/op   delta
ExecHostname-8    6.15kB ± 0%    6.13kB ± 0%  -0.38%  (p=0.000 n=20+19)

name            old allocs/op  new allocs/op  delta
ExecHostname-8      34.0 ± 0%      31.0 ± 0%  -8.82%  (p=0.000 n=20+20)

Fixes #30535

Change-Id: Idd31c7cced6e15387acc698ffc011e1b7b479903
Reviewed-on: https://go-review.googlesource.com/c/164971
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-03-04 01:22:28 +00:00
Richard Musiol
c0d82bb0ec all: rename WebAssembly instructions according to spec changes
The names of some instructions have been updated in the WebAssembly
specification to be more consistent, see
994591e51c.
This change to the spec is possible because it is still in a draft
state.

Go's support for WebAssembly is still experimental and thus excempt from
the compatibility promise. Being consistent with the spec should
warrant this breaking change to the assembly instruction names.

Change-Id: Iafb8b18ee7f55dd0e23c6c7824aa1fad43117ef1
Reviewed-on: https://go-review.googlesource.com/c/153797
Run-TryBot: Richard Musiol <neelance@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-03-03 21:10:01 +00:00
Alex Brainman
2edd559223 os: make Readlink work with symlinks with target like \??\Volume{ABCD}\
windows-arm TMP directory live inside such link (see
https://github.com/golang/go/issues/29746#issuecomment-456526811 for
details), so symlinks like that will be common at least on windows-arm.

This CL builds on current syscall.Readlink implementation. Main
difference between the two is how new code handles symlink targets,
like \??\Volume{ABCD}\.

New implementation uses Windows CreateFile API with
FILE_FLAG_OPEN_REPARSE_POINT flag to get \??\Volume{ABCD}\ file handle.
And then it uses Windows GetFinalPathNameByHandle with VOLUME_NAME_DOS
flag to convert that handle into standard Windows path.
FILE_FLAG_OPEN_REPARSE_POINT flag ensures that symlink is not followed
when CreateFile opens the file.

Fixes #30463

Change-Id: I33b18227ce36144caed694169ef2e429fd995fb4
Reviewed-on: https://go-review.googlesource.com/c/164201
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-03-01 07:44:37 +00:00
lukechampine
9d40fadb1c fmtsort: sort interfaces deterministically
Previously, the result of sorting a map[interface{}] containing
multiple concrete types was non-deterministic. To ensure consistent
results, sort first by type name, then by concrete value.

Fixes #30398

Change-Id: I10fd4b6a74eefbc87136853af6b2e689bc76ae9d
GitHub-Last-Rev: 1b07f0c275
GitHub-Pull-Request: golang/go#30406
Reviewed-on: https://go-review.googlesource.com/c/163745
Reviewed-by: Rob Pike <r@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-02-28 21:09:49 +00:00
Michael Munday
1f17d61026 internal/cpu: change s390x API to match x/sys/cpu
This CL changes the internal/cpu API to more closely match the
public version in x/sys/cpu (added in CL 163003). This will make it
easier to update the dependencies of vendored code. The most prominent
renaming is from VE1 to VXE for the vector-enhancements facility 1.
VXE is the mnemonic used for this facility in the HWCAP vector.

Change-Id: I922d6c8bb287900a4bd7af70567e22eac567b5c1
Reviewed-on: https://go-review.googlesource.com/c/164437
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
Run-TryBot: Michael Munday <mike.munday@ibm.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-02-28 13:46:45 +00:00
Daniel Martí
bd23e84b73 internal/lazytemplate: add a lazy template wrapper
Similar to internal/lazyregexp, this will allow removing unnecessary
work from init functions with trivial refactors, thanks to sync.Once.

Copy the structure. The only major difference is that a template also
carries a name.

For #29382.

Change-Id: I65d096dc2e2072b310bf59a814cd62669856b5b5
Reviewed-on: https://go-review.googlesource.com/c/164337
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-27 23:49:01 +00:00
Daniel Martí
df557fe3ea cmd/go: avoid compiling most regexes at init
These regexes are all related to commands like get and build, so they're
unnecessary for simpler commands like env. In particular, we need env to
be fast, since libraries like go/packages call it early and often. Some
external Go tools are interactive, so milliseconds matter.

lazyregexp eagerly compiles the patterns when running from within a test
binary, so there's no longer any need to do that as part of non-test
binaries.

Picking up the low-hanging fruit spotted by 'perf record' shaves off
well over a full millisecond off the benchmark on my laptop:

name         old time/op    new time/op    delta
ExecGoEnv-8    4.92ms ± 1%    3.81ms ± 0%  -22.52%  (p=0.004 n=6+5)

This CL required adding a few more methods to the lazy regexp wrapper.

Updates #29382.

Change-Id: I22417ab6258f7437a2feea0d25ceb2bb4d735a15
Reviewed-on: https://go-review.googlesource.com/c/155540
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-02-27 19:39:47 +00:00
Marcel van Lohuizen
9650726e79 internal/reflectlite: lite version of reflect package
to be used by errors package for checking assignability
and setting error values in As.

Updates #29934.

Change-Id: I8c1d02a2c6efa0919d54b286cfe8b4edc26da059
Reviewed-on: https://go-review.googlesource.com/c/161759
Run-TryBot: Marcel van Lohuizen <mpvl@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
2019-02-27 18:27:01 +00:00
Daniel Martí
f40cb19aff internal/lazyregexp: add a lazy Regexp package
This was implemented as part of go/doc, but it's going to be useful in
other packages. In particular, many packages under cmd/go like web and
vcs make somewhat heavy use of global regexes, which add a non-trivial
amount of init work to the cmd/go program.

A lazy wrapper around regexp.Regexp will make it trivial to get rid of
the extra cost with a trivial refactor, so make it possible for other
packages in the repository to make use of it. While naming the package,
give the members better names, such as lazyregexp.New and
lazyregexp.Regexp.

We're also considering adding some form of a lazy API to the public
regexp package, so this internal package will allow us to get some
initial experience across std and cmd.

For #29382.

Change-Id: I30b0e72871d5267c309786f95f4cb15c68b2393d
Reviewed-on: https://go-review.googlesource.com/c/164040
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-02-27 15:58:10 +00:00
Leon Klingele
d090429ea9 all: fix typos as reported by 'misspell'
Change-Id: I904b8655f21743189814bccf24073b6fbb9fc56d
GitHub-Last-Rev: b032c14394
GitHub-Pull-Request: golang/go#29997
Reviewed-on: https://go-review.googlesource.com/c/160421
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-02-26 23:02:05 +00:00
Brad Fitzpatrick
dbd323bb88 internal/x/net/http2/hpack: update from upstream
Updates to x/net git rev 891ebc4b82d6e74f468c533b06f983c7be918a96 for:

   http2/hpack: track the beginning of a header block
   https://go-review.googlesource.com/c/153978

Updates golang/go#29187

Change-Id: Ie2568b3f8d6aaa3f097a4ac25d3acdc794f5ff5c
Reviewed-on: https://go-review.googlesource.com/c/154118
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Matt Layher <mdlayher@gmail.com>
2018-12-14 18:15:56 +00:00
Elias Naur
9eb383e8f0 runtime,os,syscall,internal/poll: replace getdirentries on iOS
The getdirentries syscall is considered private API on iOS and is
rejected by the App Store submission checks. Replace it with the
fdopendir/readdir_r/closedir syscalls.

Fixes #28984

Change-Id: I73341b124310e9cb34834a95f946769f337ec5b7
Reviewed-on: https://go-review.googlesource.com/c/153338
Reviewed-by: Keith Randall <khr@golang.org>
2018-12-13 17:10:32 +00:00
Clément Chigot
21da8e6022 internal/xcoff: add big archive support
This commit adds support to read AIX big archive inside internal/xcoff
package.

Change-Id: I4317b40824b24312a69c918dfc6438dc3aff7be7
Reviewed-on: https://go-review.googlesource.com/c/153398
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-12-11 16:15:10 +00:00
Clément Chigot
e256afff51 all: move cmd/internal/xcoff to internal/xcoff
This commit moves cmd/internal/xcoff package to internal/xcoff because
it will be needed to add XCOFF support in go/internal/gccgoimporter.

Change-Id: Id12df0c438fb7db4a6a458fc1478480851bf7771
Reviewed-on: https://go-review.googlesource.com/c/152719
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-12-10 18:52:31 +00:00
bill_ofarrell
897e0807c3 crypto/elliptic: utilize faster z14 multiply/square instructions (when available)
In the s390x assembly implementation of NIST P-256 curve, utilize faster multiply/square
instructions introduced in the z14. These new instructions are designed for crypto
and are constant time. The algorithm is unchanged except for faster
multiplication when run on a z14 or later. On z13, the original mutiplication
(also constant time) is used.

P-256 performance is critical in many applications, such as Blockchain.

name            old time      new time     delta
BaseMultP256    24396 ns/op   21564 ns/op  1.13x
ScalarMultP256  87546 ns/op   72813 ns/op. 1.20x

Change-Id: I7e6d8b420fac56d5f9cc13c9423e2080df854bac
Reviewed-on: https://go-review.googlesource.com/c/146022
Reviewed-by: Michael Munday <mike.munday@ibm.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Michael Munday <mike.munday@ibm.com>
2018-12-05 10:58:44 +00:00
Tobias Klauser
9e277f7d55 all: use "reports whether" consistently instead of "returns whether"
Follow-up for CL 147037 and after Brad noticed the "returns whether"
pattern during the review of CL 150621.

Go documentation style for boolean funcs is to say:

    // Foo reports whether ...
    func Foo() bool

(rather than "returns whether")

Created with:

    $ perl -i -npe 's/returns whether/reports whether/' $(git grep -l "returns whether" | grep -v vendor)

Change-Id: I15fe9ff99180ad97750cd05a10eceafdb12dc0b4
Reviewed-on: https://go-review.googlesource.com/c/150918
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-12-02 15:12:26 +00:00
Bryan C. Mills
2012227b01 vendor/golang_org/x: move to internal/x
Packages in vendor/ directories have a "vendor/" path prefix in GOPATH
mode, but intentionally do not in module mode. Since the import path
is embedded in the compiled output, changing that path invalidates
cache entries and causes cmd/go to try to rebuild (and reinstall) the
vendored libraries, which will fail if the directory containing those
libraries is read-only.

If I understood correctly, this is the approach Russ suggested as an
alternative to https://golang.org/cl/136138.

Fixes #27285
Fixes #26988

Change-Id: I8a2507fa892b84cde0a803aaa79e460723da572b
Reviewed-on: https://go-review.googlesource.com/c/147443
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
2018-11-29 15:42:16 +00:00
Keith Randall
ca3749230b cmd/compile: allow bodyless function if it is linkname'd
In assembly free packages (aka "complete" or "pure go"), allow
bodyless functions if they are linkname'd to something else.

Presumably the thing the function is linkname'd to has a definition.
If not, the linker will complain. And linkname is unsafe, so we expect
users to know what they are doing.

Note this handles only one direction, where the linkname directive
is in the local package. If the linkname directive is in the remote
package, this CL won't help. (See os/signal/sig.s for an example.)

Fixes #23311

Change-Id: I824361b4b582ee05976d94812e5b0e8b0f7a18a6
Reviewed-on: https://go-review.googlesource.com/c/151318
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2018-11-26 20:00:59 +00:00
Hana Kim
6d5caf38e3 cmd/trace: revert internal/traceparser
The performance improvement is not as big as we hoped.
Until the API is feature complete, we postpone the release
and avoid added complexity.

This change was prepared by reverting all the changes affected
src/cmd/trace and src/internal/traceparser packages after
golang.org/cl/137635, and then bringing back MMU computation
APIs (originally in src/internal/traceparser) to the
src/internal/trace package.

Revert "cmd/trace: use new traceparser to parse the raw trace files"
This reverts https://golang.org/cl/145457
  (commit 08816cb8d7).

Revert "internal/traceparser: provide parser that uses less space and parses segments of runtime trace files"
This reverts https://golang.org/cl/137635
  (commit daaf361f74).

Change-Id: Ic2a068a7dbaf4053cd9674ca7bde9c58e74385b4
Reviewed-on: https://go-review.googlesource.com/c/150517
Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
2018-11-22 02:59:55 +00:00
Martin Möhrmann
2de53906e1 internal/cpu: move GODEBUGCPU options into GODEBUG
Change internal/cpu feature configuration to use
GODEBUG=cpu.feature1=value,cpu.feature2=value...
instead of GODEBUGCPU=feature1=value,feature2=value... .

This is not a backwards compatibility breaking change
since GODEBUGCPU was introduced in go1.11 as an
undocumented compiler experiment.

Fixes #28757

Change-Id: Ib21b3fed2334baeeb061a722ab1eb513d1137e87
Reviewed-on: https://go-review.googlesource.com/c/149578
Run-TryBot: Martin Möhrmann <martisch@uos.de>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-11-14 21:47:50 +00:00
Austin Clements
ef7ce57ac2 internal/bytealg, runtime: provide linknames for pushed symbols
The internal/bytealg package defines several symbols in the runtime,
bytes, and strings packages in assembly, and the runtime package
defines symbols in reflect and sync/atomic. Currently, there's no
corresponding Go prototype for these symbols in the defining package.

We're going to start depending on Go prototypes in the same package as
their assembly definitions in order to provide ABI wrappers. Plus,
these are good documentation and colocate type information with
definitions, which could be useful for vet if it learned a little
about linkname.

This CL adds linknamed Go prototypes for all pushed symbols in
internal/bytealg and runtime.

For #27539.

Change-Id: I9b0c12d935a75bb6af46b6761180d451c00f11b8
Reviewed-on: https://go-review.googlesource.com/c/146820
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
2018-11-12 20:27:16 +00:00
Keith Randall
ad4a58e315 strings,bytes: use inlineable function trampolines instead of linkname
Cleans things up quite a bit.

There's still a few more, like runtime.cmpstring, which might also
be worth fixing.

Change-Id: Ide18dd621efc129cc686db223f47fa0b044b5580
Reviewed-on: https://go-review.googlesource.com/c/148578
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
2018-11-08 20:52:47 +00:00
Keith Randall
be5f646dab internal/syscall/unix: use libc calls on Darwin
Add unexported unlinkat, openat, and fstatat calls, so that
the internal/syscall/unix package can use them.

Change-Id: I1df81ecae6427211dd392ec68c9f020fe131a526
Reviewed-on: https://go-review.googlesource.com/c/148457
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-11-08 17:26:22 +00:00
Keith Randall
a3b01440fe syscall: implement syscalls on Darwin using libSystem
There are still some references to the bare Syscall functions
in the stdlib. I will root those out in a following CL.
(This CL is big enough as it is.)
Most are in vendor directories:

cmd/vendor/golang.org/x/sys/unix/
vendor/golang_org/x/net/route/syscall.go
syscall/bpf_bsd.go
syscall/exec_unix.go
syscall/flock.go

Update #17490

Change-Id: I69ab707811530c26b652b291cadee92f5bf5c1a4
Reviewed-on: https://go-review.googlesource.com/c/141639
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Elias Naur <elias.naur@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-11-07 20:27:01 +00:00
Elias Naur
ac277d9234 internal/traceparser: skip test on iOS
Change-Id: Ifc9581ba82a13f507c288282b517ebf8a5f93b4e
Reviewed-on: https://go-review.googlesource.com/c/148058
Run-TryBot: Elias Naur <elias.naur@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
2018-11-07 13:04:50 +00:00
Austin Clements
33563d1cfc internal/trace: support for mutator utilization distributions
This adds support for computing the quantiles of a mutator utilization
distribution.

Change-Id: Ia8b3ed14bf415c234e2f567360fd1b361d28bd40
Reviewed-on: https://go-review.googlesource.com/c/60799
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
2018-11-05 19:10:27 +00:00
Austin Clements
bef4efc822 internal/trace: add "per-P" MMU analysis
The current MMU analysis considers all Ps together, so if, for
example, one of four Ps is blocked, mutator utilization is 75%.
However, this is less useful for understanding the impact on
individual goroutines because that one blocked goroutine could be
blocked for a very long time, but we still appear to have good
utilization.

Hence, this introduces a new flag that does a "per-P" analysis where
the utilization of each P is considered independently. The MMU is then
the combination of the MMU for each P's utilization function.

Change-Id: Id67b980d4d82b511d28300cdf92ccbb5ae8f0c78
Reviewed-on: https://go-review.googlesource.com/c/60797
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
2018-11-05 19:10:23 +00:00
Austin Clements
27920c8ddc internal/trace: flags for what to include in GC utilization
Change-Id: I4ba963b003cb25b39d7575d423f17930d84f3f69
Reviewed-on: https://go-review.googlesource.com/c/60796
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
2018-11-05 19:10:20 +00:00
Austin Clements
cbe04e8d70 internal/trace: track worst N mutator utilization windows
This will let the trace viewer show specifically when poor utilization
happened and link to specific instances in the trace.

Change-Id: I1f03a0f9d9a7570009bb15762e7b8b6f215e9423
Reviewed-on: https://go-review.googlesource.com/c/60793
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
2018-11-05 19:10:16 +00:00
Austin Clements
52ee654b25 internal/trace: use banding to optimize MMU computation
This further optimizes MMU construction by first computing a
low-resolution summary of the utilization curve. This "band" summary
lets us compute the worst-possible window starting in each of these
low-resolution bands (even without knowing where in the band the
window falls). This in turn lets us compute precise minimum mutator
utilization only in the worst low-resolution bands until we can show
that any remaining bands can't possibly contain a worse window.

This slows down MMU construction for small traces, but these are
reasonably fast to compute either way. For large traces (e.g.,
150,000+ utilization changes) it's significantly faster.

Change-Id: Ie66454e71f3fb06be3f6173b6d91ad75c61bda48
Reviewed-on: https://go-review.googlesource.com/c/60792
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
2018-11-05 19:10:09 +00:00
Austin Clements
c6c602a926 internal/trace: use MU slope to optimize MMU
This commit speeds up MMU construction by ~10X (and reduces the number
of windows considered by ~20X) by using an observation about the
maximum slope of the windowed mutator utilization function to advance
the window time in jumps if the window's current mean mutator
utilization is much larger than the current minimum.

Change-Id: If3cba5da0c4adc37b568740f940793e491e96a51
Reviewed-on: https://go-review.googlesource.com/c/60791
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
2018-11-05 19:10:06 +00:00
Austin Clements
3053788cac cmd/trace: add minimum mutator utilization (MMU) plot
This adds an endpoint to the trace tool that plots the minimum mutator
utilization curve using information on mark assists and GC pauses from
the trace.

This commit implements a fairly straightforward O(nm) algorithm for
computing the MMU (and tests against an even more direct but slower
algorithm). Future commits will extend and optimize this algorithm.

This should be useful for debugging and understanding mutator
utilization issues like #14951, #14812, #18155. #18534, #21107,
particularly once follow-up CLs add trace cross-referencing.

Change-Id: Ic2866869e7da1e6c56ba3e809abbcb2eb9c4923a
Reviewed-on: https://go-review.googlesource.com/c/60790
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
2018-11-05 19:10:04 +00:00
Brad Fitzpatrick
3813edf26e all: use "reports whether" consistently in the few places that didn't
Go documentation style for boolean funcs is to say:

    // Foo reports whether ...
    func Foo() bool

(rather than "returns true if")

This CL also replaces 4 uses of "iff" with the same "reports whether"
wording, which doesn't lose any meaning, and will prevent people from
sending typo fixes when they don't realize it's "if and only if". In
the past I think we've had the typo CLs updated to just say "reports
whether". So do them all at once.

(Inspired by the addition of another "returns true if" in CL 146938
in fd_plan9.go)

Created with:

$ perl -i -npe 's/returns true if/reports whether/' $(git grep -l "returns true iff" | grep -v vendor)
$ perl -i -npe 's/returns true if/reports whether/' $(git grep -l "returns true if" | grep -v vendor)

Change-Id: Ided502237f5ab0d25cb625dbab12529c361a8b9f
Reviewed-on: https://go-review.googlesource.com/c/147037
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-11-02 22:47:58 +00:00
Bryan C. Mills
3a0d6091d6 internal/syscall/windows: add LockFileEx and UnlockFileEx for use in cmd/go
Updates #26794

Change-Id: Ic1d3078176721f3d2e5d8188c234383037babbaf
Reviewed-on: https://go-review.googlesource.com/c/145177
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
2018-11-02 15:56:35 +00:00
Clément Chigot
5ee06f5471 internal/poll, os/exec, runtime: replace PollDescriptor by IsPollDescriptor
This commit changes poll.PollDescriptor by poll.IsPollDescriptor. This
is needed for OS like AIX which have more than one FD using inside their
netpoll implementation.

Change-Id: I49e12a8d74045c501e19fdd8527cf166a3c64850
Reviewed-on: https://go-review.googlesource.com/c/146938
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-11-02 13:43:23 +00:00
Tobias Klauser
cdad408069 os: add support for long path names on aix RemoveAll
Follow CL 146020 and enable RemoveAll based on Unlinkat and Openat on
aix.

Updates #27029

Change-Id: I78b34ed671166ee6fa651d5f2025b88548ee6c68
Reviewed-on: https://go-review.googlesource.com/c/146937
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Clément Chigot <clement.chigot@atos.net>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-11-02 13:21:12 +00:00
Dmitry Vyukov
21f7f01289 runtime: avoid runtimeNano call on a common netpoll path
runtimeNano is slower than nanotime, so pass the duration
to runtime_pollSetDeadline as is. netpoll can add nanotime itself.
Arguably a bit simpler because, say, a negative duration
clearly represents already expired timer, no need to compare to
nanotime again.
This may also fix an obscure corner case when a deadline in past
which happens to be nanotime 0 is confused with no deadline at all,
which are radically different things.
Also don't compute any durations and times if Time is zero
(currently we first compute everything and then reset d back to 0,
which is wasteful).

name                  old time/op  new time/op  delta
TCP4OneShotTimeout-6  17.1µs ± 0%  17.0µs ± 0%     ~     (p=0.421 n=5+5)
SetReadDeadline-6      230ns ± 0%   205ns ± 1%  -10.63%  (p=0.008 n=5+5)

Change-Id: I2aad699270289a5b9ead68f5e44ec4ec6d96baa0
Reviewed-on: https://go-review.googlesource.com/c/146344
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Dmitry Vyukov <dvyukov@google.com>
2018-11-02 12:55:58 +00:00
Dmitry Vyukov
ff51353c38 runtime: move nanotime wrappers to time and poll packages
The nanotime wrappers in runtime introduce a bunch
of unnecessary code onto hot paths, e.g.:

0000000000449d70 <time.runtimeNano>:
  449d70:       64 48 8b 0c 25 f8 ff    mov    %fs:0xfffffffffffffff8,%rcx
  449d77:       ff ff
  449d79:       48 3b 61 10             cmp    0x10(%rcx),%rsp
  449d7d:       76 26                   jbe    449da5 <time.runtimeNano+0x35>
  449d7f:       48 83 ec 10             sub    $0x10,%rsp
  449d83:       48 89 6c 24 08          mov    %rbp,0x8(%rsp)
  449d88:       48 8d 6c 24 08          lea    0x8(%rsp),%rbp
  449d8d:       e8 ae 18 01 00          callq  45b640 <runtime.nanotime>
  449d92:       48 8b 04 24             mov    (%rsp),%rax
  449d96:       48 89 44 24 18          mov    %rax,0x18(%rsp)
  449d9b:       48 8b 6c 24 08          mov    0x8(%rsp),%rbp
  449da0:       48 83 c4 10             add    $0x10,%rsp
  449da4:       c3                      retq
  449da5:       e8 56 e0 00 00          callq  457e00 <runtime.morestack_noctxt>
  449daa:       eb c4                   jmp    449d70 <time.runtimeNano>

Move them to the corresponding packages which eliminates all of this.

name                  old time/op  new time/op  delta
TCP4OneShotTimeout-6  17.1µs ± 1%  17.0µs ± 0%  -0.66%  (p=0.032 n=5+5)
SetReadDeadline-6      234ns ± 1%   232ns ± 0%  -0.77%  (p=0.016 n=5+4)

Update #25729

Change-Id: Iee05027adcdc289ba895c5f5a37f154e451bc862
Reviewed-on: https://go-review.googlesource.com/c/146342
Run-TryBot: Dmitry Vyukov <dvyukov@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-11-02 12:52:30 +00:00
Alex Brainman
f10815898c os: use CreateFile for Stat of symlinks
Stat uses Windows FindFirstFile + CreateFile to gather symlink
information - FindFirstFile determines if file is a symlink,
and then CreateFile follows symlink to capture target details.

Lstat only uses FindFirstFile.

This CL replaces current approach with just a call to CreateFile.
Lstat uses FILE_FLAG_OPEN_REPARSE_POINT flag, that instructs
CreateFile not to follow symlink. Other than that both Stat and
Lstat look the same now. New code is simpler.

CreateFile + GetFileInformationByHandle (unlike FindFirstFile)
does not report reparse tag of a file. I tried to ignore reparse
tag altogether. And it works for symlinks and mount points.
Unfortunately (see https://github.com/moby/moby/issues/37026),
files on deduped disk volumes are reported with
FILE_ATTRIBUTE_REPARSE_POINT attribute set and reparse tag set
to IO_REPARSE_TAG_DEDUP. So, if we ignore reparse tag, Lstat
interprets deduped volume files as symlinks. That is incorrect.
So I had to add GetFileInformationByHandleEx call to gather
reparse tag after calling CreateFile and GetFileInformationByHandle.

Fixes #27225
Fixes #27515

Change-Id: If60233bcf18836c147597cc17450d82f3f88c623
Reviewed-on: https://go-review.googlesource.com/c/143578
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Kirill Kolyshkin <kolyshkin@gmail.com>
2018-11-02 07:30:03 +00:00
Tobias Klauser
fa466a4f5a os: add support for long path names on freebsd RemoveAll
Follow CL 146020 and enable RemoveAll based on Unlinkat and Openat on
freebsd.

Since the layout of syscall.Stat_t changes in FreeBSD 12, Fstatat needs
a compatibility wrapper akin to Fstatat in x/sys/unix. See CL 138595 and
CL 136816 for details.

Updates #27029

Change-Id: I8851a5b7fa658eaa6e69a1693150b16d9a68f36a
Reviewed-on: https://go-review.googlesource.com/c/146597
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-11-01 17:08:32 +00:00
Tobias Klauser
4d1e95bb63 os: add support for long path names on solaris RemoveAll
Follow CL 146020 and enable RemoveAll based on Unlinkat and Openat on
solaris.

Updates #27029

Change-Id: I0b0e92f4422fa960a13dcd3e9adb57cd23f09ed4
Reviewed-on: https://go-review.googlesource.com/c/145839
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-10-31 16:52:45 +00:00
Clément Chigot
cb07f492db internal/cpu, runtime: add CPU feature detection support for AIX
AIX doesn't have HWCAP/HWCAP2 variables like Linux. Therefore, it relies on
getsystemcfg syscall which can provide some information about the CPU.

Change-Id: Ic0dc927e80890d4bf8f0bdfb43fad1e2b890d7a0
Reviewed-on: https://go-review.googlesource.com/c/144959
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Martin Möhrmann <moehrmann@google.com>
2018-10-31 16:11:20 +00:00
Oliver Stenbom
e8ffb8a74c os: add support for long path names on unix RemoveAll
On unix systems, long enough path names will fail when performing syscalls
like `Lstat`. The current RemoveAll uses several of these syscalls, and so
will fail for long paths. This can be risky, as it can let users "hide"
files from the system or otherwise make long enough paths for programs
to fail. By using `Unlinkat` and `Openat` syscalls instead, RemoveAll is
safer on unix systems. Initially implemented for linux, darwin, dragonfly,
netbsd and openbsd. Not yet implemented on freebsd due to fstatat 64-bit
inode compatibility issues.

Fixes #27029

Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io>
Co-authored-by: Julia Nedialkova <yulia.nedyalkova@sap.com>

Change-Id: I978a6a4986878fe076d3c7af86e7927675624a96
GitHub-Last-Rev: 9235489c81
GitHub-Pull-Request: golang/go#28494
Reviewed-on: https://go-review.googlesource.com/c/146020
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-10-31 13:55:01 +00:00