Document that *os.File is subject to resource limits
for concurrent operations. We aren't documenting
a specific number of concurrent operations because that
number is OS/system dependent. This limit comes from:
internal/poll/fd_mutex.go
where we use 20 bits to count locks.
Fixes#32544
Change-Id: I7d305d4aaba5b2dbc6f1ab8c447117fde5e31a66
Reviewed-on: https://go-review.googlesource.com/c/go/+/181841
Reviewed-by: Rob Pike <r@golang.org>
The old code used ~/Library/Preferences, which is documented by
Apple as:
This directory contains app-specific preference files. You
should not create files in this directory yourself. Instead, use
the NSUserDefaults class or CFPreferences API to get and set
preference values for your app.
It looks like we missed everything after the first sentence; it's
definitely not the right choice for files that Go programs and users
should be touching directly.
Instead, use ~/Library/Application Support, which is documented as:
Use this directory to store all app data files except those
associated with the user’s documents. For example, you might use
this directory to store app-created data files, configuration
files, templates, or other fixed or modifiable resources that
are managed by the app. An app might use this directory to store
a modifiable copy of resources contained initially in the app’s
bundle. A game might use this directory to store new levels
purchased by the user and downloaded from a server.
This seems in line with what UserConfigDir is for, so use it.
The documentation quotes above are obtained from the surprisingly long
link below:
https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.htmlFixes#32475.
Change-Id: Ic27a6c92d76a5d7a4d4b8eac5cd8472f67a533a4
Reviewed-on: https://go-review.googlesource.com/c/go/+/181177
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
On unix if exec.Command() is given both ExtraFiles and Ctty, and the
Ctty file descriptor overlaps the range of FDs intended for the child,
then cmd.Start() the ioctl(fd,TIOCSCTTY) call fails with an
"inappropriate ioctl for device" error.
When child file descriptors overlap the new child's ctty the ctty will
be closed in the fd shuffle before the TIOCSCTTY. Thus TIOCSCTTY is
used on one of the ExtraFiles rather than the intended Ctty file. Thus
the error.
exec.Command() callers can workaround this by ensuring the Ctty fd is
larger than any ExtraFiles destined for the child.
Fix this by doing the ctty ioctl before the fd shuffle.
Test for this issue by modifying TestTerminalSignal to use more
ExtraFiles. The test fails on linux and freebsd without this change's
syscall/*.go changes. Other platforms (e.g. darwin, aix, solaris) have
the same fd shuffle logic, so the same fix is applied to them. However,
I was only able to test on linux (32 and 64 bit) and freebsd (64 bit).
Manual runs of the test in https://golang.org/issue/29458 start passing
with this patch:
Before:
% /tmp/src/go/bin/go run t
successfully ran child process with ParentExtraFileFdNum=5, ChildExtraFileFd=6, ParentPtyFd=7
panic: failed to run child process with ParentExtraFileFdNum=10, ChildExtraFileFd=11, ParentPtyFd=11: fork/exec /bin/true: inappropriate ioctl for device
After:
% /tmp/src/go/bin/go run t
successfully ran child process with ParentExtraFileFdNum=5, ChildExtraFileFd=6, ParentPtyFd=7
successfully ran child process with ParentExtraFileFdNum=10, ChildExtraFileFd=11, ParentPtyFd=11
Fixes#29458
Change-Id: I99513de7b6073c7eb855f1eeb4d1f9dc0454ef8b
Reviewed-on: https://go-review.googlesource.com/c/go/+/178919
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Shorten some of the longest tests that run during all.bash.
Removes 7r 50u 21s from all.bash.
After this change, all.bash is under 5 minutes again on my laptop.
For #26473.
Change-Id: Ie0460aa935808d65460408feaed210fbaa1d5d79
Reviewed-on: https://go-review.googlesource.com/c/go/+/177559
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
This is CVE-2019-11888.
Previously, passing a nil environment but a non-nil token would result
in the new potentially unprivileged process inheriting the parent
potentially privileged environment, or would result in the new
potentially privileged process inheriting the parent potentially
unprivileged environment. Either way, it's bad. In the former case, it's
an infoleak. In the latter case, it's a possible EoP, since things like
PATH could be overwritten.
Not specifying an environment currently means, "use the existing
environment". This commit amends the behavior to be, "use the existing
environment of the token the process is being created for." The behavior
therefore stays the same when creating processes without specifying a
token. And it does the correct thing when creating processes when
specifying a token.
Fixes#32000
Change-Id: Ia57f6e89b97bdbaf7274d6a89c1d9948b6d40ef5
Reviewed-on: https://go-review.googlesource.com/c/go/+/176619
Run-TryBot: Jason Donenfeld <Jason@zx2c4.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
Add Unwrap methods to types which wrap an underlying error:
"encodinc/csv".ParseError
"encoding/json".MarshalerError
"net/http".transportReadFromServerError
"net".OpError
"net".DNSConfigError
"net/url".Error
"os/exec".Error
"signal/internal/pty".PtyError
"text/template".ExecError
Add os.ErrTemporary. A case could be made for putting this error
value in package net, since no exported error types in package os
include a Temporary method. However, syscall errors returned from
the os package do include this method.
Add Is methods to error types with a Timeout or Temporary method,
making errors.Is(err, os.Err{Timeout,Temporary}) equivalent to
testing the corresponding method:
"context".DeadlineExceeded
"internal/poll".TimeoutError
"net".adrinfoErrno
"net".OpError
"net".DNSError
"net/http".httpError
"net/http".tlsHandshakeTimeoutError
"net/pipe".timeoutError
"net/url".Error
Updates #30322
Updates #29934
Change-Id: I409fb20c072ea39116ebfb8c7534d493483870dc
Reviewed-on: https://go-review.googlesource.com/c/go/+/170037
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
Like GOOS=android which implies the "linux" build tag, GOOS=illumos
implies the "solaris" build tag. This lets the existing ecosystem of
packages still work on illumos, but still permits packages to start
differentiating between solaris and illumos.
Fixes#20603
Change-Id: I8f4eabf1a66060538dca15d7658c1fbc6c826622
Reviewed-on: https://go-review.googlesource.com/c/go/+/174457
Run-TryBot: Benny Siegert <bsiegert@gmail.com>
Reviewed-by: Benny Siegert <bsiegert@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Fixes#20858
Change-Id: I45c397795426aaa276b20f5cbeb80270c95b920c
Reviewed-on: https://go-review.googlesource.com/c/go/+/174320
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Follow up CL 156379.
Updates #19093
Change-Id: I5ea3177fc5911d3af71cbb32584249e419e9d4a3
Reviewed-on: https://go-review.googlesource.com/c/go/+/172937
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Though there is variation in the spelling of canceled,
cancellation is always spelled with a double l.
Reference: https://www.grammarly.com/blog/canceled-vs-cancelled/
Change-Id: I240f1a297776c8e27e74f3eca566d2bc4c856f2f
Reviewed-on: https://go-review.googlesource.com/c/go/+/170060
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The code to swap RemoveAllTestHook in and out in
TestRemoveAllWithMoreErrorThanReqSize was making a copy of the
RemoveAllTestHook pointer, then attempting to restore by loading from
the copy of that pointer. Since the two copies of the pointer aliased
the same address, the restore operation had no effect, and any
RemoveAll tests that happened to run after
TestRemoveAllWithMoreErrorThanReqSize would fail.
Fixes#31421
Change-Id: I7028475f5ceb3b0a2fa69d22af8d3379508c4531
Reviewed-on: https://go-review.googlesource.com/c/go/+/171777
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
golang.org/cl/121255 added close and re-open the directory when looping, prevent
us from missing some if previous iteration deleted files.
The CL introdued a bug. If we can not delete all entries in one request,
the looping never exits, causing RemoveAll hangs.
To fix that, simply discard the entries if we can not delete all of them
in one iteration, then continue reading entries and delete them.
Also make sure removeall_at return first error it encounters.
Fixes#29921
Change-Id: I8ec3a4c822d8d2d95d9f1ab71547879da395bc4a
Reviewed-on: https://go-review.googlesource.com/c/go/+/171099
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The underlying system call tested by TestCredentialNoSetGroups
is blocked on Android.
Discovered while running all.bash from an Android device; the syscall
is only blocked in an app context.
Change-Id: I16fd2e64636a0958b0ec86820723c0577b8f8f24
Reviewed-on: https://go-review.googlesource.com/c/go/+/170945
Run-TryBot: Elias Naur <mail@eliasnaur.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
It's not supported in an app context:
$ go test -short os
--- FAIL: TestChdirAndGetwd (0.00s)
os_test.go:1213: Open /: open /: permission denied
Change-Id: I56b951f925a50fd67715ee2f1de64951ee867e91
Reviewed-on: https://go-review.googlesource.com/c/go/+/170946
Run-TryBot: Elias Naur <mail@eliasnaur.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Getdirentries is implemented with the __getdirentries64 function
in libSystem.dylib. That function works, but it's on Apple's
can't-be-used-in-an-app-store-application list.
Implement Getdirentries using the underlying fdopendir/readdir_r/closedir.
The simulation isn't faithful, and could be slow, but it should handle
common cases.
Don't use Getdirentries in the stdlib, use fdopendir/readdir_r/closedir
instead (via (*os.File).readdirnames).
Fixes#30933
Update #28984
RELNOTE=yes
Change-Id: Ia6b5d003e5bfe43ba54b1e1d9cfa792cc6511717
Reviewed-on: https://go-review.googlesource.com/c/go/+/168479
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Using os.UserHomeDir for user.HomeDir helps us deduplicate the
logic and keep the behavior consistent.
Also make os.UserHomeDir return "/sdcard" in android.
See: https://go-review.googlesource.com/c/go/+/37960/1/src/os/user/lookup_stubs.go#48Fixes#31070
Change-Id: I521bad050bc5761ecc5c0085501374d2cf8e6897
Reviewed-on: https://go-review.googlesource.com/c/go/+/169540
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
WriteAt use pwrite syscall on *nix or WriteFile on Windows.
On Linux/Windows, these system calls always write to end of file in
append mode, regardless of offset parameter.
It is hard (maybe impossible) to make WriteAt work portably.
Making WriteAt returns an error if file is opened in append mode, we
guarantee to get consistent behavior between platforms, also prevent
user from accidently corrupting their data.
Fixes#30716
Change-Id: If83d935a22a29eed2ff8fe53d13d0b4798aa2b81
Reviewed-on: https://go-review.googlesource.com/c/go/+/166578
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
As proposed in Issue #29934, update errors produced by the os package to
work with errors.Is sentinel tests. For example,
errors.Is(err, os.ErrPermission) is equivalent to os.IsPermission(err)
with added unwrapping support.
Move the definition for os.ErrPermission and others into the syscall
package. Add an Is method to syscall.Errno and others. Add an Unwrap
method to os.PathError and others.
Updates #30322
Updates #29934
Change-Id: I95727d26c18a5354c720de316dff0bffc04dd926
Reviewed-on: https://go-review.googlesource.com/c/go/+/163058
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
Add an Unwrap method to PathError so it works with the errors.Is/As
functions.
Change-Id: Ia6171c0418584f3cd53ee99d97c687941a9e3109
Reviewed-on: https://go-review.googlesource.com/c/go/+/168097
Reviewed-by: Damien Neil <dneil@google.com>
On Windows, GetFileAttributesEx fails with ERROR_SHARING_VIOLATION for
some files, like c:\pagefile.sys. In this case,
newFileStatFromWin32finddata is used to fill file info, but it does not fill
name and path.
After getting file stat from newFileStatFromWin32finddata, just set file info
name and path before return fixes the issue.
Fixes#30883
Change-Id: I654e96c634e8a9bf5ce7e1aaa93968e88953620d
Reviewed-on: https://go-review.googlesource.com/c/go/+/167779
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
When closing a pipe, use CancelIoEx to cancel pending I/O.
This makes concurrent Read and Write calls return os.ErrClosed.
This change also enables some pipe tests on Windows.
Fixes#28477Fixes#25835
Change-Id: If52bb7d80895763488a61632e4682a78336e8420
Reviewed-on: https://go-review.googlesource.com/c/go/+/164721
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
UserHomeDir always returns "/" for platforms where the home directory
is not always well defined. However, the user might set HOME before
running a Go program on those platforms and on at least iOS, HOME
is actually set to something useful (the root of the app specific
writable directory).
This CL changes UserHomeDir to use the root directory "/" only if
$HOME is empty.
Change-Id: Icaa01de53cd585d527d9a23b1629375d6b7f67e9
Reviewed-on: https://go-review.googlesource.com/c/go/+/167802
Run-TryBot: Elias Naur <mail@eliasnaur.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Support for FreeBSD 10 will be dropped with Go 1.13, so revert the
workaround introduced in CL 157099.
Updates #29633
Updates #27619
Change-Id: I1a2e50d3f807a411389f3db07c0f4535a590da02
Reviewed-on: https://go-review.googlesource.com/c/go/+/165801
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Before this change, if a directory was closed twice on Windows,
the returning error would be "use of closed network connection".
Some code assumes FD.isFile means whether the fd isn't a network
socket, which is true on Unix. But isFile reports whether
the fd is a normal file rather than directory or console on Windows.
With this change, isFile will have the same meaning on different
platforms. And the change adds a new field kind to replace isConsole
and isDir.
Change-Id: Ib12265f1e12fa3d0239ae925291128a84be59cc2
GitHub-Last-Rev: 3f031756de
GitHub-Pull-Request: golang/go#30589
Reviewed-on: https://go-review.googlesource.com/c/go/+/165257
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
After UserCacheDir and UserHomeDir, the only remaining piece which is
commonly needed and portable is a per-user directory to store persistent
files.
For that purpose, UserCacheDir is wrong, as it's meant only for
temporary files. UserHomeDir is also far from ideal, as that clutters
the user's home directory.
Add UserConfigDir, which is implemented in a similar manner to
UserConfigDir.
Fixes#29960.
Change-Id: I7d7a56615103cf76e2b5e2bab2029a6b09d19f0b
Reviewed-on: https://go-review.googlesource.com/c/go/+/160877
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
When Stdin, Stdout, and Stderr are nil, there are no goroutines to keep
track of, so we don't need a channel.
And in startProcess, preallocate the right size for sysattr.Files,
saving a bit of space and a couple of slice growth allocs.
name old time/op new time/op delta
ExecHostname-8 419µs ± 0% 417µs ± 1% ~ (p=0.093 n=6+6)
name old alloc/op new alloc/op delta
ExecHostname-8 6.40kB ± 0% 6.28kB ± 0% -1.86% (p=0.002 n=6+6)
name old allocs/op new allocs/op delta
ExecHostname-8 34.0 ± 0% 31.0 ± 0% -8.82% (p=0.002 n=6+6)
Change-Id: Ic1d617f29e9c6431cdcadc7f9bb992750a6d5f48
Reviewed-on: https://go-review.googlesource.com/c/164801
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Most notably, it's missing on Windows machines. For example,
windows-amd64-race started failing consistently:
--- FAIL: BenchmarkExecEcho
bench_test.go:15: could not find echo: exec: "echo": executable file not found in %PATH%
We can also reproduce this from Linux with Wine:
$ GOOS=windows go test -bench=. -benchtime=1x -run=- -exec wine
--- FAIL: BenchmarkExecEcho
bench_test.go:15: could not find echo: exec: "echo": executable file not found in %PATH%
Instead, use the "hostname" program, which is available on Windows too.
Interestingly enough, it's also slightly faster than "echo". Any program
is fine as long as it's close enough to a no-op, though.
name old time/op new time/op delta
ExecEcho-8 422µs ± 0% 395µs ± 0% -6.39% (p=0.004 n=6+5)
name old alloc/op new alloc/op delta
ExecEcho-8 6.39kB ± 0% 6.42kB ± 0% +0.53% (p=0.002 n=6+6)
name old allocs/op new allocs/op delta
ExecEcho-8 36.0 ± 0% 36.0 ± 0% ~ (all equal)
Change-Id: I772864d69979172b5cf807552c84d0e165e73051
Reviewed-on: https://go-review.googlesource.com/c/164704
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
We're always going to add stdin, stdout, and stderr to childFiles, so
its length will be at least three. The final length will be those three
elements plus however many files were given via ExtraFiles.
Allocate for that final length directly, saving two slice growth allocs
in the common case where ExtraFiles is empty.
name old time/op new time/op delta
ExecEcho-8 435µs ± 0% 435µs ± 0% ~ (p=0.394 n=6+6)
name old alloc/op new alloc/op delta
ExecEcho-8 6.39kB ± 0% 6.37kB ± 0% -0.39% (p=0.002 n=6+6)
name old allocs/op new allocs/op delta
ExecEcho-8 36.0 ± 0% 34.0 ± 0% -5.56% (p=0.002 n=6+6)
Change-Id: Ib702c0da1e43f0a55ed937af6d45fca6a170e8f3
Reviewed-on: https://go-review.googlesource.com/c/164898
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The common case is that most env vars are distinct;
optimize for that.
name old time/op new time/op delta
ExecEcho-8 2.16ms ± 3% 2.14ms ± 1% ~ (p=0.315 n=10+10)
name old alloc/op new alloc/op delta
ExecEcho-8 7.87kB ± 0% 6.35kB ± 0% -19.31% (p=0.000 n=9+10)
name old allocs/op new allocs/op delta
ExecEcho-8 72.0 ± 0% 69.0 ± 0% -4.17% (p=0.000 n=10+10)
Change-Id: I42bb696c6862f2ea12c5cbd2f24c64336a7a759a
Reviewed-on: https://go-review.googlesource.com/c/164960
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
windows-arm TMP directory live inside such link (see
https://github.com/golang/go/issues/29746#issuecomment-456526811 for
details), so symlinks like that will be common at least on windows-arm.
This CL builds on current syscall.Readlink implementation. Main
difference between the two is how new code handles symlink targets,
like \??\Volume{ABCD}\.
New implementation uses Windows CreateFile API with
FILE_FLAG_OPEN_REPARSE_POINT flag to get \??\Volume{ABCD}\ file handle.
And then it uses Windows GetFinalPathNameByHandle with VOLUME_NAME_DOS
flag to convert that handle into standard Windows path.
FILE_FLAG_OPEN_REPARSE_POINT flag ensures that symlink is not followed
when CreateFile opens the file.
Fixes#30463
Change-Id: I33b18227ce36144caed694169ef2e429fd995fb4
Reviewed-on: https://go-review.googlesource.com/c/164201
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
