The Transport's writer to the remote server is wrapped in a
bufio.Writer to suppress many small writes while writing headers and
trailers. However, when writing the request body, the buffering may get
in the way if the request body is arriving slowly.
Because the io.Copy from the Request.Body to the writer is already
buffered, the outer bufio.Writer is unnecessary and prevents small
Request.Body.Reads from going to the server right away. (and the
io.Reader contract does say to return when you've got something,
instead of blocking waiting for more). After the body is finished, the
Transport's bufio.Writer is still used for any trailers following.
A previous attempted fix for this made the chunk writer always flush
if the underlying type was a bufio.Writer, but that is not quite
correct. This CL instead makes it opt-in by using a private sentinel
type (wrapping a *bufio.Writer) to the chunk writer that requests
Flushes after each chunk body (the chunk header & chunk body are still
buffered together into one write).
Fixes#6574
Change-Id: Icefcdf17130c9e285c80b69af295bfd3e72c3a70
Reviewed-on: https://go-review.googlesource.com/10021
Reviewed-by: Andrew Gerrand <adg@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
When running the client header timeout test, there is a race between
us timing out and waiting on the remaining requests to be serviced. If
the client times out before the server blocks on the channel in the
handler, we will be simultaneously adding to a waitgroup with the
value 0 and waiting on it when we call TestServer.Close().
This is largely a theoretical race. We have to time out before we
enter the handler and the only reason we would time out if we're
blocked on the channel. Nevertheless, make the race detector happy
by turning the close into a channel send. This turns the defer call
into a synchronization point and we can be sure that we've entered
the handler before we close the server.
Fixes#10780
Change-Id: Id73b017d1eb7503e446aa51538712ef49f2f5c9e
Reviewed-on: https://go-review.googlesource.com/9905
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The test was measuring something, assuming other goroutines had
already scheduled.
Fixes#10427
Change-Id: I2a4d3906f9d4b5ea44b57d972e303bbe2b0b1cde
Reviewed-on: https://go-review.googlesource.com/9561
Reviewed-by: David Crawshaw <crawshaw@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Fixes an issue where Response.Write writes out a Content-Length: -1
header when the corresponding Request is a POST or PUT and the
ContentLength was not previously set.
This was encountered when using httputil.DumpResponse
to write out the response from a server that responded to a PUT
request with no Content-Length header. The dumped output is
thus invalid.
Change-Id: I52c6ae8ef3443f1f9de92aeee9f9581dabb05991
Reviewed-on: https://go-review.googlesource.com/9496
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Fixes#10366 (how to set custom headers)
Fixes#9836 (PATCH in PostForm)
Fixes#9276 (generating a server-side Request for testing)
Update #8991 (clarify Response.Write for now; export ReverseProxy's copy later?)
Change-Id: I95a11bf3bb3eeeeb72775b6ebfbc761641addc35
Reviewed-on: https://go-review.googlesource.com/9410
Reviewed-by: David Crawshaw <crawshaw@golang.org>
In the brief window between getConn and persistConn.roundTrip,
a cancel could end up going missing.
Fix by making it possible to inspect if a cancel function was cleared
and checking if we were canceled before entering roundTrip.
Fixes#10511
Change-Id: If6513e63fbc2edb703e36d6356ccc95a1dc33144
Reviewed-on: https://go-review.googlesource.com/9181
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Previously all errors were 404 errors, even if the real error had
nothing to do with a file being non-existent.
Fixes#10283
Change-Id: I5b08b471a9064c347510cfcf8557373704eef7c0
Reviewed-on: https://go-review.googlesource.com/9200
Reviewed-by: Daniel Morsing <daniel.morsing@gmail.com>
There used to be a small window where if a server declared it would do
a keep-alive connection but then actually closed the connection before
the roundTrip goroutine scheduled after being sent a response from the
readLoop goroutine, then the readLoop goroutine would loop around and
block forever reading from a channel because the numExpectedResponses
accounting was done too late.
Fixes#10457
Change-Id: Icbae937ffe83c792c295b7f4fb929c6a24a4f759
Reviewed-on: https://go-review.googlesource.com/9169
Reviewed-by: Daniel Morsing <daniel.morsing@gmail.com>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Just like darwin/arm.
Change-Id: Ib9a32bb0aed5f08b27de11a93aaf273cacdf5779
Reviewed-on: https://go-review.googlesource.com/8819
Reviewed-by: Minux Ma <minux@golang.org>
There was a logical race in Transport.RoundTrip where a roundtrip with
a pending response would race with the channel for the connection
closing. This usually happened for responses with connection: close
and no body.
We handled this race by reading the close channel, setting a timer
for 100ms and if no response was returned before then, we would then
return an error.
This put a lower bound on how fast a connection could fail. We couldn't
fail a request faster than 100ms.
Reordering the channel operations gets rid of the logical race. If
the readLoop causes the connection to be closed, it would have put
its response into the return channel already and we can fetch it with
a non-blocking receive.
Change-Id: Idf09e48d7a0453d7de0120d3055d0ce5893a5428
Reviewed-on: https://go-review.googlesource.com/1787
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
When making a request to an IPv6 address with a zone identifier, for
exmaple [fe80::1%en0], RFC 6874 says HTTP clients must remove the zone
identifier "%en0" before writing the request for security reason.
This change removes any IPv6 zone identifer attached to URI in the Host
header field in requests.
Fixes#9544.
Change-Id: I7406bd0aa961d260d96f1f887c2e45854e921452
Reviewed-on: https://go-review.googlesource.com/3111
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
ServeContent ignored zero time.Time{} values when generating
Last-Modified response headers and checking If-Modified-Since request
headers. Do the same for a time.Time representing the Unix epoch zero
value, as this is a common bogus value. Callers who really want to
send that value (incredibly unlikely) can add a nanosecond to it and
it will be truncated to second granularity anyway.
Fixes#9842
Change-Id: I69f697bfc4017404a92a34e3fe57e2711c1e299d
Reviewed-on: https://go-review.googlesource.com/7915
Reviewed-by: David Symonds <dsymonds@golang.org>
The body tag in the pprof template was misplaced.
Change-Id: Icd7948b358f52df1acc7e033ab27a062990ef977
Reviewed-on: https://go-review.googlesource.com/7795
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This allows /debug/pprof/ and descendents to be used through
http.StripPrefix and other path rewriting handlers.
Change-Id: I53673876c107bbfaf430123ead78e6524b42ac21
Reviewed-on: https://go-review.googlesource.com/7351
Reviewed-by: Andrew Gerrand <adg@golang.org>
Currently the test fails if run more than once:
$ go test -v -run=TestChildServeCleansUp -cpu=1,1 net/http/fcgi
=== RUN TestChildServeCleansUp
--- PASS: TestChildServeCleansUp (0.00s)
=== RUN TestChildServeCleansUp
fatal error: all goroutines are asleep - deadlock!
The problem is that the writer mutates test input data,
so it is wrong on the second execution.
Change-Id: I4ca54dd2926c6986b2908023ac65e5e65630ed26
Reviewed-on: https://go-review.googlesource.com/6383
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Change-Id: I2fc3cf94b465bf9d7ff8d7bf935b45e334b401e3
Reviewed-on: https://go-review.googlesource.com/6180
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
Env vars were incorrectly copying whole value of http.RemoteAddr
to REMOTE_ADDR and REMOTE_HOST. They contained IP:port pair which
instead should only have IP (RFC 3875, other sources).
Module also was not setting REMOTE_PORT variable which become de-facto
standard for passing TCP client port to CGI scripts (Apache mod_cgi,
IIS, and probably others)
Fixes#9861
Change-Id: Ia73e664c48539e3c7db4997d09d957884e98d8a5
Reviewed-on: https://go-review.googlesource.com/4933
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
I think the test was meant to test requests to 'localhost:80' instead
of 'localhost:80:80'. It passes even with 'localhost:80:80' because
net.SplitHostPort fails inside useProxy. Please comment if you want to
leave old 'localhost:80' is the list too to check old code path.
Change-Id: Ic4cd21901563449e3d4e2f4c8caf723f4ca15bac
u
Reviewed-on: https://go-review.googlesource.com/4293
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Only documentation / comment changes. Update references to
point to golang.org permalinks or go.googlesource.com/go.
References in historical release notes under doc are left as is.
Change-Id: Icfc14e4998723e2c2d48f9877a91c5abef6794ea
Reviewed-on: https://go-review.googlesource.com/4060
Reviewed-by: Ian Lance Taylor <iant@golang.org>
We already checked for the prefix with strings.HasPrefix
Change-Id: I33852fd19ffa92aa33b75b94b4bb505f4043a54a
Reviewed-on: https://go-review.googlesource.com/3691
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This should fix the race builders.
Change-Id: I9c9e7393d5e29d64ab797e346b34b1fa1dfe6d96
Reviewed-on: https://go-review.googlesource.com/3441
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
BenchmarkClient is intended for profiling
the client without the HTTP server code.
The server code runs in a subprocess.
Change-Id: I9aa128604d0d4e94dc5c0372dc86f962282ed6e8
Reviewed-on: https://go-review.googlesource.com/3164
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Close the pipe for the body of a request when it is aborted and close
all pipes when child.serve terminates.
Fixes#6934
Change-Id: I1c5e7d2116e1ff106f11a1ef8e99bf70cf04162a
Reviewed-on: https://go-review.googlesource.com/1923
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Cygwin perl uses unix pathnames in windows. Include cygwin perl in the
list of special cases for unix pathname handling in test.cgi.
Change-Id: I30445a9cc79d62d022ecc232c35aa5015b7418dc
Reviewed-on: https://go-review.googlesource.com/2973
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
Skip the allocation testing (which was only used as a signal for
whether the interface was implemented by ResponseWriter), and just
test for it directly.
Fixes#9575
Change-Id: Ie230f1d21b104537d5647e9c900a81509d692469
Reviewed-on: https://go-review.googlesource.com/2720
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
If an inbound connection is closed, cancel the outbound http request.
This is particularly useful if the outbound request may consume resources
unnecessarily until it is cancelled.
Fixes#8406
Change-Id: I738c4489186ce342f7e21d0ea3f529722c5b443a
Signed-off-by: Peter Waller <p@pwaller.net>
Reviewed-on: https://go-review.googlesource.com/2320
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reader.Discard is the complement to Peek. It discards the next n bytes
of input.
We already have Reader.Buffered to see how many bytes of data are
sitting available in memory, and Reader.Peek to get that that buffer
directly. But once you're done with the Peek'd data, you can't get rid
of it, other than Reading it.
Both Read and io.CopyN(ioutil.Discard, bufReader, N) are relatively
slow. People instead resort to multiple blind ReadByte calls, just to
advance the internal b.r variable.
I've wanted this previously, several people have asked for it in the
past on golang-nuts/dev, and somebody just asked me for it again in a
private email. There are a few places in the standard library we'd use
it too.
Change-Id: I85dfad47704a58bd42f6867adbc9e4e1792bc3b0
Reviewed-on: https://go-review.googlesource.com/2260
Reviewed-by: Russ Cox <rsc@golang.org>
We already had client support for trailers, but no way for a server to
set them short of hijacking the connection.
Fixes#7759
Change-Id: Ic83976437739ec6c1acad5f209ed45e501dbb93a
Reviewed-on: https://go-review.googlesource.com/2157
Reviewed-by: Andrew Gerrand <adg@golang.org>
Currently when we get a CGI or FCGI request, the remote port of the client
is hard coded to zero, despite nearly every webserver passing down the
REMOTE_PORT variable.
This was likely originally excluded because the CGI RFC (rfc3875) does not
mention anything about the remote port of the client. However every webserver
tested does pass REMOTE_PORT down. This includes Apache 2.2, Apache 2.4,
nginx and lighttpd.
Fixes#8351
Change-Id: I4c6366cb39f0ccc05e038bd31d85f93b76e8d0c8
Reviewed-on: https://go-review.googlesource.com/1750
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Given a file of size N, a request for "Range: bytes=N-*" should
return a 416 [1]. Currently, it returns a 206 and a body of 0
bytes, with the illegal Content-Range of "bytes N-(N-1)/N" [2].
[1]: RFC 7233, sec 2.1: "If a valid byte-range-set includes at least one
byte-range-spec with a first-byte-pos that is less than the current
length of the representation, [...]". sec 3.1: "If all of the
preconditions are true, the server supports the Range header field for
the target resource, and the specified range(s) are invalid or
unsatisfiable, the server SHOULD send a 416 (Range Not Satisfiable)
response."
[2]: RFC 7233, sec 4.2: "A Content-Range field value is invalid if it
contains a byte-range-resp that has a last-byte-pos value less than its
first-byte-pos value, [...]"
Fixes#8988
Change-Id: If3e1134e7815f5d361efea01873b29aafe3de817
Reviewed-on: https://go-review.googlesource.com/1862
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Co-hacking with Dave Cheney.
Fixes#9405
Change-Id: I14fc3b6a47dcdb5e514e93d062b804bb24e89f47
Reviewed-on: https://go-review.googlesource.com/1875
Reviewed-by: Dave Cheney <dave@cheney.net>
I broke the build in https://golang.org/change/207950a when I made
http.Transport send "Connection: close" request headers when
DisableKeepAlives was set true because I didn't run all the tests
before submitting.
httputil.DumpRequestOut used Transport to get its output, and used it
with DisableKeepAlives, so this changed the output.
Rather than updating golden data in our tests (because surely others
depend on the exact bytes from these in their tests), switch to not
using DisableKeepAlives in DumpRequestOut instead, so the output is
the same as before.
Change-Id: I9fad190be8032e55872e6947802055a6d65244df
Reviewed-on: https://go-review.googlesource.com/1632
Reviewed-by: Andrew Gerrand <adg@golang.org>
No bug was open, but I found an old email to myself to investigate
when I suspected this was happening.
Change-Id: Icedefec6f15a000eaabb2693b0640b3b6c8bf82c
Reviewed-on: https://go-review.googlesource.com/1578
Reviewed-by: Andrew Gerrand <adg@golang.org>
New detection because of net/http now using TestMain.
Fixes#9033
LGTM=iant
R=golang-codereviews, iant
CC=adg, golang-codereviews, rsc
https://golang.org/cl/170210043
We force runtime.GC before WriteHeapProfile with -test.heapprofile.
Make it possible to do the same with the HTTP interface.
Some servers only run a GC every few minutes.
On such servers, the heap profile will be a few minutes stale,
which may be too old to be useful.
Requested by private mail.
LGTM=dvyukov
R=dvyukov
CC=golang-codereviews
https://golang.org/cl/161990043
The http package by default adds "Accept-Encoding: gzip" to outgoing
requests, unless it's a bad idea, or the user requested otherwise.
Only when the http package adds its own implicit Accept-Encoding header
does the http package also transparently un-gzip the response.
If the user requested part of a document (e.g. bytes 40 to 50), it appears
that Github/Varnish send:
range(gzip(content), 40, 50)
And not:
gzip(range(content, 40, 50))
The RFC 2616 set of replacements (with the purpose of
clarifying ambiguities since 1999) has an RFC about Range
requests (http://tools.ietf.org/html/rfc7233) but does not
mention the interaction with encodings.
Regardless of whether range(gzip(content)) or gzip(range(content)) is
correct, this change prevents the Go package from asking for gzip
in requests if we're also asking for Range, avoiding the issue.
If the user cared, they can do it themselves. But Go transparently
un-gzipping a fragment of gzip is never useful.
Fixes#8923
LGTM=adg
R=adg
CC=golang-codereviews
https://golang.org/cl/155420044
http.Client calls URL.String() to fill in the Referer header, which may
contain authentication info. This patch removes authentication info from
the Referer header without introducing any API changes.
A new test for net/http is also provided.
This is the polished version of Alberto García Hierro's
https://golang.org/cl/9766046/
It should handle https Referer right.
Fixes#8417
LGTM=bradfitz
R=golang-codereviews, gobot, bradfitz, mikioh.mikioh
CC=golang-codereviews
https://golang.org/cl/151430043
Bodies larger than 8KB (the default bufio reader size) weren't
being dumped. Force a read of the body so they're teed into
the response buffer.
Thanks to Steven Hartland for identifying the problem.
Fixes#8089
LGTM=r
R=golang-codereviews, r
CC=adg, golang-codereviews
https://golang.org/cl/144650044
See comment 4 of https://code.google.com/p/go/issues/detail?id=8483#c4:
"So if a user creates a http.Client, issues a bunch of
requests and then wants to shutdown it and all opened connections;
what is she intended to do? The report suggests that just waiting for
all pending requests and calling CloseIdleConnections won't do, as
there can be new racing connections. Obviously she can't do what
you've done in the test, as it uses the unexported function. If this
happens periodically, it can lead to serious resource leaks (the
transport is also preserved alive). Am I missing something?"
This CL tracks the user's intention to close all idle
connections (CloseIdleConnections sets it true; and making a
new request sets it false). If a pending dial finishes and
nobody wants it, before it's retained for a future caller, the
"wantIdle" bool is checked and it's closed if the user has
called CloseIdleConnections without a later call to make a new
request.
Fixes#8483
LGTM=adg
R=golang-codereviews, dvyukov, adg
CC=golang-codereviews, rsc
https://golang.org/cl/148970043
