The listed invariant, while technically true, was misleading, and the
invariant can be tightened. We never actually get to (d.hi ==
d.overflow), due to the "d.hi--" line in the decoder.decode method.
This is a comment-only commit, changing the comment to match the code.
A follow-up commit could restore the comment, changing the code to match
the original intented invariant. But the first step is to have the
comment and the code say the same thing.
Change-Id: Ifc9f78d5060454fc107af9be298026bf3043d400
Reviewed-on: https://go-review.googlesource.com/c/go/+/191358
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Shorten some of the longest tests that run during all.bash.
Removes 7r 50u 21s from all.bash.
After this change, all.bash is under 5 minutes again on my laptop.
For #26473.
Change-Id: Ie0460aa935808d65460408feaed210fbaa1d5d79
Reviewed-on: https://go-review.googlesource.com/c/go/+/177559
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The first biggest offender was crypto/des.init at ~1%. It's
cryptographically broken and the init function is relatively expensive,
which is unfortunate as both crypto/tls and crypto/x509 (and by
extension, cmd/go) import it. Hide the work behind sync.Once.
The second biggest offender was flag.sortFlags at just under 1%, used by
the Visit flagset methods. It allocated two slices, which made a
difference as cmd/go iterates over multiple flagsets during init.
Use a single slice with a direct sort.Interface implementation.
Another big offender is initializing global maps. Reducing this work in
cmd/go/internal/imports and net/textproto gives us close to another
whole 1% in saved work. The former can use map literals, and the latter
can hide the work behind sync.Once.
Finally, compress/flate used newHuffmanBitWriter as part of init, which
allocates many objects and slices. Yet it only used one of the slice
fields. Allocating just that slice saves a surprising ~0.3%, since we
generated a lot of unnecessary garbage.
All in all, these little pieces amount to just over 3% saved CPU time.
name old time/op new time/op delta
ExecGoEnv-8 3.61ms ± 1% 3.50ms ± 0% -3.02% (p=0.000 n=10+10)
Updates #26775.
Updates #29382.
Change-Id: I915416e88a874c63235ba512617c8aef35c0ca8b
Reviewed-on: https://go-review.googlesource.com/c/go/+/166459
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This applies the new gofmt literal normalizations to the library.
Change-Id: I8c1e8ef62eb556fc568872c9f77a31ef236348e7
Reviewed-on: https://go-review.googlesource.com/c/162539
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
This reverts commit 067bb443af.
Reason for revert:
Failing Darwin-arm builds because that testing environment does not access testdata
from sibling directories. A future change will likely be made to move this testdata
out of src/testdata to create a solution that doesn't require the single-file directory.
Updates #27151
Change-Id: I8dbf5dd9512c94a605ee749ff4655cb00b0de686
Reviewed-on: https://go-review.googlesource.com/138737
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This text is used mainly for benchmark compression testing, and in one
net test. The text was prevoiusly in a src/testdata directory, but since
that directory would only include one file, the text is moved to the
existing src/compression/testdata directory.
This does not cause any change to the benchmark results.
Updates #27151
Change-Id: I38ab5089dfe744189a970947d15be50ef1d48517
Reviewed-on: https://go-review.googlesource.com/138495
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Use the binary.{Big,Little}Endian integer encoding methods rather than
variations found in local implementations. The functions in
the binary package have been tested to ensure they inline correctly and
don't add unnecessary bounds checking.
Change-Id: Ie10111ca6edb7c11e8e5e21c58a5748ae99b7f87
Reviewed-on: https://go-review.googlesource.com/134375
Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Michael Munday <mike.munday@ibm.com>
Each URL was manually verified to ensure it did not serve up incorrect
content.
Change-Id: I4dc846227af95a73ee9a3074d0c379ff0fa955df
Reviewed-on: https://go-review.googlesource.com/115798
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
The new iOS test harness forwards environment variables to the
test program, which means that it runs builder-only tests that were
previously skipped because GO_BUILDER_NAME was missing.
Skip one such unblocked test, TestGZIPFilesHaveZeroMTimes, which
assumes a valid GOROOT.
Change-Id: I5daf0f4c1897afbeb8b1a380669a1d2aa47e764a
Reviewed-on: https://go-review.googlesource.com/111475
Run-TryBot: Elias Naur <elias.naur@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Before, if an underlying writer errored within 10 bytes (plus any gzip
header metadata), a gzip.Write would erroneously report up to 10 bytes
written that were not actually written of the input slice. This is
especially problematic when the input slice is less than 10 bytes.
The error came from counting the 10 header byte write. If writing the
header is completely successful, the 10 bytes written is overridden by
the flate write with the input slice.
This removes counting the 10 required header bytes, and also changes the
return to use zero until the slice is used.
The old Write could return one byte written when it actually was not.
This is difficult to verify because the smallest input slice is one
byte; a test checking that the input slice was the byte written would be
quite involved. Thankfully, gzip's minimum header write is 10 bytes. If
we test that two bytes are not falsely written, we indirectly cover the
one byte case.
Fixes#24625
Change-Id: I1c1f8cd791e0c4cffc22aa8acd95186582c832ba
Reviewed-on: https://go-review.googlesource.com/103861
Reviewed-by: Joe Tsai <joetsai@google.com>
Run-TryBot: Joe Tsai <joetsai@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Previously, the read method checked whether the current block
was fully consumed or not based on whether the buffer could be filled
with a non-zero number of bytes. This check is problematic because
zero bytes could be read if the provided buffer is empty.
We fix this case by simply checking for whether the input buffer
provided by the user was empty or not. If empty, we assume that
we could not read any bytes because the buffer was too small,
rather than indicating that the current block was fully exhausted.
This check causes bzip2.Reader to be unable to make progress
on the next block unless a non-empty buffer is provided.
However, that is an entirely reasonable expectation since a
non-empty buffer needs to be provided eventually anyways to
read the actual contents of subsequent blocks.
Fixes#22028
Change-Id: I2bb1b2d54e78567baf2bf7b490a272c0853d7bfe
Reviewed-on: https://go-review.googlesource.com/66110
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Some constants were added to flate that seem to be an experimental
attempt at increasing the window size. However, according to RFC1951,
the largest window size is 32KiB, so these constants are non-standard.
Delete them.
Fixes#18458
Change-Id: Ia94989637ca031a56bce2548624fa48044caa7b9
Reviewed-on: https://go-review.googlesource.com/60490
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
The logic performs a series of shifts, which are useless given
that they are followed by an assignment that overrides the
value of the previous computation.
I suspect (but cannot prove) that this is leftover logic from an
original approach that attempted to store both the Huffman code
and the length within the same variable instead of using two
different variables as it currently does now.
Fixes#17949
Change-Id: Ibf6c807c6cef3b28bfdaf2b68d9bc13503ac21b2
Reviewed-on: https://go-review.googlesource.com/44091
Reviewed-by: Ian Lance Taylor <iant@golang.org>
This reverts commit 467109bf56.
Replaced by a improved strategy later in the CL relation chain.
Change-Id: Ib90813b5a6c4716b563c8496013d2d57f9c022b8
Reviewed-on: https://go-review.googlesource.com/36066
Reviewed-by: David Crawshaw <crawshaw@golang.org>
Run-TryBot: Elias Naur <elias.naur@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
The working directory is now adjusted to match the typical Go test
working directory in main, as the old trick for adjusting earlier
stopped working with the latest version of LLDB bugs.
That means the small number of places where testdata files are
read before main is called no longer work. This CL adjusts those
reads to happen after main is called. (This has the bonus effect of
not reading some benchmark testdata files in all.bash.)
Fixes compress/bzip2, go/doc, go/parser, os, and time package
tests on the iOS builder.
Change-Id: If60f026aa7848b37511c36ac5e3985469ec25209
Reviewed-on: https://go-review.googlesource.com/35255
Run-TryBot: David Crawshaw <crawshaw@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Ranging over an array causes the array to be copied over to the
stack, which cause large re-growths. Instead, we should iterate
over slices of the array.
Also, assigning a large struct literal uses the stack even
though the actual fields being populated are small in comparison
to the entirety of the struct (see #18636).
Fixing the stack growth does not alter CPU-time performance much
since the stack-growth and copying was such a tiny portion of the
compression work:
name old time/op new time/op delta
Encode/Digits/Default/1e4-8 332µs ± 1% 332µs ± 1% ~ (p=0.796 n=10+10)
Encode/Digits/Default/1e5-8 5.07ms ± 2% 5.05ms ± 1% ~ (p=0.815 n=9+8)
Encode/Digits/Default/1e6-8 53.7ms ± 1% 53.9ms ± 1% ~ (p=0.075 n=10+10)
Encode/Twain/Default/1e4-8 380µs ± 1% 380µs ± 1% ~ (p=0.684 n=10+10)
Encode/Twain/Default/1e5-8 5.79ms ± 2% 5.79ms ± 1% ~ (p=0.497 n=9+10)
Encode/Twain/Default/1e6-8 61.5ms ± 1% 61.8ms ± 1% ~ (p=0.247 n=10+10)
name old speed new speed delta
Encode/Digits/Default/1e4-8 30.1MB/s ± 1% 30.1MB/s ± 1% ~ (p=0.753 n=10+10)
Encode/Digits/Default/1e5-8 19.7MB/s ± 2% 19.8MB/s ± 1% ~ (p=0.795 n=9+8)
Encode/Digits/Default/1e6-8 18.6MB/s ± 1% 18.5MB/s ± 1% ~ (p=0.072 n=10+10)
Encode/Twain/Default/1e4-8 26.3MB/s ± 1% 26.3MB/s ± 1% ~ (p=0.616 n=10+10)
Encode/Twain/Default/1e5-8 17.3MB/s ± 2% 17.3MB/s ± 1% ~ (p=0.484 n=9+10)
Encode/Twain/Default/1e6-8 16.3MB/s ± 1% 16.2MB/s ± 1% ~ (p=0.238 n=10+10)
Updates #18636Fixes#18625
Change-Id: I471b20339bf675f63dc56d38b3acdd824fe23328
Reviewed-on: https://go-review.googlesource.com/35122
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
I used the slowtests.go tool as described in
https://golang.org/cl/32684 on packages that stood out.
go test -short std drops from ~56 to ~52 seconds.
This isn't a huge win, but it was mostly an exercise.
Updates #17751
Change-Id: I9f3402e36a038d71e662d06ce2c1d52f6c4b674d
Reviewed-on: https://go-review.googlesource.com/32751
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Previously, we were off by one.
Also fix a comment typo.
Change-Id: Ib94d23acc56d5fccd44144f71655481f98803ac8
Reviewed-on: https://go-review.googlesource.com/32149
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
The GZIP format records the ModTime as an uint32 counting seconds since
the Unix epoch. The zero value is explicitly defined in section 2.3.1
as meaning no timestamp is available.
Currently, the Writer always encodes the ModTime even if it is the zero
time.Time value, which causes the Writer to try and encode the value
-62135596800 into the uint32 MTIME field. This causes an overflow and
results in our GZIP files having MTIME fields indicating a date in 2042-07-13.
We alter the Writer to only encode ModTime if the value does not underflow
the MTIME field (i.e., it is newer than the Unix epoch). We do not attempt
to fix what happens when the timestamp overflows in the year 2106.
We alter the Reader to only decode ModTime if the value is non-zero.
There is no risk of overflowing time.Time when decoding.
Fixes#17663
Change-Id: Ie1b65770c6342cd7b14aeebe10e5a49e6c9eb730
Reviewed-on: https://go-review.googlesource.com/32325
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Tests for determinism was not working as intended since io.Copybuffer
uses the io.WriterTo if available.
This exposed that level 0 (no compression) changed output
based on the number of writes and buffers given to the
writer.
Previously, Write would emit a new raw block (BTYPE=00) for
every non-empty call to Write.
This CL fixes it such that a raw block is only emitted upon
the following conditions:
* A full window is obtained (every 65535 bytes)
* Flush is called
* Close is called
Change-Id: I807f866d97e2db7820f11febab30a96266a6cbf1
Reviewed-on: https://go-review.googlesource.com/31174
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
This exposes HuffmanOnly in zlib and gzip packages, which is currently
unavailable.
Change-Id: If5d103bbc8b5fce2f5d740fd103a235c5d1ed7cd
Reviewed-on: https://go-review.googlesource.com/31186
Reviewed-by: Nigel Tao <nigeltao@golang.org>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
The incorrect table was used for estimating output size.
This can give suboptimal selection of entropy encoder in rare cases.
Change-Id: I8b358200f2d1f9a3f9b79a44269d7be704e1d2d9
Reviewed-on: https://go-review.googlesource.com/31172
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
In the event of an unexpected error, we should always flush available
decompressed data to the user.
Fixes#16924
Change-Id: I0bc0824c3201f3149e84e6a26e3dbcba72a1aae5
Reviewed-on: https://go-review.googlesource.com/28216
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
For persistent error handling, the methods of huffmanBitWriter have to be
consistent about how they check errors. It must either consistently
check error *before* every operation OR immediately *after* every
operation. Since most of the current logic uses the previous approach,
we apply the same style of error checking to writeBits and all calls
to Write such that they only operate if w.err is already nil going
into them.
The error handling approach is brittle and easily broken by future commits to
the code. In the near future, we should switch the logic to use panic at the
lowest levels and a recover at the edge of the public API to ensure
that errors are always persistent.
Fixes#16749
Change-Id: Ie1d83e4ed8842f6911a31e23311cd3cbf38abe8c
Reviewed-on: https://go-review.googlesource.com/27200
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
