Motivation:
* Previously, the size of the compressed data was used for metrics,
rather than the uncompressed size. This causes the library to appear
to perform poorly relative to C or other implementation. Switch it
to use the uncompressed size so that it matches how decompression
benchmarks are usually done (like in compress/flate). This also makes
it easier to compare bzip2 rates to other algorithms since they measure
performance in this way.
* Also, reset the timer after doing initialization work.
Change-Id: I32112c2ee8e7391e658c9cf31039f70a689d9b9d
Reviewed-on: https://go-review.googlesource.com/17611
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
The bzip2 block size is a multiple of 100*1000 not 100*1024.
Thus, the bzip2 decoder would incorrectly decode files with larger
block sizes when it should have otherwise failed.
Fortunately, we can correct this in a backwards compatible way since
Go has no implementation of a bzip2 encoder to produce bad blocks :)
To confirm that the C bzip2 utlity chokes on this data:
$ echo "425a683131415926535936dc55330063ffc0006000200020a40830008b00
08b8bb9229c28481b6e2a998" | xxd -r -p | bzip2 -d
bzip2: Data integrity error when decompressing.
Fixes#13941
Change-Id: I2402e8829a8027ef94dd4fac050b200440a3d4e4
Reviewed-on: https://go-review.googlesource.com/20011
Run-TryBot: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
The LZ77 portion of DEFLATE is relatively self-contained. For the
decompression side of things, we extract this logic out for the
following reasons:
* It is easier to test just the LZ77 portion of the logic.
* It reduces the noise in the inflate.go
Also, we adjust the way that callbacks are handled in the inflate.
Instead of using functions to abstract the logical componets of
huffmanBlock(), use goto statements to jump between the necessary
sections. This is faster since it avoids a function call and is
arguably more readable.
benchmark old MB/s new MB/s speedup
BenchmarkDecodeDigitsSpeed1e4-4 53.62 60.11 1.12x
BenchmarkDecodeDigitsSpeed1e5-4 61.90 69.07 1.12x
BenchmarkDecodeDigitsSpeed1e6-4 63.24 70.58 1.12x
BenchmarkDecodeDigitsDefault1e4-4 54.10 59.00 1.09x
BenchmarkDecodeDigitsDefault1e5-4 69.50 74.07 1.07x
BenchmarkDecodeDigitsDefault1e6-4 71.54 75.85 1.06x
BenchmarkDecodeDigitsCompress1e4-4 54.39 58.94 1.08x
BenchmarkDecodeDigitsCompress1e5-4 69.21 73.96 1.07x
BenchmarkDecodeDigitsCompress1e6-4 71.14 75.75 1.06x
BenchmarkDecodeTwainSpeed1e4-4 53.15 58.13 1.09x
BenchmarkDecodeTwainSpeed1e5-4 66.56 72.29 1.09x
BenchmarkDecodeTwainSpeed1e6-4 69.13 75.11 1.09x
BenchmarkDecodeTwainDefault1e4-4 56.00 60.23 1.08x
BenchmarkDecodeTwainDefault1e5-4 77.84 82.27 1.06x
BenchmarkDecodeTwainDefault1e6-4 82.07 86.85 1.06x
BenchmarkDecodeTwainCompress1e4-4 56.13 60.38 1.08x
BenchmarkDecodeTwainCompress1e5-4 78.23 82.62 1.06x
BenchmarkDecodeTwainCompress1e6-4 82.38 86.73 1.05x
Change-Id: I8c6ae0e6bed652dd0570fc113c999977f5e71636
Reviewed-on: https://go-review.googlesource.com/16528
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Takes 3% off my all.bash run time.
For #10571.
Change-Id: I8f00f523d6919e87182d35722a669b0b96b8218b
Reviewed-on: https://go-review.googlesource.com/18087
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Commit 7a1fb95d50 strips non-free license
from Mark.Twain-Tom.Sawyer.txt, but forgot to remove it from the compressed
version of the file.
Update #13216
Change-Id: I60f53275d56ba5baa6898db47b1d41f85e985c00
Reviewed-on: https://go-review.googlesource.com/17264
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This change strips non-free license from Mark.Twain-Tom.Sawyer.txt along with all reference to Project Gutenberg in the file and the whole source tree. Making the file public domain again.
Fixes#13216
Change-Id: I2f41b0de225f627dde152efe93c006a4c24be668
Reviewed-on: https://go-review.googlesource.com/17196
Reviewed-by: Andrew Gerrand <adg@golang.org>
The gzip package is asymmetrical in the way it handles headers.
In Writer, the Header is written on the first call to Write, Flush, or Close.
In Reader, the Header is read on calls to NewReader or Reset as opposed to
after the first Read. Thus, we document this difference.
Fixes#13211
Change-Id: I5f87beff036e5e2fd68a02a15fdb7137e9ca4c37
Reviewed-on: https://go-review.googlesource.com/16838
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The flate library guarantees that the Reader will never read more
bytes than is necessary. This way, the underlying io.Reader will
be left exactly after the last byte of the DEFLATE stream.
Formats like gzip depend on this behavior being true.
As such, inflate conservatively reads the minimum symbol length in
huffSym leading to many individual calls to moreBits. However, if we
take advantage of the fact that every block *must* end with the EOB
symbol, we can choose to read the length of the EOB symbol.
Since the EOB symbol is also the most rare symbol (occuring exactly
once) in a block, we can hypothesize that it is almost as long as
the max symbol length, allowing huffSym to ask for more bits at the
start of every loop. This increases the probabilty that the Huffman
code is decoded on the first iteration of the outer for-loop.
benchmark old MB/s new MB/s speedup
BenchmarkDecodeDigitsSpeed1e4-4 51.05 54.31 1.06x
BenchmarkDecodeDigitsSpeed1e5-4 58.86 62.24 1.06x
BenchmarkDecodeDigitsSpeed1e6-4 59.63 63.13 1.06x
BenchmarkDecodeDigitsDefault1e4-4 51.94 54.61 1.05x
BenchmarkDecodeDigitsDefault1e5-4 63.70 69.13 1.09x
BenchmarkDecodeDigitsDefault1e6-4 66.08 71.43 1.08x
BenchmarkDecodeDigitsCompress1e4-4 52.25 54.56 1.04x
BenchmarkDecodeDigitsCompress1e5-4 63.34 68.30 1.08x
BenchmarkDecodeDigitsCompress1e6-4 66.84 70.64 1.06x
BenchmarkDecodeTwainSpeed1e4-4 50.74 53.40 1.05x
BenchmarkDecodeTwainSpeed1e5-4 60.77 67.03 1.10x
BenchmarkDecodeTwainSpeed1e6-4 62.08 69.78 1.12x
BenchmarkDecodeTwainDefault1e4-4 53.45 56.40 1.06x
BenchmarkDecodeTwainDefault1e5-4 73.54 79.05 1.07x
BenchmarkDecodeTwainDefault1e6-4 77.68 83.65 1.08x
BenchmarkDecodeTwainCompress1e4-4 53.21 56.15 1.06x
BenchmarkDecodeTwainCompress1e5-4 73.82 77.76 1.05x
BenchmarkDecodeTwainCompress1e6-4 79.23 83.30 1.05x
Change-Id: Ie194925c827988a380b8c2fdd13b13c4faa5d397
Reviewed-on: https://go-review.googlesource.com/15651
Reviewed-by: Nigel Tao <nigeltao@golang.org>
A vast majority of the time, ReadError isn't even returned during
IO operations. Instead, an unwrapped error will be returned because
of the ReadByte call on L705. Because DEFLATE streams are primarily
compressed and require byte for byte Huffman decoding, most of the
data read from a data stream will go through ReadByte.
Although this is technically an API change, any user reliant on
this error would not have worked properly anyways due to the fact
that most IO error are not wrapped. We might as well deprecate
ReadError. It is useless and actually makes clients that do
depend on catching IO errors more difficult.
Fixes#11856Fixes#12724
Change-Id: Ib5fec5ae215e977c4e85de5701ce6a473d400af8
Reviewed-on: https://go-review.googlesource.com/14834
Reviewed-by: Nigel Tao <nigeltao@golang.org>
Reader failed to detect truncated streams since calls to
io.ReadFull did not check if the error is io.EOF.
Change-Id: I86c497519daaaccefc6eb5617ddcd8fd3b99f51b
Reviewed-on: https://go-review.googlesource.com/14835
Reviewed-by: Nigel Tao <nigeltao@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reader fails to detect truncated streams since calls to io.ReadFull
do not check if the error is io.EOF.
Change-Id: I052cd03161e43fec17e3d328106c40e17923e52b
Reviewed-on: https://go-review.googlesource.com/14832
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reader failed to detect truncated streams since calls to
io.ReadFull did not check if the error is io.EOF.
Change-Id: I0634e0d8de1ab04e8f93242c27a9f89e57743e87
Reviewed-on: https://go-review.googlesource.com/14833
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
The flate library contains generator code, which is used to generate
the fixed huffman table. This is done so that fixed blocks can be
processed quicker since there is no need generate the decoder table
for fixed codes.
Instead, delete the precomputed table, and use sync.Once to generate
it at runtime when used.
Advantages:
* Reduces duplicated logic in flate package
* Reduces binary size by approximately 2KiB
Disadvantages:
* For the simplest possible program that simply decodes the fixed
block "\x03\x00" once, the modified code takes 4.7% longer for the
first decode. Compression performance for subsequent blocks afterwards
has no noticeable slow down.
Change-Id: I8f351218debf7d732118808859eda481b01011f6
Reviewed-on: https://go-review.googlesource.com/14181
Reviewed-by: Nigel Tao <nigeltao@golang.org>
Issue 6754 reports that Go bzip2 Decode function is much slower
(about 2.5x in go1.5) than the Python equivalent (which is
actually just a wrapper around the usual C library) on random data.
Profiling the code shows that half a dozen of CMP instructions in a
tight loop are responsibile for most of the execution time.
This patch reduces the number of branches of the loop, greatly
improving performance on random data and speeding up decoding of
real data.
name old time/op new time/op delta
DecodeDigits-4 9.28ms ± 1% 8.05ms ± 1% -13.18% (p=0.000 n=15+14)
DecodeTwain-4 28.9ms ± 2% 26.4ms ± 1% -8.57% (p=0.000 n=15+14)
DecodeRand-4 3.94ms ± 1% 3.06ms ± 1% -22.45% (p=0.000 n=15+14)
name old speed new speed delta
DecodeDigits-4 4.65MB/s ± 1% 5.36MB/s ± 1% +15.21% (p=0.000 n=13+14)
DecodeTwain-4 4.32MB/s ± 2% 4.72MB/s ± 1% +9.36% (p=0.000 n=15+14)
DecodeRand-4 4.27MB/s ± 1% 5.51MB/s ± 1% +28.86% (p=0.000 n=15+14)
I've run some benchmark comparing Go bzip2 implementation with the
usual Linux bzip2 command (which is written in C). On my machine
this patch brings go1.5
from ~2.26x to ~1.50x of bzip2 time (on 64MB random data)
from ~1.70x to ~1.50x of bzip2 time (on 100MB english text)
from ~2.00x to ~1.88x of bzip2 time (on 64MB /dev/zero data)
Fixes#6754
Change-Id: I3cb12d2c0c2243c1617edef1edc88f05f91d26d1
Reviewed-on: https://go-review.googlesource.com/13853
Reviewed-by: Nigel Tao <nigeltao@golang.org>
The one in misc/makerelease/makerelease.go is particularly bad and
probably warrants rotating our keys.
I didn't update old weekly notes, and reverted some changes involving
test code for now, since we're late in the Go 1.5 freeze. Otherwise,
the rest are all auto-generated changes, and all manually reviewed.
Change-Id: Ia2753576ab5d64826a167d259f48a2f50508792d
Reviewed-on: https://go-review.googlesource.com/12048
Reviewed-by: Rob Pike <r@golang.org>
Add sentences to the docs explaining the limit on input
bytes implicit in the choice of litWidth, and the fact that
compress and decompress litWidth must match.
Fixes#11142.
Change-Id: I20cfb4df35739f7bfeb50b92c78249df3d47942c
Reviewed-on: https://go-review.googlesource.com/11063
Reviewed-by: Nigel Tao <nigeltao@golang.org>
All the other error messages in this package are "lzw: etc".
Change-Id: Ie359a8912d213f92b15f02abc953127380503048
Reviewed-on: https://go-review.googlesource.com/11228
Reviewed-by: David Symonds <dsymonds@golang.org>
Corrected several issues:
* RFC1951 section 3.2.7 dictates that it is okay for the HDist tree to have a
single code of zero bits. Furthermore, the behavior of the C zlib library
permits empty trees even when there are more than one codes.
* RFC1951 section 3.2.5 shows that HLit codes 286 and 287 are invalid. Thus,
Go's implementation should choke on inputs using these codes.
* RFC1951 section 3.2.5 and 3.2.7 are ambiguous about whether the number of
HDist codes can be greater than 30. The C zlib library (which is the canonical
reference implementation) performs this check here:
62d6112a79/inflate.c (L906)
In addition, a number of test cases were added to the unit tests that exercises
these edge cases. The test cases listed in TestStreams will either fail or
succeed in a manner matching the behaviour of the C zlib version. Given that the
C zlib implementation is the reference for the world, Go's implementation should
match C zlib behaviour.
Fixes#11030
Change-Id: Ic24e4e40ce5832c7e1930249246e86d34bfedaa6
Reviewed-on: https://go-review.googlesource.com/11000
Reviewed-by: Nigel Tao <nigeltao@golang.org>
If the requested coding bit sizes don't result in a full binary tree,
then reject the input as invalid.
Exception: We still need to allow degenerate Huffman codings with a
single 1-bit code to be compatible with zlib and files compressed with
Go's compress/flate package.
Update #10426.
Change-Id: I171b98d12e65b4deb9f4031cd802407ebb5e266c
Reviewed-on: https://go-review.googlesource.com/8922
Reviewed-by: Nigel Tao <nigeltao@golang.org>
This code's test coverage is ad hoc at best, and it's easy to make
changes that accidentally regress invariants. This CL adds a "sanity"
constant that can be changed to "true" during development to add extra
runtime checking that the Huffman decoder tables are sane.
Change-Id: I0d0ca53ad7c9566be18046d9b255e1a30059f28b
Reviewed-on: https://go-review.googlesource.com/8974
Reviewed-by: Nigel Tao <nigeltao@golang.org>
When decoding Huffman codes, if an invalid bit sequence is discovered,
reject the input instead of treating it as a 0-length code.
Fixes#10426.
Change-Id: Ie2f1a3a718afd7c6bee73a67480d4b84936c21c9
Reviewed-on: https://go-review.googlesource.com/8893
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Nigel Tao <nigeltao@golang.org>
Only documentation / comment changes. Update references to
point to golang.org permalinks or go.googlesource.com/go.
References in historical release notes under doc are left as is.
Change-Id: Icfc14e4998723e2c2d48f9877a91c5abef6794ea
Reviewed-on: https://go-review.googlesource.com/4060
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Allows parsing some file formats that assign special
meaning to which stream data is found in.
Will do the same for compress/bzip2 once this is
reviewed and submitted.
Fixes#6486.
LGTM=nigeltao
R=nigeltao, dan.kortschak
CC=adg, bradfitz, golang-codereviews, r
https://golang.org/cl/159120044
This adds a Reset() to compress/flate's decompressor and plumbs that through
to compress/zlib and compress/gzip's Readers so callers can avoid large
allocations when performing many inflate operations. In particular this
preserves the allocation of the decompressor.hist buffer, which is 32kb and
overwritten as needed while inflating.
On the benchmark described in issue 6317, produces the following speedup on
my 2.3ghz Intel Core i7 MBP with go version devel +6b696a34e0af Sun Aug 03
15:14:59 2014 -0700 darwin/amd64:
blocked.text w/out patch vs blocked.text w/ patch:
benchmark old ns/op new ns/op delta
BenchmarkGunzip 8371577533 7927917687 -5.30%
benchmark old allocs new allocs delta
BenchmarkGunzip 176818 148519 -16.00%
benchmark old bytes new bytes delta
BenchmarkGunzip 292184936 12739528 -95.64%
flat.text vs blocked.text w/patch:
benchmark old ns/op new ns/op delta
BenchmarkGunzip 7939447827 7927917687 -0.15%
benchmark old allocs new allocs delta
BenchmarkGunzip 90702 148519 +63.74%
benchmark old bytes new bytes delta
BenchmarkGunzip 9959528 12739528 +27.91%
Similar speedups to those bradfitz saw in https://golang.org/cl/13416045.
Fixes#6317.
Fixes#7950.
LGTM=nigeltao
R=golang-codereviews, bradfitz, dan.kortschak, adg, nigeltao, jamesr
CC=golang-codereviews
https://golang.org/cl/97140043
