The flate library contains generator code, which is used to generate
the fixed huffman table. This is done so that fixed blocks can be
processed quicker since there is no need generate the decoder table
for fixed codes.
Instead, delete the precomputed table, and use sync.Once to generate
it at runtime when used.
Advantages:
* Reduces duplicated logic in flate package
* Reduces binary size by approximately 2KiB
Disadvantages:
* For the simplest possible program that simply decodes the fixed
block "\x03\x00" once, the modified code takes 4.7% longer for the
first decode. Compression performance for subsequent blocks afterwards
has no noticeable slow down.
Change-Id: I8f351218debf7d732118808859eda481b01011f6
Reviewed-on: https://go-review.googlesource.com/14181
Reviewed-by: Nigel Tao <nigeltao@golang.org>
Issue 6754 reports that Go bzip2 Decode function is much slower
(about 2.5x in go1.5) than the Python equivalent (which is
actually just a wrapper around the usual C library) on random data.
Profiling the code shows that half a dozen of CMP instructions in a
tight loop are responsibile for most of the execution time.
This patch reduces the number of branches of the loop, greatly
improving performance on random data and speeding up decoding of
real data.
name old time/op new time/op delta
DecodeDigits-4 9.28ms ± 1% 8.05ms ± 1% -13.18% (p=0.000 n=15+14)
DecodeTwain-4 28.9ms ± 2% 26.4ms ± 1% -8.57% (p=0.000 n=15+14)
DecodeRand-4 3.94ms ± 1% 3.06ms ± 1% -22.45% (p=0.000 n=15+14)
name old speed new speed delta
DecodeDigits-4 4.65MB/s ± 1% 5.36MB/s ± 1% +15.21% (p=0.000 n=13+14)
DecodeTwain-4 4.32MB/s ± 2% 4.72MB/s ± 1% +9.36% (p=0.000 n=15+14)
DecodeRand-4 4.27MB/s ± 1% 5.51MB/s ± 1% +28.86% (p=0.000 n=15+14)
I've run some benchmark comparing Go bzip2 implementation with the
usual Linux bzip2 command (which is written in C). On my machine
this patch brings go1.5
from ~2.26x to ~1.50x of bzip2 time (on 64MB random data)
from ~1.70x to ~1.50x of bzip2 time (on 100MB english text)
from ~2.00x to ~1.88x of bzip2 time (on 64MB /dev/zero data)
Fixes#6754
Change-Id: I3cb12d2c0c2243c1617edef1edc88f05f91d26d1
Reviewed-on: https://go-review.googlesource.com/13853
Reviewed-by: Nigel Tao <nigeltao@golang.org>
The one in misc/makerelease/makerelease.go is particularly bad and
probably warrants rotating our keys.
I didn't update old weekly notes, and reverted some changes involving
test code for now, since we're late in the Go 1.5 freeze. Otherwise,
the rest are all auto-generated changes, and all manually reviewed.
Change-Id: Ia2753576ab5d64826a167d259f48a2f50508792d
Reviewed-on: https://go-review.googlesource.com/12048
Reviewed-by: Rob Pike <r@golang.org>
Add sentences to the docs explaining the limit on input
bytes implicit in the choice of litWidth, and the fact that
compress and decompress litWidth must match.
Fixes#11142.
Change-Id: I20cfb4df35739f7bfeb50b92c78249df3d47942c
Reviewed-on: https://go-review.googlesource.com/11063
Reviewed-by: Nigel Tao <nigeltao@golang.org>
All the other error messages in this package are "lzw: etc".
Change-Id: Ie359a8912d213f92b15f02abc953127380503048
Reviewed-on: https://go-review.googlesource.com/11228
Reviewed-by: David Symonds <dsymonds@golang.org>
Corrected several issues:
* RFC1951 section 3.2.7 dictates that it is okay for the HDist tree to have a
single code of zero bits. Furthermore, the behavior of the C zlib library
permits empty trees even when there are more than one codes.
* RFC1951 section 3.2.5 shows that HLit codes 286 and 287 are invalid. Thus,
Go's implementation should choke on inputs using these codes.
* RFC1951 section 3.2.5 and 3.2.7 are ambiguous about whether the number of
HDist codes can be greater than 30. The C zlib library (which is the canonical
reference implementation) performs this check here:
62d6112a79/inflate.c (L906)
In addition, a number of test cases were added to the unit tests that exercises
these edge cases. The test cases listed in TestStreams will either fail or
succeed in a manner matching the behaviour of the C zlib version. Given that the
C zlib implementation is the reference for the world, Go's implementation should
match C zlib behaviour.
Fixes#11030
Change-Id: Ic24e4e40ce5832c7e1930249246e86d34bfedaa6
Reviewed-on: https://go-review.googlesource.com/11000
Reviewed-by: Nigel Tao <nigeltao@golang.org>
If the requested coding bit sizes don't result in a full binary tree,
then reject the input as invalid.
Exception: We still need to allow degenerate Huffman codings with a
single 1-bit code to be compatible with zlib and files compressed with
Go's compress/flate package.
Update #10426.
Change-Id: I171b98d12e65b4deb9f4031cd802407ebb5e266c
Reviewed-on: https://go-review.googlesource.com/8922
Reviewed-by: Nigel Tao <nigeltao@golang.org>
This code's test coverage is ad hoc at best, and it's easy to make
changes that accidentally regress invariants. This CL adds a "sanity"
constant that can be changed to "true" during development to add extra
runtime checking that the Huffman decoder tables are sane.
Change-Id: I0d0ca53ad7c9566be18046d9b255e1a30059f28b
Reviewed-on: https://go-review.googlesource.com/8974
Reviewed-by: Nigel Tao <nigeltao@golang.org>
When decoding Huffman codes, if an invalid bit sequence is discovered,
reject the input instead of treating it as a 0-length code.
Fixes#10426.
Change-Id: Ie2f1a3a718afd7c6bee73a67480d4b84936c21c9
Reviewed-on: https://go-review.googlesource.com/8893
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Nigel Tao <nigeltao@golang.org>
Only documentation / comment changes. Update references to
point to golang.org permalinks or go.googlesource.com/go.
References in historical release notes under doc are left as is.
Change-Id: Icfc14e4998723e2c2d48f9877a91c5abef6794ea
Reviewed-on: https://go-review.googlesource.com/4060
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Allows parsing some file formats that assign special
meaning to which stream data is found in.
Will do the same for compress/bzip2 once this is
reviewed and submitted.
Fixes#6486.
LGTM=nigeltao
R=nigeltao, dan.kortschak
CC=adg, bradfitz, golang-codereviews, r
https://golang.org/cl/159120044
This adds a Reset() to compress/flate's decompressor and plumbs that through
to compress/zlib and compress/gzip's Readers so callers can avoid large
allocations when performing many inflate operations. In particular this
preserves the allocation of the decompressor.hist buffer, which is 32kb and
overwritten as needed while inflating.
On the benchmark described in issue 6317, produces the following speedup on
my 2.3ghz Intel Core i7 MBP with go version devel +6b696a34e0af Sun Aug 03
15:14:59 2014 -0700 darwin/amd64:
blocked.text w/out patch vs blocked.text w/ patch:
benchmark old ns/op new ns/op delta
BenchmarkGunzip 8371577533 7927917687 -5.30%
benchmark old allocs new allocs delta
BenchmarkGunzip 176818 148519 -16.00%
benchmark old bytes new bytes delta
BenchmarkGunzip 292184936 12739528 -95.64%
flat.text vs blocked.text w/patch:
benchmark old ns/op new ns/op delta
BenchmarkGunzip 7939447827 7927917687 -0.15%
benchmark old allocs new allocs delta
BenchmarkGunzip 90702 148519 +63.74%
benchmark old bytes new bytes delta
BenchmarkGunzip 9959528 12739528 +27.91%
Similar speedups to those bradfitz saw in https://golang.org/cl/13416045.
Fixes#6317.
Fixes#7950.
LGTM=nigeltao
R=golang-codereviews, bradfitz, dan.kortschak, adg, nigeltao, jamesr
CC=golang-codereviews
https://golang.org/cl/97140043
