Fixes#24929
Change-Id: Icc426068cd73b75b78001f55e1e5d81ccebbe854
Reviewed-on: https://go-review.googlesource.com/c/go/+/127120
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
This reverts CL 176098.
Reason for revert: added complexity, but did not completely fix the
underlying problem. A complete solution would not be worth the
complexity, and as a partial solution this is probably not worth the
complexity either.
Updates #31859
Change-Id: Ifd34c292fd1b811c60afe3c339e5edd3f37190c8
Reviewed-on: https://go-review.googlesource.com/c/go/+/186817
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Caleb Spare <cespare@gmail.com>
It was added in CL 83956 but never used.
Updates #23129
Change-Id: I70b50e974a56620069a77658386722af314cc857
Reviewed-on: https://go-review.googlesource.com/c/go/+/179138
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Shorten some of the longest tests that run during all.bash.
Removes 7r 50u 21s from all.bash.
After this change, all.bash is under 5 minutes again on my laptop.
For #26473.
Change-Id: Ie0460aa935808d65460408feaed210fbaa1d5d79
Reviewed-on: https://go-review.googlesource.com/c/go/+/177559
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
callerName requested 2 PCs from Callers, and that causes
both to be looked up in the file/line mapping.
We really only need to do the work for one PC.
(And in fact the caller doesn't need file/line at all, but
the Callers API can't express that.)
We used to request 2 PCs because in 1.11 and earlier we
stored an inline skip count in the second entry.
That's not necessary any more (as of 1.12).
Fixes#32093
Change-Id: I7b272626ef6496e848ee8af388cdaafd2556857b
Reviewed-on: https://go-review.googlesource.com/c/go/+/177858
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Caleb Spare <cespare@gmail.com>
This cuts the time for 'go test -short testing' from 0.9s to < 0.1s.
Change-Id: Ib8402f80239e1e96ea5221dfd5cd0db08170d85b
Reviewed-on: https://go-review.googlesource.com/c/go/+/177420
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
CL 121936 added this diagnostic to avoid a panic accessing *short.
(Hence the "This shouldn't really be a panic" comment.)
That CL was right to produce a clearer error than a plain memory fault,
but I think wrong to print+exit instead of panicking. I just ran into
one of these in a real program, and there is no indication anywhere
of how the program reached this point. The panic will show that.
So change print+exit to a panic with a helpful message, in contrast
to the original panic with an unhelpful message and the current
helpful message without stack trace.
Change-Id: Ib2bae1dead4ccde92f00fa3a34c05241ff7690c6
Reviewed-on: https://go-review.googlesource.com/c/go/+/177419
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
In CL 173722, we moved the flag registration in the testing package into
an Init function. In order to avoid needing changes to user code, we
called Init automatically as part of testing.MainStart.
However, that isn't early enough if flag.Parse is called before the
tests run, as part of package initialization.
Fix this by injecting a bit of code to call testing.Init into test
packages. This runs before any other initialization code in the user's
test package, so testing.Init will be called before any user code can
call flag.Parse.
Fixes#31859
Change-Id: Ib42cd8d3819150c49a3cecf7eef2472319d0c7e9
Reviewed-on: https://go-review.googlesource.com/c/go/+/176098
Run-TryBot: Caleb Spare <cespare@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Any code that imports the testing package forces the testing flags to be
defined, even in non-test binaries. People work around this today by
defining a copy of the testing.TB interface just to avoid importing
testing.
Fix this by moving flag registration into a new function, testing.Init.
Delay calling Init until the testing binary begins to run, in
testing.MainStart.
Init is exported for cases where users need the testing flags to be
defined outside of a "go test" context. In particular, this may be
needed where testing.Benchmark is called outside of a test.
Fixes#21051
Change-Id: Ib7e02459e693c26ae1ba71bbae7d455a91118ee3
Reviewed-on: https://go-review.googlesource.com/c/go/+/173722
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
CL 172698 documented this flag but the description was missing
punctuation and could be clearer.
Change-Id: I310d91ae8c6b947ce7d1ae7559882f49778f770a
Reviewed-on: https://go-review.googlesource.com/c/go/+/172817
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Document that the default quickcheck configuration is to run 100 times
and that there is a flag that configures it called "quickchecks".
Change-Id: I46fdab9d572e132ccc23ef907f9cc6b2d06b37c4
Reviewed-on: https://go-review.googlesource.com/c/go/+/172698
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
CL 166717 changed the way ns/op benchmark results were printed and
inadvertently rounded all ns/op results down to an integer, even if
they were small enough to print with digits after the decimal place.
For example, prior to this change, we got output like
BenchmarkFast-8 380491575 3.12 ns/op
CL 166717 changed this to
BenchmarkFast-8 380491575 3.00 ns/op
This had the further side-effect that ns/op values between 0 and 1
would not be printed at all because they would be rounded down to 0.
This CL fixes this by always recomputing the float64 value of ns/op
instead of using the int64 truncation from BenchmarkResult.NsPerOp.
Fixes#30997. Fixes#31005.
Change-Id: I21f73b9d5cc5ad41e7ff535675d07ca00051ecd7
Reviewed-on: https://go-review.googlesource.com/c/go/+/168937
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
The original goal of rounding to readable b.N
was to make it easier to eyeball times.
However, proper analysis requires tooling
(such as benchstat) anyway.
Instead, take b.N as it comes.
This will reduce the impact of external noise
such as GC on benchmarks.
This requires reworking our iteration estimates.
We used to calculate the estimated ns/op
and then divide our target ns by that estimate.
However, this order of operations was destructive
when the ns/op was very small; rounding could
hide almost an order of magnitude of variation.
Instead, multiply first, then divide.
Also, make n an int64 to avoid overflow.
Prior to this change, we attempted to cap b.N at 1e9.
Due to rounding up, it was possible to get b.N as high as 2e9.
This change consistently enforces the 1e9 cap.
This change also reduces the wall time required to run benchmarks.
Here's the impact of this change on the wall time to run
all benchmarks once with benchtime=1s on some std packages:
name old time/op new time/op delta
bytes 306s ± 1% 238s ± 1% -22.24% (p=0.000 n=10+10)
encoding/json 112s ± 8% 99s ± 7% -11.64% (p=0.000 n=10+10)
net/http 54.7s ± 7% 44.9s ± 4% -17.94% (p=0.000 n=10+9)
runtime 957s ± 1% 714s ± 0% -25.38% (p=0.000 n=10+9)
strings 262s ± 1% 201s ± 1% -23.27% (p=0.000 n=10+10)
[Geo mean] 216s 172s -20.23%
Updates #24735
Change-Id: I7e38efb8e23c804046bf4fc065b3f5f3991d0a15
Reviewed-on: https://go-review.googlesource.com/c/go/+/112155
Reviewed-by: Austin Clements <austin@google.com>
This adds a ReportMetric method to testing.B that lets the user report
custom benchmark metrics and override built-in metrics.
Fixes#26037.
Change-Id: I8236fbde3683fc27bbe45cbbedfd377b435edf64
Reviewed-on: https://go-review.googlesource.com/c/go/+/166717
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
os.Pipe is not implemented on wasm/js so for that purpose use
a temporary file for js/wasm. This change creates two versions
of runExample:
* runExample verbatim that still uses os.Pipe for non js/wasm
* runExample that uses a temporary file
Also added a TODO to re-unify these function versions back into
example.go wasm/js gets an os.Pipe implementation.
Change-Id: I9f418a49b2c397e1667724c7442b7bbe8942225e
Reviewed-on: https://go-review.googlesource.com/c/go/+/165357
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
I had been finding these over a year or so, but none were big enough
changes to warrant CLs. They're a handful now, so clean them all up in a
single commit.
The smaller bodies get a bit simpler, but most importantly, the larger
bodies get unindented.
Change-Id: I5707a6fee27d4c9ff9efd3d363af575d7a4bf2aa
Reviewed-on: https://go-review.googlesource.com/c/go/+/165340
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Fixes StartTimer's doc with the verb 'be'
that was previously missing in 'can also used'.
Change-Id: I4b3e6103fbf62d676056d32fcce4618536b7c05c
Reviewed-on: https://go-review.googlesource.com/c/go/+/165117
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The package doc for the testing package doesn't have a simple
example demonstrating how to write a test with an expectation. The doc
has simple examples for benchmarks, examples, and skipping, and it would be
useful for people new to writing tests in Go.
Also moved the skip example further down because it references tests and
benchmarks but benchmarks haven't been discussed in detail until the
next section. Skip is also a less used feature and it seems misplaced to
sit so high up in the package documentation. As an example, Skip is used
570 times the Go code repository which is significantly less than Error
and Fatal that are used 23,303 times.
Also changed 'sample' to 'simple' in other places in the package documentation
to keep the language used consistent when describing the small examples.
Fixes#27839
Change-Id: Ie01a3751986ee61adf2a2f2eda59cc182342baa7
GitHub-Last-Rev: 7357bfdcd2
GitHub-Pull-Request: golang/go#27840
Reviewed-on: https://go-review.googlesource.com/c/137175
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
When running benchmarks with profilers and trying to
compare one run against another, it is very useful to be
able to force each run to execute exactly the same number
of iterations.
Discussion on the proposal issue #24735 led to the decision
to overload -benchtime, so that instead of saying
-benchtime 10s to run a benchmark for 10 seconds,
you say -benchtime 100x to run a benchmark 100 times.
Fixes#24735.
Change-Id: Id17c5bd18bd09987bb48ed12420d61ae9e200fd7
Reviewed-on: https://go-review.googlesource.com/c/139258
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
If the test has already completed when a go routine with a panic
handler reports an error the location of the error call is lost.
Added logDepth to be able to log location of failure at different
depths down the stack.
Fixes#26720
Change-Id: I8b7789ddae757ef6f4bd315cb20356709f4fadec
Reviewed-on: https://go-review.googlesource.com/c/127596
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
I omitted vendor directories and anything necessary for bootstrapping.
(Tested by bootstrapping with Go 1.4)
Updates #27864
Change-Id: I7d9b68d0372d3a34dee22966cca323513ece7e8a
Reviewed-on: https://go-review.googlesource.com/137856
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The previous CL (https://go-review.googlesource.com/c/go/+/96756)
added comments that didn't really say much, but there is something
so say: what the units are and that they are indexed starting at 1.
Add a more helpful comment on the type, and also follow proper
style by using initial capitals and a period.
Change-Id: Id19cd5f392faf7c7bac034073f276cc770589075
Reviewed-on: https://go-review.googlesource.com/121875
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Fill in the missing descriptions for the CoverBlock struct fields
Change-Id: I9257881a19b01e5cfe61cf19a91375b6d7cc68ef
GitHub-Last-Rev: f5b9e1d49d
GitHub-Pull-Request: golang/go#24079
Reviewed-on: https://go-review.googlesource.com/96756
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Run-TryBot: Andrew Bonventre <andybons@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This reverts golang.org/cl/110775
Reason for revert: this is causing huge slow-dows on every run after
the 1st, on various benchmarks, on multiple architectures (see Issue
25622 for details). It's just a nice-to-have little optimization, and
we're near the 1st go1.11 beta release, so I'm reverting it.
Fixes#25622
Change-Id: I758ade4af4abf764abd8336d404396992d11a0c6
Reviewed-on: https://go-review.googlesource.com/115535
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
Since subtests and subbenchmarks run in a separate goroutine, and thus
a separate stack, this entails capturing the stack trace at the point
tb.Run is called. The work of getting the file and line information from
this stack is only done when needed, however.
Continuing the search into the parent test also requires temporarily
holding its mutex. Since Run does not hold it while waiting for the
subtest to complete, there should be no risk of a deadlock due to this.
Fixes#24128
Change-Id: If0bb169f3ac96bd48794624e619ade7edb599f83
Reviewed-on: https://go-review.googlesource.com/108658
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
When running a benchmark multiple times, instead of re-computing the
value of b.N each time, use the value found by the first run.
For
go test -bench=. -benchtime 3s -count 2 p_test.go
on the benchmark in the linked issue; before:
BenchmarkBenchmark-4 500 10180593 ns/op
--- BENCH: BenchmarkBenchmark-4
p_test.go:13: single call took 10.111079ms
p_test.go:13: single call took 1.017298685s
p_test.go:13: single call took 5.090096124s
BenchmarkBenchmark-4 500 10182164 ns/op
--- BENCH: BenchmarkBenchmark-4
p_test.go:13: single call took 10.098169ms
p_test.go:13: single call took 1.017712905s
p_test.go:13: single call took 5.090898517s
PASS
ok command-line-arguments 12.244s
and after:
BenchmarkBenchmark-4 500 10177076 ns/op
--- BENCH: BenchmarkBenchmark-4
p_test.go:13: single call took 10.091301ms
p_test.go:13: single call took 1.016943125s
p_test.go:13: single call took 5.088376028s
BenchmarkBenchmark-4 500 10171497 ns/op
--- BENCH: BenchmarkBenchmark-4
p_test.go:13: single call took 10.140245ms
p_test.go:13: single call took 5.085605921s
PASS
ok command-line-arguments 11.218s
Fixes#23423
Change-Id: Ie66a8c5ac43881eb8741e14105db28745b4d56d3
Reviewed-on: https://go-review.googlesource.com/110775
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
We need to grab the mutex before we can access it.
Fixes#24438
Change-Id: Idd6130036691acec5bc5f8b40d6884f8db1d9d3c
Reviewed-on: https://go-review.googlesource.com/101283
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The Go's heap profile contains four kinds of samples
(inuse_space, inuse_objects, alloc_space, and alloc_objects).
The pprof tool by default chooses the inuse_space (the bytes
of live, in-use objects). When analyzing the current memory
usage the choice of inuse_space as the default may be useful,
but in some cases, users are more interested in analyzing the
total allocation statistics throughout the program execution.
For example, when we analyze the memory profile from benchmark
or program test run, we are more likely interested in the whole
allocation history than the live heap snapshot at the end of
the test or benchmark.
The pprof tool provides flags to control which sample type
to be used for analysis. However, it is one of the less-known
features of pprof and we believe it's better to choose the
right type of samples as the default when producing the profile.
This CL introduces a new type of profile, "allocs", which is
the same as the "heap" profile but marks the alloc_space
as the default type unlike heap profiles that use inuse_space
as the default type.
'go test -memprofile=...' command is changed to use the new
"allocs" profile type instead of the traditional "heap" profile.
Fixes#24443
Change-Id: I012dd4b6dcacd45644d7345509936b8380b6fbd9
Reviewed-on: https://go-review.googlesource.com/102696
Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
Reviewed-by: Russ Cox <rsc@golang.org>
When a test calls t.Fatal()/t.Fatalf(), only deferred code will execute.
Increment the failure count as part of a deferred call.
Fixes#24412
Change-Id: Ibb154015fcd3d0fb7739718fdda8c9ad22f9e896
Reviewed-on: https://go-review.googlesource.com/101035
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Fixed a broken link to a section in the documentation for the
test flags for the go command.
Change-Id: Ic4bdd4965aac7856dd13a2adda9d774b9bae4113
GitHub-Last-Rev: 15bda34067
GitHub-Pull-Request: golang/go#24613
Reviewed-on: https://go-review.googlesource.com/103835
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
-race sets a hard cap of 8,192, which is easily hit while testing.
Fixes#23611
Change-Id: I0f720ec39c82c2194a485d437d6373f4bdc8a9c1
Reviewed-on: https://go-review.googlesource.com/103160
Reviewed-by: Rob Pike <r@golang.org>
I grepped for "bytes.Buffer" and "buf.String" and mostly ignored test
files. I skipped a few on purpose and probably missed a few others,
but otherwise I think this should be most of them.
Updates #18990
Change-Id: I5a6ae4296b87b416d8da02d7bfaf981d8cc14774
Reviewed-on: https://go-review.googlesource.com/102479
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Don’t panic if a subtest inadvertently calls FailNow
on a parent’s T. Instead, report the offending subtest
while still reporting the error with the ancestor test and
keep exiting goroutines.
Note that this implementation has a race if parallel
subtests are failing the parent concurrently.
This is fine:
Calling FailNow on a parent is considered an error
in principle, at the moment, and is reported if it is
detected. Having the race allows the race detector
to detect the error as well.
Fixes#22882
Change-Id: Ifa6d5e55bb88f6bcbb562fc8c99f1f77e320015a
Reviewed-on: https://go-review.googlesource.com/97635
Run-TryBot: Marcel van Lohuizen <mpvl@golang.org>
Reviewed-by: Kunpei Sakai <namusyaka@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
In func TestXxxx(*testing.T) the Xxxx can be anything that can appear
in an identifier, but can't start with a lowercase letter. Clarify the docs.
Fixes#23322
Change-Id: I5c297916981f7e3890ee955d12bc7422a75488e2
Reviewed-on: https://go-review.googlesource.com/86001
Reviewed-by: Rob Pike <r@golang.org>
Tests exist that call m.Run in a loop‽
Now we have one too.
Fixes#23129.
Change-Id: I8cbecb724f239ae14ad45d75e67d12c80e41c994
Reviewed-on: https://go-review.googlesource.com/83956
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
t.Run(f) does not wait for f after f calls t.Parallel.
Otherwise it would be impossible to create new
parallel sibling subtests for f.
Fixes#22993.
Change-Id: I27e1555ab1ff608eb8155db261d5e7ee8f486aef
Reviewed-on: https://go-review.googlesource.com/83880
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
When we write a cached test result, we now also write a log of the
environment variables and files inspected by the test run,
along with a hash of their content. Before reusing a cached test result,
we recompute the hash of the content specified by the log, and only
use the result if that content has not changed.
This makes test caching behave correctly for tests that consult
environment variables or stat or read files or directories.
Fixes#22593.
Change-Id: I8608798e73c90e0c1911a38bf7e03e1232d784dc
Reviewed-on: https://go-review.googlesource.com/81895
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The test binaries accept -timeout=0 to mean no timeout,
but then the backup timer in cmd/go kills the test after 1 minute.
Make cmd/go understand this special case and change
behavior accordingly.
Fixes#14780.
Change-Id: I66bf517173a4ad21d53a5ee88d163f04b8929fb6
Reviewed-on: https://go-review.googlesource.com/81499
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
It's not safe (it crashes), and it's also useless: if you run
multiple benchmarks in parallel you will not get reliable
timing results from any of them.
Fixes#18603.
Change-Id: I00e5a72f7c98151543cf7d5573c38383276e391a
Reviewed-on: https://go-review.googlesource.com/80841
Reviewed-by: Ian Lance Taylor <iant@golang.org>
When -test.failfast flag is provided to go test,
no new tests get started after the first failure.
Fixes#21700
Change-Id: I0092e72f25847af05e7c8e1b811dcbb65a00cbe7
Reviewed-on: https://go-review.googlesource.com/74450
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
