This is part 1 of a 2 part changelist. Part 2 contains the mechanical
change to parse.go to compare atoms (ints) instead of strings.
The overall effect of the two changes are:
benchmark old ns/op new ns/op delta
BenchmarkParser 4462274 4058254 -9.05%
BenchmarkRawLevelTokenizer 913202 912917 -0.03%
BenchmarkLowLevelTokenizer 1268626 1267836 -0.06%
BenchmarkHighLevelTokenizer 1947305 1968944 +1.11%
R=rsc
CC=andybalholm, golang-dev, r
https://golang.org/cl/6305053
Use perfect cuckoo hash, to avoid binary search.
Define Atom bits as offset+len in long string instead
of enumeration, to avoid string headers.
Before: 1909 string bytes + 6060 tables = 7969 total data
After: 1406 string bytes + 2048 tables = 3454 total data
benchmark old ns/op new ns/op delta
BenchmarkLookup 83878 64681 -22.89%
R=nigeltao, r
CC=golang-dev
https://golang.org/cl/6262051
Implement the (3-per-family) Noah's Ark clause (i.e. don't put
more than three identical elements on the list of active formatting
elements.
Also, when running tests, sort attributes by name before dumping
them.
Pass 4 additional tests with Noah's Ark clause (including one
that needs attributes to be sorted).
Pass 5 additional, unrelated tests because of sorting attributes.
R=nigeltao, rsc
CC=golang-dev
https://golang.org/cl/6247056
Remove redundant checks for integration points.
Ignore null bytes in text.
Don't break out of foreign content for a <font> tag unless it
has a color, face, or size attribute.
Check for MathML text integration points when breaking out of
foreign content.
Pass two new tests.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6256045
There's no need for the 16-bit arithmetic here,
and it tickles a long-standing compiler bug.
Fix the exp code not to use 16-bit math and
create an explicit test for the compiler bug.
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/6256048
Detect HTML integration points and MathML text integration points.
At these points, process tokens as HTML, not as foreign content.
Pass 33 more tests.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6249044
Import updated test data from the WebKit Subversion repository (SVN revision 118111).
Some of the old tests were failing because we were HTML5 compliant, but the tests weren't.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6228049
Handle text, comment, and doctype tokens in afterBodyIM, afterAfterBodyIM,
and afterAfterFramesetIM.
Pass three more tests.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6231043
Clean up flow of control.
Ignore </table>, </tbody>, </tfoot>, </thead>, </tr> if there is not
an appropriate element in table scope.
Pass 3 more tests.
R=golang-dev, nigeltao
CC=golang-dev
https://golang.org/cl/6206093
Delete cases that just fall down to "anything else" action.
Handle </tbody>, </tfoot>, and </thead>.
R=golang-dev, nigeltao
CC=golang-dev
https://golang.org/cl/6203061
key and simple comparisson. Search is not yet implemented in this CL.
Changed some of the types of table_test.go to allow reuse in the new test.
Also reduced number of primary values for illegal runes to 1 (both map to
the same).
R=r
CC=golang-dev
https://golang.org/cl/6202062
Also set maxContractLen automatically.
Note that the table size is much bigger than it needs to be.
Optimization is best done, though, when the language specific
tables are added.
R=r
CC=golang-dev
https://golang.org/cl/6167044
dictates a CJK rune is only part of a certain specified range if it
is explicitly defined in the Unicode Codepoint Database.
Fixed the code and some of the tests accordingly.
R=r
CC=golang-dev
https://golang.org/cl/6160044
The first bug was that tertiary ignorables had the same colElem as
implicit colElems, yielding unexpected results. The current encoding
ensures that a non-implicit colElem is never 0. This fix uncovered
another bug of the trie that indexed incorrectly into the null block.
This was caused by an unfinished optimization that would avoid the
need to max out the most-significant bits of continuation bytes.
This bug was also present in the trie used in exp/norm and has been
fixed there as well. The appearence of the bug was rare, as the lower
blocks happened to be nearly nil.
R=r
CC=golang-dev
https://golang.org/cl/6127070
context for change lists of lower-level types. The public APIs are defined
in builder.go and collate.go. Type table is the glue between the lower and
higher level code and might be a good starting point for understanding the
collation code.
R=r, r
CC=golang-dev
https://golang.org/cl/5999053
The trie code looks a lot like the trie in exp/norm. It uses different
types, however. Also, there is only a lookup for []byte and the unsafe
lookup methods have been dropped, as well as sparse mode.
There is now a method for generating a trie. To output Go code, one now needs
to first generate a trie and then call print() on it.
R=r, r, mpvl
CC=golang-dev
https://golang.org/cl/5966064
Don't foster-parent text nodes that consist only of whitespace.
(I implemented this entirely in inTableIM instead of creating an
inTableTextIM, because the sole purpose of inTableTextIM seems to be
to combine character tokens into a string, which our tokenizer does
already.)
Use parseImpliedToken to clarify a couple of cases.
Handle <style>, <script>, <input>, and <form>.
Ignore doctype tokens.
Pass 20 additional tests.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6117048
This CL corrects the remaining differences that I could find between the
implementation of inBodyIM and the spec:
Handle <rp> and <rt>.
Adjust SVG and MathML attributes.
Reconstruct active formatting elements in the "any other start tag" case.
Pass 7 additional tests.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6101055
Clean up the flow of control.
Fix the TODO for handling <html> tags.
Add a case to ignore doctype declarations.
Pass one additional test.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6072047
This method will allow us to be explicit about what we're doing when
we insert an implied token, and avoid repeating the logic involved in
multiple places.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6060048
Add a case to ignore doctype tokens.
Clean up the flow of control to more clearly match the spec.
Pass one more test.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6062047
Add case for doctype tokens (which are ignored).
This CL does not change the status of any tests.
R=golang-dev, nigeltao
CC=golang-dev
https://golang.org/cl/6061047
Currently, the html package only runs a limited subset of the tests
in the testdata directory. This tends to limit development of the
parser to fixing the bug that causes the first failing test.
This CL gives it the ability to run all the tests and produce a
log showing the status of each test. (It does it when tests are run with
'go test --update-logs') The status is listed as PASS, FAIL, or PARSE
(PARSE means that parsing produced the correct tree, but rendering and
re-parsing does not produce the same tree).
When 'go test' is run without --update-logs, it runs the tests marked
'PASS' in the logs (and the parsing portion of the tests marked 'PARSE').
Thus it will fail if there has been a regression since the last
time the logs were updated.
My goal for this CL is to allow develoment of the html package to
be less test-driven, while still having the advantages of regression
tests. In other words, one can work on any portion of the parser
and quickly see whether he is breaking things or improving them.
Current statistics of the tests:
$ grep ^PASS *.log|wc -l
1017
$ grep ^PARSE *.log|wc -l
46
$ grep ^FAIL *.log|wc -l
181
R=nigeltao
CC=golang-dev
https://golang.org/cl/6031049
below do not support '.
This makes package html consistent with package text/template's
HTMLEscape function.
Fixes#3489.
R=rsc, mikesamuel, dsymonds
CC=golang-dev
https://golang.org/cl/5992071
For completeness, we also expose the Canonical Combining Class of a rune.
This does not increase the data size.
R=r
CC=golang-dev
https://golang.org/cl/5931043
- Renamed ExportData -> FindGcExportData
and base it on an a bufio.Reader rather
than a filename so it can be used in
environments where object files are
stored elsewhere.
- Factor former GcImporter into GcImportData
and GcImport. Implementations with different
storage locations for object files can build
a customized GcImport using GcImportData.
This is pkg/exp only - no impact on Go 1.
R=golang-dev, lvd, rsc
CC=golang-dev
https://golang.org/cl/5574069
They have been deleted from package unsafe.
Also delete their appearance in exp/types.
Fixes#3338.
R=golang-dev, dsymonds
CC=golang-dev
https://golang.org/cl/5847056
This is an API change, but one I have been promising would
happen when it was clear what the go command needed.
This is basically a complete replacement of what used to be here.
build.Tree is gone.
build.DirInfo is expanded and now called build.Package.
build.FindTree is now build.Import(package, srcDir, build.FindOnly).
The returned *Package contains information that FindTree returned,
but applicable only to a single package.
build.ScanDir is now build.ImportDir.
build.FindTree+build.ScanDir is now build.Import.
The new Import API allows specifying the source directory,
in order to resolve local imports (import "./foo") and also allows
scanning of packages outside of $GOPATH. They will come back
with less information in the Package, but they will still work.
The old go/build API exposed both too much and too little.
This API is much closer to what the go command needs,
and it works well enough in the other places where it is
used. Path is gone, so it can no longer be misused. (Fixes issue 2749.)
This CL updates clients of go/build other than the go command.
The go command changes are in a separate CL, to be submitted
at the same time.
R=golang-dev, r, alex.brainman, adg
CC=golang-dev
https://golang.org/cl/5713043
When debugging ASTs, it's useful to also
see the comments on occasion. Usage:
gotype -ast -comments file.go
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/5703043
Not a Go 1 issue, but appeared to be fairly easy to fix.
- Note that a few existing test cases look slightly worse but
those cases were not representative for real code. All real
code looks better now.
- Manual move of the comment in go/scanner/example_test.go
before applying gofmt.
- gofmt -w $GOROOT/src $GOROOT/misc
Fixes#3062.
R=rsc
CC=golang-dev
https://golang.org/cl/5674093
by other low-level libraries, like collate. Extra care has been given to optimize the performance
of normalizing to NFD, as this is what will be used by the collator. The overhead of checking
whether a string is normalized vs simply decomposing a string is neglible. Assuming that most
strings are in the FCD form, this iterator can be used to decompose strings and normalize with
minimal overhead.
R=r
CC=golang-dev
https://golang.org/cl/5676057
* add -work option to save temporary files (Fixes issue 2980)
* fix go test -i to work with cgo packages (Fixes issue 2936)
* do not overwrite/remove empty directories or non-object
files during build (Fixes issue 2829)
* remove package main vs package non-main heuristic:
a directory must contain only one package (Fixes issue 2864)
* to make last item workable, ignore +build tags for files
named on command line: go build x.go builds x.go even
if it says // +build ignore.
* add // +build ignore tags to helper programs
R=golang-dev, r, r
CC=golang-dev
https://golang.org/cl/5674043
This CL makes it possible to run make.bash with
GOOS and GOARCH set to something other than
the native host GOOS and GOARCH.
As part of the CL, the tool directory moves from bin/tool/
to pkg/tool/goos_goarch where goos and goarch are
the values for the host system (running the build), not
the target. pkg/ is not technically appropriate, but C objects
are there now tool (pkg/obj/) so this puts all the generated
binaries in one place (rm -rf $GOROOT/pkg cleans everything).
Including goos_goarch in the name allows different systems
to share a single $GOROOT on a shared file system.
Fixes#2920.
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/5645093
Restore package os/signal, with new API:
Notify replaces Incoming, allowing clients
to ask for certain signals only. Also, signals
go to everyone who asks, not just one client.
This could plausibly move into package os now
that there are no magic side effects as a result
of the import.
Update runtime for new API: move common Unix
signal handling code into signal_unix.c.
(It's so easy to do this now that we don't have
to edit Makefiles!)
Tested on darwin,linux 386,amd64.
Fixes#1266.
R=r, dsymonds, bradfitz, iant, borman
CC=golang-dev
https://golang.org/cl/3749041
one trie lookup per rune is needed. See forminfo.go for a description
of the new format. Also included leading and trailing canonical
combining class in decomposition information. This will often avoid
additional trie lookups.
R=r, r
CC=golang-dev
https://golang.org/cl/5616071
Issue 2856 asks for a rename of a few methods to a
more idiomatic Go style. This is a very early API
that evolved organically throughout the years.
Together with the fact that ErrorVectors were embedded
in other data structures (e.g. Parser), just renaming
methods (e.g. GetError -> Error) would lead to undesired
behavior (e.g., Parser would act like an Error). Instead,
cleaned up API a bit more:
- removed ErrorVector in favor of ErrorList (already
present)
- simplified Scanner.Init by making the error handler a
function instead of requiring an ErrorHandler implementation
- adjusted helper functions accordingly
- updated Go 1 doc
Fixes#2856.
R=rsc
CC=golang-dev
https://golang.org/cl/5624047
The practice encourages people to think this is the way to
create a bytes.Buffer when new(bytes.Buffer) or
just var buf bytes.Buffer work fine.
(html/token.go was missing the point altogether.)
R=golang-dev, bradfitz, r
CC=golang-dev
https://golang.org/cl/5637043
The go- is redundant now that the directory is required
to be inside $GOROOT. Rob LGTMed the idea.
R=golang-dev, gri
CC=golang-dev
https://golang.org/cl/5618044
- Unified bounary conditions for NFC and NFD and removed some indirections.
This enforces boundaries at the character level, which is typically what
the user expects. (NFD allows a boundary between 'a' and '`', for example,
which may give unexpected results for collation. The current implementation
is already stricter than the standard, so nothing much changes. This change
just formalizes it.
- Moved methods of qcflags to runeInfo.
- Swapped YesC and YesMaybe bits in qcFlags. This is to aid future changes.
- runeInfo return values use named fields in preperation for struct change.
- Replaced some left-over uint32s with rune.
R=r
CC=golang-dev
https://golang.org/cl/5607050
Consequently, remove many package Makefiles,
and shorten the few that remain.
gomake becomes 'go tool make'.
Turn off test phases of run.bash that do not work,
flagged with $BROKEN. Future CLs will restore these,
but this seemed like a big enough CL already.
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/5601057
Also delete gotest, since it's messy to fix and slated for deletion anyway.
A couple of things outside src can't be tested any more. "go test" will be
fixed and these tests will be re-enabled. They're noisy for now.
Fixes#284.
R=rsc
CC=golang-dev
https://golang.org/cl/5598049
work in progress, and we are not ready to freeze its API for Go 1.
Package html still exists, containing just two functions: EscapeString
and UnescapeString.
Both the packages at exp/html and html are "package html". The former
is a superset of the latter.
At some point in the future, the exp/html code will move back into
html, once we have finalized the parser API.
R=rsc, dsymonds
CC=golang-dev
https://golang.org/cl/5571059
Remove NewMD5, NewSHA1 and NewSHA256 in favor of using New and
explicitly importing the used hash-function. This way when using, for
example, HMAC with RIPEMD there's no md5, sha1 and sha256 linked in
through the hmac package.
A gofix rule is included, and applied to the standard library (3 files
altered).
This change is the result of a discussion at
https://golang.org/cl/5550043/ to pull the discussion about
deprecating these functions out of that issue.
R=golang-dev, agl
CC=golang-dev, r, rsc
https://golang.org/cl/5556058
Previously, a timeout (in int64 nanoseconds) applied to a granularity
even smaller than one operation: a 100 byte read with a 1 second timeout
could take 100 seconds, if the bytes all arrived on the network 1 second
apart. This was confusing.
Rather than making the timeout granularity be per-Read/Write,
this CL makes callers set an absolute deadline (in time.Time)
after which operations will fail. This makes it possible to
set deadlines at higher levels, without knowing exactly how
many read/write operations will happen in e.g. reading an HTTP
request.
Fixes#2723
R=r, rsc, dave
CC=golang-dev
https://golang.org/cl/5555048
- Use a faster closure-based iterator rather than a channel-based one.
- Otherwise: better code organization, but no other API changes.
R=r, r
CC=golang-dev
https://golang.org/cl/5557051
First was, apart from adding tests, a single line of code (to add the
constructor function). Adding SHA512-based hashing to crypto/hmac
required minor rework of the package because of a previously hardcoded
block-size in it's implementation. Instead of using a hash.Hash
generator function the constructor function now uses a crypto.Hash
type, which was extended to expose information about block size.
The only standard library package impacted by the change is
crypto/tls, for which the fix is included in this patch. It might be
useful to extend gofix to include this API change too.
R=agl, r, rsc, r
CC=golang-dev
https://golang.org/cl/5550043
The terminal code in exp/terminal was forked from the code in exp/ssh.
This change removes the duplicated code from exp/ssh in favour of
using exp/terminal.
R=rsc
CC=golang-dev
https://golang.org/cl/5375064
(This was missing in the last change because I uploaded it from the
wrong machine.)
Large pastes previously misbehaved because the code tried reading from
the terminal before checking whether an line was already buffered.
Large pastes can cause multiples lines to be read at once from the
terminal.
R=bradfitz
CC=golang-dev
https://golang.org/cl/5542049
exp/proxy provides client support for tunneling connections through
various proxies.
This is an initial, incomplete sketch of the code to lay down an
API.
R=golang-dev, r, r, bradfitz, rsc
CC=golang-dev
https://golang.org/cl/5490062
Also verified in external test suite that this fixes MySQL
resource exhaustion problems, and also exposed a double-free
bug in the gosqlite3 driver (where gosqlite3 either got lucky
before, or was working around this bug)
R=golang-dev, rsc
CC=golang-dev
https://golang.org/cl/5544057
This is a relic from the times when we switched
to automatic semicolon insertion. It's still use-
ful to have a non-exported switch for testing.
R=golang-dev, r, rsc
CC=golang-dev
https://golang.org/cl/5528077
This CL fixes two issues sending data to the remote peer.
The first bug occurs when the size of the buffer passed to
Write is larger than the current window, in this case, w.rwin
can become negative.
The second issue is more problematic than the first as the
amount of data passed to writePacket was not limited to w.rwin.
In this case the remote peer could silently drop the additional
data, or drop the connection.
Credit to Jacek Masiulaniec for the bug report.
R=agl, jacek.masiulaniec
CC=golang-dev
https://golang.org/cl/5511043
transport.go:
* remove unused nil check.
doc.go:
* improve documentation about supported auth
methods and update Run example.
Thanks Jacek Masiulaniec for both reports.
R=jacek.masiulaniec, agl
CC=golang-dev
https://golang.org/cl/5501075
1) incorrect length given for out buffer in String.
2) patchTail bug that could cause characters to be lost
when crossing into the out-buffer boundary.
Added tests to expose these bugs. Also slightly improved
performance of Bytes() and String() by sharing the reorderBuffer
across operations.
Fixes#2567.
R=r
CC=golang-dev
https://golang.org/cl/5502069
1) Add EscapeCodes to the terminal so that applications don't wire
them in.
2) Add a callback for auto-complete
3) Fix an issue with input lines longer than the width of the
terminal.
4) Have Write() not stomp the current line. It now erases the current
input, writes the output and reprints the prompt and partial input.
5) Support prompting without local echo in Terminal.
6) Add GetSize to report the size of terminal.
R=bradfitz
CC=golang-dev
https://golang.org/cl/5479043
- no empty lines inside empty structs and interfaces
- top-level declarations are separated by a blank line if
a) they are of different kind (e.g. const vs type); or
b) there are documentation comments associated with a
declaration (this is new)
- applied gofmt -w misc src
The actual changes are in go/printer/nodes.go:397-400 (empty structs/interfaces),
and go/printer/printer.go:307-309 (extra line break). The remaining
changes are cleanups w/o changing the existing functionality.
Fixes issue 2570.
R=rsc
CC=golang-dev
https://golang.org/cl/5493057
If a Pipe method is called, return the underlying
reader/writer from session.clientChan, bypassing the
io.Copy and io.Pipe harness.
StdoutPipe and StderrPipe now return an io.Reader not
an io.ReadCloser as SSH cannot signal the close of the
local reader to the remote process.
R=rsc, agl, gustav.paul, cw
CC=golang-dev
https://golang.org/cl/5493047
I added the clientChan's msg channel to the list of channels that are closed in mainloop when the server sends a channelCloseMsg.
I added an ExitError type that wraps a Waitmsg similar to that of os/exec. I fill ExitStatus with the data returned in the 'exit-status' channel message and Msg with the data returned in the 'exit-signal' channel message.
Instead of having Wait() return on the first 'exit-status'/'exit-signal' I have it return an ExitError containing the status and signal when the clientChan's msg channel is closed.
I added two tests cases to session_test.go that test for exit status 0 (in which case Wait() returns nil) and exit status 1 (in which case Wait() returns an ExitError with ExitStatus 1)
R=dave, agl, rsc, golang-dev, bradfitz
CC=golang-dev
https://golang.org/cl/5452051
This CL cleans up the client auth tests, making the
individual test body more manageable.
Also, adds tests for rsa and dsa key negotiation.
Finally, remove the package level use of the variable
strings, which avoids conflicting with the strings pkg.
R=gustav.paul, agl, n13m3y3r, rsc
CC=golang-dev
https://golang.org/cl/5447049
This is part one of a small set of CL's that aim to resolve
the outstanding TODOs relating to channel close and blocking
behavior.
Firstly, the hairy handling of assigning the peersId is now
done in one place. The cost of this change is the slightly
paradoxical construction of the partially created clientChan.
Secondly, by creating clientChan.stdin/out/err when the channel
is opened, the creation of consumers like tcpchan and Session
is simplified; they just have to wire themselves up to the
relevant readers/writers.
R=agl, gustav.paul, rsc
CC=golang-dev
https://golang.org/cl/5448073
All but 3 cases (in gcimporter.go and hixie.go)
are automatic conversions using gofix.
No attempt is made to use the new Append functions
even though there are definitely opportunities.
R=golang-dev, gri
CC=golang-dev
https://golang.org/cl/5447069
The SSH spec allows for the server to send a banner message to the client at any point during the authentication process. Currently the ssh client auth types all assume that the first response from the server after issuing a userAuthRequestMsg will be one of a couple of possible authentication success/failure messages. This means that client authentication breaks if the ssh server being connected to has a banner message configured.
This changeset refactors the noneAuth, passwordAuth and publickeyAuth types' auth() function and allows for msgUserAuthBanner during authentication.
R=golang-dev, rsc, dave, agl
CC=golang-dev
https://golang.org/cl/5432065
