This is more in sync with the rest of the package;
for instance, we have functions (not methods) to
deref or find the underlying type of a Type.
In the process use a single bytes.Buffer to create
the string representation for a type rather than
the (occasional) string concatenation.
R=r
CC=golang-dev
https://golang.org/cl/6458057
If an end tag has an attribute that is a quoted string containing '>',
the tokenizer would end the tag prematurely. Now it reads the attributes
on end tags just as it does on start tags, but the high-level interface
still doesn't return them, because their presence is a parse error.
Pass 1 additional test.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6457060
- Allow handles into the trie for different locales. Multiple tables share the same
try to allow for reuse of blocks.
- Significantly improved memory footprint and reduced allocations of trieNodes.
This speeds up generation by about 30% and allows keeping trieNodes around
for multiple locales during generation.
- Renamed print method to fprint.
R=r
CC=golang-dev
https://golang.org/cl/6408052
If NUL bytes occur inside certain elements, convert them to U+FFFD
replacement character.
Pass 1 additional test.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6452047
Fixed creation of Func's, taking IsVariadic from parameter list rather
than results.
Updated checkObj to process ast.Fun objects.
R=gri
CC=golang-dev
https://golang.org/cl/6402046
If the body of an HTML document contains text, the <frameset> tag is
ignored. But not if the text is only whitespace.
Pass 4 additional tests.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6442043
Don't unescape entities in attributes when they don't end with
a semicolon and they are followed by '=', a letter, or a digit.
Pass 6 more tests from the WebKit test suite, plus one that was
commented out in token_test.go.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6405073
- Changed the representation of colElem to support a few cases
for some languages not supported by the current format.
- Changed offsets for implicit primary values. This makes the
values both easier to read and debug (last 4 nibbles are identical to
implicit primary value) and also results in better packing.
- Fixed bug in weight conversion code that did not pop up yet by
sheer luck.
Note that tables.go also includes changes to the contraction trie
from CL 6346092.
R=r, mpvl
CC=golang-dev
https://golang.org/cl/6392060
which has a rather large contraction table. The value of the next state
offset now starts after the current block, instead of before. This is
slightly less efficient (on extra addition per state change), but gives
some extra range for the offsets.
Also introduced constants for final (0) and noIndex (0xFF).
tables.go is updated in a separate CL.
R=r
CC=golang-dev
https://golang.org/cl/6346092
This is part 1 of a 2 part changelist. Part 2 contains the mechanical
change to parse.go to compare atoms (ints) instead of strings.
The overall effect of the two changes are:
benchmark old ns/op new ns/op delta
BenchmarkParser 4462274 4058254 -9.05%
BenchmarkRawLevelTokenizer 913202 912917 -0.03%
BenchmarkLowLevelTokenizer 1268626 1267836 -0.06%
BenchmarkHighLevelTokenizer 1947305 1968944 +1.11%
R=rsc
CC=andybalholm, golang-dev, r
https://golang.org/cl/6305053
Use perfect cuckoo hash, to avoid binary search.
Define Atom bits as offset+len in long string instead
of enumeration, to avoid string headers.
Before: 1909 string bytes + 6060 tables = 7969 total data
After: 1406 string bytes + 2048 tables = 3454 total data
benchmark old ns/op new ns/op delta
BenchmarkLookup 83878 64681 -22.89%
R=nigeltao, r
CC=golang-dev
https://golang.org/cl/6262051
Implement the (3-per-family) Noah's Ark clause (i.e. don't put
more than three identical elements on the list of active formatting
elements.
Also, when running tests, sort attributes by name before dumping
them.
Pass 4 additional tests with Noah's Ark clause (including one
that needs attributes to be sorted).
Pass 5 additional, unrelated tests because of sorting attributes.
R=nigeltao, rsc
CC=golang-dev
https://golang.org/cl/6247056
Remove redundant checks for integration points.
Ignore null bytes in text.
Don't break out of foreign content for a <font> tag unless it
has a color, face, or size attribute.
Check for MathML text integration points when breaking out of
foreign content.
Pass two new tests.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6256045
There's no need for the 16-bit arithmetic here,
and it tickles a long-standing compiler bug.
Fix the exp code not to use 16-bit math and
create an explicit test for the compiler bug.
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/6256048
Detect HTML integration points and MathML text integration points.
At these points, process tokens as HTML, not as foreign content.
Pass 33 more tests.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6249044
Import updated test data from the WebKit Subversion repository (SVN revision 118111).
Some of the old tests were failing because we were HTML5 compliant, but the tests weren't.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6228049
Handle text, comment, and doctype tokens in afterBodyIM, afterAfterBodyIM,
and afterAfterFramesetIM.
Pass three more tests.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6231043
Clean up flow of control.
Ignore </table>, </tbody>, </tfoot>, </thead>, </tr> if there is not
an appropriate element in table scope.
Pass 3 more tests.
R=golang-dev, nigeltao
CC=golang-dev
https://golang.org/cl/6206093
Delete cases that just fall down to "anything else" action.
Handle </tbody>, </tfoot>, and </thead>.
R=golang-dev, nigeltao
CC=golang-dev
https://golang.org/cl/6203061
key and simple comparisson. Search is not yet implemented in this CL.
Changed some of the types of table_test.go to allow reuse in the new test.
Also reduced number of primary values for illegal runes to 1 (both map to
the same).
R=r
CC=golang-dev
https://golang.org/cl/6202062
Also set maxContractLen automatically.
Note that the table size is much bigger than it needs to be.
Optimization is best done, though, when the language specific
tables are added.
R=r
CC=golang-dev
https://golang.org/cl/6167044
dictates a CJK rune is only part of a certain specified range if it
is explicitly defined in the Unicode Codepoint Database.
Fixed the code and some of the tests accordingly.
R=r
CC=golang-dev
https://golang.org/cl/6160044
The first bug was that tertiary ignorables had the same colElem as
implicit colElems, yielding unexpected results. The current encoding
ensures that a non-implicit colElem is never 0. This fix uncovered
another bug of the trie that indexed incorrectly into the null block.
This was caused by an unfinished optimization that would avoid the
need to max out the most-significant bits of continuation bytes.
This bug was also present in the trie used in exp/norm and has been
fixed there as well. The appearence of the bug was rare, as the lower
blocks happened to be nearly nil.
R=r
CC=golang-dev
https://golang.org/cl/6127070
context for change lists of lower-level types. The public APIs are defined
in builder.go and collate.go. Type table is the glue between the lower and
higher level code and might be a good starting point for understanding the
collation code.
R=r, r
CC=golang-dev
https://golang.org/cl/5999053
The trie code looks a lot like the trie in exp/norm. It uses different
types, however. Also, there is only a lookup for []byte and the unsafe
lookup methods have been dropped, as well as sparse mode.
There is now a method for generating a trie. To output Go code, one now needs
to first generate a trie and then call print() on it.
R=r, r, mpvl
CC=golang-dev
https://golang.org/cl/5966064
Don't foster-parent text nodes that consist only of whitespace.
(I implemented this entirely in inTableIM instead of creating an
inTableTextIM, because the sole purpose of inTableTextIM seems to be
to combine character tokens into a string, which our tokenizer does
already.)
Use parseImpliedToken to clarify a couple of cases.
Handle <style>, <script>, <input>, and <form>.
Ignore doctype tokens.
Pass 20 additional tests.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6117048
This CL corrects the remaining differences that I could find between the
implementation of inBodyIM and the spec:
Handle <rp> and <rt>.
Adjust SVG and MathML attributes.
Reconstruct active formatting elements in the "any other start tag" case.
Pass 7 additional tests.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6101055
Clean up the flow of control.
Fix the TODO for handling <html> tags.
Add a case to ignore doctype declarations.
Pass one additional test.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6072047
This method will allow us to be explicit about what we're doing when
we insert an implied token, and avoid repeating the logic involved in
multiple places.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6060048
Add a case to ignore doctype tokens.
Clean up the flow of control to more clearly match the spec.
Pass one more test.
R=nigeltao
CC=golang-dev
https://golang.org/cl/6062047
Add case for doctype tokens (which are ignored).
This CL does not change the status of any tests.
R=golang-dev, nigeltao
CC=golang-dev
https://golang.org/cl/6061047
Currently, the html package only runs a limited subset of the tests
in the testdata directory. This tends to limit development of the
parser to fixing the bug that causes the first failing test.
This CL gives it the ability to run all the tests and produce a
log showing the status of each test. (It does it when tests are run with
'go test --update-logs') The status is listed as PASS, FAIL, or PARSE
(PARSE means that parsing produced the correct tree, but rendering and
re-parsing does not produce the same tree).
When 'go test' is run without --update-logs, it runs the tests marked
'PASS' in the logs (and the parsing portion of the tests marked 'PARSE').
Thus it will fail if there has been a regression since the last
time the logs were updated.
My goal for this CL is to allow develoment of the html package to
be less test-driven, while still having the advantages of regression
tests. In other words, one can work on any portion of the parser
and quickly see whether he is breaking things or improving them.
Current statistics of the tests:
$ grep ^PASS *.log|wc -l
1017
$ grep ^PARSE *.log|wc -l
46
$ grep ^FAIL *.log|wc -l
181
R=nigeltao
CC=golang-dev
https://golang.org/cl/6031049
below do not support '.
This makes package html consistent with package text/template's
HTMLEscape function.
Fixes#3489.
R=rsc, mikesamuel, dsymonds
CC=golang-dev
https://golang.org/cl/5992071
For completeness, we also expose the Canonical Combining Class of a rune.
This does not increase the data size.
R=r
CC=golang-dev
https://golang.org/cl/5931043
- Renamed ExportData -> FindGcExportData
and base it on an a bufio.Reader rather
than a filename so it can be used in
environments where object files are
stored elsewhere.
- Factor former GcImporter into GcImportData
and GcImport. Implementations with different
storage locations for object files can build
a customized GcImport using GcImportData.
This is pkg/exp only - no impact on Go 1.
R=golang-dev, lvd, rsc
CC=golang-dev
https://golang.org/cl/5574069
They have been deleted from package unsafe.
Also delete their appearance in exp/types.
Fixes#3338.
R=golang-dev, dsymonds
CC=golang-dev
https://golang.org/cl/5847056
This is an API change, but one I have been promising would
happen when it was clear what the go command needed.
This is basically a complete replacement of what used to be here.
build.Tree is gone.
build.DirInfo is expanded and now called build.Package.
build.FindTree is now build.Import(package, srcDir, build.FindOnly).
The returned *Package contains information that FindTree returned,
but applicable only to a single package.
build.ScanDir is now build.ImportDir.
build.FindTree+build.ScanDir is now build.Import.
The new Import API allows specifying the source directory,
in order to resolve local imports (import "./foo") and also allows
scanning of packages outside of $GOPATH. They will come back
with less information in the Package, but they will still work.
The old go/build API exposed both too much and too little.
This API is much closer to what the go command needs,
and it works well enough in the other places where it is
used. Path is gone, so it can no longer be misused. (Fixes issue 2749.)
This CL updates clients of go/build other than the go command.
The go command changes are in a separate CL, to be submitted
at the same time.
R=golang-dev, r, alex.brainman, adg
CC=golang-dev
https://golang.org/cl/5713043
When debugging ASTs, it's useful to also
see the comments on occasion. Usage:
gotype -ast -comments file.go
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/5703043
Not a Go 1 issue, but appeared to be fairly easy to fix.
- Note that a few existing test cases look slightly worse but
those cases were not representative for real code. All real
code looks better now.
- Manual move of the comment in go/scanner/example_test.go
before applying gofmt.
- gofmt -w $GOROOT/src $GOROOT/misc
Fixes#3062.
R=rsc
CC=golang-dev
https://golang.org/cl/5674093
by other low-level libraries, like collate. Extra care has been given to optimize the performance
of normalizing to NFD, as this is what will be used by the collator. The overhead of checking
whether a string is normalized vs simply decomposing a string is neglible. Assuming that most
strings are in the FCD form, this iterator can be used to decompose strings and normalize with
minimal overhead.
R=r
CC=golang-dev
https://golang.org/cl/5676057
* add -work option to save temporary files (Fixes issue 2980)
* fix go test -i to work with cgo packages (Fixes issue 2936)
* do not overwrite/remove empty directories or non-object
files during build (Fixes issue 2829)
* remove package main vs package non-main heuristic:
a directory must contain only one package (Fixes issue 2864)
* to make last item workable, ignore +build tags for files
named on command line: go build x.go builds x.go even
if it says // +build ignore.
* add // +build ignore tags to helper programs
R=golang-dev, r, r
CC=golang-dev
https://golang.org/cl/5674043
This CL makes it possible to run make.bash with
GOOS and GOARCH set to something other than
the native host GOOS and GOARCH.
As part of the CL, the tool directory moves from bin/tool/
to pkg/tool/goos_goarch where goos and goarch are
the values for the host system (running the build), not
the target. pkg/ is not technically appropriate, but C objects
are there now tool (pkg/obj/) so this puts all the generated
binaries in one place (rm -rf $GOROOT/pkg cleans everything).
Including goos_goarch in the name allows different systems
to share a single $GOROOT on a shared file system.
Fixes#2920.
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/5645093
Restore package os/signal, with new API:
Notify replaces Incoming, allowing clients
to ask for certain signals only. Also, signals
go to everyone who asks, not just one client.
This could plausibly move into package os now
that there are no magic side effects as a result
of the import.
Update runtime for new API: move common Unix
signal handling code into signal_unix.c.
(It's so easy to do this now that we don't have
to edit Makefiles!)
Tested on darwin,linux 386,amd64.
Fixes#1266.
R=r, dsymonds, bradfitz, iant, borman
CC=golang-dev
https://golang.org/cl/3749041
one trie lookup per rune is needed. See forminfo.go for a description
of the new format. Also included leading and trailing canonical
combining class in decomposition information. This will often avoid
additional trie lookups.
R=r, r
CC=golang-dev
https://golang.org/cl/5616071
Issue 2856 asks for a rename of a few methods to a
more idiomatic Go style. This is a very early API
that evolved organically throughout the years.
Together with the fact that ErrorVectors were embedded
in other data structures (e.g. Parser), just renaming
methods (e.g. GetError -> Error) would lead to undesired
behavior (e.g., Parser would act like an Error). Instead,
cleaned up API a bit more:
- removed ErrorVector in favor of ErrorList (already
present)
- simplified Scanner.Init by making the error handler a
function instead of requiring an ErrorHandler implementation
- adjusted helper functions accordingly
- updated Go 1 doc
Fixes#2856.
R=rsc
CC=golang-dev
https://golang.org/cl/5624047
The practice encourages people to think this is the way to
create a bytes.Buffer when new(bytes.Buffer) or
just var buf bytes.Buffer work fine.
(html/token.go was missing the point altogether.)
R=golang-dev, bradfitz, r
CC=golang-dev
https://golang.org/cl/5637043
The go- is redundant now that the directory is required
to be inside $GOROOT. Rob LGTMed the idea.
R=golang-dev, gri
CC=golang-dev
https://golang.org/cl/5618044
- Unified bounary conditions for NFC and NFD and removed some indirections.
This enforces boundaries at the character level, which is typically what
the user expects. (NFD allows a boundary between 'a' and '`', for example,
which may give unexpected results for collation. The current implementation
is already stricter than the standard, so nothing much changes. This change
just formalizes it.
- Moved methods of qcflags to runeInfo.
- Swapped YesC and YesMaybe bits in qcFlags. This is to aid future changes.
- runeInfo return values use named fields in preperation for struct change.
- Replaced some left-over uint32s with rune.
R=r
CC=golang-dev
https://golang.org/cl/5607050
