The typical UNIX method for controlling long running process is to
send the process signals. Since this doesn't get you very far, various
ad-hoc, remote-control protocols have been used over time by programs
like Apache and BIND.
Implementing an SSH server means that Go code will have a standard,
secure way to do this in the future.
R=bradfitz, borman, dave, gustavo, dsymonds, r, adg, rsc, rogpeppe, lvd, kevlar, raul.san
CC=golang-dev
https://golang.org/cl/4962064
I found a simple test case that does require doing the fixed point TODO
in computeOutCtx.
I found a way though to do this and simplify away the escapeRange
hackiness that was added in https://golang.org/cl/5012044/
R=nigeltao
CC=golang-dev
https://golang.org/cl/5015052
This replaces the errStr & errLine members of context with a single err
*Error, and introduces a number of const error codes, one per
escape-time failure mode, that can be separately documented.
The changes to the error documentation moved from doc.go to error.go
are cosmetic.
R=r, nigeltao
CC=golang-dev
https://golang.org/cl/5026041
Not all content is plain text. Sometimes content comes from a trusted
source, such as another template invocation, an HTML tag whitelister,
etc.
Template authors can deal with over-escaping in two ways.
1) They can encapsulate known-safe content via
type HTML, type CSS, type URL, and friends in content.go.
2) If they know that the for a particular action never needs escaping
then they can add |noescape to the pipeline.
{{.KnownSafeContent | noescape}}
which will prevent any escaping directives from being added.
This CL defines string type aliases: HTML, CSS, JS, URI, ...
It then modifies stringify to unpack the content type.
Finally it modifies the escaping functions to use the content type and
decline to escape content that does not require it.
There are minor changes to escapeAction and helpers to treat as
equivalent explicit escaping directives such as "html" and "urlquery"
and the escaping directives defined in the contextual autoescape module
and to recognize the special "noescape" directive.
The html escaping functions are rearranged. Instead of having one
escaping function used in each {{.}} in
{{.}} : <textarea title="{{.}}">{{.}}</textarea>
a slightly different escaping function is used for each.
When {{.}} binds to a pre-sanitized string of HTML
`one < <i>two</i> & two < "3"`
we produces something like
one < <i>two</i> & two < "3" :
<textarea title="one < two & two < "3"">
one < <i>two</i> & two < "3"
</textarea>
Although escaping is not required in <textarea> normally, if the
substring </textarea> is injected, then it breaks, so we normalize
special characters in RCDATA and do the same to preserve attribute
boundaries. We also strip tags since developers never intend
typed HTML injected in an attribute to contain tags escaped, but
do occasionally confuse pre-escaped HTML with HTML from a
tag-whitelister.
R=golang-dev, nigeltao
CC=golang-dev
https://golang.org/cl/4962067
This moots a caveat in the proposed package documentation by
rendering useless any template that could not be escaped.
From https://golang.org/cl/4969078/
> If EscapeSet returns an error, do not Execute the set; it is not
> safe against injection.
r: [but isn't the returned set nil? i guess you don't overwrite the
r: original if there's a problem, but i think you're in your rights to
r: do so]
R=r
CC=golang-dev
https://golang.org/cl/5020043
This adds support for {{template "callee"}} calls.
It recognizes that calls can appear in many contexts.
{{if .ImageURL}}
<img src="{{.ImageURL}}" alt="{{template "description"}}">
{{else}}
<p>{{template "description"}}</p>
{{end}}
calls a template in two different contexts, first in an HTML attribute
context, and second in an HTML text context.
Those two contexts aren't very different, but when linking text
to search terms, the escaping context can be materially different:
<a href="/search?q={{template "tags"}}">{{template "tags"}}</a>
This adds API:
EscapeSet(*template.Set, names ...string) os.Error
takes a set of templates and the names of those which might be called
in the default context as starting points.
It changes the escape* functions to be methods of an object which
maintains a conceptual mapping of
(template names*input context) -> output context.
The actual mapping uses as key a mangled name which combines the
template name with the input context.
The mangled name when the input context is the default context is the
same as the unmangled name.
When a template is called in multiple contexts, we clone the template.
{{define "tagLink"}}
<a href="/search?q={{template "tags"}}">{{template "tags"}}</a>
{{end}}
{{define "tags"}}
{{range .Tags}}{{.}},{{end}}
{{end}}
given []string{ "foo", "O'Reilly", "bar" } produces
<a href="/search?q=foo,O%27Reilly,bar">foo,O'Reilly,bar</a>
This involves rewriting the above to something like
{{define "tagLink"}}
<a href="/search?q={{template "tags$1"}}">{{template "tags"}}</a>
{{end}}
{{define "tags"}}
{{range .Tags}}{{. | html}},{{end}}
{{end}}
{{define "tags$1"}}
{{range .Tags}}{{. | urlquery}},{{end}}
{{end}}
clone.go provides a mechanism for cloning template "tags" to produce
"tags$1".
changes to escape.go implement the new API and context propagation
around the call graph.
context.go includes minor changes to support name mangling and
context_test.go tests those.
js.go contains a bug-fix.
R=nigeltao, r
CC=golang-dev
https://golang.org/cl/4969072
Often, division/regexp ambiguity doesn't matter in JS because the next
token is not a slash.
For example, in
<script>var global{{if .InitVal}} = {{.InitVal}}{{end}}</script>
When there is an initial value, the {{if}} ends with jsCtxDivOp
since a '/' following {{.InitVal}} would be a division operator.
When there is none, the empty {{else}} branch ends with jsCtxRegexp
since a '/' would start a regular expression. A '/' could result
in a valid program if it were on a new line to allow semicolon
insertion to terminate the VarDeclaration.
There is no '/' though, so we can ignore the ambiguity.
There are cases where a missing semi can result in ambiguity that
we should report.
<script>
{{if .X}}var x = {{.X}}{{end}}
/...{{.Y}}
</script>
where ... could be /foo/.test(bar) or /divisor. Disambiguating in
this case is hard and is required to sanitize {{.Y}}.
Note, that in the case where there is a '/' in the script tail but it
is not followed by any interpolation, we already don't care. So we
are already tolerant of
<script>{{if .X}}var x = {{.X}}{{end}}/a-bunch-of-text</script>
because tJS checks for </script> before looking in /a-bunch-of-text.
This CL
- Adds a jsCtx value: jsCtxUnknown
- Changes joinContext to join contexts that only differ by jsCtx.
- Changes tJS to return an error when a '/' is seen in jsCtxUnknown.
- Adds tests for both the happy and sad cases.
R=nigeltao
CC=golang-dev
https://golang.org/cl/4956077
Augments type context and adds grammatical rules to handle special HTML constructs:
<!-- comments -->
<script>raw text</script>
<textarea>no tags here</textarea>
This CL does not elide comment content. I recommend we do that but
have not done it in this CL.
I used a codesearch tool over a codebase in another template language.
Based on the below I think we should definitely recognize
<script>, <style>, <textarea>, and <title>
as each of these appears frequently enough that there are few
template using apps that do not use most of them.
Of the other special tags,
<xmp>, <noscript>
are used but infrequently, and
<noframe> and friend, <listing>
do not appear at all.
We could support <xmp> even though it is obsolete in HTML5
because we already have the machinery, but I suggest we do not
support noscript since it is a normal tag in some browser
configurations.
I suggest recognizing and eliding <!-- comments -->
(but not escaping text spans) as they are widely used to
embed comments in template source. Not eliding them increases
the size of content sent over the network, and risks leaking
code and project internal details.
The template language I tested elides them so there are
no instance of IE conditional compilation directives in the
codebase but that could be a source of confusion.
The codesearch does the equivalent of
$ find . -name \*.file-extension \
| perl -ne 'print "\L$1\n" while s@<([a-z][a-z0-9])@@i' \
| sort | uniq -c | sort
The 5 uses of <plaintext> seem to be in tricky code and can be ignored.
The 2 uses of <xmp> appear in the same tricky code and can be ignored.
I also ignored end tags to avoid biasing against unary
elements and threw out some nonsense names since since the
long tail is dominated by uses of < as a comparison operator
in the template languages expression language.
I have added asterisks next to abnormal elements.
26765 div
7432 span
7414 td
4233 a
3730 tr
3238 input
2102 br
1756 li
1755 img
1674 table
1388 p
1311 th
1064 option
992 b
891 label
714 script *
519 ul
446 tbody
412 button
381 form
377 h2
358 select
353 strong
318 h3
314 body
303 html
266 link
262 textarea *
261 head
258 meta
225 title *
189 h1
176 col
156 style *
151 hr
119 iframe
103 h4
101 pre
100 dt
98 thead
90 dd
83 map
80 i
69 object
66 ol
65 em
60 param
60 font
57 fieldset
51 string
51 field
51 center
44 bidi
37 kbd
35 legend
30 nobr
29 dl
28 var
26 small
21 cite
21 base
20 embed
19 colgroup
12 u
12 canvas
10 sup
10 rect
10 optgroup
10 noscript *
9 wbr
9 blockquote
8 tfoot
8 code
8 caption
8 abbr
7 msg
6 tt
6 text
6 h5
5 svg
5 plaintext *
5 article
4 shortquote
4 number
4 menu
4 ins
3 progress
3 header
3 content
3 bool
3 audio
3 attribute
3 acronym
2 xmp *
2 overwrite
2 objects
2 nobreak
2 metadata
2 description
2 datasource
2 category
2 action
R=nigeltao
CC=golang-dev
https://golang.org/cl/4964045
This does not wire up <style> elements as that is pending support
for raw text content in CL https://golang.org/cl/4964045/
This CL allows actions to appear in contexts like
selectors: {{.Tag}}{{.Class}}{{.Id}}
property names: border-{{.BidiLeadingEdge}}
property values: color: {{.Color}}
strings: font-family: "{{font-name}}"
URL strings: background: "/foo?image={{.ImgQuery}}"
URL literals: background: url("{{.Image}}")
but disallows actions inside CSS comments and disallows
embedding of JS in CSS entirely.
It is based on the CSS3 lexical grammar with affordances for
common browser extensions including line comments.
R=nigeltao
CC=golang-dev
https://golang.org/cl/4968058
Also add exp/regexp to build (forgot before).
At this point I am very confident in exp/regexp's
behavior. It should be usable as a drop-in
replacement for regexp now.
Later CLs could introduce a CompilePOSIX
to get at traditional POSIX ``extended regular expressions''
as in egrep and also an re.MatchLongest method to
change the matching mode to leftmost longest
instead of leftmost first. On the other hand, I expect
very few people to use either.
R=r, r, gustavo
CC=golang-dev
https://golang.org/cl/4990041
Needed to ensure that finding the last boundary does not result in O(n^2)-like behavior.
Now prevents lookbacks beyond 31 characters across the board (starter + 30 non-starters).
composition.go:
- maxCombiningCharacters now means exactly that.
- Bug fix.
- Small performance improvement/ made code consistent with other code.
forminfo.go:
- Bug fix: ccc needs to be 0 for inert runes.
normalize.go:
- A few bug fixes.
- Limit the amount of combining characters considered in FirstBoundary.
- Ditto for LastBoundary.
- Changed semantics of LastBoundary to not consider trailing illegal runes a boundary
as long as adding bytes might still make them legal.
trie.go:
- As utf8.UTFMax is 4, we should treat UTF-8 encodings of size 5 or greater as illegal.
This has no impact on the normalization process, but it prevents buffer overflows
where we expect at most UTFMax bytes.
R=r
CC=golang-dev
https://golang.org/cl/4963041
This transitions into a JS state when entering any attribute whose
name starts with "on".
It does not yet enter a JS on entry into a <script> element as script
element handling is introduced in another CL.
R=nigeltao
CC=golang-dev
https://golang.org/cl/4968052
1. adds a urlPart field to context
2. implements tURL to figure out the URL part
3. modifies joinContext to allow common context mismatches
around branches to be ignored when not material as in
<a href="/foo{{if .HasQuery}}?q={{.Query}}{{/if}}">
4. adds a pipeline function that filters dynamically inserted
protocols to prevent code injection via URLs.
R=nigeltao
CC=golang-dev
https://golang.org/cl/4957041
The following testcases now pass:
`<a href=x` tests that we do not error on partial unquoted attrs.
`<a href=x ` tests that spaces do end unquoted attrs on spaces.
`<a href=''` tests that we recognize the end of single quoted attrs.
`<a href=""` tests that we recognize the end of double quoted attrs.
R=golang-dev, nigeltao
CC=golang-dev
https://golang.org/cl/4932051
maketables.go/tables.go
- Properly set combinesForward flag for JamoL and JamoV.
- Fixed Printf bug.
composition.go
- Make insertString use the same control flow as insert.
- Better Hangul and non-Hangul mixing.
forminfo.go
- Fixed bug in compBoundaryBefore that affected a few esoteric cases.
- Buffer overflow now tested in normalize_test.go (other CL).
R=r
CC=golang-dev
https://golang.org/cl/4924041
This defines just enough context to distinguish HTML URI attributes
from parsed character data.
It does not affect any public module API as I thought I would get
early comment on style for defining enumerations and tables.
R=rsc, r, nigeltao, r
CC=golang-dev
https://golang.org/cl/4906043
forminfo.go:
- Wrappers for table data.
- Per Form dispatch table.
composition.go:
- reorderBuffer type. Implements decomposition, reordering, and composition.
- Note: decompose and decomposeString fields in formInfo could be replaced by
a pointer to the trie for the respective form. The proposed design makes
testing easier, though.
normalization.go:
- Temporarily added panic("not implemented") methods to make the tests run.
These will be removed again with the next CL, which will introduce the
implementation.
R=r, rogpeppe, mpvl, rsc
CC=golang-dev
https://golang.org/cl/4875043
Replaces the toy func Reverse(*Template) with one that implements
naive autoescaping.
Now Escape(*Template) walks a template parse tree to find all
template actions and adds the |html command to them if it is not
already present.
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/4867049
This is just moving the URL code from package http into its own package,
which has been planned for a while.
Besides clarity, this also breaks a nascent dependency cycle the new template
package was about to introduce.
Add a gofix module, url, and use it to generate changes outside http and url.
Sadness about the churn, gladness about some of the naming improvements.
R=dsymonds, bradfitz, rsc, gustavo, r
CC=golang-dev
https://golang.org/cl/4893043
While using exp/template in practice, the syntax for
indexing values using the "index" action was found to be
very inconvenient for frequent use when handling dynamic
data from maps such as the ones used with json and yaml,
that use a type like map[string]interface{}.
For these kinds of maps, the default handling of fields as
{{.Field}} makes the task of handling the several references
significantly more pleasant and elegant, and is equivalent
to what's currently done in the "template" package and in
other external packages (e.g. mustache).
Even with this change, the index action is still relevant
as it allows indexing maps in other scenarios where keys
wouldn't be valid field names.
R=golang-dev, r, gustavo
CC=golang-dev
https://golang.org/cl/4898043
This avoids a non-obvious panic when range is used on a
nil interface, and fixes it by behaving as if the range
was empty.
The new behavior is equivalent to the outcome of iterating
on a nil map or slice, and is useful because it allows
generic structures such as used in json (map[string]interface{})
to behave correctly if a key generally set to a list or map
isn't present.
R=golang-dev, r, gustavo
CC=golang-dev
https://golang.org/cl/4876046
func Reverse(*Template) *Template
returns a template that produces the reverse of the original
for any input.
Changes outside exp/template/html include:
- Adding a getter for a template's FuncMap so that derived templates
can inherit function definitions.
- Exported one node factory function, newIdentifier.
Deriving tempaltes requires constructing new nodes, but I didn't
export all of them because I think shallow copy functions might
be more useful for this kind of work.
- Bugfix: Template's Name() method ignores the name field so
template.New("foo") is a nil dereference instead of "foo".
Caveats: Reverse is a toy. It is not UTF-8 safe, and does not
preserve order of calls to funcs in FuncMap.
For context, see http://groups.google.com/group/golang-nuts/browse_thread/thread/e8bc7c771aae3f20/b1ac41dc6f609b6e?lnk=gst
R=rsc, r, nigeltao, r
CC=golang-dev
https://golang.org/cl/4808089
IMPORTANT: Previous usage of *Files will continue to compile
fine but misbehave since the interface is compatible.
The following functions have been renamed:
ParseFiles => ParseGlob
ParseFile => ParseFiles
ParseSetFiles => ParseSetGlob
ParseSetFile => ParseSetFiles
ParseTemplateFiles => ParseTemplateGlob
ParseTemplateFile => ParseTemplateFiles
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/4867041
- triegen.go: Factored out trie generation code from maketables.go
(only renamed printTrieTables to printTables and made it a method).
- maketesttables.go: new tool to generate data for the trie unit test.
- Makefile: changed accordingly.
- trie.go: trie lookup code.
- trietest_data.go: generated by maketesttables.go.
- trie_test.go: unit test for trie.go.
R=r
CC=golang-dev
https://golang.org/cl/4844053
- triegen.go: Factored out trie generation code from maketables.go
(only renamed printTrieTables to printTables and made it a method).
- maketesttables.go: new tool to generate data for the trie unit test.
- Makefile: changed accordingly.
- trie.go: trie lookup code.
- trietest_data.go: generated by maketesttables.go.
- trie_test.go: unit test for trie.go.
R=r
CC=golang-dev
https://golang.org/cl/4844053
Mostly a mechanical change, with a few cleanups to make the split easier.
The external interface to exp/template is unaffected.
In another round I will play with the function map setup to see if I can
avoid exposing reflect across the boundary, but that will require some
structural changes I did not want to mix into this CL.
R=golang-dev, dsymonds
CC=golang-dev
https://golang.org/cl/4849049
Before this change, syscall package would load
all dlls used anywhere in the go tree on startup.
For example, this program:
package main
import "fmt"
func main() {
fmt.Printf("Hello world\n")
}
would load these dlls
kernel32.dll
advapi32.dll
shell32.dll
wsock32.dll
ws2_32.dll
dnsapi.dll
iphlpapi.dll
Most of these dlls are network related and are not used
in this program. Now the same program loads only
kernel32.dll
shell32.dll
This decreases start times somewhat.
This also relaxes the rules of which dlls can be included
in the standard library. We could now include system calls
that are not available on all versions of Windows, because
we could decide if we should call them during runtime.
R=rsc, vcc.163
CC=golang-dev
https://golang.org/cl/4815046
- create a pair of "Must" functions, one for templates, one for sets
- regularize the return values of the parsers so they can be wrapped by Must
- delete all the old Must functions and methods.
R=golang-dev, gri, r
CC=golang-dev
https://golang.org/cl/4826052
This brings them into line with the Template.MustParse* methods
and makes it possible to use them in global variable initializations.
R=r
CC=golang-dev
https://golang.org/cl/4798059
Still need to write tests for new syntax
and fix bugs that the tests find, but this
is a good check point.
All tests pass.
Compared against existing regexp:
benchmark old ns/op new ns/op delta
regexp.BenchmarkLiteral 1869 620 -66.83%
regexp.BenchmarkNotLiteral 9489 7823 -17.56%
regexp.BenchmarkMatchClass 10372 8386 -19.15%
regexp.BenchmarkMatchClass_InRange 10800 7750 -28.24%
regexp.BenchmarkReplaceAll 13492 8519 -36.86%
regexp.BenchmarkAnchoredLiteralShortNonMatch 747 339 -54.62%
regexp.BenchmarkAnchoredLiteralLongNonMatch 599 335 -44.07%
regexp.BenchmarkAnchoredShortMatch 2137 917 -57.09%
regexp.BenchmarkAnchoredLongMatch 2029 917 -54.81%
R=r, r
CC=golang-dev, sam.thorogood
https://golang.org/cl/4820046
This is the last piece (I hope) of the set creation code.
These helpers create sets from files containing individual
template definitions, free of {{define}} clauses. This
design is helpful if the templates live one per file,
undecorated.
R=golang-dev, dsymonds
CC=golang-dev
https://golang.org/cl/4801052
Also make the Set.Parse methods variadic so you can parse static lists of files without loops.
R=rsc, dsymonds, r
CC=golang-dev
https://golang.org/cl/4802051
This simplifies the API and makes it easier to make the template
invocation statically secure, at the cost of some minor flexibility.
R=golang-dev, dsymonds, r
CC=golang-dev
https://golang.org/cl/4794045
interface field can be unpacked. We don't have type assertions here so we
must be forthright.
R=golang-dev, adg
CC=golang-dev
https://golang.org/cl/4757047
The previous CL doicumented and diagnosed the old situation.
This one changes it to something more traditional: any action
may declare a variable, and the block structure of scopes
applies only to control seequences.
R=golang-dev, dsymonds
CC=golang-dev
https://golang.org/cl/4748047
{{range $key, $element := pipeline}}
This CL is smaller than it looks due to some rearrangement and renaming.
R=rsc, r
CC=golang-dev
https://golang.org/cl/4709047
Angle brackets can trigger some browser sniffers,
causing some output to be interpreted as HTML.
Escaping angle brackets closes that security hole.
R=r
CC=golang-dev
https://golang.org/cl/4714044
Russ suggested this technique, making the "and" and "or" functions handier.
But it's hacky, and I can be talked out of it.
R=dsymonds, rsc
CC=golang-dev
https://golang.org/cl/4698044
- template invocation is by string constant only.
- NewSet is gone.
- no global Funcs
- writer is now first arg to Execute
R=rsc, r
CC=golang-dev
https://golang.org/cl/4700043
Must keep dot and the receiver separate - variables broke that symmetry.
Also clean up function invocation and rename "data" to "dot" for clarity.
R=golang-dev, dsymonds
CC=golang-dev
https://golang.org/cl/4678048
Not strictly necessary (you could achieve the same, clumsily,
via with blocks) but great to have: $x.Field, $y.Method.
R=golang-dev, adg
CC=golang-dev
https://golang.org/cl/4678047
We can avoid the check against empty constants (''),
which UnquoteChar doesn't handle well, by leaving on
the trailing quote and seeing that's all we have left at the end.
R=golang-dev, dsymonds
CC=golang-dev
https://golang.org/cl/4657090
of the origin.
image/png and image/jpeg benchmarks show no significant changes.
The image/draw changes suggest to me that making a gofix for this is not
feasible. People are just going to have to make manual fixes.
R=r
CC=golang-dev
https://golang.org/cl/4681044
1) Make the value optional ({{template "foo"}})
2) Allow the template identifier to be a thing of type *Template.
The second makes it easy to drop templates in to a set dynamically
during invocation.
R=golang-dev, dsymonds
CC=golang-dev
https://golang.org/cl/4671056
It was an ill-advised carryover from the previous template package.
Also clean up function evaluation.
Also add a Name method to Template.
R=golang-dev, gri
CC=golang-dev
https://golang.org/cl/4657088
- fix line numbers - forgot to update state.line during execution
- add a comment convention {{/* comment */}}
- set.Template returns the named template in the set
- set.Execute executes the named template in the set
- use a local methodByName so this package can be used with earlier release of reflect.
- use initial cap to detect exported names
R=golang-dev, adg
CC=golang-dev
https://golang.org/cl/4668054
Add the ability to attach functions to template and template sets.
Make variadic functions and methods work.
Still to come: static checking of function names during parse.
R=golang-dev, dsymonds
CC=golang-dev
https://golang.org/cl/4643068
Also reuse of *Regexp nodes.
I believe this is the end of the parser.
The only non-execution code that remains is
the code to expand x{3,5} into simpler operations.
R=sam.thorogood, r
CC=golang-dev
https://golang.org/cl/4629078
including evaluation up the data tree (in this code all fields must be
in dot itself), plus more control structure, but the basics are in place.
R=rsc, r
CC=golang-dev
https://golang.org/cl/4665041
Change the signature of Split to have no count,
assuming a full split, and rename the existing
Split with a count to SplitN.
Do the same to package bytes.
Add a gofix module.
R=adg, dsymonds, alex.brainman, rsc
CC=golang-dev
https://golang.org/cl/4661051
Still TODO: parsing optimizations
make_perl_groups.pl is copied with minimal modifications
(just to generate Go syntax instead of C++) from RE2.
Google Inc is "The RE2 Author" of that file and is one of
the Go Authors, so copyright changed to the Go Authors instead.
R=sam.thorogood, r, fvbommel, robert.hencke
CC=golang-dev
https://golang.org/cl/4612041
Parser is a work in progress but can populate most of the
interesting parts of the data structure, so a good checkpoint.
All the complicated Perl syntax is missing, as are various
important optimizations made during parsing to the
syntax tree.
The plan is that exp/regexp's API will mimic regexp,
and exp/regexp/syntax provides the parser directly
for programs that need it (and for implementing exp/regexp).
Once finished, exp/regexp will replace regexp.
R=r, sam.thorogood, kevlar, edsrzf
CC=golang-dev
https://golang.org/cl/4538123
An unusual design using slice and a goroutine makes for a
compact scanner with very little allocation.
R=rsc, r
CC=golang-dev, kevlar
https://golang.org/cl/4610041
On my laptop, I had an 800x600 jpeg and an 800x600 png (with
transparency). I timed how long it took to draw each image onto an
equivalently sized, zeroed RGBA image.
Previously, the jpeg took 75ms and the png took 70ms, going through
the medium-fast path, i.e. func drawRGBA in draw.go.
After this CL, the jpeg took 14ms, and the png took 21ms with the
Over operator and 12ms with the Src operator.
It's only a rough estimate basd on one image file, but it should
give an idea of the order of magnitude of improvement.
R=rsc, r
CC=adg, golang-dev
https://golang.org/cl/4468044
We replace the current Open with:
OpenFile(name, flag, perm) // same as old Open
Open(name) // same as old Open(name, O_RDONLY, 0)
Create(name) // same as old Open(name, O_RDWR|O_TRUNC|O_CREAT, 0666)
This CL includes a gofix module and full code updates: all.bash passes.
(There may be a few comments I missed.)
The interesting packages are:
gofix
os
Everything else is automatically generated except for hand tweaks to:
src/pkg/io/ioutil/ioutil.go
src/pkg/io/ioutil/tempfile.go
src/pkg/crypto/tls/generate_cert.go
src/cmd/goyacc/goyacc.go
src/cmd/goyacc/units.y
R=golang-dev, bradfitzwork, rsc, r2
CC=golang-dev
https://golang.org/cl/4357052
The parser populates all scopes for a given file (except
type-related scopes for structs, interfaces, and methods
of types) at parse time.
A new parser flag, DeclarationErrors, enables error messages
related to declaration errors (as far as it is possible to
provide them).
The resulting AST has all (non-field, non-method) identifiers
resolved that can be resolved w/o doing imports or parsing
missing package files.
The ast.File node contains the (partially complete)
package scope and a list of unresolved global identifiers.
All type-specific data structures have been removed from the AST.
The existing typechecker is functional but needs to be adjusted
(simplified) accordingly. Utility functions to resolve all
identifiers for a package (after handling imports and parsing
all package files) are missing.
Unrelated changes:
- Rename typechecker/testdata files to that they are not considered
by gofmt.
- Minor cleanups/simplifications.
Parses all .go files in src and misc without declaration errors.
Runs all tests. Changes do not affect gofmt output.
R=rsc
CC=golang-dev
https://golang.org/cl/4244049
There are some minor irregularities in the printer
output (some paren's are present that should be
removed), but these are unrelated issues.
Will review in a 2nd step.
R=rsc
CC=golang-dev
https://golang.org/cl/4188068
In line with other functions such as Fprintf, put the
thing to be written first.
Apologies for the breakages this is sure to cause.
R=rsc, gri, adg, eds, r2, aam
CC=golang-dev
https://golang.org/cl/4169042
The callback mechanism has been made more flexible.
Eliminated one round of argument copying in Syscall.
Faster Get/SetLastError implemented.
Added gettime for gc perf profiling.
R=rsc, brainman, mattn, rog
CC=golang-dev
https://golang.org/cl/4058046
Until now, each scan of a file added a new file to the file set.
With this change, a file can be re-scanned using the same *token.File
w/o changing the file set. Eventually this will enable the re-use of
cached source code in godoc (for the fulltext index). At the moment,
source files are read over and over again from disk.
This is the first step in that direction.
R=r, rsc, r2
CC=golang-dev
https://golang.org/cl/4001041
- add End() method to all nodes; the text range of a node n is [n.Pos(), n.End())
- various small bug fixes in the process
- fixed several comments
R=r, rsc
CC=golang-dev
https://golang.org/cl/3769042
It was only used by exp/4s, and even if it is general purpose, I think
it belongs in a graphics library atop exp/draw, not in exp/draw itself.
R=rsc
CC=golang-dev
https://golang.org/cl/3705041
The recent linker changes broke NaCl support
a month ago, and there are no known users of it.
The NaCl code can always be recovered from the
repository history.
R=adg, r
CC=golang-dev
https://golang.org/cl/3671042
Specifically:
* lib/godoc:
- provide file set (FSet) argument to formatters where needed
* src/cmd:
- cgo, ebnflint, godoc, gofmt, goinstall: provide file set (fset) where needed
- godoc: remove local binary search with sort.Search (change by rsc),
extract file set for formatters
* src/pkg:
- exp/eval: remove embedded token.Position fields from nodes and replace
with named token.Pos fields; add corresponding Pos() accessor methods
- go/token: added file.Line(), changed signature of File.Position()
* test/fixedbugs/:
- bug206.go: change test to not rely on token.Pos details
* added various extra comments
* Runs all.bash
* gofmt formats all of src, misc w/o changes
* godoc runs
* performance:
- The new version of godoc consumes about the same space after indexing
has completed, but indexing is half the speed. Significant space savings
are expected from smaller ASTs, but since they are thrown away after a
file has been indexed, this is not visible anymore. The slower indexing
time is due to the much more expensive computation of line information.
However, with the new compressed position information, indexing can be
rewritten and simplified. Furthermore, computing the line info can be
done more efficiently.
New godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
44381 godoc 0.0% 0:38.00 4 19 149 145M 184K 148M 176M
2010/12/03 17:58:35 index updated (39.231s, 18505 unique words, 386387 spots)
2010/12/03 17:58:35 bytes=90858456 footprint=199182584
2010/12/03 17:58:36 bytes=47858568 footprint=167295224
Old godoc, immediately after indexing completed (best of three runs):
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
23167 godoc 0.0% 0:22.02 4 17 132 129M 184K 132M 173M
2010/12/03 14:51:32 index updated (24.892s, 18765 unique words, 393830 spots)
2010/12/03 14:51:32 bytes=66404528 footprint=163907832
2010/12/03 14:51:32 bytes=46282224 footprint=163907832
The different numbers for unique words/spots stem from the fact the the
two workspaces are not exactly identical. The new godoc maintains a large
file set data structure during indexing which (probably) is the reason
for the larger heap (90858456 vs 66404528) before garbage collection.
R=rsc, r
CC=golang-dev
https://golang.org/cl/3050041
Rather than updating the stripped-down regexp implementation embedded
in testing, delete it by passing the one function we need from the package
main file created by gotest.
R=rsc
CC=golang-dev
https://golang.org/cl/2761043
New logging interface simplifies and generalizes.
1) Loggers now have only one output.
2) log.Stdout, Stderr, Crash and friends are gone.
Logging is now always to standard error by default.
3) log.Panic* replaces log.Crash*.
4) Exiting and panicking are not part of the logger's state; instead
the functions Exit* and Panic* simply call Exit or panic after
printing.
5) There is now one 'standard logger'. Instead of calling Stderr,
use Print etc. There are now triples, by analogy with fmt:
Print, Println, Printf
What was log.Stderr is now best represented by log.Println,
since there are now separate Print and Println functions
(and methods).
6) New functions SetOutput, SetFlags, and SetPrefix allow global
editing of the standard logger's properties. This is new
functionality. For instance, one can call
log.SetFlags(log.Lshortfile|log.Ltime|log.Lmicroseconds)
to get all logging output to show file name, line number, and
time stamp.
In short, for most purposes
log.Stderr -> log.Println or log.Print
log.Stderrf -> log.Printf
log.Crash -> log.Panicln or log.Panic
log.Crashf -> log.Panicf
log.Exit -> log.Exitln or log.Exit
log.Exitf -> log.Exitf (no change)
This has a slight breakage: since loggers now write only to one
output, existing calls to log.New() need to delete the second argument.
Also, custom loggers with exit or panic properties will need to be
reworked.
All package code updated to new interface.
The test has been reworked somewhat.
The old interface will be removed after the new release.
For now, its elements are marked 'deprecated' in their comments.
Fixes#1184.
R=rsc
CC=golang-dev
https://golang.org/cl/2419042
Package iterable has outlived its utility.
It is an interesting demonstration, but it encourages
people to use iteration over channels where simple
iteration over array indices or a linked list would be
cheaper, simpler, and have fewer races.
R=dsymonds, r
CC=golang-dev
https://golang.org/cl/2436041
- change ast.Ident back to contain the name and adjust all dependent code
- identifier object information will be added again through an optional
typechecker phase (in the works).
- remove tracking of scopes in parser - it's easier to do this in a separate
phase (in the works)
- in godoc, generate popup info table directly instead of through a formatter
for simpler data flow (at the expense of a little bit more code)
Runs all tests.
As a result of this change, the currently shown popup information
(const, var, type, func, followed by identifier name) will not be
shown anymore temporarily.
R=rsc
CC=golang-dev
https://golang.org/cl/1994041
Rect types.
The actual image representation is unchanged. A future change will
replace the {[][]color} with {[]color, stride int, r Rectangle} and
possibly a clip region.
The draw.Color, draw.Point and draw.Rect types will be removed in a
future change. Trying to do it in this one polluted the diff with
trivia.
R=r, rsc
CC=golang-dev
https://golang.org/cl/1918047
Also in this CL:
* Removed util.go, as its functionality is in big
* Removed some semicolons from the code generated by gen.go
* Added a generate target to Makefile
* Removed an outdated TODO from value.go
R=gri
CC=golang-dev
https://golang.org/cl/1780042
Time (us) to draw a 200x200 src onto an image.RGBA dst with the Over
operator. Each case was measured three separate times, and the sorted
times are reported.
Fill case (where src is an image.ColorImage):
Before:
7438, 7511, 7526
After:
3073, 3087, 3102
Copy case (where src is an image.RGBA):
Before:
9506, 9510, 9563
After:
5262, 5300, 5344
R=rsc
CC=golang-dev
https://golang.org/cl/1532041
Fix Rectangle.Clip.
It could return a non-canonical rectangle if its arguments
did not overlap.
e.g. Rect(0, 0, 10, 10).Clip(Rect(0, 20, 10, 30)) -> Rect(0, 20, 10, 10)
R=rsc, r
CC=golang-dev, golang-dev
https://golang.org/cl/1290041
On my laptop, time to prepare and write 800x600 pixels over the
socket falls from 125-ish ms to 80-ish ms.
Thanks to Roger Peppe for the suggestion.
R=r
CC=golang-dev
https://golang.org/cl/1228044
Time to draw.Draw a 200x200 image fell from 18.4ms (and 1 malloc) to
5.6ms (and 0 mallocs). It's still relatively slow since it assumes
nothing about the src or mask images, but it does remove the malloc.
There are existing faster, more specialized paths for copies, fills
and image glyph masks.
Also added a "compare to a slow but obviously correct implementation"
check to draw_test.go.
R=rsc, r
CC=golang-dev
https://golang.org/cl/1223044
Timings (as for change 1122043) go from 49ms to 48ms ish. It's
mostly lost in the noise, but it probably doesn't hurt.
R=r
CC=golang-dev
https://golang.org/cl/1179041
To draw.Draw a 32x32 image.Alpha 10000 times,
Before: 633ms with 10000 mallocs
After: 49ms with 0 mallocs
These times are just blitting an image.Alpha, and do not include
rasterizing a glyph's vector contours to an image.Alpha.
The "generic" test case in draw_test.go tests this fast path.
R=rsc
CC=golang-dev
https://golang.org/cl/1122043
note that sortmain.go has been run through hg gofmt;
only the formatting of the day initializers changed.
i'm happy to revert that formatting if you'd prefer.
stop on error in doc/progs/run
R=r
CC=golang-dev
https://golang.org/cl/850041
- only manual changes are in src/pkg/go/printer/nodes.go
- use a heuristic to determine "outliers" such that not entire composites are
forced to align with them
- improves several places that were not unligned before due too simple heuristic
- unalignes some cases that contain "outliers"
- gofmt -w src misc
Fixes#644.
R=rsc, r
CC=golang-dev
https://golang.org/cl/241041
- gofmt -w src misc
- improves several lists and fixes minor degradation introduced
with the fix for issue 628
- removed some dead code (stringList)
R=rsc
CC=golang-dev
https://golang.org/cl/223058
- applied gofmt to src and misc
Note: This fix improved formatting of src/pkg/math/all_test.go but leads
to a degradation in src/pkg/exp/4s/xs.go. The latter happened to "work"
before accidentally. Fixing the alignment in that case in general will
be a separate CL.
Fixes#628.
R=rsc
CC=golang-dev
https://golang.org/cl/223054
- always include position information about opening/closing parens/braces
- replace uses of []*ast.Field with *ast.FieldList
Fixes#473.
R=rsc
CC=golang-dev
https://golang.org/cl/223043
- go/ast: removed StringList (not needed anymore)
- go/ast: changed import path and field list tag to a single string
- updated all dependencies
R=rsc
CC=golang-dev
https://golang.org/cl/217056
Averaged times (in microseconds) for drawing an 800x600 rectangle
are listed below. The summary is: around a 100x improvement.
draw.Draw call times were typically linear in the number of pixels
touched (i.e. drawing an 800x600 rectangle took 100x as much time as
drawing an 80x60 rectangle).
Before this change, there was only the general-but-slow code path.
When drawing any src with a 50%-opaque mask: 237300 us
When drawing any src with a nil mask: 50100 us
After this change, the 50%-opaque mask case is unchanged.
For an *image.RGBA dst and nil mask and...
...a uniform color (i.e. an image.ColorImage) src: 282 us
...another *image.RGBA src: 615 us.
For the curious, an intermediate implementation detected the special
cases but used simple nested for loops instead of the built-in copy
function. The respective times (compared to 282 and 615 for the
final implementation, or 50100 for the original) were 3110 and 3573.
Times were measured with 8g/8l on my laptop. I haven't tried gccgo
or other architectures.
R=r, rsc
CC=golang-dev
https://golang.org/cl/201048
- provide scope to parse functions; if non-nil, parser uses the scope
to declare and lookup identifiers
- resolve forward references where possible
R=rsc
CC=golang-dev
https://golang.org/cl/194098
- Identifiers refer now to the language entity (Object)
that they denote. At the moment this is at best an
approximation.
- Initial data structures for language entities (Objects)
and expression types (Type) independent of the actual
type notations.
- Initial support for declaring and looking up identifiers.
- Updated various dependent files and added support functions.
- Extensively tested to avoid breakage. This is an AST change.
R=rsc
CC=golang-dev, rog
https://golang.org/cl/189080
Manual changes to the following files:
src/pkg/Makefile
src/pkg/exp/vector/Makefile (now: src/pkg/container/vector/Makefile)
R=rsc, r
CC=golang-dev
https://golang.org/cl/181041
parsing and printing to new syntax.
Use -oldparser to parse the old syntax,
use -oldprinter to print the old syntax.
2) Change default gofmt formatting settings
to use tabs for indentation only and to use
spaces for alignment. This will make the code
alignment insensitive to an editor's tabwidth.
Use -spaces=false to use tabs for alignment.
3) Manually changed src/exp/parser/parser_test.go
so that it doesn't try to parse the parser's
source files using the old syntax (they have
new syntax now).
4) gofmt -w src misc test/bench
2nd set of files.
R=rsc
CC=golang-dev
https://golang.org/cl/179067
This change removes the necessity to have GOBIN in $PATH,
and also doesn't assume that the build is being run from
$GOROOT/src. This is a minimal set of necessary changes
to get Go to build happily from the FreeBSD ports
collection.
R=rsc
CC=golang-dev
https://golang.org/cl/171044
- oldparser parse old syntax (required semicolons)
- oldprinter print old syntax (required semicolons)
By default, these flags are enabled for now.
Setting -oldparser=false has no effect until go/parser is changed
to accept the new syntax.
Enabled exp/parser in Makefile; update dependent exp/eval.
R=rsc
https://golang.org/cl/174051
this is the exact same thing issue #115 is about. fix makefiles to use relative
path to work in the case we have whitespaces as part of GOROOT.
R=rsc
https://golang.org/cl/162055
This provides an experimental X11 backend for the exp/draw interface.
It does not aim to provide a complete implementation of the X11 client protocol.
This works for me (Ubuntu Hardy 8.04, GOARCH=386). Your mileage my vary.
R=r, rsc, r1
CC=golang-dev
https://golang.org/cl/156109
the bash scripts and makefiles for building go didn't take into account
the fact $GOROOT / $GOBIN could both be directories containing whitespaces,
and was not possible to build it in such a situation.
this commit adjusts the various makefiles/scripts to make it aware of that
possibility, and now it builds successfully when using a path with whitespaces
as well.
Fixes#115.
R=rsc, dsymonds1
https://golang.org/cl/157067
Meant as illustration of the Go pattern that is using
goroutines and channels to handle exceptional situations.
Note: There is no need for "Finally" since the
"try block" (the function f supplied to Try)
cannot do a Smalltalk-style non-local return
and terminate the function surrounding Try.
Replaces CL 157083.
R=r, rsc
https://golang.org/cl/157087
- enabled for function declarations (not just function literals)
- applied gofmt -w $GOROOT/src
(look for instance at src/pkg/debug/elf/elf.go)
R=r, rsc
CC=go-dev
http://go/go-review/1026006