Consequently, remove many package Makefiles,
and shorten the few that remain.
gomake becomes 'go tool make'.
Turn off test phases of run.bash that do not work,
flagged with $BROKEN. Future CLs will restore these,
but this seemed like a big enough CL already.
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/5601057
Also delete gotest, since it's messy to fix and slated for deletion anyway.
A couple of things outside src can't be tested any more. "go test" will be
fixed and these tests will be re-enabled. They're noisy for now.
Fixes#284.
R=rsc
CC=golang-dev
https://golang.org/cl/5598049
work in progress, and we are not ready to freeze its API for Go 1.
Package html still exists, containing just two functions: EscapeString
and UnescapeString.
Both the packages at exp/html and html are "package html". The former
is a superset of the latter.
At some point in the future, the exp/html code will move back into
html, once we have finalized the parser API.
R=rsc, dsymonds
CC=golang-dev
https://golang.org/cl/5571059
Remove NewMD5, NewSHA1 and NewSHA256 in favor of using New and
explicitly importing the used hash-function. This way when using, for
example, HMAC with RIPEMD there's no md5, sha1 and sha256 linked in
through the hmac package.
A gofix rule is included, and applied to the standard library (3 files
altered).
This change is the result of a discussion at
https://golang.org/cl/5550043/ to pull the discussion about
deprecating these functions out of that issue.
R=golang-dev, agl
CC=golang-dev, r, rsc
https://golang.org/cl/5556058
Previously, a timeout (in int64 nanoseconds) applied to a granularity
even smaller than one operation: a 100 byte read with a 1 second timeout
could take 100 seconds, if the bytes all arrived on the network 1 second
apart. This was confusing.
Rather than making the timeout granularity be per-Read/Write,
this CL makes callers set an absolute deadline (in time.Time)
after which operations will fail. This makes it possible to
set deadlines at higher levels, without knowing exactly how
many read/write operations will happen in e.g. reading an HTTP
request.
Fixes#2723
R=r, rsc, dave
CC=golang-dev
https://golang.org/cl/5555048
- Use a faster closure-based iterator rather than a channel-based one.
- Otherwise: better code organization, but no other API changes.
R=r, r
CC=golang-dev
https://golang.org/cl/5557051
First was, apart from adding tests, a single line of code (to add the
constructor function). Adding SHA512-based hashing to crypto/hmac
required minor rework of the package because of a previously hardcoded
block-size in it's implementation. Instead of using a hash.Hash
generator function the constructor function now uses a crypto.Hash
type, which was extended to expose information about block size.
The only standard library package impacted by the change is
crypto/tls, for which the fix is included in this patch. It might be
useful to extend gofix to include this API change too.
R=agl, r, rsc, r
CC=golang-dev
https://golang.org/cl/5550043
The terminal code in exp/terminal was forked from the code in exp/ssh.
This change removes the duplicated code from exp/ssh in favour of
using exp/terminal.
R=rsc
CC=golang-dev
https://golang.org/cl/5375064
(This was missing in the last change because I uploaded it from the
wrong machine.)
Large pastes previously misbehaved because the code tried reading from
the terminal before checking whether an line was already buffered.
Large pastes can cause multiples lines to be read at once from the
terminal.
R=bradfitz
CC=golang-dev
https://golang.org/cl/5542049
exp/proxy provides client support for tunneling connections through
various proxies.
This is an initial, incomplete sketch of the code to lay down an
API.
R=golang-dev, r, r, bradfitz, rsc
CC=golang-dev
https://golang.org/cl/5490062
Also verified in external test suite that this fixes MySQL
resource exhaustion problems, and also exposed a double-free
bug in the gosqlite3 driver (where gosqlite3 either got lucky
before, or was working around this bug)
R=golang-dev, rsc
CC=golang-dev
https://golang.org/cl/5544057
This is a relic from the times when we switched
to automatic semicolon insertion. It's still use-
ful to have a non-exported switch for testing.
R=golang-dev, r, rsc
CC=golang-dev
https://golang.org/cl/5528077
This CL fixes two issues sending data to the remote peer.
The first bug occurs when the size of the buffer passed to
Write is larger than the current window, in this case, w.rwin
can become negative.
The second issue is more problematic than the first as the
amount of data passed to writePacket was not limited to w.rwin.
In this case the remote peer could silently drop the additional
data, or drop the connection.
Credit to Jacek Masiulaniec for the bug report.
R=agl, jacek.masiulaniec
CC=golang-dev
https://golang.org/cl/5511043
transport.go:
* remove unused nil check.
doc.go:
* improve documentation about supported auth
methods and update Run example.
Thanks Jacek Masiulaniec for both reports.
R=jacek.masiulaniec, agl
CC=golang-dev
https://golang.org/cl/5501075
1) incorrect length given for out buffer in String.
2) patchTail bug that could cause characters to be lost
when crossing into the out-buffer boundary.
Added tests to expose these bugs. Also slightly improved
performance of Bytes() and String() by sharing the reorderBuffer
across operations.
Fixes#2567.
R=r
CC=golang-dev
https://golang.org/cl/5502069
1) Add EscapeCodes to the terminal so that applications don't wire
them in.
2) Add a callback for auto-complete
3) Fix an issue with input lines longer than the width of the
terminal.
4) Have Write() not stomp the current line. It now erases the current
input, writes the output and reprints the prompt and partial input.
5) Support prompting without local echo in Terminal.
6) Add GetSize to report the size of terminal.
R=bradfitz
CC=golang-dev
https://golang.org/cl/5479043
- no empty lines inside empty structs and interfaces
- top-level declarations are separated by a blank line if
a) they are of different kind (e.g. const vs type); or
b) there are documentation comments associated with a
declaration (this is new)
- applied gofmt -w misc src
The actual changes are in go/printer/nodes.go:397-400 (empty structs/interfaces),
and go/printer/printer.go:307-309 (extra line break). The remaining
changes are cleanups w/o changing the existing functionality.
Fixes issue 2570.
R=rsc
CC=golang-dev
https://golang.org/cl/5493057
If a Pipe method is called, return the underlying
reader/writer from session.clientChan, bypassing the
io.Copy and io.Pipe harness.
StdoutPipe and StderrPipe now return an io.Reader not
an io.ReadCloser as SSH cannot signal the close of the
local reader to the remote process.
R=rsc, agl, gustav.paul, cw
CC=golang-dev
https://golang.org/cl/5493047
I added the clientChan's msg channel to the list of channels that are closed in mainloop when the server sends a channelCloseMsg.
I added an ExitError type that wraps a Waitmsg similar to that of os/exec. I fill ExitStatus with the data returned in the 'exit-status' channel message and Msg with the data returned in the 'exit-signal' channel message.
Instead of having Wait() return on the first 'exit-status'/'exit-signal' I have it return an ExitError containing the status and signal when the clientChan's msg channel is closed.
I added two tests cases to session_test.go that test for exit status 0 (in which case Wait() returns nil) and exit status 1 (in which case Wait() returns an ExitError with ExitStatus 1)
R=dave, agl, rsc, golang-dev, bradfitz
CC=golang-dev
https://golang.org/cl/5452051
This CL cleans up the client auth tests, making the
individual test body more manageable.
Also, adds tests for rsa and dsa key negotiation.
Finally, remove the package level use of the variable
strings, which avoids conflicting with the strings pkg.
R=gustav.paul, agl, n13m3y3r, rsc
CC=golang-dev
https://golang.org/cl/5447049
This is part one of a small set of CL's that aim to resolve
the outstanding TODOs relating to channel close and blocking
behavior.
Firstly, the hairy handling of assigning the peersId is now
done in one place. The cost of this change is the slightly
paradoxical construction of the partially created clientChan.
Secondly, by creating clientChan.stdin/out/err when the channel
is opened, the creation of consumers like tcpchan and Session
is simplified; they just have to wire themselves up to the
relevant readers/writers.
R=agl, gustav.paul, rsc
CC=golang-dev
https://golang.org/cl/5448073
All but 3 cases (in gcimporter.go and hixie.go)
are automatic conversions using gofix.
No attempt is made to use the new Append functions
even though there are definitely opportunities.
R=golang-dev, gri
CC=golang-dev
https://golang.org/cl/5447069
The SSH spec allows for the server to send a banner message to the client at any point during the authentication process. Currently the ssh client auth types all assume that the first response from the server after issuing a userAuthRequestMsg will be one of a couple of possible authentication success/failure messages. This means that client authentication breaks if the ssh server being connected to has a banner message configured.
This changeset refactors the noneAuth, passwordAuth and publickeyAuth types' auth() function and allows for msgUserAuthBanner during authentication.
R=golang-dev, rsc, dave, agl
CC=golang-dev
https://golang.org/cl/5432065
This is the result of running `gofix -r hashsum` over the tree, changing
the hash function implementations by hand and then fixing a couple of
instances where gofix didn't catch something.
The changed implementations are as simple as possible while still
working: I'm not trying to optimise in this CL.
R=rsc, cw, rogpeppe
CC=golang-dev
https://golang.org/cl/5448065
Exec() has been renamed to Run() in keeping with the os/exec API.
Added func (*Session) Start(cmd string) which starts a remote process but unlike Run() doesn't wait for it to finish before returning.
Run() has been refactored to use Start internally. Its really just a refactoring, no new code but some extra functionality was won.
Also added func (*Session) Signal(sig signal) which sends a UNIX signal to a remote process. This is espcially useful in conjunction with Start() as the two allow you to start a remote process, monitor its stdout/stderr, and send it a TERM/HUP/etc signal when you want it to close.
R=dave, rsc, agl, bradfitz, n13m3y3r, gustavo
CC=golang-dev
https://golang.org/cl/5437058
According to http://www.ietf.org/rfc/rfc4254.txt most channel messages contain the channel id of the recipient channel, not the sender id. This allows the recipient connection multiplexer to route the message to the correct channel.
This changeset fixes several messages that incorrectly send the local channel id instead of the remote channel's id.
While sessions were being created and closed in sequence channels in the channel pool were freed and reused on the server side of the connection at the same rate as was done on the client, so the channel local and remote channel ids always corresponded. As soon as I had concurrent sessions on the same clientConn the server started to complain of 'uknown channel id N' where N is the local channel id, which is actually paired with server channel id K.
R=golang-dev, dave, rsc, agl
CC=golang-dev
https://golang.org/cl/5433063
IIRC, package sql used to be called db. There was one occurrence
of the old name in a comment.
R=golang-dev, bradfitz
CC=golang-dev
https://golang.org/cl/5431075
This CL inverts the direction of the Stdin/out/err members of the
Session struct so they reflect the API of the exec.Cmd. In doing so
it borrows heavily from the exec package.
Additionally Shell now returns immediately, wait for completion using
Wait. Exec calls Wait internally and so blocks until the remote
command is complete.
Credit to Gustavo Niemeyer for the impetus for this CL.
R=rsc, agl, n13m3y3r, huin, bradfitz
CC=cw, golang-dev
https://golang.org/cl/5322055
deps.bash does not understand current format,
so make.bash fails (with MAKEFLAGS=-j20).
R=golang-dev, bradfitz
CC=golang-dev
https://golang.org/cl/5410046
Fixes use of c after Dial failure (causes crash).
May fix Dial failure by listening to 127.0.0.1:0
instead of 0.0.0.0:0 (tests should only listen on
localhost).
R=golang-dev, gri
CC=golang-dev
https://golang.org/cl/5395052
Ensure that empty NameLists always return
a zero length []string, not nil.
In practice NameLists are only used in a few
message types and always consumed by a for
range function so the difference between nil
and []string{} is not significant.
Also, add exp/ssh to pkg/Makefile as suggested
by rsc.
R=rsc, agl
CC=golang-dev
https://golang.org/cl/5400042
- syscall (not os) now defines the Errno type.
- the low-level assembly functions Syscall, Syscall6, and so on
return Errno, not uintptr
- syscall wrappers all return error, not uintptr.
R=golang-dev, mikioh.mikioh, r, alex.brainman
CC=golang-dev
https://golang.org/cl/5372080
This CL adds experimental support for making proxied
net.Conn connections via the remote server.
nb. Functional tests exist for this feature but CL
5320053 or similar needs to be committed to support them.
R=rsc, agl, n13m3y3r
CC=cw, golang-dev, huin
https://golang.org/cl/5371081
client.go/client_auth.go:
* add support for publickey key auth using the interface
outlined by rsc in the previous auth CL
client_auth_test.go:
* password and publickey tests against server.go
common.go/server.go:
* move some helper methods from server.go into common.go
* generalise serializeRSASignature
R=rsc, agl, huin
CC=cw, golang-dev, n13m3y3r
https://golang.org/cl/5373055
Some remote servers send a 0 window size in the channel
open confirm msg, others send a non zero window size. Make
sure this initial advertisement is not lost.
R=agl, rsc, cw
CC=golang-dev
https://golang.org/cl/5372083
It was never really a shell, but the name carried
over from SSH's ServerShell.
Two small functional changes:
Add SetSize, which allows the size of the terminal
to be changed in response, say, to an SSH message.
Don't write the prompt if there's already something
on the current line.
R=rsc
CC=golang-dev
https://golang.org/cl/5376066
(This is part of removing the duplicate code between exp/terminal and
exp/ssh, but hg is having a very hard time keeping up so I'm doing it
in small steps.)
R=bradfitz, rsc
CC=golang-dev
https://golang.org/cl/5373061
This CL adds an API for handling the various SSH
authenticaton methods. None and password continue
to be the only supported methods.
R=bradfitz, agl, n13m3y3r, rsc, cw
CC=golang-dev
https://golang.org/cl/5328045
There are three classes of methods/functions called Error:
a) The Error method in the just introduced error interface
b) Error methods that create or report errors (http.Error, etc)
c) Error methods that return errors previously associated with
the receiver (Tokenizer.Error, rows.Error, etc).
This CL introduces the convention that methods in case (c)
should be named Err.
The reasoning for the change is:
- The change differentiates the two kinds of APIs based on
names rather than just on signature, unloading Error a bit
- Err is closer to the err variable name that is so commonly
used with the intent of verifying an error
- Err is shorter and thus more convenient to be used often
on error verifications, such as in iterators following the
convention of the sql package.
R=bradfitz, rsc
CC=golang-dev
https://golang.org/cl/5327064
This is Go 1 package renaming CL #2.
This one merely moves the source; the import strings will be
changed after the next weekly release.
exp/template/html -> html/template
big -> math/big
cmath -> math/cmplx
rand -> math/rand
syslog -> log/syslog
The only edits are in Makefiles and deps.bash.
Note that this CL moves exp/template/html out of exp. I decided
to do that so all the renamings can be done together, even though
the API (and that of template, for that matter) is still fluid.
R=r, rsc
CC=golang-dev
https://golang.org/cl/5332053
(The definition of ErrorList is in another file, so gofix
has no hope of getting this right.)
R=golang-dev, iant
CC=golang-dev
https://golang.org/cl/5330043
This CL grew the archive file name length from 16 to 64:
changeset: 909:58574851d792
user: Russ Cox <rsc@golang.org>
date: Mon Oct 20 13:53:56 2008 -0700
Back then, every x.go file in a package became an x.6 file
in the archive. It was important to be able to allow the
use of long Go source file names, hence the increase in size.
Today, all Go source files compile into a single _go_.6 file
regardless of their names, so the archive file name length
no longer needs to be long. The longer name causes some
problems on Plan 9, where the native archive format is the
same but with 16-byte names, so revert back to 16.
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/5333050
Fixed error checking in exec.go to give a sensible error message when
execution is attempted before a successful parse (rather than an
outright panic).
R=r
CC=golang-dev
https://golang.org/cl/5306065
The payload of a data message is defined as an SSH string type,
which uses the first four bytes to encode its length. When channelData
and channelExtendedData were added I defined Payload as []byte to
be able to use it directly without a string to []byte conversion. This
resulted in the length data leaking into the payload data.
This CL fixes the bug, and restores agl's original fast path code.
Additionally, a bug whereby s.lock was not released if a packet arrived
for an invalid channel has been fixed.
Finally, as they were no longer used, I have removed
the channelData and channelExtedendData structs.
R=agl, rsc
CC=golang-dev
https://golang.org/cl/5330053
I found these by adding a check to govet, but the check
produces far too many false positives to be useful.
Even so, these few seem worth cleaning up.
R=golang-dev, bradfitz, iant
CC=golang-dev
https://golang.org/cl/5311067
Nothing terribly interesting here. (!)
Since the public APIs are all in terms of UTF-8,
the changes are all internal only.
R=mpvl, gri, r
CC=golang-dev
https://golang.org/cl/5309042
API question: is a scanner token an int or a rune?
Since the rune is the common case and the token values
are the special (negative) case, I chose rune. But it could
easily go the other way.
R=gri
CC=golang-dev
https://golang.org/cl/5301049
This CL replaces the Cmd type with a Session type representing
interactive channels. This lays the foundation for supporting
other kinds of channels like direct-tcpip or x11.
client.go:
* replace chanlist map with slice.
* generalize stdout and stderr into a single type.
* unexport ClientChan to clientChan.
doc.go:
* update ServerConfig/ServerConn documentation.
* update Client example for Session.
message.go:
* make channelExtendedData more like channelData.
session.go:
* added Session which replaces Cmd.
R=agl, rsc, n13m3y3r, gustavo
CC=golang-dev
https://golang.org/cl/5302054
server.go/channel.go:
* rename Server to ServerConfig to match Client.
* rename ServerConnection to ServeConn to match Client.
* add Listen/Listener.
* ServerConn.Handshake(), general cleanups.
client.go:
* fix bug where fmt.Error was not assigned to err
R=rsc, agl
CC=golang-dev
https://golang.org/cl/5265049
cssEscaper escapes using the CSS convention: `\` + hex + optional-space
It outputs the space when the escape could be followed by
a hex digit to distinguish a "\na" from "\u00aa".
It did not output a space when the escape is followed by a space
character so did not distinguish "\n " from "\n".
Currently when doing lookahead, it does not distinguish spaces that
will be escaped later by the same function from ones that will not.
This is correct but suboptimal.
R=nigeltao
CC=golang-dev
https://golang.org/cl/5306042
This change splits terminal handling from exp/ssh, as suggested
several times in the ssh code review.
shell.go and shell_test.go are copies from exp/ssh with minimal
changes, so don't need another full review. A future CL will remove
that code from exp/ssh.
R=bradfitz, r, dave, rsc
CC=golang-dev
https://golang.org/cl/5278049
(more are possible but omitted for now as they are part of
specific tests where rather than changing what is there we
should probably expand the tests to cover the new case)
R=rsc, dvyukov
CC=golang-dev
https://golang.org/cl/5247058
therefore unlikely that there is a good use for its string version
LastBoundaryInString. Yet, the implemenation of this method would complicate
things a bit as it would require the introduction for another interface and
some duplication of code. Removing it seems a better choice.
R=r
CC=golang-dev
https://golang.org/cl/5182044
This is a possible optimization. I'm not sure the complexity is worth it.
The new benchmark in escape_test is 46us without and 35us with the optimization.
R=nigeltao
CC=golang-dev
https://golang.org/cl/5168041
This removes a few cases from escapeAction and clarifies the
responsibilities of urlFilter which no longer does any
escaping or normalization. It is now solely a filter.
R=nigeltao
CC=golang-dev
https://golang.org/cl/5162043
HTML5 allows embedded SVG and MathML.
Code searches show SVG is used for graphing.
This changes transition to deal with constructs like
<svg xmlns:xlink="http://www.w3.org/1999/xlink">
It changes attr and clients to call a single function that combines
the name lookup and "on" prefix check to determine an attribute
value type given an attribute name.
That function uses heuristics to recognize that
xlink:href and svg:href
have URL content, and that data-url is likely contains URL content,
since "javascript:" injection is such a problem.
I did a code search over a closure templates codebase to determine
patterns of custom attribute usage. I did something like
$ find . -name \*.soy | \
xargs egrep perl -ne 'while (s/\b((data-|\w+:)\w+)\s*=//) { print "$1\n"; }' | \
sort | uniq
to produce the list at the bottom.
Filtering that by egrep -i 'src|url|uri' produces
data-docConsumptionUri
data-docIconUrl
data-launchUrl
data-lazySrc
data-pageUrl
data-shareurl
data-suggestServerUrl
data-tweetUrl
g:secondaryurls
g:url
which seem to match all the ones that are likely URL content.
There are some short words that match that heuristic, but I still think it decent since
any custom attribute that has a numeric or enumerated keyword value will be unaffected by
the URL assumption.
Counterexamples from /usr/share/dict:
during, hourly, maturity, nourish, purloin, security, surly
Custom attributes present in existing closure templates codebase:
buzz:aid
data-a
data-action
data-actor
data-allowEqualityOps
data-analyticsId
data-bid
data-c
data-cartId
data-categoryId
data-cid
data-command
data-count
data-country
data-creativeId
data-cssToken
data-dest
data-docAttribution
data-docConsumptionUri
data-docCurrencyCode
data-docIconUrl
data-docId
data-docPrice
data-docPriceMicros
data-docTitle
data-docType
data-docid
data-email
data-entityid
data-errorindex
data-f
data-feature
data-fgid
data-filter
data-fireEvent
data-followable
data-followed
data-hashChange
data-height
data-hover
data-href
data-id
data-index
data-invitable
data-isFree
data-isPurchased
data-jid
data-jumpid
data-launchUrl
data-lazySrc
data-listType
data-maxVisiblePages
data-name
data-nid
data-nodeid
data-numItems
data-numPerPage
data-offerType
data-oid
data-opUsesEquality
data-overflowclass
data-packageName
data-pageId
data-pageUrl
data-pos
data-priceBrief
data-profileIds
data-query
data-rating
data-ref
data-rentalGrantPeriodDays
data-rentalactivePeriodHours
data-reviewId
data-role
data-score
data-shareurl
data-showGeLe
data-showLineInclude
data-size
data-sortval
data-suggestServerType
data-suggestServerUrl
data-suggestionIndex
data-tabBarId
data-tabBarIndex
data-tags
data-target
data-textColor
data-theme
data-title
data-toggletarget
data-tooltip
data-trailerId
data-transactionId
data-transition
data-ts
data-tweetContent
data-tweetUrl
data-type
data-useAjax
data-value
data-width
data-x
dm:index
dm:type
g:aspects
g:decorateusingsecondary
g:em
g:entity
g:groups
g:id
g:istoplevel
g:li
g:numresults
g:oid
g:parentId
g:pl
g:pt
g:rating_override
g:secondaryurls
g:sortby
g:startindex
g:target
g:type
g:url
g:value
ga:barsize
ga:css
ga:expandAfterCharsExceed
ga:initialNumRows
ga:nocancelicon
ga:numRowsToExpandTo
ga:type
ga:unlockwhenrated
gw:address
gw:businessname
gw:comment
gw:phone
gw:source
ng:controller
xlink:href
xml:lang
xmlns:atom
xmlns:dc
xmlns:jstd
xmlns:ng
xmlns:og
xmlns:webstore
xmlns:xlink
R=nigeltao
CC=golang-dev
https://golang.org/cl/5119041
The normalization that prevents element name and comment injection in
<{{.}}
by converting it to
<{{.}}
breaks
<!DOCTYPE html>
Instead of splitting states to have a start of document state and a text
state, I whitelist <!DOCTYPE.
R=nigeltao
CC=golang-dev
https://golang.org/cl/5131051
In
{{$x := . | foo}}
{{$x}}
the first action is a variable assignment that contributes
nothing to the output while the first is a use that needs
to be escaped.
This CL fixes escapeAction to distinguish assignments from
interpolations and to only modify interpolations.
R=nigeltao, r
CC=golang-dev
https://golang.org/cl/5143048
Does some TODOs and changes the term "div" in an error message
to "division" to avoid confusion with "<div>".
R=nigeltao, r
CC=golang-dev
https://golang.org/cl/5141047
This makes sure that all JS newlines are encoded in JSON.
It also moots a TODO about possibly escaping supplemental codepoints.
I served:
Content-Type: text/javascript;charset=UTF-8
var s = "%s";
document.write("<p>", s, "</p><ol>");
for (var i = 0; i < s.length; i++) {
document.write("<li>", s.charCodeAt(i).toString(16), "</li>");
}
document.write("</l>");
where %s was replaced with bytes "\xf0\x9d\x84\x9e" to test
straight UTF-8 instead of encoding surrogates separately.
Recent Firefox, Chrome, and Safari all decoded it properly.
I have yet to try it on IE or older versions.
R=nigeltao
CC=golang-dev
https://golang.org/cl/5129042
The template
"<a="
caused an infinite loop in escape text.
The change to tTag fixes that and the change to escape.go causes
escapeText to panic on any infinite loop that does not involve
a state cycle.
R=nigeltao
CC=golang-dev
https://golang.org/cl/5115041
HTML parsers may differ on whether
<input id= onchange=f( ends in id's or onchange's value,
<a class=`foo ends inside a value,
<input style=font:'Arial' needs open-quote fixup.
Per
http://www.w3.org/TR/html5/tokenization.html#attribute-value-unquoted-state
this treats the error cases in 8.2.4.40 Attribute value (unquoted) state
as fatal errors.
\> U+0022 QUOTATION MARK (")
\> U+0027 APOSTROPHE (')
\> U+003C LESS-THAN SIGN (<)
\> U+003D EQUALS SIGN (=)
\> U+0060 GRAVE ACCENT (`)
Parse error. Treat it as per the "anything else" entry below.
and emits ErrBadHTML.
R=nigeltao
CC=golang-dev
https://golang.org/cl/5085050
When templates are stored in external files, developers often embed
comments to explain&|disable code.
<!-- Oblique reference to project code name here -->
{{if .C}}...{{else}}<!-- commented out default -->{{end}}
This unnecessarily increases the size of shipped HTML and can leak
information.
This change elides all comments of the following types:
1. <!-- ... --> comments found in source.
2. /*...*/ and // comments found in <script> elements.
3. /*...*/ and // comments found in <style> elements.
It does not elide /*...*/ or // comments found in HTML attributes:
4. <button onclick="/*...*/">
5. <div style="/*...*/">
I can find no examples of comments in attributes in Closure Templates
code and doing so would require keeping track of character positions
post decode in
<button onclick="/*...*/">
To prevent token joining, /*comments*/ are JS and CSS comments are
replaced with a whitespace char.
HTML comments are not, but to prevent token joining we could try to
detect cases like
<<!---->b>
</<!---->b>
which has a well defined meaning in HTML but will cause a validator
to barf. This is difficult, and this is a very minor case.
I have punted for now, but if we need to address this case, the best
way would be to normalize '<' in stateText to '<' consistently.
The whitespace to replace a JS /*comment*/ with depends on whether
there is an embedded line terminator since
break/*
*/foo
...
is equivalent to
break;
foo
...
while
break/**/foo
...
is equivalent to
break foo;
...
Comment eliding can interfere with IE conditional comments.
http://en.wikipedia.org/wiki/Conditional_comment
<!--[if IE 6]>
<p>You are using Internet Explorer 6.</p>
<![endif]-->
/*@cc_on
document.write("You are using IE4 or higher");
@*/
I have not encountered these in production template code, and
the typed content change in CL 4962067 provides an escape-hatch
if conditional comments are needed.
R=nigeltao
CC=golang-dev
https://golang.org/cl/4999042
This simplifies transition functions to make it easier to reliably
elide comments in a later CL.
Before:
- transition functions are responsible for detecting special end tags.
After:
- the code to detect special end tags is done in one place.
We were relying on end tags being skipped which meant we were
not noticing comments inside script/style elements that contain no
substitutions.
This change means we will notice all such comments where necessary,
but stripTags will notice none since it does not need to. This speeds
up stripTags.
R=nigeltao
CC=golang-dev
https://golang.org/cl/5074041
This CL generalises the pair of halfConnection members that the
serverConn holds into a single transport struct that is shared by
both Server and Client, see also CL 5037047.
This CL is a replacement for 5040046 which I closed by accident.
R=agl, bradfitz
CC=golang-dev
https://golang.org/cl/5075042
Formulaic changes to transition functions in preparation for CL 5074041.
This should be completely semantics preserving.
R=nigeltao
CC=golang-dev
https://golang.org/cl/5091041
Instead of erroring on actions inside comments, use existing escaping
pipeline to quash the output of actions inside comments.
If a template maintainer uses a comment to disable template code:
{{if .}}Hello, {{.}}!{{end}}
->
<!--{{if true}}Hello, {{.}}!{{end}}-->
will result in
<!--Hello, !-->
regardless of the value of {{.}}.
In a later CL, comment elision will result in the entire commented-out
section being dropped from the template output.
Any side-effects in pipelines, such as panics, will still be realized.
R=nigeltao
CC=golang-dev
https://golang.org/cl/5078041
This addresses several use cases:
(1) <h{{.HeaderLevel}}> used to build hierarchical documents.
(2) <input on{{.EventType}}=...> used in widgets.
(3) <div {{" dir=ltr"}}> used to embed bidi-hints.
It also makes sure that we treat the two templates below the same:
<img src={{if .Avatar}}"{{.Avatar}}"{{else}}"anonymous.png"{{end}}>
<img src="{{if .Avatar}}{{.Avatar}}{{else}}anonymous.png{{end}}">
This splits up tTag into a number of sub-states and adds testcases.
R=nigeltao
CC=golang-dev
https://golang.org/cl/5043042
The template
<{{.}}
would violate the structure preservation property if allowed and not
normalized, because when {{.}} emitted "", the "<" would be part of
a text node, but if {{.}} emitted "a", the "<" would not be part of
a text node.
This change rewrites '<' in text nodes and RCDATA text nodes to
'<' allowing template authors to write the common, and arguably more
readable:
Your price: {{.P1}} < list price {{.P2}}
while preserving the structure preservation property.
It also lays the groundwork for comment elision, rewriting
Foo <!-- comment with secret project details --> Bar
to
Foo Bar
R=nigeltao
CC=golang-dev
https://golang.org/cl/5043043
The typical UNIX method for controlling long running process is to
send the process signals. Since this doesn't get you very far, various
ad-hoc, remote-control protocols have been used over time by programs
like Apache and BIND.
Implementing an SSH server means that Go code will have a standard,
secure way to do this in the future.
R=bradfitz, borman, dave, gustavo, dsymonds, r, adg, rsc, rogpeppe, lvd, kevlar, raul.san
CC=golang-dev
https://golang.org/cl/4962064
I found a simple test case that does require doing the fixed point TODO
in computeOutCtx.
I found a way though to do this and simplify away the escapeRange
hackiness that was added in https://golang.org/cl/5012044/
R=nigeltao
CC=golang-dev
https://golang.org/cl/5015052
This replaces the errStr & errLine members of context with a single err
*Error, and introduces a number of const error codes, one per
escape-time failure mode, that can be separately documented.
The changes to the error documentation moved from doc.go to error.go
are cosmetic.
R=r, nigeltao
CC=golang-dev
https://golang.org/cl/5026041
Not all content is plain text. Sometimes content comes from a trusted
source, such as another template invocation, an HTML tag whitelister,
etc.
Template authors can deal with over-escaping in two ways.
1) They can encapsulate known-safe content via
type HTML, type CSS, type URL, and friends in content.go.
2) If they know that the for a particular action never needs escaping
then they can add |noescape to the pipeline.
{{.KnownSafeContent | noescape}}
which will prevent any escaping directives from being added.
This CL defines string type aliases: HTML, CSS, JS, URI, ...
It then modifies stringify to unpack the content type.
Finally it modifies the escaping functions to use the content type and
decline to escape content that does not require it.
There are minor changes to escapeAction and helpers to treat as
equivalent explicit escaping directives such as "html" and "urlquery"
and the escaping directives defined in the contextual autoescape module
and to recognize the special "noescape" directive.
The html escaping functions are rearranged. Instead of having one
escaping function used in each {{.}} in
{{.}} : <textarea title="{{.}}">{{.}}</textarea>
a slightly different escaping function is used for each.
When {{.}} binds to a pre-sanitized string of HTML
`one < <i>two</i> & two < "3"`
we produces something like
one < <i>two</i> & two < "3" :
<textarea title="one < two & two < "3"">
one < <i>two</i> & two < "3"
</textarea>
Although escaping is not required in <textarea> normally, if the
substring </textarea> is injected, then it breaks, so we normalize
special characters in RCDATA and do the same to preserve attribute
boundaries. We also strip tags since developers never intend
typed HTML injected in an attribute to contain tags escaped, but
do occasionally confuse pre-escaped HTML with HTML from a
tag-whitelister.
R=golang-dev, nigeltao
CC=golang-dev
https://golang.org/cl/4962067
This moots a caveat in the proposed package documentation by
rendering useless any template that could not be escaped.
From https://golang.org/cl/4969078/
> If EscapeSet returns an error, do not Execute the set; it is not
> safe against injection.
r: [but isn't the returned set nil? i guess you don't overwrite the
r: original if there's a problem, but i think you're in your rights to
r: do so]
R=r
CC=golang-dev
https://golang.org/cl/5020043
This adds support for {{template "callee"}} calls.
It recognizes that calls can appear in many contexts.
{{if .ImageURL}}
<img src="{{.ImageURL}}" alt="{{template "description"}}">
{{else}}
<p>{{template "description"}}</p>
{{end}}
calls a template in two different contexts, first in an HTML attribute
context, and second in an HTML text context.
Those two contexts aren't very different, but when linking text
to search terms, the escaping context can be materially different:
<a href="/search?q={{template "tags"}}">{{template "tags"}}</a>
This adds API:
EscapeSet(*template.Set, names ...string) os.Error
takes a set of templates and the names of those which might be called
in the default context as starting points.
It changes the escape* functions to be methods of an object which
maintains a conceptual mapping of
(template names*input context) -> output context.
The actual mapping uses as key a mangled name which combines the
template name with the input context.
The mangled name when the input context is the default context is the
same as the unmangled name.
When a template is called in multiple contexts, we clone the template.
{{define "tagLink"}}
<a href="/search?q={{template "tags"}}">{{template "tags"}}</a>
{{end}}
{{define "tags"}}
{{range .Tags}}{{.}},{{end}}
{{end}}
given []string{ "foo", "O'Reilly", "bar" } produces
<a href="/search?q=foo,O%27Reilly,bar">foo,O'Reilly,bar</a>
This involves rewriting the above to something like
{{define "tagLink"}}
<a href="/search?q={{template "tags$1"}}">{{template "tags"}}</a>
{{end}}
{{define "tags"}}
{{range .Tags}}{{. | html}},{{end}}
{{end}}
{{define "tags$1"}}
{{range .Tags}}{{. | urlquery}},{{end}}
{{end}}
clone.go provides a mechanism for cloning template "tags" to produce
"tags$1".
changes to escape.go implement the new API and context propagation
around the call graph.
context.go includes minor changes to support name mangling and
context_test.go tests those.
js.go contains a bug-fix.
R=nigeltao, r
CC=golang-dev
https://golang.org/cl/4969072
Often, division/regexp ambiguity doesn't matter in JS because the next
token is not a slash.
For example, in
<script>var global{{if .InitVal}} = {{.InitVal}}{{end}}</script>
When there is an initial value, the {{if}} ends with jsCtxDivOp
since a '/' following {{.InitVal}} would be a division operator.
When there is none, the empty {{else}} branch ends with jsCtxRegexp
since a '/' would start a regular expression. A '/' could result
in a valid program if it were on a new line to allow semicolon
insertion to terminate the VarDeclaration.
There is no '/' though, so we can ignore the ambiguity.
There are cases where a missing semi can result in ambiguity that
we should report.
<script>
{{if .X}}var x = {{.X}}{{end}}
/...{{.Y}}
</script>
where ... could be /foo/.test(bar) or /divisor. Disambiguating in
this case is hard and is required to sanitize {{.Y}}.
Note, that in the case where there is a '/' in the script tail but it
is not followed by any interpolation, we already don't care. So we
are already tolerant of
<script>{{if .X}}var x = {{.X}}{{end}}/a-bunch-of-text</script>
because tJS checks for </script> before looking in /a-bunch-of-text.
This CL
- Adds a jsCtx value: jsCtxUnknown
- Changes joinContext to join contexts that only differ by jsCtx.
- Changes tJS to return an error when a '/' is seen in jsCtxUnknown.
- Adds tests for both the happy and sad cases.
R=nigeltao
CC=golang-dev
https://golang.org/cl/4956077
Augments type context and adds grammatical rules to handle special HTML constructs:
<!-- comments -->
<script>raw text</script>
<textarea>no tags here</textarea>
This CL does not elide comment content. I recommend we do that but
have not done it in this CL.
I used a codesearch tool over a codebase in another template language.
Based on the below I think we should definitely recognize
<script>, <style>, <textarea>, and <title>
as each of these appears frequently enough that there are few
template using apps that do not use most of them.
Of the other special tags,
<xmp>, <noscript>
are used but infrequently, and
<noframe> and friend, <listing>
do not appear at all.
We could support <xmp> even though it is obsolete in HTML5
because we already have the machinery, but I suggest we do not
support noscript since it is a normal tag in some browser
configurations.
I suggest recognizing and eliding <!-- comments -->
(but not escaping text spans) as they are widely used to
embed comments in template source. Not eliding them increases
the size of content sent over the network, and risks leaking
code and project internal details.
The template language I tested elides them so there are
no instance of IE conditional compilation directives in the
codebase but that could be a source of confusion.
The codesearch does the equivalent of
$ find . -name \*.file-extension \
| perl -ne 'print "\L$1\n" while s@<([a-z][a-z0-9])@@i' \
| sort | uniq -c | sort
The 5 uses of <plaintext> seem to be in tricky code and can be ignored.
The 2 uses of <xmp> appear in the same tricky code and can be ignored.
I also ignored end tags to avoid biasing against unary
elements and threw out some nonsense names since since the
long tail is dominated by uses of < as a comparison operator
in the template languages expression language.
I have added asterisks next to abnormal elements.
26765 div
7432 span
7414 td
4233 a
3730 tr
3238 input
2102 br
1756 li
1755 img
1674 table
1388 p
1311 th
1064 option
992 b
891 label
714 script *
519 ul
446 tbody
412 button
381 form
377 h2
358 select
353 strong
318 h3
314 body
303 html
266 link
262 textarea *
261 head
258 meta
225 title *
189 h1
176 col
156 style *
151 hr
119 iframe
103 h4
101 pre
100 dt
98 thead
90 dd
83 map
80 i
69 object
66 ol
65 em
60 param
60 font
57 fieldset
51 string
51 field
51 center
44 bidi
37 kbd
35 legend
30 nobr
29 dl
28 var
26 small
21 cite
21 base
20 embed
19 colgroup
12 u
12 canvas
10 sup
10 rect
10 optgroup
10 noscript *
9 wbr
9 blockquote
8 tfoot
8 code
8 caption
8 abbr
7 msg
6 tt
6 text
6 h5
5 svg
5 plaintext *
5 article
4 shortquote
4 number
4 menu
4 ins
3 progress
3 header
3 content
3 bool
3 audio
3 attribute
3 acronym
2 xmp *
2 overwrite
2 objects
2 nobreak
2 metadata
2 description
2 datasource
2 category
2 action
R=nigeltao
CC=golang-dev
https://golang.org/cl/4964045
This does not wire up <style> elements as that is pending support
for raw text content in CL https://golang.org/cl/4964045/
This CL allows actions to appear in contexts like
selectors: {{.Tag}}{{.Class}}{{.Id}}
property names: border-{{.BidiLeadingEdge}}
property values: color: {{.Color}}
strings: font-family: "{{font-name}}"
URL strings: background: "/foo?image={{.ImgQuery}}"
URL literals: background: url("{{.Image}}")
but disallows actions inside CSS comments and disallows
embedding of JS in CSS entirely.
It is based on the CSS3 lexical grammar with affordances for
common browser extensions including line comments.
R=nigeltao
CC=golang-dev
https://golang.org/cl/4968058
