1
0
mirror of https://github.com/golang/go synced 2024-11-21 16:04:45 -07:00

Clarify section on tokens.

Fixes #457.

R=r
CC=golang-dev
https://golang.org/cl/181083
This commit is contained in:
Robert Griesemer 2009-12-28 14:40:42 -08:00
parent 33311a7945
commit eb109a765d

View File

@ -151,12 +151,13 @@ Comments do not nest.
<p>
Tokens form the vocabulary of the Go language.
There are four classes: identifiers, keywords, operators
and delimiters, and literals. <i>White space</i>, formed from
There are four classes: <i>identifiers</i>, <i>keywords</i>, <i>operators
and delimiters</i>, and <i>literals</i>. <i>White space</i>, formed from
spaces (U+0020), horizontal tabs (U+0009),
carriage returns (U+000D), and newlines (U+000A),
is ignored except as it separates tokens
that would otherwise combine into a single token.
that would otherwise combine into a single token. Also, a newline
may trigger the insertion of a <a href="#Semicolons">semicolon</a>.
While breaking the input into tokens,
the next token is the longest sequence of characters that form a
valid token.