Writing Web Applications

Introduction

Covered in this codelab:

Assumed knowledge:

Getting Started

At present, you need to have a Linux, OS X, or FreeBSD machine to run Go. If you don't have access to one, you could set up a Linux Virtual Machine (using VirtualBox or similar) or a Virtual Private Server.

Install Go (see the Installation Instructions).

Make a new directory for this codelab and cd to it: 

$ mkdir ~/gowiki
$ cd ~/gowiki

Create a file named wiki.go, open it in your favorite editor, and add the following lines: 

package main

import (
	"fmt"
	"io/ioutil"
	"os"
)

We import the fmt, ioutil and os packages from the Go standard library. Later, as we implement additional functionality, we will add more packages to this import declaration.

Data Structures

Let's start by defining the data structures. A wiki consists of a series of interconnected pages, each of which has a title and a body (the page content). Here, we define page as a struct with two fields representing the title and body. 

type page struct {
	title	string
	body	[]byte
}

The type []byte means "a byte slice". (See Effective Go for more on slices.) The body element is a []byte rather than string because that is the type expected by the io libraries we will use, as you'll see below.

The page struct describes how page data will be stored in memory. But what about persistent storage? We can address that by creating a save method on page: 

func (p *page) save() os.Error {
	filename := p.title + ".txt"
	return ioutil.WriteFile(filename, p.body, 0600)
}

This method's signature reads: "This is a method named save that takes as its receiver p, a pointer to page . It takes no parameters, and returns a value of type os.Error."

This method will save the page's body to a text file. For simplicity, we will use the title as the file name.

The save method returns an os.Error value because that is the return type of WriteFile (a standard library function that writes a byte slice to a file). The save method returns the error value, to let the application handle it should anything go wrong while writing the file. If all goes well, page.save() will return nil (the zero-value for pointers, interfaces, and some other types).

The octal integer constant 0600, passed as the third parameter to WriteFile, indicates that the file should be created with read-write permissions for the current user only. (See the Unix man page open(2) for details.)

We will want to load pages, too: 

func loadPage(title string) *page {
	filename := title + ".txt"
	body, _ := ioutil.ReadFile(filename)
	return &page{title: title, body: body}
}

The function loadPage constructs the file name from title, reads the file's contents into a new page, and returns a pointer to that new page.

Functions can return multiple values. The standard library function io.ReadFile returns []byte and os.Error. In loadPage, error isn't being handled yet; the "blank identifier" represented by the underscore (_) symbol is used to throw away the error return value (in essence, assigning the value to nothing).

But what happens if ReadFile encounters an error? For example, the file might not exist. We should not ignore such errors. Let's modify the function to return *page and os.Error. 

func loadPage(title string) (*page, os.Error) {
	filename := title + ".txt"
	body, err := ioutil.ReadFile(filename)
	if err != nil {
		return nil, err
	}
	return &page{title: title, body: body}, nil
}

Callers of this function can now check the second parameter; if it is nil then it has successfully loaded a page. If not, it will be an os.Error that can be handled by the caller (see the os package documentation for details).

At this point we have a simple data structure and the ability to save to and load from a file. Let's write a main function to test what we've written: 

func main() {
	p1 := &page{title: "TestPage", body: []byte("This is a sample page.")}
	p1.save()
	p2, _ := loadPage("TestPage")
	fmt.Println(string(p2.body))
}

After compiling and executing this code, a file named TestPage.txt would be created, containing the contents of p1. The file would then be read into the struct p2, and its body element printed to the screen.

You can compile and run the program like this: 

$ 8g wiki.go
$ 8l wiki.8
$ ./8.out
This is a sample page.

(The 8g and 8l commands are applicable to GOARCH=386. If you're on an amd64 system, substitute 6's for the 8's.)

Click here to view the code we've written so far.

Introducing the http package (an interlude)

Here's a full working example of a simple web server: 

package main

import (
	"fmt"
	"http"
)

func handler(c *http.Conn, r *http.Request) {
	fmt.Fprintf(c, "Hi there, I love %s!", r.URL.Path[1:])
}

func main() {
	http.HandleFunc("/", handler)
	http.ListenAndServe(":8080", nil)
}

The main function begins with a call to http.HandleFunc, which tells the http package to handle all requests to the web root ("/") with handler.

It then calls http.ListenAndServe, specifying that it should listen on port 8080 on any interface (":8080"). (Don't worry about its second parameter, nil, for now.) This function will block until the program is terminated.

The function handler is of the type http.HandlerFunc. It takes an http.Conn and http.Request as its arguments.

An http.Conn is the server end of an HTTP connection; by writing to it, we send data to the HTTP client.

An http.Request is a data structure that represents the client HTTP request. The string r.URL.Path is the path component of the request URL. The trailing [1:] means "create a sub-slice of Path from the 1st character to the end." This drops the leading "/" from the path name.

If you run this program and access the URL: 

http://localhost:8080/monkeys

the program would present a page containing: 

Hi there, I love monkeys!

Using http to serve wiki pages

To use the http package, it must be imported: 

import (
	"fmt"
	"http"
	"io/ioutil"
	"os"
)

Let's create a handler to view a wiki page: 

const lenPath = len("/view/")

func viewHandler(c *http.Conn, r *http.Request) {
	title := r.URL.Path[lenPath:]
	p, _ := loadPage(title)
	fmt.Fprintf(c, "<h1>%s</h1><div>%s</div>", p.title, p.body)
}

First, this function extracts the page title from r.URL.Path, the path component of the request URL. The global constant lenPath is the length of the leading "/view/" component of the request path. The Path is re-sliced with [lenPath:] to drop the first 6 characters of the string. This is because the path will invariably begin with "/view/", which is not part of the page title.

The function then loads the page data, formats the page with a string of simple HTML, and writes it to c, the http.Conn.

Again, note the use of _ to ignore the os.Error return value from loadPage. This is done here for simplicity and generally considered bad practice. We will attend to this later.

To use this handler, we create a main function that initializes http using the viewHandler to handle any requests under the path /view/. 

func main() {
	http.HandleFunc("/view/", viewHandler)
	http.ListenAndServe(":8080", nil)
}

Click here to view the code we've written so far.

Let's create some page data (as test.txt), compile our code, and try serving a wiki page: 

$ echo "Hello world" > test.txt
$ 8g wiki.go
$ 8l wiki.8
$ ./8.out

With this web server running, a visit to http://localhost:8080/view/test should show a page titled "test" containing the words "Hello world".

Editing pages

A wiki is not a wiki without the ability to edit pages. Let's create two new handlers: one named editHandler to display an 'edit page' form, and the other named saveHandler to save the data entered via the form.

First, we add them to main(): 

func main() {
	http.HandleFunc("/view/", viewHandler)
	http.HandleFunc("/edit/", editHandler)
	http.HandleFunc("/save/", saveHandler)
	http.ListenAndServe(":8080", nil)
}

The function editHandler loads the page (or, if it doesn't exist, create an empty page struct), and displays an HTML form. 

func editHandler(c *http.Conn, r *http.Request) {
	title := r.URL.Path[lenPath:]
	p, err := loadPage(title)
	if err != nil {
		p = &page{title: title}
	}
	fmt.Fprintf(c, "<h1>Editing %s</h1>"+
		"<form action=\"/save/%s\" method=\"POST\">"+
		"<textarea name=\"body\">%s</textarea><br>"+
		"<input type=\"submit\" value=\"Save\">"+
		"</form>",
		p.title, p.title, p.body)
}

This function will work fine, but all that hard-coded HTML is ugly. Of course, there is a better way.

The template package

The template package is part of the Go standard library. We can use template to keep the HTML in a separate file, allowing us to change the layout of our edit page without modifying the underlying Go code.

First, we must add template to the list of imports: 

import (
	"http"
	"io/ioutil"
	"os"
	"template"
)

Let's create a template file containing the HTML form. Open a new file named edit.html, and add the following lines: 

<h1>Editing {title}</h1>

<form action="/save/{title}" method="POST">
<div><textarea name="body" rows="20" cols="80">{body|html}</textarea></div>
<div><input type="submit" value="Save"></div>
</form>

Modify editHandler to use the template, instead of the hard-coded HTML: 

func editHandler(c *http.Conn, r *http.Request) {
	title := r.URL.Path[lenPath:]
	p, err := loadPage(title)
	if err != nil {
		p = &page{title: title}
	}
	t, _ := template.ParseFile("edit.html", nil)
	t.Execute(p, c)
}

The function template.ParseFile will read the contents of edit.html and return a *template.Template.

The method t.Execute replaces all occurrences of {title} and {body} with the values of p.title and p.body, and writes the resultant HTML to the http.Conn.

Note that we've used {body|html} in the above template. The |html part asks the template engine to pass the value body through the html formatter before outputting it, which escapes HTML characters (such as replacing > with &gt;). This will prevent user data from corrupting the form HTML.

Now that we've removed the fmt.Sprintf statement, we can remove "fmt" from the import list.

While we're working with templates, let's create a template for our viewHandler called view.html: 

<h1>{title}</h1>

<p>[<a href="/edit/{title}">edit</a>]</p>

<div>{body}</div>

Modify viewHandler accordingly: 

func viewHandler(c *http.Conn, r *http.Request) {
	title := r.URL.Path[lenPath:]
	p, _ := loadPage(title)
	t, _ := template.ParseFile("view.html", nil)
	t.Execute(p, c)
}

Notice that we've used almost exactly the same templating code in both handlers. Let's remove this duplication by moving the templating code to its own function: 

func viewHandler(c *http.Conn, r *http.Request) {
	title := r.URL.Path[lenPath:]
	p, _ := loadPage(title)
	renderTemplate(c, "view", p)
}

func editHandler(c *http.Conn, r *http.Request) {
	title := r.URL.Path[lenPath:]
	p, err := loadPage(title)
	if err != nil {
		p = &page{title: title}
	}
	renderTemplate(c, "edit", p)
}

func renderTemplate(c *http.Conn, tmpl string, p *page) {
	t, _ := template.ParseFile(tmpl+".html", nil)
	t.Execute(p, c)
}

The handlers are now shorter and simpler.

Handling non-existent pages

What if you visit /view/APageThatDoesntExist? The program will crash. This is because it ignores the error return value from loadPage. Instead, if the requested page doesn't exist, it should redirect the client to the edit page so the content may be created: 

func viewHandler(c *http.Conn, r *http.Request, title string) {
	p, err := loadPage(title)
	if err != nil {
		http.Redirect(c, "/edit/"+title, http.StatusFound)
		return
	}
	renderTemplate(c, "view", p)
}

The http.Redirect function adds an HTTP status code of http.StatusFound (302) and a Location header to the HTTP response.

Saving pages

The function saveHandler will handle the form submission. 

func saveHandler(c *http.Conn, r *http.Request) {
	title := r.URL.Path[lenPath:]
	body := r.FormValue("body")
	p := &page{title: title, body: []byte(body)}
	p.save()
	http.Redirect(c, "/view/"+title, http.StatusFound)
}

The page title (provided in the URL) and the form's only field, body, are stored in a new page. The save() method is then called to write the data to a file, and the client is redirected to the /view/ page.

The value returned by FormValue is of type string. We must convert that value to []byte before it will fit into the page struct. We use []byte(body) to perform the conversion.

Error handling

There are several places in our program where errors are being ignored. This is bad practice, not least because when an error does occur the program will crash. A better solution is to handle the errors and return an error message to the user. That way if something does go wrong, the server will continue to function and the user will be notified.

First, let's handle the errors in renderTemplate: 

func renderTemplate(c *http.Conn, tmpl string, p *page) {
	t, err := template.ParseFile(tmpl+".html", nil)
	if err != nil {
		http.Error(c, err.String(), http.StatusInternalServerError)
		return
	}
	err = t.Execute(p, c)
	if err != nil {
		http.Error(c, err.String(), http.StatusInternalServerError)
	}
}

The http.Error function sends a specified HTTP response code (in this case "Internal Server Error") and error message. Already the decision to put this in a separate function is paying off.

Now let's fix up saveHandler: 

func saveHandler(c *http.Conn, r *http.Request, title string) {
	body := r.FormValue("body")
	p := &page{title: title, body: []byte(body)}
	err := p.save()
	if err != nil {
		http.Error(c, err.String(), http.StatusInternalServerError)
		return
	}
	http.Redirect(c, "/view/"+title, http.StatusFound)
}

Any errors that occur during p.save() will be reported to the user.

Template caching

There is an inefficiency in this code: renderTemplate calls ParseFile every time a page is rendered. A better approach would be to call ParseFile once for each template at program initialization, and store the resultant *Template values in a data structure for later use.

First we create a global map named templates in which to store our *Template values, keyed by string (the template name): 

var templates = make(map[string]*template.Template)

Then we create an init function, which will be called before main at program initialization. The function template.MustParseFile is a convenience wrapper around ParseFile that does not return an error code; instead, it panics if an error is encountered. A panic is appropriate here; if the templates can't be loaded the only sensible thing to do is exit the program.

func init() { for _, tmpl := range []string{"edit", "view"} { templates[tmpl] = template.MustParseFile(tmpl+".html", nil) } }

A for loop is used with a range statement to iterate over an array constant containing the names of the templates we want parsed. If we were to add more templates to our program, we would add their names to that array.

We then modify our renderTemplate function to call the Execute method on the appropriate Template from templates: 

func renderTemplate(c *http.Conn, tmpl string, p *page) {
	err := templates[tmpl].Execute(p, c)
	if err != nil {
		http.Error(c, err.String(), http.StatusInternalServerError)
	}
}

Validation

As you may have observed, this program has a serious security flaw: a user can supply an arbitrary path to be read/written on the server. To mitigate this, we can write a function to validate the title with a regular expression.

First, add "regexp" to the import list. Then we can create a global variable to store our validation regexp: 

var titleValidator = regexp.MustCompile("^[a-zA-Z0-9]+$")

The function regexp.MustCompile will parse and compile the regular expression, and return a regexp.Regexp. MustCompile, like template.MustParseFile, is distinct from Compile in that it will panic if the expression compilation fails, while Compile returns an os.Error as a second parameter.

Now, let's write a function that extracts the title string from the request URL, and tests it against our titleValidator expression: 

func getTitle(c *http.Conn, r *http.Request) (title string, err os.Error) {
	title = r.URL.Path[lenPath:]
	if !titleValidator.MatchString(title) {
		http.NotFound(c, r)
		err = os.NewError("Invalid Page Title")
	}
	return
}

If the title is valid, it will be returned along with a nil error value. If the title is invalid, the function will write a "404 Not Found" error to the HTTP connection, and return an error to the handler.

Let's put a call to getTitle in each of the handlers: 

func viewHandler(c *http.Conn, r *http.Request) {
	title, err := getTitle(c, r)
	if err != nil {
		return
	}
	p, err := loadPage(title)
	if err != nil {
		http.Redirect(c, "/edit/"+title, http.StatusFound)
		return
	}
	renderTemplate(c, "view", p)
}

func editHandler(c *http.Conn, r *http.Request) {
	title, err := getTitle(c, r)
	if err != nil {
		return
	}
	p, err := loadPage(title)
	if err != nil {
		p = &page{title: title}
	}
	renderTemplate(c, "edit", p)
}

func saveHandler(c *http.Conn, r *http.Request) {
	title, err := getTitle(c, r)
	if err != nil {
		return
	}
	body := r.FormValue("body")
	p := &page{title: title, body: []byte(body)}
	err = p.save()
	if err != nil {
		http.Error(c, err.String(), http.StatusInternalServerError)
		return
	}
	http.Redirect(c, "/view/"+title, http.StatusFound)
}

Introducing Function Literals and Closures

Catching the error condition in each handler introduces a lot of repeated code. What if we could wrap each of the handlers in a function that does this validation and error checking? Go's function literals provide a powerful means of abstracting functionality that can help us here.

First, we re-write the function definition of each of the handlers to accept a title string: 

func viewHandler(c, *http.Conn, r *http.Request, title string)
func editHandler(c, *http.Conn, r *http.Request, title string)
func saveHandler(c, *http.Conn, r *http.Request, title string)

Now let's define a wrapper function that takes a function of the above type, and returns a function of type http.HandlerFunc (suitable to be passed to the function http.HandleFunc): 

func makeHandler(fn func (*http.Conn, *http.Request, string)) http.HandlerFunc {
	return func(c *http.Conn, r *http.Request) {
		// Here we will extract the page title from the Request,
		// and call the provided handler 'fn'
	}
}

The returned function is called a closure because it encloses values defined outside of it. In this case, the variable fn (the single argument to makeHandler) is enclosed by the closure. The variable fn will be one of our save, edit, or view handlers.

Now we can take the code from getTitle and use it here (with some minor modifications): 

func makeHandler(fn func(*http.Conn, *http.Request, string)) http.HandlerFunc {
	return func(c *http.Conn, r *http.Request) {
		title := r.URL.Path[lenPath:]
		if !titleValidator.MatchString(title) {
			http.NotFound(c, r)
			return
		}
		fn(c, r, title)
	}
}

The closure returned by makeHandler is a function that takes an http.Conn and http.Request (in other words, an http.HandlerFunc). The closure extracts the title from the request path, and validates it with the titleValidator regexp. If the title is invalid, an error will be written to the Conn using the http.NotFound function. If the title is valid, the enclosed handler function fn will be called with the Conn, Request, and title as arguments.

Now we can wrap the handler functions with makeHandler in main, before they are registered with the http package: 

func main() {
	http.HandleFunc("/view/", makeHandler(viewHandler))
	http.HandleFunc("/edit/", makeHandler(editHandler))
	http.HandleFunc("/save/", makeHandler(saveHandler))
	http.ListenAndServe(":8080", nil)
}

Finally we remove the calls to getTitle from the handler functions, making them much simpler: 

func viewHandler(c *http.Conn, r *http.Request, title string) {
	p, err := loadPage(title)
	if err != nil {
		http.Redirect(c, "/edit/"+title, http.StatusFound)
		return
	}
	renderTemplate(c, "view", p)
}

func editHandler(c *http.Conn, r *http.Request, title string) {
	p, err := loadPage(title)
	if err != nil {
		p = &page{title: title}
	}
	renderTemplate(c, "edit", p)
}

func saveHandler(c *http.Conn, r *http.Request, title string) {
	body := r.FormValue("body")
	p := &page{title: title, body: []byte(body)}
	err := p.save()
	if err != nil {
		http.Error(c, err.String(), http.StatusInternalServerError)
		return
	}
	http.Redirect(c, "/view/"+title, http.StatusFound)
}

Try it out!

Click here to view the final code listing.

Recompile the code, and run the app: 

$ 8g wiki.go
$ 8l wiki.8
$ ./8.out

Visiting http://localhost:8080/view/ANewPage should present you with the page edit form. You should then be able to enter some text, click 'Save', and be redirected to the newly created page.

Other tasks

Here are some simple tasks you might want to tackle on your own: