A Solid Foundation to Start a New Gin Web App

This is the first in a series of articles about writing a small reading list app in Go for personal use.

This is the first part of a series of articles about writing a small web app in Go for personal use. When we’re done you’ll be able to deploy your version on your home lab, or maybe on one of the fine services that hosts small apps like this.

I’m assuming you’re familiar with Go and have some clue about web things in general.

At the end of this article we will have:

• a skeleton “hello, world” web app in Go using the Gin framework
• a unit test covering this code
• a continuous integration pipeline that:
  • checks for lint (using golangci-lint)
  • runs the tests
  • collects and reports test coverage information
  • builds the executable

Defining the Goal

Rather than building a blog or a todo app let’s make a clone cheap imitation of the essential functionality of LibraryThing or Goodreads. My version of the essential functionality is:

• I want to track lists of books — mainly the list of books that I’ve read and the list of books that I someday want to read.
• But I also have books categorized into other lists (this is a secondary feature) — e.g. “programming”, “recommended”, “owned”.
• I also like that I can have ratings (i.e. 1-5 stars) and reviews or notes attached to books (another secondary feature).

If we build those few small features then we can export all of our info from one of those other sites and keep it our own personal installations. Which reminds me — I almost forgot one last “feature”: it would be nice to have a way to bulk-import data — we can probably just do this via a cli command that performs a one-shot import to the database. Since it’s really just a way to bootstrap a bunch of data, and we’re only going to do it once, it doesn’t need a nice UI.

Naming things is always hard. For projects like this I spend a few minutes looking up translations of the obvious English name into other languages and pick one that sounds pleasing. “Aklatan” is the Filipino word for “library”, so that’s what this app is going to be called.

Devising a Plan

I like to set out a rough plan for a project like this. It seems like a simple and obvious project goal, but there are always complications along the way and having a plan helps with decision-making when those things come up. It’s also helpful for having a way to measure how close it is to being done.

Tools

Let’s use Go 1.18 (beta), and we might find an excuse to use new language features somewhere along the way.

I do all of my coding in vim. I have a couple of ideas for future articles that include some editor-fu with details for vim. But the thinking behind it all will generalize across whatever editor you prefer.

My preferred CI environment is GitLab.

We’ll use a little basic Docker knowledge along the way to enhance our CI experience.

I like the Gin web framework so that’s what we’ll use to build this app.

Since this is really just a basic CRUD app, an ORM will help us avoid tedious boilerplate to map SQL tables onto structs. Gorm is the obvious package to use for this.

We’ll use sqlite for the database – for convenience of deployment and because I plan on hosting this internally for myself only.

Bootstrap will suffice for styling, and we’ll just use straight HTML with minimal javascript.

If your favorite stack is different, please drop me a note. I’m always curious to hear what other people are using and why they like different parts of the stack or dev toolchain.

Architecture

A simple MVC architecture is the obvious choice for Aklatan.

We only need a small number of models:

• books
• tags (for the lists/categories)
• reviews

If we wanted a lot of functionality around works by the same author, books in a series, editions of books, etc, we could split the book model up so that authors are managed independently. But for our purposes a single model/table will work fine — the author and other relevant metadata can just be tracked as fields on the book model. Just three models — easy!

Sizing

If we just add basic CRUD for each, we have five endpoints for each model: create, detail, listing, edit, and delete. The create and edit pages need to both present a form and handle POST data, but the delete page only needs to handle POST, so there are seven handlers per model. Add a default route and there are 22 handlers total.

In terms of sizing, two other small projects I’m nearly finished have:

• 6 models, 36 handlers → 1200 LOC Go, 800 LOC HTML
• 7 models, 12 handlers → 1300 LOC Go, 2K LOC HTML

(not including test code).

Aklatan is going to be more like the first project than the second in terms of simplicity, and handlers contribute more to code size than models, so I’d estimate this project is going to be about ⅔ the size of the first — let’s say 800 LOC of Go. We won’t go too deep in our data collection per book, so our forms will be fairly simple — let’s say 600 LOC of HTML.

We can make four main iterations:

• the skeleton
• CRUD for books
• CRUD for reviews
• CRUD for tags

When we’re finished with these, Aklatan will be usable as long as it’s deployed in a private/secure network — because we haven’t talked about any kind of authentication for it. But we can add that in a later iteration.

Let’s Get Started

Head on over to GitLab and create a new blank project. You can choose public or private, add a description, and optionally add a README. Here’s mine. (I’m sharing it publicly, but beware that I may rewrite history without warning or do other strange things because it’s meant as a demo project and not really intended to be actually usable as an app.)

Clone your project, edit the README as you see fit, and commit the changes. Go ahead and push your commit. One more change will be the last time we’ll just push straight to main! If you go to the project page, there’s an “Add LICENSE” button. Click that, choose the license you prefer, and submit.

Add the Code Skeleton

The code is pretty straightfoward. Create go.mod:

$ go mod init gitlab.com/yourname/aklatan

Create main.go:

package main

import (
	"log"
	"net/http"

	"github.com/gin-gonic/gin"
)

We import log for handling an error message, net/http for status codes, and gin to set up the server.

func defaultHandler(c *gin.Context) {
	c.HTML(http.StatusOK, "default.html", gin.H{})
}

This is just a stub handler for our default route. We’ll change this later to redirect to the book list, but for now this will just return a template that says “Hello, gin!”.

func setupRouter(r *gin.Engine) {
	r.LoadHTMLGlob("templates/**/*.html")
	r.GET("/", defaultHandler)
}

This sets up our routes and other miscellaneous housekeeping like loading templates. This is broken out into a separate function instead of having it inline in main so that we can call it from test code.

func main() {
	r := gin.Default()
	setupRouter(r)
	err := r.Run(":3000")
	if err != nil {
		log.Fatalf("gin Run error: %s", err)
	}
}

Our main function just creates a gin.Engine, sets it up, and runs it.

Before we can build we have to add gin to the project:

$ go get github.com/gin-gonic/gin

And before we can run anything we have to create a template. First create a directory structure for it:

$ mkdir -p templates/base

And then create the “default.html” template that we referenced above as templates/base/default.html:

{{ define "default.html" }}
Hello, gin!
{{ end }}

Again, we’ll end up changing this later, but this gives us a starting point.

Now we can build it:

$ go build .

And run it:

$ ./aklatan

And browsing to http://localhost:3000/ will show “Hello, gin!”

In your console you will see a log like:

[GIN] 2022/01/30 - 12:58:51 | 200 | 633.138µs |   127.0.0.1 | GET      "/"

At this point I like to commit my changes to git, but I don’t want to put this on main until I’ve had a chance to add a test and a CI pipeline. So let’s add this on a branch:

$ git checkout -b initial-skeleton
$ git add go.mod go.sum main.go templates/base/default.html

And then let’s add ./aklatan to our .gitignore so that we don’t see it in git status output:

$ echo aklatan > .gitignore
$ git add .gitignore

And now we can commit this:

$ git commit -m “Initial router setup”

Add a Test

In main_test.go:

package main

import (
	"net/http"
	"net/http/httptest"
	"strings"
	"testing"

	"github.com/gin-gonic/gin"
)

func TestDefaultRoute(t *testing.T) {
	t.Parallel()

I like to define all of my tests as parallel right from the start, and then run them with the race detector. This helps to immediately signal when we introduce either race conditions in the main product code or things in the test code that will prevent tests from being able to run in parallel. Parallelizing tests, if you have any significant number of http tests especially, makes a big difference in the runtime of the test suite.

	w := httptest.NewRecorder()
	ctx, r := gin.CreateTestContext(w)
	setupRouter(r)

Here we get a test recorder from httptest, get a test context and router from Gin, and then use our setupRouter function above to add routes to it.

	req, err := http.NewRequestWithContext(ctx, "GET", "/", nil)
	if err != nil {
		t.Errorf("got error: %s", err)
	}

This creates a new request for a GET on our default route.

	r.ServeHTTP(w, req)
	if http.StatusOK != w.Code {
		t.Fatalf("expected response code %d, got %d", http.StatusOK, w.Code)
	}

And then we run the request through our router, recording the result on w. I always perform a quick check of the status code to make sure that the handler at least responded with success (or failure if that’s what we’re testing).

	body := w.Body.String()

Then we extract the body from the recorder so that we can make some test comparisons — or in this case, exactly one test comparison.

	expected := "Hello, gin!"

	if expected != strings.Trim(body, " \r\n") {
		t.Fatalf("expected response body '%s', got '%s'", expected, body)
	}
} // End TestDefaultRoute

And then we can run this test:

$ go test -race .
ok      gitlab.com/bstpierre/aklatan    0.051s

“Meh, I don’t really need the race detector or parallel tests yet…”

I didn’t really think I needed it either, and then I was using testify in a project, and my tests were really slow, so I parallelized them. I immediately started getting weird errors. So I ran the race detector and got some funky data race reports. After a spending a bunch of time (and fixing some of my own racy test code), I managed to narrow it down and filed this issue against testify. And spent the better part of an afternoon ripping out testify in favor of just using the builtin testing library. I could have avoided all of that (and had faster tests all along) if I had just started with t.Parallel() and the race detector. But you may have different needs on your project, so you’re free to choose whatever approach you want.

Add the test and commit it:

$ git add main_test.go
$ git commit -m “Add unit test”

Add a Continuous Integration Pipeline

We could click the “Set up CI/CD” button, but I prefer to write my own pipeline, mostly because I copy bits of config from previous projects.

Create .gitlab-ci.yml with these contents:

---
stages:
    - build

build:
    stage: build
    image: golang:1.17.6-alpine3.15
    script:
        - apk update
        - apk add build-base
        - wget -O- -nv https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s v1.43.0
        - PATH=./bin/:$PATH
        - golangci-lint run -v --timeout 180s
        - go test -race -coverprofile=cover.out ./...
        - “go tool cover -func cover.out | sed -n -e '/^total/s/:.*statements)[^0-9]*/: /p'“
        - go build .
        - go install github.com/jstemmer/[email protected]
        - go test -v ./...  2>&1 | go-junit-report report.xml
    artifacts:
        when: always
        reports:
            junit: report.xml

Breaking this down:

• This defines a single-stage pipeline with just a build job. For larger projects it could make sense to split this into two or three stages, build, test, and perhaps a quick linting job to run first. But for this small project, and especially if you are running on a free gitlab hosted account, you’re about as likely to end up waiting for containers to spin up as you are to save time by parallelizing the jobs.
• I like to use a docker image that’s fairly tightly tagged — if you wanted to be a little more loose and just use whatever 1.17 image is available on whatever alpine is current, you could instead use 1.17-alpine or some other variant. This way I don’t get a surprise failure when some new image magically appears. (A later article in the series will use a customized image so that we can get somewhat faster startup times.)
• I always reach for alpine first, but sometimes if a project needs a whole pile of packages I’ll use an ubuntu image.
• We need to apk add build-base so that cgo will work, because cgo is required to install golangci-lint.
• We have to install golangci-lint as a binary bundle instead of via go get because of how they bundle things (see their website for details).
• The linter runs with a three minute timeout, because in the past I’ve had issues with the initial pull taking longer than the default one minute timeout.
• Verbose linter output is noisier than what you’d want if you were running manually, but it’s helpful for troubleshooting a failed pipeline (especially if it times out!).
• Then we run tests, with both the race detector and coverage enabled.
• Running go tool cover will output the overall coverage percentage — including subpackages if our package grows to that level of complexity. (Note that this line has to be quoted because of yaml syntax issues.) For this project we could get away with just using the output of go test but as noted above, I like to reuse pipeline code from other projects, and this will grow with us.
• We can get gitlab to display this coverage percentage in merge requests by going to your project on gitlab.com » Settings » CI/CD » General » Test Coverage Parsing, and setting total:.*\s(\d+(?:\.\d+)?%) as the regex. Be sure to click “Save Changes”.
• Then we build the app. This should always succeed if the linting & tests succeed, but we run it anyway just as a sanity check. For now, we’re not actually doing anything with the built program, but eventually we will use this for deployment.
• Finally we install a package to help generate a report.xml in the JUnit format that gitlab wants in order to display pipeline test results on merge requests. We have to mark it as an artifact so that gitlab knows to parse this file.

Commit this file and push to gitlab:

$ git add .gitlab-ci.yml
$ git ci -m "Add ci pipeline"
$ git push --set-upstream origin initial-skeleton

Click on the link to create a merge request. Fill in the description, review your changes, and click “Create Merge Request”. Wait for the pipeline to pass, and then verify the following:

1. The pipeline should succeed. If it fails, check the job log and see what went wrong.
2. It should report “Test summary contained no changed test results out of 1 total test”.
3. Right under the pipeline display in the merge request it should show “Test coverage 37.50% from 1 job”. (If it doesn’t display, and you realize that you didn’t set the coverage regex shown above: set that regex and then try rerunning a pipeline or, if that doesn’t show the coverage you can push a new change to the MR to trigger a new pipeline and get the coverage indication. Sometimes it takes me a couple of tries before I get it right.)

Note: That’s a pretty low coverage rate, but it’s ok at this point because most of our code so far is inside main() and we don’t unit test that.

If these items and everything else look good, click the blue “Merge” button to merge this branch to main.

Success! We’ve got the basic structure of our app, with a unit test, and a CI pipeline that works and reports our test coverage.

Next: How to Add a Database to our Gin Web App

Next we’ll start the database integration and set up a template to display a list of books — even if we don’t have any UI for adding books yet. Free bonuses: a Makefile to build the app, and CI status badges for the readme CI status badges for the README.

• gin