Using Go's Fuzz Testing with Gin
- 10 minutes read - 2025 wordsThis is the tenth in a series of articles about writing a small reading list app in Go for personal use.
In the last article we looked at pagination, and I promised to share an approach to testing the pagination support with Go 1.18’s fuzzing support. That’s what we’re going to look at today.
At the end of this article you will:
- see how to run fuzz testing against a function
- see why you might want to refactor a function for more effective fuzz testing
Photo of peaches by Neil Conway, CC-BY-2.0
Required Upgrades
In order to use the fuzzer, we have to upgrade to Go
1.18. Download the update for your OS if you haven’t
already, and change the 1.17 at the top of your go.mod to say 1.18.
You will also need to upgrade your golangci-lint
installation so that you’re
running a set of linters that understand some of the new syntax in 1.18 –
or at least the ones that don’t work with 1.18 are disabled.
Fuzzing the Handler
Writing a fuzz test is fairly straightforward. See the code shown below.
First, we define a function using a name that starts with Fuzz, and that
takes a single *testing.F argument.
Then we have some setup code. Note in our setup code that we are using the
existing function freshDb. I haven’t shown the change here, but we need
to change freshDb so that instead of taking a *testing.T it takes a
testing.TB – this is an interface that is common to both *testing.T
and *testing.F so that it can be called from our regular unit tests and
from fuzz tests.
In the setup code we create 150 book rows in the database. This makes it so that the pagination code has something to work with. It’s important to note here that since we’re setting up a static number of books in the database as part of the setup, this fuzz function won’t be randomizing that aspect of the operation of the function we’re fuzzing.
Then we call f.Add three times with different inputs. This seeds the
corpus with some initial values. We don’t have to do this, but it can be
helpful. Note that the argument(s) to f.Add has to match the type of the
argument(s) to f.Fuzz below.
Finally we call f.Fuzz, passing a test function that takes a *testing.T
and some number of arguments. Note that the set of types that are allowed
for those arguments are restricted, and are documented in the package
docs.
Also note that this test function can’t call any methods on F, it can
only call methods on the T that is passed in.
In this test function we perform a similar sort of test that I’ve shown here previously. I’ve taken some shortcuts with the verifications so that the function isn’t too long and tedious here.
func FuzzBookIndexPaginate(f *testing.F) {
db := freshDb(f)
for i := 0; i < 150; i++ {
b := Book{
Title: fmt.Sprintf("Book%d", i),
Author: fmt.Sprintf("Author%d", i),
}
if err := db.Create(&b).Error; err != nil {
f.Fatalf("error creating book: %s", err)
}
}
f.Add(1)
f.Add(5)
f.Add(500)
f.Fuzz(func(t *testing.T, page int) {
w := httptest.NewRecorder()
ctx, r := gin.CreateTestContext(w)
setupRouter(r, db)
req, err := http.NewRequestWithContext(ctx, "GET",
fmt.Sprintf("/books0/?page=%d", page),
nil,
)
if err != nil {
t.Errorf("got error: %s", err)
}
r.ServeHTTP(w, req)
expected := http.StatusOK
if page > 10 || page < 1 {
expected = http.StatusBadRequest
}
if expected != w.Code {
t.Fatalf("expected response code %d, got %d", http.StatusOK, w.Code)
}
if expected == http.StatusOK {
body := w.Body.String()
fragments := make([]string, 2)
fragments[0] = "<h1>My Books</h1>"
if page == 4 {
fragments[0] = `<strong>4</strong>`
} else {
fragments[0] = `<a href="/books/?page=4">4</a>`
}
for _, fragment := range fragments {
if !strings.Contains(body, fragment) {
t.Fatalf("expected body to contain '%s', got %s", fragment, body)
}
}
}
})
}
With that function added, we can rebuild, and then run go test -fuzz=FuzzBook -run FuzzBook . This will perform fuzzing on the test
matching the argument to -fuzz and it will only run the test matching the
argument to -run.
You will see output something like:
% go test -fuzz=FuzzBook -run FuzzBook .
fuzz: elapsed: 0s, gathering baseline coverage: 0/17 completed
fuzz: elapsed: 0s, gathering baseline coverage: 17/17 completed, now
fuzzing with 8 workers
fuzz: elapsed: 3s, execs: 15879 (5287/sec), new interesting: 3 (total: 20)
fuzz: elapsed: 6s, execs: 32755 (5631/sec), new interesting: 3 (total: 20)
fuzz: elapsed: 9s, execs: 49534 (5594/sec), new interesting: 4 (total: 21)
fuzz: elapsed: 12s, execs: 66550 (5671/sec), new interesting: 4 (total: 21)
^Cfuzz: elapsed: 13s, execs: 70648 (5565/sec), new interesting: 4 (total: 21)
PASS
ok gitlab.com/bstpierre/aklatan 12.763s
Note that this will run until you press control-c (^C in the output
above). You can also specify -fuzztime 30s to fuzz for thirty seconds and
then exit.
The output above shows it running about 5500 iterations per second, with different random inputs on each iteration. If it encounters a failure it will stop, record the input that failed in a file and provide instructions for rerunning that test case.
Here’s an example output file from a defect I found while fuzzing the refactoring shown below:
% cat testdata/fuzz/FuzzPaginate/5f99b48ef2fc40df9ec2103b1c17fa4c123e5d02e683f4f5c147f5865a1f530c
go test fuzz v1
string("0")
int(303)
int(96)
I know from previous experience that 5k runs/second is kind of slow. Which isn’t surprising because the handler has to go through Gin, Gorm, Sqlite, and render the template. This is fine if we want to fuzz all of that logic.
But let’s say we really just want to fuzz the pagination logic, and we’d like get get more iterations per second out of our fuzzing instead of wasting lots of time with all those other layers. In order to focus on that lgoic, we need to do a little refactoring to isolate the pagination code in a single function.
Refactored Pagination
First let’s look at the updated handler.
func bookIndexGet(c *gin.Context) {
db := c.Value("database").(*gorm.DB)
pageStr := c.DefaultQuery("page", "1")
var bookCount int64
if err := db.Table("books").Count(&bookCount).Error; err != nil {
c.AbortWithStatus(http.StatusInternalServerError)
return
}
const booksPerPage = 15
p, err := paginate(pageStr, int(bookCount), booksPerPage)
if err != nil {
c.AbortWithStatus(http.StatusBadRequest)
return
}
books := []Book{}
if err := db.Limit(booksPerPage).Offset(p.Offset).Find(&books).Error; err != nil {
c.AbortWithStatus(http.StatusInternalServerError)
return
}
c.HTML(http.StatusOK, "books/index.html", gin.H{
"books": books,
"p": p,
})
}
The code dealing with pagination has moved into a function called
paginate. And this function takes only the page from the query string,
the number of books in the database, and the number of books per page.
Notably, it does not require access to the database or direct access to the
gin.Context. It returns a struct p which has an Offset field, and
gets passed to the template. Here is the new pagiation section at the
bottom of templates/books/index.html to make use of this new struct:
<div>
Pages:
{{ if ne .p.Prev 0 }}
<a href="/books/?page={{ .p.Prev }}">«</a>
{{ end }}
{{ range .p.Pages }}
{{ if eq . $.p.Page }}<strong>{{ . }}</strong>
{{ else }}<a href="/books/?page={{ . }}">{{ . }}</a>
{{ end }}
{{ end }}
{{ if ne .p.Next 0 }}
<a href="/books/?page={{ .p.Next }}">»</a>
{{ end }}
</div>
That’s how we’re going to use the struct, and here’s the struct definition:
type Pagination struct {
Page int
Count int
Offset int
Prev int
Next int
}
We provide a Pages method that returns a slice of all the pages as ints.
Note that we number pages from 1–N because that’s apparently how normal
humans like to number things instead of 0–(N-1). This is used by the
range loop in the template:
func (p *Pagination) Pages() []int {
pages := make([]int, p.Count)
for i := 0; i < p.Count; i++ {
pages[i] = i + 1
}
return pages
}
And finally we have our paginate function, which returns a pointer to
that struct if it is successful, or an error if it fails. This is the
same logic that was in the handler function last week, but now it’s
populating a struct with the information instead of discrete variables.
func paginate(pageStr string, n, per int) (*Pagination, error) {
if n < 0 || per <= 0 {
return nil, errors.New("invalid quantity or per-page")
}
p := &Pagination{}
var err error
p.Page, err = strconv.Atoi(pageStr)
if err != nil {
return nil, err
}
p.Count = int(math.Ceil(float64(n) / float64(per)))
if p.Count == 0 {
p.Count = 1
}
if p.Page < 1 || p.Page > p.Count {
return nil, errors.New("invalid page")
}
p.Offset = (p.Page - 1) * per
if p.Page > 1 {
p.Prev = p.Page - 1
}
if p.Page < p.Count {
p.Next = p.Page + 1
}
return p, nil
}
Fuzzing Faster
Now we can write a fuzzer that just exercises the pagination logic. This time, our fuzzer is randomizing the count, which was static in the previous fuzzer because it was too cumbersome to reinitialize the database on every round. It’s also randomizing the number of items per page because the function supports different page sizes even though it was previously held constant (and the handler currently uses a constant value).
func FuzzPaginate(f *testing.F) {
f.Add("1", 100, 10)
f.Add("5", 0, 50)
f.Add("10", 250, 50)
f.Fuzz(func(t *testing.T, pageStr string, n, per int) {
p, err := paginate(pageStr, n, per)
if err != nil {
// TODO: verify pageStr is invalid int
return
}
if p == nil {
t.Fatal("p is nil")
}
if p.Page < 1 || p.Page > p.Count {
f.Fatalf("p.Page is %d (count %d)", p.Page, p.Count)
}
if p.Count <= 0 {
t.Fatalf("p.Count is %d", p.Count)
}
if p.Offset < 0 || p.Offset > n {
t.Fatalf("p.Offset is %d (n=%d, per=%d)", p.Offset, n, per)
}
if p.Page == 1 {
if p.Prev != 0 {
t.Fatalf("p.Page is %d but p.Prev is not zero (%d)", p.Page, p.Prev)
}
} else if p.Prev+1 != p.Page {
t.Fatalf("prev %d+1 != %d", p.Prev, p.Page)
}
if p.Page == p.Count {
if p.Next != 0 {
t.Fatalf("p.Page is %d but p.Next is not zero (%d)", p.Page, p.Next)
}
} else if p.Next-1 != p.Page {
t.Fatalf("next %d-1 != %d", p.Next, p.Page)
}
})
}
Now when I run it I see:
% go test -fuzz=FuzzPag -fuzztime 12s -run FuzzPag .
fuzz: elapsed: 0s, gathering baseline coverage: 0/180 completed
fuzz: elapsed: 1s, gathering baseline coverage: 180/180 completed, now fuzzing with 8 workers
fuzz: elapsed: 3s, execs: 32152 (10716/sec), new interesting: 0 (total: 180)
fuzz: elapsed: 6s, execs: 78531 (15460/sec), new interesting: 0 (total: 180)
fuzz: elapsed: 9s, execs: 124014 (15158/sec), new interesting: 0 (total: 180)
fuzz: elapsed: 12s, execs: 170019 (15338/sec), new interesting: 0 (total: 180)
fuzz: elapsed: 12s, execs: 170019 (0/sec), new interesting: 0 (total: 180)
PASS
ok gitlab.com/bstpierre/aklatan 12.138s
At 15k runs/sec this is about three times faster than fuzzing the handler.
Closing Thoughts
Fuzzing is handy: as I was performing the refactoring to the paginate
function shown above the fuzzer that I had for the handler function helped
me find three defects. (It also found a few flaws in how the test itself
handled “weird” inputs generated by the fuzzer.)
It won’t replace the normal deterministic tests that I would usually write, but I will definitely incorporate this into my testing arsenal as a way to help identify edge cases and probe parts of the code that I might not think about. The inputs that it identifies
I don’t think I’ll add any timed fuzzing in normal CI runs. It seems to make more sense to run the fuzzers locallly for a bit after changing some code.
Next Week
In addition to this website, I am working on a book about building web applications with Go. Since writing for the book is eating into my time for writing articles here, I’m changing to a once-a-week schedule for articles here.
Next Friday we’ll look at session management and error reporting with flash messages.