Automating Go Builds With Goyek

This is the second in a series of articles about build tools in the Go ecosystem.

Goyek is a simple library for creating build automation in Go.

Goyek’s focus is on simplicity and portability: it lacks sophistication found in other tools. For very simple builds, especially those that are only using Go tools, and that need to be portable across different platforms, Goyek might work for you. For anything even slightly sophisticated, I think there are better options.

In this article I will show how I ported my Makefile to Goyek and provide a brief review of Goyek. I’m going to share some code, but no deep explanations – this is not a Goyek tutorial.

Disclaimer: I’m a Skeptic

As I said when I reviewed mage, I’m skeptical of build tools that claim to improve on make.

I like the idea of using something simpler than GNU Make, and the idea of being able to write the automation in Go is an attractive premise.

I’d even be willing to give up some of the conciseness of make syntax for a Go-based tool that provides significant other benefits.

However, I’m less willing to give up some of the things that make provides out of the box, like skipping targets that don’t need to be rebuilt, or sensible default error handling.

Goyek Is Simple

Porting aklatan’s Makefile to Goyek was simple. I roughly translated each make rule to a goyek.Task. However, Goyek doesn’t really have any concept of targets and prerequisites like make. It just has tasks, where each task can have other dependent tasks, and an action, which is equivalent to a Makefile recipe. Tasks have no knowledge of inputs or outputs, so there’s no built-in support for skipping an action because the outputs are up-to-date.

This file lives in ./build/main.go:

package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"os"

	"github.com/goyek/goyek/v2"
	"github.com/goyek/x/boot"
	"github.com/goyek/x/cmd"
)

var (
	// This var doesn't have a name because it's not used as a dependency
	// by anything else. We just need the registration side effect.
	_ = goyek.Define(goyek.Task{
		Name:  "semgrep",
		Usage: "semgrep",
		Action: func(a *goyek.A) {
			cmd.Exec(a, "semgrep --config rules/ --metrics=off")
		},
	})

	// This var doesn't have a name because it's not used as a dependency
	// by anything else. We just need the registration side effect.
	_ = goyek.Define(goyek.Task{
		Name:  "report.xml",
		Usage: "go-junit-report",
		Action: func(a *goyek.A) {
			testflags := os.Getenv("TESTFLAGS")
			if !cmd.Exec(a, "go test "+testflags+" -v .") {
				return
			}
		},
		Deps: goyek.Deps{
			check,
		},
	})

	aklatan = goyek.Define(goyek.Task{
		Name:  "aklatan",
		Usage: "go build .",
		Action: func(a *goyek.A) {
			cmd.Exec(a, "go build .")
		},
	})

	check = goyek.Define(goyek.Task{
		Name:  "check",
		Usage: "go test",
		Action: func(a *goyek.A) {
			cmd.Exec(a, "mkdir -p .coverage")

			testflags := os.Getenv("TESTFLAGS")
			if !cmd.Exec(a, "go test "+testflags+" -coverprofile=./.coverage/aklatan.out .") {
				return
			}

			// There's no point in touching the flag file, because goyek
			// doesn't handle file-level dependencies.
			// cmd.Exec(a, "touch .lint")

			cmd.Exec(a, "go tool cover -html=./.coverage/aklatan.out -o ./.coverage/aklatan.html")
		},
	})

	cover = goyek.Define(goyek.Task{
		Name:  "cover",
		Usage: "go tool cover",
		Deps: goyek.Deps{
			check,
		},
		Action: func(a *goyek.A) {
			cmd.Exec(a, "go tool cover -func .coverage/aklatan.out "+
				"| sed -n -e '/^total/s/:.*statements)[^0-9]*/: /p'")
		},
	})

	lint = goyek.Define(goyek.Task{
		Name:  "lint",
		Usage: "golangci-lint",
		Action: func(a *goyek.A) {
			cmd.Exec(a, "golangci-lint run --timeout 180s --skip-dirs=rules")

			// There's no point in touching the flag file, because goyek
			// doesn't handle file-level dependencies.
			// cmd.Exec(a, "touch .lint")
		},
	})
	// This is a placeholder task where individual command dependencies
	// will be attached.
	commands = goyek.Define(goyek.Task{
		Name:  "commands",
		Usage: "commands",
		Deps:  commandDeps(),
	})

	all = goyek.Define(goyek.Task{
		Name:  "all",
		Usage: "build pipeline",
		Deps: goyek.Deps{
			lint,
			check,
			cover, // FIXME: make this optional, and force verbose or log output
			aklatan,
			commands,
		},
	})
)

// commandDeps() returns the list of task dependencies for all commands, by
// reading the ./cmd/ directory for subdirectories. As a side effect, it
// registers tasks for all of those dependencies, using task names of the
// form `command.<command-name>`.
func commandDeps() goyek.Deps {
	dirs, err := ioutil.ReadDir("./cmd/")
	if err != nil {
		log.Fatalf("error reading command dir: %s", err)
	}

	commandDeps := goyek.Deps{}
	for _, dir := range dirs {
		if !dir.IsDir() {
			continue
		}

		taskName := fmt.Sprintf("command.%s", dir.Name())
		defTask := goyek.Define(goyek.Task{
			Name:  taskName,
			Usage: taskName,
			Action: func(a *goyek.A) {
				cmd.Exec(a, fmt.Sprintf("go build ./cmd/%s", dir.Name()))
			},
		})

		commandDeps = append(commandDeps, defTask)
	}

	return commandDeps
}

func main() {
	goyek.SetDefault(all)
	boot.Main()
}

To build, run go run ./build -v. (Or without the -v if you want to skip verbose output.) It’s just a Go program that uses the Goyek library to run some tasks.

It’s worth noting that the Goyek core only depends on the Go standard library. However, if you want any sort of convenient way to run commands, you will want to import goyek/x, which is still lightweight on dependencies, or some other convenience package like bitfield/script

The Good Parts

Goyek’s model is simple to understand, and it’s easy to put together a build script that works. The script ends up clean, well-structured, and there are quite a few examples linked from the Goyek docs.

If you’re building a small, uncomplicated Go package, and you don’t want to mess around with make or other external tools, Goyek could be an acceptable choice.

What’s Missing

Overall, I wouldn’t recomment Goyek for anything more than a trivial Go project. Even in that case you’d be better off with mage. Goyek has gaps in functionality that render it unusable, and the design of the library makes it hard to add wrappers that fill those gaps.

No Parallelism

Goyek doesn’t support running tasks concurrently.

This one isn’t a deal-breaker, but I frequently work on projects that benefit from make or ninja’s ability to build targets in parallel. (mage also has support for parallelism.)

Both my laptop and my build server have a bunch of CPU threads… why not make use of them?

No File Dependency Support

In my opinion, the fundamental responsibility of a build tool is to figure out when some desired output has become stale with respect to its inputs, and must be rebuilt. The output should not be rebuilt unless it is stale, because sometimes the rebuilding process is expensive.

For example, I have a PDF output, and a bunch of TeX and other input files that need to be processed in order to create that PDF. Creating the PDF takes about ten seconds. This isn’t a huge amount of time, but I should be able to run my build tool and have it skip building the PDF if everything is up-to-date.

To take this example one step further: some of those TeX files are generated from their own inputs. Whatever build tool I am using should provide some way to specify that the TeX files depend on the raw files, and the PDF depends on the TeX, and there are commands to run when something is stale.

make’s syntax is built around this concept. mage has support for this with the magefile/mage/target package, even if it is somewhat cumbersome to write.

Goyek does not support this, although the docs suggest that using mage/target is possible in a Goyek script, and may be useful.

I spent some time experimenting with a couple of different ways to integrate mage/target into a Goyek script. It is possible, on a task-by-task basis, to skip the action if the output is not stale. However, writing all of this out is tedious, and it’s easier to just use mage than to try to build more elegant abstractions around file dependencies using Goyek.

Bottom Line

Use make.

If you hate make and want to write your build script in Go, use mage.

Next Time

The next article in this series will look at Task, which appears to have a more robust feature set.

• go-build