Translating a Makefile to a Magefile
- 8 minutes read - 1696 wordsThis is the first in a series of articles about build tools in the Go ecosystem.
Mage is a tool that provides functionality similar
to make
. From their website:
Why?
Makefiles are hard to read and hard to write. Mostly because makefiles are essentially fancy bash scripts with significant white space and additional make-related syntax.
Mage lets you have multiple magefiles, name your magefiles whatever you want, and they’re easy to customize for multiple operating systems. Mage has no dependencies (aside from go) and runs just fine on all major operating systems, whereas make generally uses bash which is not well supported on Windows. Go is superior to bash for any non-trivial task involving branching, looping, anything that’s not just straight line execution of commands. And if your project is written in Go, why introduce another language as idiosyncratic as bash?
In this article I will show how I ported my Makefile to a magefile and provide a brief review of mage. I’m going to share some code, but no deep explanations – this is not a mage tutorial.

An actual mage. From Wellcome Images, via Wikipedia, CC-BY-4.0
I’m a Skeptic
Let’s get this out of the way: I’m skeptical of build tools that claim to
improve on what make
provides. I have tried several tools, in different
language ecosystems, over the course of the past 25+ years. I keep coming back
to GNU Make… mostly because it’s the “least worst” tool that I’ve found.
CMake is an improvement in some environments, but even then I’m usually using
CMake to generate Makefiles for GNU Make.
Part of my skepticism is that I’ve been writing Makefiles and sh/bash scripts for 25+ years so I’m familiar with a lot of the quirks of these tools, their “weird” syntax, and their limitations.
With this bias in mind, I’m going to try to fairly review mage (and other tools to come) as a potential replacement for GNU Make in aklatan which is a simple Go web app that I’m using as the basis for a series of articles.
Mage Doesn’t Suck
It was fairly straightforward to do an almost line-by-line (target-by-target?) port of aklatan’s Makefile to a magefile.
//go:build mage
package main
import (
"fmt"
"log"
"os"
"regexp"
"strings"
"github.com/bitfield/script"
"github.com/magefile/mage/mg"
"github.com/magefile/mage/sh"
"github.com/magefile/mage/target"
)
var (
commands = []string{"importer"}
goGlob = []string{
"*.go",
"*/*.go",
"cmd/*/*.go",
}
htmlGlob = []string{
"templates/*/*.html",
}
embedGlob = []string{
"static/*/*",
"templates/*/*.html",
}
)
func All() {
mg.Deps(Lint)
mg.Deps(Check)
mg.Deps(Binary) // equivalent to Makefile's $(PROJECT)
mg.Deps(Commands)
}
func Lint() {
output := ".lint"
newer, err := target.Glob(output, goGlob...)
if err != nil {
log.Fatalf("Glob error: %s", err)
}
if newer {
mustRunV("golangci-lint", "run", "--timeout", "180s", "--skip-dirs=rules")
sh.Run("touch", output)
}
}
func Check() {
mg.Deps(dotCoverageDir)
output := ".coverage/aklatan.out"
globs := [][]string{goGlob, htmlGlob}
for _, glob := range globs {
newer, err := target.Glob(output, glob...)
if err != nil {
log.Fatalf("Glob error: %s", err)
}
if newer {
// Note: sh.Exec _would_ substitute $TESTFLAGS, but it won't split it
// into separate arguments. So for equivalent functionality to the
// Makefile we have to do the splitting here. (It still might not be
// exactly 1:1.)
args := []string{"test", fmt.Sprintf("-coverprofile=%s", output), "."}
args = append(args, strings.Split(os.Getenv("TESTFLAGS"), " ")...)
mustRunV("go", args...)
return
}
}
}
func dotCoverageDir() {
sh.Run("mkdir", "-p", ".coverage")
}
func Cover() {
mg.Deps(htmlCover)
// XXX IMO the sed usage in the Makefile is a little clearer, but I went
// all-out in converting this to "pure" Go by using bitfield/script.
// However, it's worth noting that calling script.Exec here doesn't provide
// mage's logging support that we get from sh.RunV.
total := regexp.MustCompile("^total")
re := regexp.MustCompile(":.*statements\\)[^0-9]*")
script.Exec("go tool cover -func .coverage/aklatan.out").MatchRegexp(total).ReplaceRegexp(re, ": ").Stdout()
}
func htmlCover() {
mg.Deps(Check)
output := ".coverage/aklatan.html"
coverOut := ".coverage/aklatan.out"
newer, err := target.Path(output, coverOut)
if err != nil {
log.Fatalf("Path error: %s", err)
}
if newer {
mustRunV("go", "tool", "cover", fmt.Sprintf("-html=%s", coverOut), "-o", output)
}
}
func Binary() {
output := "aklatan"
globs := [][]string{goGlob, embedGlob}
for _, glob := range globs {
newer, err := target.Glob(output, glob...)
if err != nil {
log.Fatalf("Glob error: %s", err)
}
if newer {
mustRunV("go", "build", ".")
return
}
}
}
func Commands() {
// This ends up being a nicer structure than the hacky Makefile recipe.
for _, command := range commands {
buildCommand(command)
}
}
func buildCommand(command string) {
newer, err := target.Glob(command, goGlob...)
if err != nil {
log.Fatalf("Glob error: %s", err)
}
if newer {
mustRunV("go", "build", fmt.Sprintf("./cmd/%s", command))
}
}
// This corresponds to the "report.xml" Makefile target.
func CoverXml() {
mg.Deps(Check) // XXX this dependency was missing from the Makefile
output := "report.xml"
newer, err := target.Glob(output, goGlob...)
if err != nil {
log.Fatalf("Glob error: %s", err)
}
if newer {
// XXX sh.RunV vs script.Exec. Beware that script.Exec doesn't seem to have
// any simple way to implement `pipefail`, and it doesn't seem to expand
// environment variables that don't exist to an empty string. This makes
// it so that running it with `$TESTFLAGS` when that variable is unset
// causes `go test` to fail with an error message, and `go-junit-report` to
// generate an empty report ... but the overall pipeline has a successful
// exit status.
// mustRunV("bash", "-c", "go test $TESTFLAGS -v . 2>&1 | go-junit-report > report.xml")
_, err = script.Exec("go test -v .").Exec("go-junit-report").WriteFile(output)
if err != nil {
log.Fatalf("error writing report.xml: %s", err)
}
mustRunV("go", "tool", "cover", "-func", ".coverage/aklatan.out")
}
}
// mustRunV is a wrapper around sh.RunV that logs a fatal error if an error is
// returned.
func mustRunV(cmd string, args ...string) {
err := sh.RunV(cmd, args...)
if err != nil {
log.Fatalf("error running '%s %v': %s", cmd, args, err)
}
}
var Default = All
What You Gain
As advertised, mage allows you to write “all” of the magefile in Go. I’m
using “all” in scare quotes here because on my first pass through translating
Makefile to magefile I ended up calling sh.RunV("bash", "-c", "...")
. I made
a second pass using bitfield/script
which aims to provide some shell pipeline capabilities as a Go library. This
made it possible to remove all of the bash calls from the translated magefile.
I don’t use Windows or Mac, so I haven’t tested this, but theoretically having a magefile makes the build script more portable to those platforms.
I also don’t have any experience running an open source project with many contributors, so I can’t comment on feedback from contributors about writing build-related code in Go vs. make/bash. But I do have a lot of experience maintaining build systems at a variety of $DAYJOBS where nobody else wants to touch build code, and I wonder if it would be easier to get more people to work on build tasks if the system was written in a familiar language.
Theoretically I could use mage’s ability to compile the build script into
static binary and save some space and/or complexity in a CI pipeline (because
I don’t need to have make
installed). Practically speaking, almost
everything I build pulls in some cgo package so I already have to
build-essential
installed in my CI environment in order to have a compiler,
and build-essential
also comes with make
, so there’s no real savings to be
realized here.
I did like how a multi-output target in the Makefile translated into a much nicer combination of a Go loop and some sub-targets. There’s probably a way to achieve the equivalent with make, but I don’t think it would be elegant.
What You Lose
The biggest thing that leaps out is that you lose conciseness. Make’s syntax is well-suited to expressing targets, dependencies, and actions in a compact way. My 51-line Makefile (which fits entirely on my laptop’s external monitor) ends up as a 174-line magefile.
Part of this is that it’s tedious to check dependencies. Where this is just
implicit in make’s syntax, it requires calling a function (eg. Glob()
) in a
subpackage (magefile/mage/target
), and checking both a newer
return value
as well as an err
return. One of the things I like about Go – explicit
error checking – becomes a real burden in this context.
This error checking burden also comes into play when running the actions
associated with a target. Again, this requires calling a function (eg.
RunV()
) in a subpackage (magefile/mage/sh
), and checking the error status.
But – almost always – if some command fails, I just want to abort with an
error. With make
this is the default behavior.
I wrote a little wrapper to provide this behavior in my magefile:
// mustRunV is a wrapper around sh.RunV that logs a fatal error if an error is
// returned.
func mustRunV(cmd string, args ...string) {
err := sh.RunV(cmd, args...)
if err != nil {
log.Fatalf("error running '%s %v': %s", cmd, args, err)
}
}
And this is fine… but it would be cool if mage provided this functionality
as part of magefile/mage/sh
. Because if mage provided that functionality
then it could also provide the equivalent of make’s -k
/--keep-going
flag:
if a command fails, keep going as much as possible. And as a related feature,
it could also provide -n
/--dry-run
: don’t actually run anything, just
print what would get run.
The last thing isn’t directly due to mage. If you try to use bitfield/script
to replace a shell pipeline, it can be kind of hard to get a proper exit
status, mostly because I couldn’t figure out how to implement the equivalent
of bash’s set -o pipefail
. (I only invested about fifteen minutes of
searching; it probably wouldn’t be too hard to patch in.)
Bottom Line
I probably wouldn’t choose mage over make, but I also wouldn’t fight about it if I had to work on a project with someone who really wanted to use mage.
Next Time
In the next article in this series, I’ll dive into goyek and give a similar review.