The Lost Art of Writing Makefiles
You may come across a
Makefile today or you may be writing one yourself. The
Makefile will have some targets such as
install. When invoked, these tasks run a bunch of commands. Sadly, such
Makefiles are no better than a task runner.
Makefiles are much, much more capable.
The problem with average Makefiles
With fast disks and CPUs, we do not have a strong need to prevent duplicate work anymore. A
webpack build takes seconds, so small that you will be mostly okay with it running over and over.
But what if your tasks are not trivial enough?
You may have seen a
Makefile like this. Here, we have a Python project with source code inside
awesome_package and tests inside
tests. We run
pylint on the source files and use
pytest to run our tests.
Makefile is no better than having a shell script or an
Pipenv command. Make is (ab)used only for running few commands when
make <something> is invoked. Also, this has another glaring defect. If I create a file called
test, Make will refuse to run the tasks. Don’t believe me? Try it out. Weird, isn’t it?
No, it isn’t. Turns out
Make was explicitly designed to behave like this.
Make was developed back in the 70s UNIX days to minimize the amount of work required after any change.
Make uses a concept of target and dependencies. A target requires a set of dependencies that need to be up-to-date. When you invoke
make <something>, it builds a dependency graph of the targets it needs to run to execute the given target. However, the targets are expected to be files.
If the target file is absent or the target file’s timestamp is older than the dependencies, the target is executed.
This explains why Make will not execute my
test task if I have a file called
test. Make finds the file and thinks the task is up-to-date and hence does nothing.
So how is this going to help me?
You can modify the
Makefile to use real files as targets and run tasks out of date. Let’s change our example.
awesome_package: awesome_package/__init__.py awesome_package/code.py
touch awesome_packagetests: awesome_package tests/test_code.py
This says that the target folder
awesome_package depends on the files
Make compares the timestamps for these files and runs
pylint with the changed files only if any of those are changed.
The tests task only runs if
test_code.py or any of the files under
awesome_package is changed. The
touch is there to update the directory timestamp so that it’s newer than the files. This pattern is called “empty target”.
But do we need to specify every file in our project?
No, that would be awful.
Make supports variables and can execute shell commands to populate them, which you can use as a list of targets. Here is an updated
Makefile that uses the find command to discover the available python files.
SRC=$(shell find awesome_package -type f -name '*.py')
TEST_SRC=$(shell find tests -type f -name '*.py').PHONY: all lint testall: lint testlint: awesome_packagetest: testsawesome_package: $(SRC)
touch awesome_packagetests: awesome_package $(TEST_SRC)
I also have set up something called a phony target. This says that the targets marked
.PHONY will always run. Phony is useful for tasks that you want to run regardless of your code state. For example, you may want to have a
clean task that deletes all cached files.
If you run this
Makefile without any target arguments, it executes the first target called
all, which in-turn performs
test, which are aliases to the actual filesystem-based targets.
So go ahead, change your
Makefiles and stop
Makefile is very powerful, and when written correctly, it will shave tons of time from your development work.