Very often, when working on a project, we want to run various tasks such as linting, tests, launching a development server, etc. Rather than running the actual commands to launch these tasks, it is not unusual to use some kind of tool to manage such tasks by encapsulating a bunch of commands, which can sometimes be long and complex.
Examples of such tools include grunt, gulp, npm scripts, Ruby's rake tool and GNU Make. Make is the oldest of these tools, having been around since 1976 and inspired many task execution and automation tools over the years. Many python developers still use Make today. I have seen it used on a lot of python projects, including pandas, wagtail, scikit-learn and youtube-dl. Even audreyfeldroy/cookiecutter-pypackage, a popular cookiecutter template for python packages, includes a Makefile in the generated project!
I have also used Make a couple of times. Here's an example of a typical Makefile I used for my LaTeX based academic assignments:
PAPER=IA_Final_Research_Proposal_Victor_Miti.tex
all: pdf clean
pdf:
arara $(PAPER)
clean:
rm -fv *.toc *.aux *.log *.out *.blg *.bbl *.run.xml *.bcf *.fls *.synctex.gz *.fdb_latexmk
While Make is a very powerful tool, I find it rather complex, and I have not reached a point where I can comfortably construct a Makefile to suit particular requirements. The above Makefile is a rather simple one, but things can get much more complicated! The other problem with Make is that it was designed to work in *nix environments, so if you're using Windows, you'd have an extra job of setting up your machine to get it to work. I use Cygwin in my windows setup, so this isn't really an issue for me.
The motivation to seek a Make alternative for python projects arose when I recently started working on a FastAPI project. Being used to Django's ./manage.py runserver, I wanted a way to quickly run the development server, without having to type uvicorn app.main:app --reload each time. Since I'm not so proficient with Make, the first idea that came to mind was to create a runserver.sh script in the project root. However, I realized that I would probably either end up with several other bash scripts for specific tasks, or redefine runserver.sh and write several BASH functions for a number of tasks, while providing argument variables ... it would be better to use Make instead! I decided to do a quick search on the internet for a python solution, and the following seemed to be quite prominent:
After having a quick look at the the homepages, repos and docs for the above projects, Invoke seemed to be a perfect fit for me, because of its relatively easy syntax and simple setup. So I decided to give it a shot.
As with most python packages, you can install via pip:
pip install invoke
Next, according to the docs:
The core use case for Invoke is setting up a collection of task functions and executing them. This is pretty easy – all you need is to make a file called tasks.py importing the task decorator and decorating one or more functions. You will also need to add an arbitrarily-named context argument (convention is to use c, ctx or context) as the first positional arg.
Here's the tasks.py file I wrote for my project (this file is located at the project root, just like a Makefile or package.json or Gulpfile.js, etc.):
import os
from invoke import task
@task
def dev(c):
"""run the uvicorn development server"""
os.environ["ENVIRONMENT"] = "dev"
c.run("uvicorn app.main:app --reload", pty=True)
@task
def test(c):
"""run tests"""
os.environ["ENVIRONMENT"] = "test"
psql_command = (
'psql -c "DROP DATABASE IF EXISTS test_postgres_database" '
'&& psql -c "DROP USER IF EXISTS test_postgres_user" '
"&& psql -c \"CREATE USER test_postgres_user PASSWORD 'testDB-password'\" "
'&& psql -c "CREATE DATABASE test_postgres_database OWNER test_postgres_user" '
'&& psql -c "GRANT ALL PRIVILEGES ON DATABASE test_postgres_database to test_postgres_user"'
)
c.run(psql_command, pty=True)
c.run("pytest", pty=True)
@task
def init_db(c):
"""use aerich to generate schema and generate app migrate location"""
os.environ["ENVIRONMENT"] = "dev"
c.run("aerich -c app/aerich.ini init-db", pty=True)
@task
def migrate(c):
"""use aerich to update models and generate migrate changes file"""
os.environ["ENVIRONMENT"] = "dev"
c.run("aerich -c app/aerich.ini migrate", pty=True)
@task
def upgrade(c):
"""use aerich to upgrade db to latest version"""
os.environ["ENVIRONMENT"] = "dev"
c.run("aerich -c app/aerich.ini upgrade", pty=True)
@task(help={"fix": "let black and isort format your files"})
def lint(c, fix=False):
"""flake8, black, isort and mypy"""
if fix:
c.run("black .", pty=True)
c.run("isort --profile black .", pty=True)
else:
c.run("mypy app", pty=True)
c.run("black . --check", pty=True)
c.run("isort --check-only --profile black .", pty=True)
c.run("flake8", pty=True)
Pretty straightforward, right? I was able to do this within a short time, and I know there's room for further improvement, once I dig deeper into the docs. For now, this works well for me 😄. So, if I want to run the development server, I just hit invoke dev in my terminal. Similarly, for tests, invoke test. Notice that the lint task has a fix flag, whose invocation is either invoke lint -f or invoke lint --fix.
Running invoke --list or invoke -l lists all available tasks. In this case the output is as follows:
Available tasks:
dev run the uvicorn development server
init-db use aerich to generate schema and generate app migrate location
lint flake8, black, isort and mypy
migrate use aerich to update models and generate migrate changes file
test run tests
upgrade use aerich to upgrade db to latest version
Notice how a task's docstring provides the task-level help! This is super cool!
Well, I'm loving Invoke so far and I think this will be my go-to task execution tool for python projects. It has an intuitive syntax, making it very easy to get up and running quickly. In addition, It is well documented and comes packed with a lot of very useful and powerful features such as shell tab completion, namespacing, task aliasing, before/after hooks, parallel execution, automatically responding to program output and more.
At the time of writing this post, I had just scratched the surface and yet I was able to achieve much! As I incorporate Invoke into my python development workflow, I will be prompted to explore it further and get to know it better. If you haven't used Invoke before, I'd suggest you give it a try and see how it fits into your workflow.
Wallpaper image by Gradient on Unsplash 👍
