# Task execution and automation using Invoke

Very often, when working on a project, we want to run various tasks such as linting, tests, launching a development server, etc. Rather than running the actual commands to launch these tasks, it is not unusual to use some kind of tool to manage such tasks by encapsulating a bunch of commands, which can sometimes be long and complex.

Examples of such tools include [grunt](https://gruntjs.com/), [gulp](https://gulpjs.com/), [npm scripts](https://www.freecodecamp.org/news/introduction-to-npm-scripts-1dbb2ae01633/), Ruby's [rake](https://github.com/ruby/rake) tool and [GNU Make](https://www.gnu.org/software/make/). Make is the oldest of these tools, having been around since 1976 and inspired many task execution and automation tools over the years. Many python developers still use Make today. I have seen it used on a lot of python projects, including [pandas](https://github.com/pandas-dev/pandas), [wagtail](https://github.com/wagtail/wagtail), [scikit-learn](https://github.com/scikit-learn/scikit-learn) and [youtube-dl](https://github.com/ytdl-org/youtube-dl). Even [audreyfeldroy/cookiecutter-pypackage](https://github.com/audreyfeldroy/cookiecutter-pypackage), a popular [cookiecutter](https://cookiecutter.readthedocs.io/) template for python packages, includes a `Makefile` in the generated project!

I have also used Make a couple of times. Here's an example of a typical `Makefile` I used for my [LaTeX](https://www.latex-project.org/) based academic assignments: 

```sh
# LaTeX Makefile

## the path to your TeX file
PAPER=IA_Final_Research_Proposal_Victor_Miti.tex 

all:  pdf clean

pdf:  ## Compile paper
	arara $(PAPER)

clean:  ## Clean output files
	rm -fv *.toc *.aux *.log *.out *.blg *.bbl *.run.xml *.bcf *.fls *.synctex.gz *.fdb_latexmk
```

While Make is a very powerful tool, I find it rather complex, and I have not reached a point where I can comfortably construct a `Makefile` to suit particular requirements. The above `Makefile` is a rather simple one, but things can get much more complicated! The other problem with Make is that it was designed to work in [\*nix](https://en.wikipedia.org/wiki/Unix-like) environments, so if you're using Windows, you'd have an extra job of [setting up your machine](https://stackoverflow.com/questions/2532234/how-to-run-a-makefile-in-windows) to get it to work. I use [Cygwin](https://www.cygwin.com/) in [my windows setup](https://importthis.tech/windows-setup-for-a-linux-enthusiast), so this isn't really an issue for me.

The motivation to seek a Make alternative for python projects arose when I recently started working on a [FastAPI](https://fastapi.tiangolo.com/) project. Being used to Django's `./manage.py runserver`, I wanted a way to quickly run the development server, without having to type `uvicorn app.main:app --reload` each time. Since I'm not so proficient with Make, the first idea that came to mind was to create a `runserver.sh` script in the project root. However, I realized that I would probably either end up with several other bash scripts for specific tasks, or redefine `runserver.sh` and write several BASH functions for a number of tasks, while providing argument variables ... it would be better to use Make instead! I decided to do a quick search on the internet for a python solution, and the following seemed to be quite prominent:

- [Doit](https://pydoit.org/)
- [Invoke](http://www.pyinvoke.org/)
- [SCons](https://www.scons.org/)

After having a quick look at the the homepages, repos and docs for the above projects, Invoke seemed to be a perfect fit for me, because of its relatively easy syntax and simple setup. So I decided to give it a shot.

As with most python packages, you can install via pip:

```sh
pip install invoke
```

Next, according to the docs:

> The core use case for Invoke is setting up a collection of task functions and executing them. This is pretty easy – all you need is to make a file called `tasks.py` importing the **task** decorator and decorating one or more functions. You will also need to add an arbitrarily-named context argument (convention is to use `c`, `ctx` or `context`) as the first positional arg.

Here's the `tasks.py` file I wrote for my project (this file is located at the project root, just like a `Makefile` or `package.json` or `Gulpfile.js`, etc.):

```python
import os

from invoke import task


@task
def dev(c):
    """run the uvicorn development server"""
    os.environ["ENVIRONMENT"] = "dev"
    c.run("uvicorn app.main:app --reload", pty=True)


@task
def test(c):
    """run tests"""
    os.environ["ENVIRONMENT"] = "test"
    # setup the test database
    psql_command = (
        'psql -c "DROP DATABASE IF EXISTS test_postgres_database" '
        '&& psql -c "DROP USER IF EXISTS test_postgres_user" '
        "&& psql -c \"CREATE USER test_postgres_user PASSWORD 'testDB-password'\" "
        '&& psql -c "CREATE DATABASE test_postgres_database OWNER test_postgres_user" '
        '&& psql -c "GRANT ALL PRIVILEGES ON DATABASE test_postgres_database to test_postgres_user"'
    )
    c.run(psql_command, pty=True)
    # run pytest
    c.run("pytest", pty=True)


@task
def init_db(c):
    """use aerich to generate schema and generate app migrate location"""
    os.environ["ENVIRONMENT"] = "dev"
    c.run("aerich -c app/aerich.ini init-db", pty=True)


@task
def migrate(c):
    """use aerich to update models and generate migrate changes file"""
    os.environ["ENVIRONMENT"] = "dev"
    c.run("aerich -c app/aerich.ini migrate", pty=True)


@task
def upgrade(c):
    """use aerich to upgrade db to latest version"""
    os.environ["ENVIRONMENT"] = "dev"
    c.run("aerich -c app/aerich.ini upgrade", pty=True)


@task(help={"fix": "let black and isort format your files"})
def lint(c, fix=False):
    """flake8, black, isort and mypy"""

    if fix:
        c.run("black .", pty=True)
        c.run("isort --profile black .", pty=True)
    else:
        c.run("mypy app", pty=True)
        c.run("black . --check", pty=True)
        c.run("isort --check-only --profile black .", pty=True)
        c.run("flake8", pty=True)
```

Pretty straightforward, right? I was able to do this within a short time, and I know there's room for further improvement, once I dig deeper into the [docs](http://docs.pyinvoke.org). For now, this works well for me 😄. So, if I want to run the development server, I just hit `invoke dev` in my terminal. Similarly, for tests, `invoke test`. Notice that the _lint_ task has a `fix` flag, whose invocation is either `invoke lint -f` or `invoke lint --fix`.

Running `invoke --list` or `invoke -l` lists all available tasks. In this case the output is as follows:

```txt
Available tasks:

  dev       run the uvicorn development server
  init-db   use aerich to generate schema and generate app migrate location
  lint      flake8, black, isort and mypy
  migrate   use aerich to update models and generate migrate changes file
  test      run tests
  upgrade   use aerich to upgrade db to latest version
```

Notice how a task's docstring provides the task-level help! This is super cool!

Well, I'm loving Invoke so far and I think this will be my go-to task execution tool for python projects. It has an intuitive syntax, making it very easy to get up and running quickly. In addition, It is well documented and comes packed with a lot of very useful and powerful features such as shell tab completion, namespacing, task aliasing, before/after hooks, parallel execution, automatically responding to program output and more.

At the time of writing this post, I had just scratched the surface and yet I was able to achieve much! As I incorporate Invoke into my python development workflow, I will be prompted to explore it further and get to know it better. If you haven't used Invoke before, I'd suggest you give it a try and see how it fits into your workflow.

---

Wallpaper image by [Gradient](https://unsplash.com/@gradienta?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on [Unsplash](https://unsplash.com/s/photos/abstract-background?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) 👍